Greedy Algorithms Course: CS 5130 - Advanced Data Structures and - - PowerPoint PPT Presentation

Greedy Algorithms

Course: CS 5130 - Advanced Data Structures and Algorithms Instructor: Dr. Badri Adhikari

Motivation

For many optimization problems, dynamic programming is overkill. A greedy algorithm always makes the choice that looks best at the moment. It makes a locally optimal choice in the hope that the choice will lead to globally

• ptimal solution.

Greedy algorithms do not always yield optimal solutions, but for many problems they do. It works well for many problems, and many algorithms are based on greedy approach - minimum-spanning-tree algorithms and Dijkstra’s algorithm for shortest path.

Activity-selection problem

• several competing activities require exclusive use of a common resource
• we are required to schedule the activities so that we have a maximum-size

mutually compatible activities Example: a lecture hall can serve only one activity at a time

Each activity has a start time si and finish time fi, where 0 ≤ si ≤ fi ≤ ∞. If selected, activity ai takes place during the half-open interval [si, fi). Activities ai and aj are compatible if the intervals [si,fi) and [sj,fj) do not overlap, i.e. if si ≥ fj or sj ≥ fi. Assume that the activities are sorted by their finish time: f1 ≤ f2 ≤ f3 ≤ ... ≤ fn-1 ≤ fn The problem: Select a maximum-size subset of mutually compatible activities.

Activity-selection problem

si fi sj fj

Example

The subset {a3, a9, a11} consists of mutually compatible activities (but is not the maximum subset). Classwork: What is an optimal solution? (i.e. maximum-size subset)

Optimal substructure property of A-S problem

Let Sij is the set of activities that start after ai finishes and finish before aj starts. We wish to find the maximum set of mutually compatible activities in Sij and let that be Aij. Aij will include some activity ak, such that we are left with two subproblems: finding mutually compatible activities in the set Sik and Skj. If Aij is the optimal solution then Sik and Skj also must be the optimal solution. If not then, then we would use the ‘other’ solution in Aij. The activity-selection problem exhibits optimal substructure.

Solution using dynamic programming

If we denote the size of an optimal solution for the set Sij by c[i,j], then c[i,j] = c[i,k] + c[k,j] + 1 if we know that the optimal solution includes activity ak Otherwise,

ai ak aj

Running time of the DP solution

The DP solution saves all solutions between the time slots 1 & 2, 1 & 3, 1 & 4, …, 2 & 3, 2 & 4, …, 10 & 11. What is the time complexity of the DP solution?

Making a greedy choice

• DP solves all the subproblems to solve the main problem. Can we find the optimal solution

without solving all subproblems?

• In the activity selection problem - we need to consider only one (greedy) choice.

Intuitively:

• we should select an activity such that we leave maximum resource available for other

activities.

• the activity we choose must be the one that finishes first.

Therefore, always choose an activity in S that has the earliest finish time. Once we make this greedy choice of a1, we have only one remaining subproblem - finding activities that start after a1 finishes.

Do we need a DP solution?

We don’t need a DP solution. What does this mean? Instead, we can repeatedly choose the activity that finishes first, keep only the activities that are compatible with this activity, and repeat. Because we always choose the activity with the earliest finish time - the increasing finish time input favors This way, we can consider each activity just once overall, in monotonically increasing order of finish time.

A recursive greedy algorithm

• s and f are arrays with start and finish time
• k is the index of current finish time - defines the subproblem Sk it is to solve
• n is the size of the original problem = s.length = f.length
• We add a fictitious activity a0 with f0 = 0

find an activity am that is compatible with ak - such that sm >= fk

Iterative greedy algorithm

m represents the index our our current start time k indexes the most recent addition to A. (k represents the index of our current finish time)

• Initialize with activity a1 into A
• Find the earliest activity in Sk to finish
• Consider each activity am and add it to

A if found compatible What will be the output? (class-work)

Time complexity

The greedy version = Θ(n) Two questions: Suppose that the activities are not sorted by finish time! (a) Can our algorithm still be used? (b) What will be the running time?

Elements of greedy strategy

(a) Optimal substructure property - if an optimal solution to Sij includes activity ak, then it must also contain optimal solutions to problems Sik and Skj. (b) Greedy-choice property - we can assemble a globally optimal solution by making a locally optimal (greedy) choices. When we are considering which choice to make, we make the choice that looks best in the current problem, without considering results from subproblems.

Greedy strategy vs dynamic programming

0-1 knapsack problem: A thief robbing a store finds n items. The ith item is worth vi dollars and weighs wi pounds (vi and wi are integers). The thief wants to take as valuable a load as possible, but he can carry at most W pounds in his knapsack (W is integer). Which items should he take? Cannot take fractional items or one item more than once. “Gold ingot problem” Fractional knapsack problem: The thief can take fractions of items, rather than having to make a binary (0-1) choice for each item. “Gold dust problem” Which is a more difficult problem?

Greedy Strategy does not work for 0-1 knapsack problem

Example: Thief must select a subset of 3 items and W = 50 pounds.

• ‘item 1’ has greatest value per pound.
• The optimal subset includes items 2 and 3 (any solution with item 1 is not optimal).

The problem Possible solutions If fractions were allowed!

Tree Basics (exempt from exam)

A tree consists of a finite set of elements, called nodes (vertices), and a finite set of directed lines, called branches (edges), that connect the nodes. It has components named after natural trees – root, branches, and leaves. But we draw with root at the top. Root – if the tree is not empty, then the first node is root; it does not have any parent. Degree of a node – the number of branches associated with the node. Indegree – number of branches directed towards the node. Outdegree – number of branches directed away from the node. Parent – a node is parent if it has successor nodes, i.e. if outdegree > 0. Child – a node with a predecessor, i.e. indegree = 1. Ancestor – an ancestor is any node in the path from the root to the node. Descendant – a descendant is any node in the path below the parent node. Leaf (external node) – a node with outdegree = 0. Siblings – two or more node with same parent are called siblings. Path – a path is a sequence of nodes in which each node is adjacent to the next one. Level – the level of a node is its distance from the root. Level of root is 0. Height – the height of a tree is the level of the leaf in the longest path from the root PLUS 1. Subtree – a subtree is any connected structure below the root. Internal node – a node that is not a root or a leaf is called internal node Ordered tree – tree with defined order of children (enables ordered traversal)

Binary Tree (exempt from exam)

A binary tree is a tree in which no node can have more than two subtrees. In other words, a node can have zero, one, or two subtrees. These subtrees are designated as left subtree and right subtree. Symmetry is not a requirement. Null tree is a tree with no nodes. Properties of Binary Tree: (1) Height Given that we need to store N nodes in a binary tree, the maximum height, Hmax, is N. A tree with maximum height is rare. It

• ccurs only when the entire tree is built in one direction.

Given that we need to store N nodes in a binary tree, the minimum height, Hmin, is [log2N] + 1. (2) Number of Nodes Given height H of a binary tree, Nmin = H Given height H of a binary tree, Nmax = 2H – 1 (3) Balance - Balance Factor of a tree It is the difference in height between its left and right subtrees, B = HL - HR When is a tree balanced? A binary tree is balanced if its balance factor is 0 and its subtrees are also balanced (recursive definition). However, this is a very strict definition, so a looser definition is proposed. A binary tree is balanced if the height of its subtrees differs by no more than 1 (i.e. its balance factor is either -1, 0, or 1) and its subtrees are also balanced. Such trees are also called AVL trees.

Huffman Codes

Suppose we have to store a 100,000-character data file. Only 6 different characters appear; with following frequency: How to represent such a file of information (compactly)? Use binary character code - i.e. represent each character by a unique binary string. Option 1: Use fixed-length code => a = 000, b = 001, etc. With this method how many bits do we need to code the entire file?

Huffman Codes

Option 2: Use variable-length code - give frequent characters short codewords and infrequent characters long code words Number of total codes required to represent the file with this technique = 45 * 1 + 13 * 3 + 12 * 3 + 16 * 3 + 9 * 4 + 5 * 4 = 224,000 bits = 25% savings (compared to 300K) When would the savings be higher/lower?

Encoding and Decoding using the prefix codes

We consider only the codes where no codeword is a prefix of any other codes - such codes are prefix codes. Prefix codes always achieve the optimal data compression. Prefix codes are desirable because they simplify decoding. Encoding: Just concatenate the codewords representing each character of the file. Example: The coding of “abc” will be 0.101.100 = “0101100” Decoding: Identify the initial codeword, translate it back to the original character & repeat for remainder of encoded file.

Decoding made convenient with binary tree

The decoding process needs a convenient representation for the prefix code so that we can easily pick off the initial codeword. A binary tree whose leaves are the given characters provides one such representation. The binary codeword for a character = a simple path from the root to that character 0 means “go to the left child” and 1 means “go to the right child”

Optimal code using full binary tree

An optimal code for a file is always represented by a full binary tree. A binary tree is full if every non-leaf node has two children. Tree with fixed length coding(not optimal) Optimal prefix coding

Bits required (optimal prefix coding)

Given a tree T corresponding to a prefix code For each character c in the alphabet C, c.freq denotes the frequency of c in the file. dT(c) denotes the depth of c’s leaf in the tree (also the length of the codeword) Number of bits required, or the cost of the tree T:

Constructing Huffman code

Huffman invented a greedy algorithm that constructs an optimal prefix code. Optimal prefix code = Huffman code. Huffman codes compress data very effectively - savings of 20% to 90%

Constructing Huffman code

C is a set of n characters and each character c ∈ C c.freq gives the frequency of c. The algorithm builds a tree T corresponding to the optimal code in a bottom-up manner. It begins with a set of |C| leaves and performs a sequence of |C|-1 “merging”

• perations.

Uses a min-priority queue Q. When we ‘merge’ two objects, the result is a new object whose frequency is the sum of the two.

Class-work: What is an optimal Huffman code for the following set

• f frequencies, based on the first 8 Fibonacci numbers?

a:1 b:1 c:2 d:3 e:5 f:8 g:13 h:21 Then, encode the following sequence: “abcdefgh” Expected results: (a) a table with codeword for each character (b) encoded string for the input sequence. Hint: Draw the full binary tree.

Summary

Dynamic programming is an overkill for some problems. Greedy algorithms do not solve all optimization problems, but for some problems they are perfect fit. The fractional knapsack problem can be solved using greedy approach but the 0-1 knapsack problem requires dynamic programming. Huffman coding algorithm is an example of a greedy algorithm which allows up to 90% data compression.