CSC373 Algorithm Design, Analysis & Complexity Nisarg Shah - - PowerPoint PPT Presentation
CSC373 Algorithm Design, Analysis & Complexity Nisarg Shah 373F19 - Nisarg Shah 1 Introduction Instructors Karan Singh o dgp.toronto.edu/~karan, karan@dgp, BA 5258 o SEC 5101 and 5201 Nisarg Shah o cs.toronto.edu/~nisarg,
373F19 - Nisarg Shah 1
373F19 - Nisarg Shah 2
➢ Karan Singh
➢ Nisarg Shah
pertaining to our section (5301).
373F19 - Nisarg Shah 3
➢ Every Wed 6-9pm ➢ BA 1190
➢ Every Mon 6-7pm ➢ Divided by birth month ➢ Jan-Jun: UC 244, Jul-Dec: UC 261
373F19 - Nisarg Shah 4
373F19 - Nisarg Shah 5
www.cs.toronto.edu/~nisarg/teaching/373f19/
➢ All the information below is in the course information
sheet, available on the course page
piazza.com/utoronto.ca/fall2019/csc373
➢ Link will be distributed after about two weeks ➢ LaTeX preferred, scans are OK! ➢ An arbitrary subset of questions may be graded…
373F19 - Nisarg Shah 6
➢ A problem sheet will be posted ahead of the tutorial ➢ Easier problems that are warm-up to assignments/exams ➢ You’re expected to try them before coming to the tutorial ➢ TAs will solve the problems on the board ➢ No written/typed solutions will be posted
373F19 - Nisarg Shah 7
➢ 4 assignments ➢ In groups of up to three students ➢ Final marks will be taken from best 3 out of 4 ➢ Questions will be more difficult
start and finish the assignment on the same day!
➢ Submit a single PDF on MarkUs
373F19 - Nisarg Shah 8
➢ Two term tests, one final exam ➢ Details will be posted on the course webpage ➢ In each exam, you’ll be allowed to bring one 8.5” x 11”
sheet of handwritten notes on one side
373F19 - Nisarg Shah 9
* 10% = 30%
* 20% = 40%
* 30% = 30%
your final course grade will be reduced below 50
373F19 - Nisarg Shah 10
➢ [CLRS] Cormen, Leiserson, Rivest, Stein: Introduction to
Algorithms.
➢ [DPV] Dasgupta, Papadimitriou, Vazirani: Algorithms. ➢ [KT] Kleinberg; Tardos: Algorithm Design.
373F19 - Nisarg Shah 11
➢ Free to discuss with classmates or read online material ➢ Must write solutions in your own words
➢ For each question, must cite the peer (write the name) or
the online sources (provide links), if you obtained a significant insight directly pertinent to the question
➢ Failing to do this is plagiarism!
373F19 - Nisarg Shah 12
➢ Borrowed from: Prof. Allan Borodin (citation!)
approach the question (“I do not know how to approach this question”)
373F19 - Nisarg Shah 13
➢ 4 total late days across all 4 assignments ➢ Managed by MarkUs ➢ At most 2 late days can be applied to a single assignment ➢ Already covers legitimate reasons such as illness,
university activities, etc.
covered by this
373F19 - Nisarg Shah 14
373F19 - Nisarg Shah 15
373F19 - Nisarg Shah 16
Muhammad ibn Musa al-Khwarizmi
373F19 - Nisarg Shah 17
➢ Ubiquitous in the real world
➢ Important to be able to design and analyze algorithms ➢ For some problems, good algorithms are hard to find
results
are suddenly hard
373F19 - Nisarg Shah 18
➢ Algorithms in specialized environments or using advanced
techniques
➢ Other concerns with algorithms
➢ …mostly beyond the scope of this course
373F19 - Nisarg Shah 19
➢ Divide and Conquer ➢ Greedy ➢ Dynamic programming ➢ Network flow ➢ Linear programming ➢ Approximation algorithms ➢ Randomized algorithms
373F19 - Nisarg Shah 20
given problem?
➢ A very interesting question! ➢ Subject of much ongoing research…
➢ Proof of correctness ➢ Proof of running time
373F19 - Nisarg Shah 21
in the worst case?
➢ Polynomial time ➢ It should use at most poly(n) steps on any n-bit input
➢ How much is too much?
373F19 - Nisarg Shah 22
373F19 - Nisarg Shah 23
373F19 - Nisarg Shah 24
problem?
➢ Try to prove that the problem is hard ➢ Formally establish complexity results ➢ NP-completeness, NP-hardness, …
its simple variants may suddenly become hard
➢ Minimum spanning tree (MST) vs bounded degree MST ➢ 2-colorability vs 3-colorability
373F19 - Nisarg Shah 25
373F19 - Nisarg Shah 26
373F19 - Nisarg Shah 27
➢ You’ll be working with abstract notations, proving
correctness of algorithms, analyzing the running time of algorithms, designing new algorithms, and proving complexity results.
➢ How many of you are somewhat scared going into the
course?
➢ How many of you feel comfortable with proofs, and want
challenging problems to solve?
➢ How many prefer concrete examples to abstract symbols? ➢ We’ll have something for everyone to enjoy this course
373F19 - Nisarg Shah 28
➢ CSC473: Advanced Algorithms ➢ CSC438: Computability and Logic ➢ CSC463: Computational Complexity and Computability
➢ CSC304: Algorithmic Game Theory and Mechanism
Design (self promotion!)
➢ CSC384: Introduction to Artificial Intelligence ➢ CSC436: Numerical Algorithms ➢ CSC418: Computer Graphics
373F19 - Nisarg Shah 29
373F19 - Nisarg Shah 30
algorithms in, say, CSC236/CSC240 and/or CSC263/CSC265?
➢ Mergesort - 𝑃 𝑜 log 𝑜 ➢ Karatsuba algorithm for fast multiplication - 𝑃 𝑜log2 3
rather than 𝑃 𝑜2
➢ Largest subsequence sum in 𝑃 𝑜 ➢ …
373F19 - Nisarg Shah 31
➢ Break (a large chunk of) a problem into two smaller
subproblems of the same type
➢ Solve each subproblem recursively ➢ At the end, quickly combine solutions from the two
subproblems and/or solve any remaining part of the
divide-and-conquer…
373F19 - Nisarg Shah 32
➢ Useful for analyzing divide-and-conquer running time ➢ If you haven’t already seen it, please spend some time
understanding it
373F19 - Nisarg Shah 33
Intuition:
Compare the function f(n) with the function nlog
b
373F19 - Nisarg Shah 34
➢ Given an array 𝑏 of length 𝑜, count the number of pairs
(𝑗, 𝑘) such that 𝑗 < 𝑘 but 𝑏 𝑗 > 𝑏[𝑘]
➢ Voting theory ➢ Collaborative filtering ➢ Measuring the “sortedness” of an array ➢ Sensitivity analysis of Google's ranking function ➢ Rank aggregation for meta-searching on the Web ➢ Nonparametric statistics (e.g., Kendall's tau distance)
373F19 - Nisarg Shah 35
➢ Count (𝑗, 𝑘) such that 𝑗 < 𝑘 but 𝑏 𝑗 > 𝑏[𝑘]
➢ Check all Θ 𝑜2 pairs
➢ Divide: break array into two equal halves 𝑦 and 𝑧 ➢ Conquer: count inversions in each half recursively ➢ Combine:
373F19 - Nisarg Shah 36
373F19 - Nisarg Shah 37
373F19 - Nisarg Shah 38
373F19 - Nisarg Shah 39
➢ Induction on 𝑜 is usually very helpful ➢ Allows you to assume correctness of subproblems
➢ Suppose 𝑈(𝑜) is the running time for inputs of size 𝑜 ➢ Our algorithm satisfies 𝑈 𝑜 = 2 𝑈
Τ
𝑜 2 + 𝑃(𝑜)
➢ Master theorem says this is 𝑈 𝑜 = 𝑃(𝑜 log 𝑜)
373F19 - Nisarg Shah 40
Let’s say 𝑈 𝑜 = 2 𝑈 Τ
𝑜 2 + 2𝑜
373F19 - Nisarg Shah 41
➢ Given 𝑜 points of the form (𝑦𝑗, 𝑧𝑗) in the plane, find the
closest pair of points.
➢ Basic primitive in graphics and computer vision ➢ Geographic information systems, molecular modeling, air
traffic control
➢ Special case of nearest neighbor
373F19 - Nisarg Shah 42
➢ Sort and check!
➢ Find closest points by x coordinate ➢ Find closest points by y coordinate
➢ No two points have the same x or y coordinate
373F19 - Nisarg Shah 43
➢ Find closest points by x or y coordinate ➢ Doesn’t work!
1 + 𝜗 1 1 + 𝜗 1 2
373F19 - Nisarg Shah 44
➢ Divide: points in equal halves by drawing a vertical line 𝑀 ➢ Conquer: solve each half recursively ➢ Combine: find closest pair with one point on each side of 𝑀 ➢ Return the best of 3 solutions
Seems like Ω(𝑜2)
373F19 - Nisarg Shah 45
➢ We can restrict our attention to points within 𝜀 of 𝑀 on
each side, where 𝜀 = best of the solutions in two halves
373F19 - Nisarg Shah 46
➢ Only need to look at points within 𝜀 of 𝑀 on each side, ➢ Sort points on the strip by 𝑧 coordinate ➢ Only need to check each point with next 11 points in
sorted list!
Wait, what? Why 11?
373F19 - Nisarg Shah 47
➢ If two points are at least 12
positions apart in the sorted list, their distance is at least 𝜀
➢ No two points lie in the same
𝜀/2 × 𝜀/2 box
➢ Two points that are more than two
rows apart are at distance at least 𝜀
373F19 - Nisarg Shah 48
➢ Divide each integer into two parts
𝑜 2 + 𝑦2, 𝑧 = 𝑧1 ∗ 10 Τ 𝑜 2 + 𝑧2
𝑜 2 + (𝑦2𝑧2)
➢ Four Τ
𝑜 2-digit multiplications can be replaced by three
𝑧1 + 𝑧2 − 𝑦1𝑧1 − 𝑦2𝑧2
➢ Running time
Τ
𝑜 2 + 𝑃(𝑜) ⇒ 𝑈 𝑜 = 𝑃 𝑜log2 3
373F19 - Nisarg Shah 49
algorithm for multiplying two 𝑜 × 𝑜 matrices
➢ Call 𝑜 the “size” of the problem
𝐷11 𝐷12 𝐷21 𝐷22 = 𝐵11 𝐵12 𝐵21 𝐵22 ∗ 𝐶11 𝐶12 𝐶21 𝐶22
➢ Naively, this requires 8 multiplications of size 𝑜/2
➢ Strassen’s insight: replace 8 multiplications by 7
Τ
𝑜 2 + 𝑃(𝑜2) ⇒ 𝑈 𝑜 = 𝑃 𝑜log2 7
373F19 - Nisarg Shah 50
𝐷11 𝐷12 𝐷21 𝐷22 = 𝐵11 𝐵12 𝐵21 𝐵22 ∗ 𝐶11 𝐶12 𝐶21 𝐶22
373F19 - Nisarg Shah 51
➢ Given array 𝐵 of 𝑜 comparable elements, find 𝑙th smallest ➢ 𝑙 = 1 is min, 𝑙 = 𝑜 is max, 𝑙 =
Τ 𝑜 + 1 2 is median
➢ 𝑃 𝑜 is easy for min/max
➢ 𝑃(𝑜𝑙) by modifying bubble sort ➢ 𝑃 𝑜 log 𝑜 by sorting ➢ 𝑃 𝑜 + 𝑙 log 𝑜 using min-heap ➢ 𝑃(𝑙 + 𝑜 log 𝑙) using max-heap
373F19 - Nisarg Shah 52
➢ 𝐵𝑚𝑓𝑡𝑡 = elements ≤ 𝑞, 𝐵𝑛𝑝𝑠𝑓 = elements > 𝑞 ➢ If 𝐵𝑚𝑓𝑡𝑡 ≥ 𝑙, return 𝑙th smallest in 𝐵𝑚𝑓𝑡𝑡, otherwise return
(𝑙 − 𝐵𝑚𝑓𝑡𝑡 )th smallest in 𝐵𝑛𝑝𝑠𝑓
➢ If pivot is close to the min or the max, then we basically get
𝑈 𝑜 = 𝑈 𝑜 − 1 + 𝑃(𝑜), which is just 𝑜2
➢ Want to reduce 𝑜 − 1 to a fraction of 𝑜 (like 𝑜/2, 5𝑜/6, etc)
373F19 - Nisarg Shah 53
𝑜 5 groups of 5 each
373F19 - Nisarg Shah 54
𝑜 5 groups of 5 each
373F19 - Nisarg Shah 55
𝑜 5 groups of 5 each
373F19 - Nisarg Shah 56
𝑜 5 groups of 5 each
quickselect
373F19 - Nisarg Shah 57
➢ Out of 𝑜/5 medians, 𝑜/10 are > 𝑞∗
373F19 - Nisarg Shah 58
➢ Out of 𝑜/5 medians, 𝑜/10 are > 𝑞∗
373F19 - Nisarg Shah 59
➢ Out of 𝑜/5 medians, 𝑜/10 are > 𝑞∗ ➢ Each such median brings 3 elements > 𝑞∗
373F19 - Nisarg Shah 60
➢ ≈
Τ
𝑜 10 ⋅ 3 ≈
Τ
3𝑜 10 elements are guaranteed to be > 𝑞∗
➢ So 𝐵𝑚𝑓𝑡𝑡 ≤
Τ
7𝑜 10
➢ Similarly, 𝐵𝑛𝑝𝑠𝑓 ≤
Τ
7𝑜 10
➢ (These are rough calculations…)
analysis?
373F19 - Nisarg Shah 61
𝑜 5 groups of 5 each
Τ
𝑜 5 + 𝑈
Τ
7𝑜 10 + 𝑃(𝑜)
𝑜 5 +
Τ
7𝑜 10 =
Τ
9𝑜 10
➢ Only a fraction of 𝑜, so by Master theorem, 𝑈 𝑜 = 𝑃(𝑜)
𝑃(𝑜) 𝑃(𝑜) 𝑈(𝑜/5) 𝑈(7𝑜/10)
373F19 - Nisarg Shah 62
➢ Typically hard to determine ➢ We still don’t know best algorithms for multiplying two 𝑜-
digit integers or two 𝑜 × 𝑜 matrices
373F19 - Nisarg Shah 63
➢ Usually, we design an algorithm and then analyze its
running time
➢ Sometimes we can do the reverse:
𝑈 𝑜 = 4 𝑈 ൗ 𝑜 2 + 𝑃 𝑜2
𝑜/2 each, and then do 𝑃(𝑜2) computation to combine
373F19 - Nisarg Shah 64
➢ For much of this analysis, we are assuming random access
to elements of input
➢ So we’re ignoring underlying data structures (e.g. doubly
linked list, binary tree, etc.)
➢ We’re only counting comparison or arithmetic operations ➢ So we’re ignoring issues like how real numbers will be
represented in closest pair problem
➢ When we get to P vs NP, representation will matter
373F19 - Nisarg Shah 65
➢ Can be any reasonable parameter of the problem ➢ E.g., for matrix multiplication, we used 𝑜 as the size ➢ But an input consists of two matrices with 𝑜2 entries ➢ It doesn’t matter whether we call 𝑜 or 𝑜2 the size of the
problem
➢ The actual running time of the algorithm won’t change