CSC373 Week 11: Randomized Algorithms 373F19 - Nisarg Shah & - - PowerPoint PPT Presentation

CSC373 Week 11: Randomized Algorithms

373F19 - Nisarg Shah & Karan Singh 1

Randomized Algorithms

373F19 - Nisarg Shah & Karan Singh 2

Deterministic Algorithm Input Output Randomized Algorithm Input Output Randomness

Randomized Algorithms

373F19 - Nisarg Shah & Karan Singh 3

• Running time

➢ Sometimes, we want the algorithm to always take a small

amount of time

• Regardless of both the input and the random coin flips

➢ Sometimes, we want the algorithm to take a small

amount of time in expectation

• Expectation over random coin flips
• Still regardless of the input

Randomized Algorithms

373F19 - Nisarg Shah & Karan Singh 4

• Efficiency

➢ We want the algorithm to return a solution that is, in

expectation, close to the optimum according to the

• bjective under consideration
• Once again, the expectation is over random coin flips
• We want this to hold for every input

Derandomization

373F19 - Nisarg Shah & Karan Singh 5

• For some problems, it is easy to come up with a

very simple randomized approximation algorithm

• Later, one can ask whether this algorithm can be

“derandomized”

➢ Informally, the randomized algorithm is making some

random choices, and sometimes they turn out to be good

➢ Can we make these “good” choices deterministically?

Recap: Probability Theory

373F19 - Nisarg Shah & Karan Singh 6

• Random variable 𝑌

➢ Discrete

• Takes value 𝑤1 with probability 𝑞1, 𝑤2 w.p. 𝑞2, …
• Expected value 𝐹 𝑌 = 𝑞1 ⋅ 𝑤1 + 𝑞2 ⋅ 𝑤2 + ⋯
• Examples: the roll of a six-sided die (takes values 1 through 6 with

probability 1/6 each)

➢ Continuous

• Has a probability density function (pdf) 𝑔
• Its integral is the cumulative density function (cdf) 𝐺
• 𝐺 𝑦 = Pr[𝑌 ≤ 𝑦]
• Expected value 𝐹 𝑌 = ∫ 𝑦 𝑔 𝑦 𝑒𝑦
• Examples: normal distribution, exponential distribution, uniform

distribution over [0,1], …

Recap: Probability Theory

373F19 - Nisarg Shah & Karan Singh 7

• Things you should be aware of…

➢ Conditional probabilities ➢ Independence among random variables ➢ Conditional expectations ➢ Moments of random variables ➢ Standard discrete distributions: uniform over a finite set,

Bernoulli, binomial, geometric, Poisson, …

➢ Standard continuous distributions: uniform over intervals,

Gaussian/normal, exponential, …

Three Pillars

373F19 - Nisarg Shah & Karan Singh 8

• Deceptively simple, but incredibly powerful!
• Many many many many probabilistic results are

just interesting applications of these three results

Linearity of Expectation Union Bound Chernoff Bound

Three Pillars

373F19 - Nisarg Shah & Karan Singh 9

• Linearity of expectation

➢ 𝐹 𝑌 + 𝑍 = 𝐹 𝑌 + 𝐹[𝑍] ➢ This does not require any independence assumptions

about 𝑌 and 𝑍

➢ E.g. if you want to find out how many people will attend

your party on average, just ask each person the probability with which they will attend and add up

• It does not matter that some of them are friends, and will either

attend together or not attend together

Three Pillars

373F19 - Nisarg Shah & Karan Singh 10

• Union bound

➢ For any two events 𝐵 and 𝐶, Pr 𝐵 ∪ 𝐶 ≤ Pr 𝐵 + Pr[𝐶] ➢ “Probability that at least one of the 𝑜 events 𝐵1, … , 𝐵𝑜

will occur is at most σ𝑗 Pr 𝐵𝑗 ”

➢ Typically, 𝐵1, … , 𝐵𝑜 are “bad events”

• You do not want any of them to occur
• If you can individually bound Pr 𝐵𝑗 ≤ Τ

1 2𝑜 for each 𝑗, then

probability that at least one them occurs ≤ Τ 1 2

• So with probability ≥ Τ

1 2, none of the bad events will occur

• Chernoff bound & Hoeffding’s inequality

➢ Read up!

373F19 - Nisarg Shah & Karan Singh 11

Exact Max-𝑙-SAT

Exact Max-𝑙-SAT

373F19 - Nisarg Shah & Karan Singh 12

• Problem (recall)

➢ Input: An exact 𝑙-SAT formula 𝜒 = 𝐷1 ∧ 𝐷2 ∧ ⋯ ∧ 𝐷𝑛,

where each clause 𝐷𝑗 has exactly 𝑙 literals, and a weight 𝑥𝑗 ≥ 0 of each clause 𝐷𝑗

➢ Output: A truth assignment 𝜐 maximizing the number (or

total weight) of clauses satisfied under 𝜐

➢ Let us denote by 𝑋(𝜐) the total weight of clauses

satisfied under 𝜐

Exact Max-𝑙-SAT

373F19 - Nisarg Shah & Karan Singh 13

• Recall our local search

➢ 𝑂𝑒(𝜐) = set of all truth assignments which can be obtained

by changing the value of at most 𝑒 variables in 𝜐

• Result 1: Neighborhood 𝑂1(𝜐) ⇒ Τ

2 3-apx for Exact

Max-2-SAT.

• Result 2: Neighborhood 𝑂1 𝜐 ∪ 𝜐𝑑 ⇒ Τ

3 4-apx for

Exact Max-2-SAT.

• Result 3: Neighborhood 𝑂1 𝜐 + oblivious local

search ⇒ Τ

3 4-apx for Exact Max-2-SAT.

Exact Max-𝑙-SAT

373F19 - Nisarg Shah & Karan Singh 14

• Recall our local search

➢ 𝑂𝑒(𝜐) = set of all truth assignments which can be obtained

by changing the value of at most 𝑒 variables in 𝜐

• We claimed that ¾-apx for Exact Max-2-SAT can be

generalized to

2𝑙−1 2𝑙 -apx for Exact Max-𝑙-SAT

➢ Algorithm becomes slightly more complicated

• What can we do with randomized algorithms?

Exact Max-𝑙-SAT

373F19 - Nisarg Shah & Karan Singh 15

• Recall:

➢ We have a formula 𝜒 = 𝐷1 ∧ 𝐷2 ∧ ⋯ ∧ 𝐷𝑛 ➢ Variables = 𝑦1, … , 𝑦𝑜, literals = variables or their negations ➢ Each clause contains exactly 𝑙 literals

• The most naïve randomized algorithm

➢ Set each variable to TRUE with probability ½ and to FALSE

with probability ½

• How good is this?

Exact Max-𝑙-SAT

373F19 - Nisarg Shah & Karan Singh 16

• Recall:

➢ We have a formula 𝜒 = 𝐷1 ∧ 𝐷2 ∧ ⋯ ∧ 𝐷𝑛 ➢ Variables = 𝑦1, … , 𝑦𝑜, literals = variables or their negations ➢ Each clause contains exactly 𝑙 literals

• For each clause 𝐷𝑗:

➢ Pr[𝐷𝑗 is not satisfied] = 1/2𝑙 (WHY?) ➢ Hence, Pr[𝐷𝑗 is satisfied] = (2𝑙 − 1)/2𝑙

Exact Max-𝑙-SAT

373F19 - Nisarg Shah & Karan Singh 17

• For each clause 𝐷𝑗:

➢ Pr[𝐷𝑗 is not satisfied] = 1/2𝑙 (WHY?) ➢ Hence, Pr[𝐷𝑗 is satisfied] = (2𝑙 − 1)/2𝑙

• Let 𝜐 denote the random assignment

➢ 𝐹 𝑋 𝜐

= σ𝑗=1

𝑛 𝑥𝑗 ⋅ Pr[𝐷𝑗 is satisfied]

(Which pillar did we just use?)

➢ 𝐹 𝑋 𝜐

=

2𝑙−1 2𝑙 ⋅ σ𝑗=1 𝑛 𝑥𝑗 ≥ 2𝑙−1 2𝑙 ⋅ 𝑃𝑄𝑈

Derandomization

373F19 - Nisarg Shah & Karan Singh 18

• Can we derandomize this algorithm?

➢ What are the choices made by the algorithm?

• Setting the values of 𝑦1, 𝑦2, … , 𝑦𝑜

➢ How do we know which set of choices is good?

• Idea:

➢ Do not think about all the choices at once. ➢ Think about them one by one.

Derandomization

373F19 - Nisarg Shah & Karan Singh 19

• Say you want to deterministically make the right

choice for 𝑦1

➢ Choices of 𝑦2, … , 𝑦𝑜 are still random

𝐹 𝑋 𝜐 = Pr 𝑦1 = 𝑈 ⋅ 𝐹 𝑋 𝜐 𝑦1 = 𝑈 + Pr 𝑦1 = 𝐺 ⋅ 𝐹 𝑋 𝜐 𝑦1 = 𝐺 = ൗ 1 2 ⋅ 𝐹 𝑋 𝜐 𝑦1 = 𝑈 + ൗ 1 2 ⋅ 𝐹 𝑋 𝜐 𝑦1 = 𝐺

➢ This means at least one of 𝐹[𝑋(𝜐)|𝑦1 = 𝑈] and

𝐹[𝑋(𝜐)|𝑦1 = 𝐺] must be at least as much as 𝐹 𝑋 𝜐

• Moreover, both quantities can be computed, so we can take the

better of the two!

• For now, forget about the running time…

• Once we have made the right choice for 𝑦1 (say T),

then we can apply the same logic to 𝑦2

➢

𝐹 𝑋 𝜐 |𝑦1 = 𝑈 = ൗ 1 2 ⋅ 𝐹 𝑋 𝜐 𝑦1 = 𝑈, 𝑦2 = 𝑈 + ൗ 1 2 ⋅ 𝐹 𝑋 𝜐 𝑦1 = 𝑈, 𝑦2 = 𝐺

➢ And then we can pick the choice that leads to a better

conditional expectation

• Derandomized Algorithm:

➢ For 𝑗 = 1, … , 𝑜

• Let 𝑨𝑗 = 𝑈 if 𝐹 𝑋 𝜐 𝑦1 = 𝑨1, … , 𝑦𝑗−1 = 𝑨𝑗−1, 𝑦𝑗 = 𝑈 ≥

𝐹 𝑋 𝜐 𝑦1 = 𝑨1, … , 𝑦𝑗−1 = 𝑨𝑗−1, 𝑦𝑗 = 𝐺 , and 𝑨𝑗 = 𝐺 otherwise

• Set 𝑦𝑗 = 𝑨𝑗

Derandomization

373F19 - Nisarg Shah & Karan Singh 20

• This is called the method of conditional expectations

➢ If we’re happy when making a choice at random, we

should be at least as happy conditioned on at least one of the possible values of that choice

• Derandomized Algorithm:

➢ For 𝑗 = 1, … , 𝑜

• Let 𝑨𝑗 = 𝑈 if 𝐹 𝑋 𝜐 𝑦1 = 𝑨1, … , 𝑦𝑗−1 = 𝑨𝑗−1, 𝑦𝑗 = 𝑈 ≥

𝐹 𝑋 𝜐 𝑦1 = 𝑨1, … , 𝑦𝑗−1 = 𝑨𝑗−1, 𝑦𝑗 = 𝐺 , and 𝑨𝑗 = 𝐺 otherwise

• Set 𝑦𝑗 = 𝑨𝑗

➢ How do we compare the two conditional expectations?

Derandomization

373F19 - Nisarg Shah & Karan Singh 21

Derandomization

373F19 - Nisarg Shah & Karan Singh 22

• 𝐹 𝑋 𝜐 𝑦1 = 𝑨1, … , 𝑦𝑗−1 = 𝑨𝑗−1, 𝑦𝑗 = 𝑈

➢ = σ𝑠 𝑥𝑠 ⋅ Pr[𝐷𝑠 is satisfied 𝑦1 = 𝑨1, … , 𝑦𝑗−1 = 𝑨𝑗−1, 𝑦𝑗 = 𝑈

➢ Set the values of 𝑦1, … , 𝑦𝑗−1, 𝑦𝑗 ➢ If 𝐷𝑠 resolves to TRUE already, the corresponding probability is 1 ➢ If 𝐷𝑠 resolves to FALSE already, the corresponding probability is 0 ➢ Otherwise, if there are ℓ literals left in 𝐷𝑠 after setting 𝑦1, … , 𝑦𝑗−1, 𝑦𝑗,

the corresponding probability is

2ℓ−1 2ℓ

• Compute 𝐹 𝑋 𝜐 𝑦1 = 𝑨1, … , 𝑦𝑗−1 = 𝑨𝑗−1, 𝑦𝑗 = 𝐺

similarly

Max-SAT

373F19 - Nisarg Shah & Karan Singh 23

• Simple randomized algorithm

➢

2𝑙−1 2𝑙 −approximation for Max-𝑙-SAT

➢ Max-3-SAT ⇒ Τ

7 8

• [Håstad]: This is the best possible assuming P ≠ NP

➢ Max-2-SAT ⇒ Τ

3 4 = 0.75

• The best known approximation is 0.9401 using semi-definite

programming and randomized rounding

➢ Max-SAT ⇒ Τ

1 2

• Max-SAT = no restriction on the number of literals in each clause
• The best known approximation is 0.7968, also using semi-definite

programming and randomized rounding

Max-SAT

373F19 - Nisarg Shah & Karan Singh 24

• Better approximations for Max-SAT

➢ Semi-definite programming is out of the scope ➢ But we will see the simpler “LP + randomized rounding”

approach that gives 1 − Τ

1 𝑓 ≈ 0.6321 approximation

• Max-SAT:

➢ Input: 𝜒 = 𝐷1 ∧ 𝐷2 ∧ ⋯ ∧ 𝐷𝑛, where each clause 𝐷𝑗 has

weight 𝑥𝑗 ≥ 0 (and can have any number of literals)

➢ Output: Truth assignment that approximately maximizes

the weight of clauses satisfied

LP Formulation of Max-SAT

373F19 - Nisarg Shah & Karan Singh 25

• First, IP formulation:

➢ Variables:

• 𝑧1, … , 𝑧𝑜 ∈ {0,1}
• 𝑧𝑗 = 1 iff variable 𝑦𝑗 = TRUE in Max-SAT
• 𝑨1, … , 𝑨𝑛 ∈ {0,1}
• 𝑨

𝑘 = 1 iff clause 𝐷 𝑘 is satisfied in Max-SAT

• Program:

Maximize Σ𝑘 𝑥

𝑘 ⋅ 𝑨 𝑘

s.t. Σ𝑦𝑗∈𝐷𝑘 𝑧𝑗 + Σ ҧ

𝑦𝑗∈𝐷𝑘 1 − 𝑧𝑗 ≥ 𝑨 𝑘

∀𝑘 ∈ 1, … , 𝑛 𝑧𝑗, 𝑨

𝑘 ∈ 0,1

∀𝑗 ∈ 1, … , 𝑜 , 𝑘 ∈ 1, … , 𝑛

LP Formulation of Max-SAT

373F19 - Nisarg Shah & Karan Singh 26

• LP relaxation:

➢ Variables:

• 𝑧1, … , 𝑧𝑜 ∈ [0,1]
• 𝑧𝑗 = 1 iff variable 𝑦𝑗 = TRUE in Max-SAT
• 𝑨1, … , 𝑨𝑛 ∈ [0,1]
• 𝑨

𝑘 = 1 iff clause 𝐷 𝑘 is satisfied in Max-SAT

• Program:

Maximize Σ𝑘 𝑥

𝑘 ⋅ 𝑨 𝑘

s.t. Σ𝑦𝑗∈𝐷𝑘 𝑧𝑗 + Σ ҧ

𝑦𝑗∈𝐷𝑘 1 − 𝑧𝑗 ≥ 𝑨 𝑘

∀𝑘 ∈ 1, … , 𝑛 𝑧𝑗, 𝑨

𝑘 ∈ [0,1]

∀𝑗 ∈ 1, … , 𝑜 , 𝑘 ∈ 1, … , 𝑛

Randomized Rounding

373F19 - Nisarg Shah & Karan Singh 27

• Randomized rounding

➢ Find the optimal solution (𝑧∗, 𝑨∗) of the LP ➢ Compute a random IP solution ො

𝑧 such that

• Each ො

𝑧𝑗 = 1 with probability 𝑧𝑗

∗ and 0 with probability 1 − 𝑧𝑗 ∗

• Independently of other ො

𝑧𝑗’s

• The output of the algorithm is the corresponding truth assignment

➢ What is Pr[𝐷

𝑘 is satisfied] if 𝐷 𝑘 has 𝑙 literals?

1 − Π𝑦𝑗∈𝐷𝑘 1 − 𝑧𝑗

∗ ⋅ Π ҧ 𝑦𝑗∈𝐷𝑘 𝑧𝑗 ∗

≥ 1 − Σ𝑦𝑗∈𝐷𝑘 1 − 𝑧𝑗

∗ + Σ ҧ 𝑦𝑗∈𝐷𝑘 𝑧𝑗 ∗

𝑙

𝑙

≥ 1 − 𝑙 − 𝑨

𝑘 ∗

𝑙

𝑙

AM-GM inequality LP constraint

Randomized Rounding

373F19 - Nisarg Shah & Karan Singh 28

• Claim

➢ 1 − 1 −

𝑨 𝑙 𝑙

≥ 1 − 1 −

1 𝑙 𝑙

⋅ 𝑨 for all 𝑨 ∈ [0,1] and 𝑙 ∈ ℕ

• Assuming the claim:

Pr 𝐷

𝑘 is satisfied ≥ 1 − 𝑙−𝑨𝑘

∗

𝑙 𝑙

≥ 1 − 1 − 1

𝑙 𝑙

⋅ 𝑨𝑘

∗ ≥ 1 − 1 𝑓 ⋅ 𝑨𝑘 ∗

• Hence,

𝔽[#weight of clauses satisfied] ≥ 1 −

1 e σ𝑘 𝑥 𝑘 ⋅ 𝑨 𝑘 ∗ ≥ 1 − 1 𝑓 ⋅ 𝑃𝑄𝑈

Standard inequality

Randomized Rounding

373F19 - Nisarg Shah & Karan Singh 29

• Claim

➢ 1 − 1 −

𝑨 𝑙 𝑙

≥ 1 − 1 −

1 𝑙 𝑙

⋅ 𝑨 for all 𝑨 ∈ [0,1] and 𝑙 ∈ ℕ

• Proof of claim:

➢ True at 𝑨 = 0 and 𝑨 = 1 (same quantity on both sides) ➢ For 0 ≤ 𝑨 ≤ 1:

• LHS is a convex function
• RHS is a linear function
• Hence, LHS ≥ RHS ∎

Improving Max-SAT Apx

373F19 - Nisarg Shah & Karan Singh 30

• Claim without proof:

➢ Running both “LP + randomized rounding” and “naïve

randomized algorithm”, and returning the best of the two solutions gives Τ

3 4 = 0.75 approximation!

➢ This algorithm can be derandomized. ➢ Recall:

• “naïve randomized” = independently set each variable to

TRUE/FALSE with probability 0.5 each, which only gives Τ

1 2 = 0.5

approximation by itself

Back to 2-SAT

373F19 - Nisarg Shah & Karan Singh 31

• Max-2-SAT is NP-hard (we didn’t prove this!)
• But 2-SAT can be efficiently solved

➢ “Given a 2-CNF formula, check whether all clauses can be

satisfied simultaneously.”

• Algorithm:

➢ Eliminate all unit clauses, set the corresponding literals. ➢ Create a graph with 2𝑜 literals as vertices. ➢ For every clause (𝑦 ∨ 𝑧), add two edges: ҧ

𝑦 → 𝑧 and ത 𝑧 → 𝑦.

• 𝑣 → 𝑤 means if 𝑣 is true, 𝑤 must be true.

➢ Formula is satisfiable iff no path from 𝑦 to ҧ

𝑦 or ҧ 𝑦 to 𝑦 for any 𝑦

➢ Solve 𝑡 − 𝑢 connectivity problem in polynomial time

Random Walk + 2-SAT

373F19 - Nisarg Shah & Karan Singh 32

• Here’s a cute randomized algorithm by

Papadimitriou [1991]

• Algorithm:

➢ Start with an arbitrary assignment. ➢ While there is an unsatisfied clause 𝐷 = (𝑦 ∨ 𝑧)

• Pick one of the two literals with equal probability.
• Flip the variable value so that 𝐷 is satisfied.
• But, but, this can hurt other clauses?

Random Walk + 2-SAT

373F19 - Nisarg Shah & Karan Singh 33

• Theorem:

➢ If there is a satisfying assignment 𝜐∗, then the expected

time to reach some satisfying assignment is at most 𝑃 𝑜2 .

• Proof:

➢ Fix 𝜐∗. Let 𝜐0 be the starting assignment. Let 𝜐𝑗 be the

assignment after 𝑗 iterations.

➢ Consider the “hamming distance” 𝑒𝑗 between 𝜐𝑗 and 𝜐∗

• Number of coordinates in which the two differ
• 𝑒𝑗 ∈ {0,1, … , 𝑜}.

➢ To show: in expectation, we will hit 𝑒𝑗 = 0 in 𝑃 𝑜2

iterations, unless the algorithm stops before that.

Random Walk + 2-SAT

373F19 - Nisarg Shah & Karan Singh 34

• Observation: 𝑒𝑗+1 = 𝑒𝑗 − 1 or 𝑒𝑗+1 = 𝑒𝑗 + 1

➢ Because we change one variable in each iteration.

• Claim: Pr 𝑒𝑗+1 = 𝑒𝑗 − 1 ≥ 1/2
• Proof:

➢ Iteration 𝑗 considers an unsatisfied clause 𝐷 = (𝑦 ∨ 𝑧) ➢ 𝜐∗ satisfies at least one of 𝑦 or 𝑧, while 𝜐𝑗 satisfies neither ➢ Because we pick a literal randomly, w.p. at least ½ we

pick one where 𝜐𝑗 and 𝜐∗ differ, and decrease distance.

➢ Q: Why did we need an unsatisfied clause? What if we

pick one of 𝑜 variables randomly, and flip it?

Random Walk 2-SAT

373F19 - Nisarg Shah & Karan Singh 35

• Answer:

➢ We want the distance to decrease with probability at

least

1 2 no matter how close or far we are from 𝜐∗.

➢ If we are already close, choosing a variable at random will

likely choose one where 𝜐 and 𝜐∗ already match.

➢ Flipping this variable will increase the distance with high

probability.

➢ An unsatisfied clause narrows it down to two variables

s.t. 𝜐 and 𝜐∗ differ on at least one of them

Random Walk + 2-SAT

373F19 - Nisarg Shah & Karan Singh 36

• Observation: 𝑒𝑗+1 = 𝑒𝑗 − 1 or 𝑒𝑗+1 = 𝑒𝑗 + 1
• Claim: Pr 𝑒𝑗+1 = 𝑒𝑗 − 1 ≥ 1/2
• How does this help?

1 2 3 4 𝑜 5

≥ 1 2 ≤ 1 2 ≥ 1 2 ≤ 1 2

Random Walk + 2-SAT

373F19 - Nisarg Shah & Karan Singh 37

• How does this help?

➢ Can view this as Markov chain and use hitting time results ➢ But let’s prove it with elementary methods.

1 2 3 4 𝑜 5

≥ 1 2 ≤ 1 2 ≥ 1 2 ≤ 1 2

Random Walk + 2-SAT

373F19 - Nisarg Shah & Karan Singh 38

• For 𝑙 > ℓ, define:

➢ 𝑈𝑙,ℓ = expected number of iterations it takes to hit distance ℓ

for the first time when you start at distance 𝑙

• 𝑈𝑗+1,𝑗 ≤

1 2 ∗ 1 + 1 2 ∗ 1 + 𝑈𝑗+2,𝑗

=

1 2 ∗ (1) + 1 2 ∗ 1 + 𝑈𝑗+2,𝑗+1 + 𝑈𝑗+1,𝑗

• Simplifying:

➢ 𝑈

𝑗+1,𝑗 ≤ 2 + 𝑈 𝑗+2,𝑗+1 ≤ 4 + 𝑈 𝑗+3,𝑗+2 ≤ ⋯ ≤ 𝑃 𝑜 + 𝑈 𝑜,𝑜−1 ≤ 𝑃 𝑜

• Uses 𝑈

𝑜,𝑜−1 = 1 (Why?)

• 𝑈𝑜,0 ≤ 𝑈𝑜,𝑜−1 + ⋯ + 𝑈1,0 = 𝑃 𝑜2

Random Walk + 2-SAT

373F19 - Nisarg Shah & Karan Singh 39

• Can view this algorithm as a “drunken local search”

➢ We are searching the local neighborhood ➢ But we don’t ensure that we necessarily improve. ➢ We just ensure that in expectation, we aren’t hurt. ➢ Hope to reach a feasible solution in polynomial time

• Schöning extended this technique to 𝑙-SAT

➢ Schöning’s algorithm no longer runs in polynomial time,

but this is okay because 𝑙-SAT is NP-hard

➢ It still improves upon the naïve 2𝑜 ➢ Later derandomized by Moser and Scheder [2011]

Schöning’s Algorithm for 𝑙-SAT

373F19 - Nisarg Shah & Karan Singh 40

• Algorithm:

➢ Choose a random assignment 𝜐. ➢ Repeat 3𝑜 times (𝑜 = #variables)

• If 𝜐 satisfies the CNF, stop.
• Else, pick an arbitrary unsatisfied clause, and flip a random literal

in the clause.

Schöning’s Algorithm

373F19 - Nisarg Shah & Karan Singh 41

• Randomized algorithm with one-sided error

➢ If the CNF is satisfiable, it finds an assignment with

probability at least

1 2 ⋅ 𝑙 𝑙−1 𝑜

➢ If the CNF is unsatisfiable, it surely does not find an

assignment.

• Expected # times we need to repeat = 2 1 −

1 𝑙 𝑜

➢ For 𝑙 = 3, this gives 𝑃(1.3333𝑜) ➢ For 𝑙 = 4, this gives 𝑃 1.5𝑜

Best Known Results

373F19 - Nisarg Shah & Karan Singh 42

• 3-SAT
• Deterministic

➢ Derandomized Schöning’s algorithm: 𝑃(1.3333𝑜) ➢ Best known: 𝑃(1.3303𝑜) [HSSW]

• If there is a unique satisfying assignment: 𝑃(1.3071𝑜) [PPSZ]
• Randomized

➢ Nothing better known without one-sided error ➢ With one-sided error, best known is 𝑃 1.30704𝑜

[Modified PPSZ]

Random Walk + 2-SAT

373F19 - Nisarg Shah & Karan Singh 43

• Random walks are not only of theoretical interest

➢ WalkSAT is a practical SAT algorithm ➢ At each iteration, pick an unsatisfied clause at random ➢ Pick a variable in the unsatisfied clause to flip:

• With some probability, pick at random.
• With the remaining probability, pick one that will make the fewest

previously satisfied clauses unsatisfied.

➢ Restart a few times (avoids being stuck in local minima)

• Faster than “intelligent local search” (GSAT)

➢ Flip the variable that satisfies most clauses

Random Walks on Graphs

373F19 - Nisarg Shah & Karan Singh 44

• Aleliunas et al. [1979]

➢ Let 𝐻 be a connected undirected graph. Then a random

walk starting from any vertex will cover the entire graph (visit each vertex at least once) in 𝑃(𝑛𝑜) steps.

• Also care about limiting probability distribution

➢ In the limit, the random walk with spend

𝑒𝑗 2𝑛 fraction of

the time on vertex with degree 𝑒𝑗

• Markov chains

➢ Generalize to directed (possibly infinite) graphs with

unequal edge probabilities

373F19 - Nisarg Shah & Karan Singh 45

Randomization for Sublinear Running Time

Sublinear Running Time

373F19 - Nisarg Shah & Karan Singh 46

• Given an input of length 𝑜, we want an algorithm

that runs in time 𝑝(𝑜)

➢ 𝑝(𝑜) examples: log 𝑜 ,

𝑜, 𝑜0.999,

𝑜 log 𝑜 , …

➢ The algorithm doesn’t even get to read the full input! ➢ There are four possibilities:

• Exact vs inexact: whether the algorithm always returns the

correct/optimal solution or only does so with high probability (or gives some approximation)

• Worst-case versus expected running time: whether the algorithm

always takes 𝑝(𝑜) time or only does so in expectation (but still on every instance)

373F19 - Nisarg Shah & Karan Singh 47

Exact algorithms, expected sublinear time

Searching in Sorted List

373F19 - Nisarg Shah & Karan Singh 48

• Input: A sorted doubly linked list with 𝑜 elements.

➢ Imagine you have an array 𝐵 with 𝑃(1) access to 𝐵[𝑗] ➢ 𝐵[𝑗] is a tuple (𝑦𝑗, 𝑞𝑗, 𝑜𝑗)

• Value, index of previous element, index of next element.

➢ Sorted: 𝑦𝑞𝑗 ≤ 𝑦𝑗 ≤ 𝑦𝑜𝑗

• Task: Given 𝑦, check if there exists 𝑗 s.t. 𝑦 = 𝑦𝑗
• Goal: We will give a randomized + exact algorithm

with expected running time 𝑃 𝑜 !

Searching in Sorted List

373F19 - Nisarg Shah & Karan Singh 49

• Motivation:

➢ Often we deal with large datasets that are stored in a

large file on disk, or possibly broken into multiple files

➢ Creating a new, sorted version of the dataset is expensive ➢ It is often preferred to “implicitly sort” the data by simply

adding previous-next pointers along with each element

➢ Would like algorithms that can operate on such implicitly

sorted versions and yet achieve sublinear running time

• Just like binary search achieves for an explicitly sorted array

Searching in Sorted List

373F19 - Nisarg Shah & Karan Singh 50

Algorithm:

➢ Select 𝑜 random indices 𝑆 ➢ Access 𝑦𝑘 for each 𝑘 ∈ 𝑆 ➢ Find “accessed 𝑦𝑘 nearest to 𝑦 in either direction”

• Either largest among all 𝑦𝑘 ≤ 𝑦 or smallest among all 𝑦𝑘 ≥ 𝑦
• At least one direction must be possible (WHY?)

➢ If you take the largest 𝑦𝑘 ≤ 𝑦, start from there and keep

going “next” until you find 𝑦 or go past its value

➢ If you take the smallest 𝑦𝑘 ≥ 𝑦, start from there and keep

going “previous” until you find 𝑦 or go past its value

Searching in Sorted List

373F19 - Nisarg Shah & Karan Singh 51

• Analysis sketch:

➢ Suppose you find the largest 𝑦𝑘 ≤ 𝑦 and keep going

“next”

➢ Let 𝑦𝑗 be smallest value ≥ 𝑦 ➢ Algorithm stops when it hits 𝑦𝑗 ➢ Algorithm throws 𝑜 random “darts” on the sorted list ➢ Chernoff bound:

• Expected distance of 𝑦𝑗 to the closest dart to its left is 𝑃

𝑜

• We’ll assume this without proof!

➢ Hence, the algorithm only does “next” 𝑃

𝑜 times in expectation

Searching in Sorted List

373F19 - Nisarg Shah & Karan Singh 52

• Note:

➢ We don’t really require the list to be doubly linked. Just

“next” pointer suffices if we have a pointer to the first element of the list (a.k.a. “anchored list”).

• This algorithm is optimal!
• Theorem: No algorithm that always returns the

correct answer can run in 𝑝 𝑜 expected time.

➢ Can be proved using Yao’s minimax principle ➢ Beyond the scope of the course, but this is a fundamental

result with wide-ranging applications

Sublinear Geometric Algorithms

373F19 - Nisarg Shah & Karan Singh 53

• Chazelle, Liu, and Magen [2003] proved the Θ

𝑜 bound for searching in a sorted linked list

➢ Their main focus was to generalize these ideas to come

up with sublinear algorithms for geometric problems

➢ Polygon intersection: Given two convex polyhedra, check

if they intersect.

➢ Point location: Given a Delaunay triangulation (or Voronoi

diagram) and a point, find the cell in which the point lies.

➢ They provided optimal 𝑃

𝑜 algorithms for both these problems.

373F19 - Nisarg Shah & Karan Singh 54

Inexact algorithms, expected sublinear time

Estimating Avg Degree in Graph

373F19 - Nisarg Shah & Karan Singh 55

• Input: Graph 𝐻 with 𝑜 vertices, and access to an
• racle that returns the degree of a queried vertex

in 𝑃 1 time.

• Goal: (2 + 𝜗)-approximation in expected time

𝑃 𝜗−𝑃 1 𝑜

➢ 𝜗 is constant ⇒ sublinear in input size 𝑜

Estimating Avg Degree in Graph

373F19 - Nisarg Shah & Karan Singh 56

• Wait!

➢ Isn’t this equivalent to “given an array of 𝑜 numbers

between 1 and 𝑜 − 1, estimate their average”?

➢ No! That requires Ω(𝑜) time for constant approximation!

• Consider an instance with constantly many 𝑜 − 1’s, and all other

1’s: you may not discover any 𝑜 − 1 until you query Ω(𝑜) numbers

➢ Why are degree sequences more special?

• Erdős–Gallai theorem: 𝑒1 ≥ ⋯ ≥ 𝑒𝑜 is a degree sequence iff their

sum is even and σ𝑗=1

𝑙

𝑒𝑗 ≤ 𝑙 𝑙 − 1 + σ𝑗=𝑙+1

𝑜

𝑒𝑗.

• Intuitively, we will sample 𝑃

𝑜 vertices

• We may not discover the few high degree vertices, but we’ll

find their neighbors, and thus account for their edges anyway!

Estimating Avg Degree in Graph

373F19 - Nisarg Shah & Karan Singh 57

• Algorithm:

➢ Take Τ

8 𝜗 random subsets 𝑇𝑗 ⊆ 𝑊 with 𝑇𝑗 = 𝑡

➢ Compute the average degree 𝑒𝑇𝑗 in each 𝑇𝑗. ➢ Return ෢

𝑒 = min𝑗 𝑒𝑇𝑗

• Analysis beyond the scope of this course

➢ But doesn’t use anything other than Hoeffding’s

inequality, Markov’s inequality, linearity of expectation, and union bound