On Medians of (Randomized) Pairwise Means Pierre Laforgue 1 , Stephan - - PowerPoint PPT Presentation

On Medians of (Randomized) Pairwise Means

Pierre Laforgue1, Stephan Cl´ emenc ¸on1, Patrice Bertail2

1 LTCI, T´

el´ ecom Paris, Institut Polytechnique de Paris, France

2 Modal’X, UPL, Universit´

e Paris-Nanterre, France 1

The Median of Means

x1 . . . xB . . . xn−B+1 . . . xn mean mean ˆ θ1 . . . ˆ θK ˆ θMoM median

x1, . . . , xn i.i.d. realizations of r.v. X s.t. E[X] = θ, Var(X) = σ2. ∀δ ∈ [e1− n

2 , 1[, set K := ⌈log(1/δ)⌉, it holds [Devroye et al., 2016]:

P

  

• ˆ

θMoM − θ

• > 2

√ 2eσ

• 1 + log(1/δ)

n

   ≤ δ.

2

The Median of Randomized Means (1st contribution)

x1 . . . xB xB+1 . . . xn−1 xn mean mean ¯ θ1 . . . ¯ θK ¯ θMoRM median

With blocks formed by SWoR, ∀ τ ∈]0, 1/2[, ∀ δ ∈ [2e− 8τ2n

9 , 1[, set

K :=

log(2/δ)

2(1/2−τ)2

• , and B :=
• 8τ 2n

9 log(2/δ)

• , it holds:

P

  

• ¯

θMoRM − θ

• > 3

√ 3 σ 2 τ 3/2

• log(2/δ)

n

   ≤ δ.

3

U-statistics & Pairwise Learning

Estimate E[h(X1, X2)] from an i.i.d. sample x1, . . . , xn: Un(h) = 2 n(n − 1)

• 1≤i<j≤n

h(xi, xj). Ex: the empirical variance when h(x, x′) = (x−x′)2

2

. Encountered e.g. in pairwise ranking or in metric learning:

• Rn(r) =

2 n(n − 1)

• 1≤i<j≤n

✶ {r(xi, xj) · (yi − yj) ≤ 0} .

4

The Median of (Randomized) U-statistics (2nd contribution)

Blocks are formed either by partitioning or by SWoR. Medians of the (randomized) U-statistics verify ∀τ ∈]0, 1/2[ w.p.a.l. 1 − δ:

• ˆ

θMoU − θ(h)

• ≤
• C1 log 1

δ

n + C2 log2(1

δ)

n

• 2n − 9 log 1

δ

,

• ¯

θMoRU − θ(h)

• ≤
• C1(τ) log 2

δ

n + C2(τ) log2(2

δ)

n

• 8n − 9 log 2

δ

,

with C1(τ) − − − →

τ→ 1

2

C1 and C2(τ) − − − →

τ→ 1

2

C2.

5

The Pairwise Tournament Procedure (3rd contribution)

Adapted from [Lugosi and Mendelson, 2016]. We want to find f ∗ ∈ argmin

f ∈F

R(f ) = E[ℓ(f , (X, X ′))]. For f ∈ F, let Hf :=

• ℓ(f , X, X ′). For any pair (f , g) ∈ F2:

1) Compute the MoU estimate of Hf − HgL1 ΦS(f , g) = median

ˆ

U1|Hf − Hg|, . . . , ˆ UK|Hf − Hg|

• .

2) If it is large enough, compute the match ΨS′(f , g) = median

ˆ

U1(H2

f − H2 g), . . . , ˆ

UK ′(H2

f − H2 g)

• .

ˆ f winning all its matches verify w.p.a.l. 1 − exp(c0n min{1, r 2}) R(ˆ f ) − R(f ∗) ≤ cr.

6

Conclusion

• MoM exhibits good guarantees with few assumptions

7

Conclusion

• MoM exhibits good guarantees with few assumptions
• 1st contrib. Guarantees preserved through randomization

7

Conclusion

• MoM exhibits good guarantees with few assumptions
• 1st contrib. Guarantees preserved through randomization
• 2nd contrib. Extension to (randomized) U-statistics

7

Conclusion

• MoM exhibits good guarantees with few assumptions
• 1st contrib. Guarantees preserved through randomization
• 2nd contrib. Extension to (randomized) U-statistics
• 3rd contrib. Pairwise tournament procedure

7

References

Devroye, L., Lerasle, M., Lugosi, G., Oliveira, R. I., et al. (2016). Sub-gaussian mean estimators. The Annals of Statistics, 44(6):2695–2725. Lugosi, G. and Mendelson, S. (2016). Risk minimization by median-of-means tournaments. arXiv preprint arXiv:1608.00757. Minsker, S. et al. (2015). Geometric Median and Robust Estimation in Banach Spaces. Bernoulli, 21(4):2308–2335.

8