Machine Learning Classification over Encrypted Data Raphael Bost, - PowerPoint PPT Presentation

Machine Learning Classification over Encrypted Data Raphael Bost, Raluca Ada Popa, Stephen Tu, Shafi Goldwasser

Classification (Machine Learning) • Supervised learning (training) • Classification data classification training model data set phase phase prediction server client

Secure Classification • The provider’s model is sensible financial model, genetic sequences, … • Client’s private data medical records, credit history, …

Secure Classification • The provider’s model is sensible financial model, genetic sequences, … • Client’s private data medical records, credit history, … MPC / 2PC

Using General 2PC ? + Works for every circuit + Constant number of interactions - Have to build circuits - Hard to ‘compose’ - Not easily reusable ➡ Ad Hoc protocols

Scope of our work • Secure classification, no learning the model is already known • Differential privacy is out of scope can be treated separately • Classifiers as specialized 2PC, but not a specialized classifier

Approach • Security model: passive (honest-but-curious) adversary • Identify and construct reusable building blocks • Practical performance as a primary goal • Choose the best fitted primitives Homomorphic Encryption, FHE, Garbled Circuits, …

Building Blocks • Dot product • Encrypted Comparison • Encrypted (arg)max • Decision trees • Encryption scheme switching

Argmax • Alice ( J a 1 K , . . . , J a n K , PK ) • Bob SK • The comparison pattern must not depend on the values

Argmax • Alice ( J a 1 K , . . . , J a n K , PK ) • Bob SK • The comparison pattern must not depend on the values • Compare everything

Argmax • Alice ( J a 1 K , . . . , J a n K , PK ) • Bob SK • The comparison pattern must not depend on the values • Compare everything ⇒ O ( n 2 )

Argmax • Alice ( J a 1 K , . . . , J a n K , PK ) • Bob SK • The comparison pattern must not depend on the values • Compare everything ⇒ O ( n 2 )

Argmax • Alice ( J a 1 K , . . . , J a n K , PK ) • Bob SK • The comparison pattern must not depend on the values • Compare everything ⇒ O ( n 2 ) • ‘Classical’ algorithm

Argmax • Alice ( J a 1 K , . . . , J a n K , PK ) • Bob SK • The comparison pattern must not depend on the values • Compare everything ⇒ O ( n 2 ) • ‘Classical’ algorithm ⇒ O ( n )

Compare & Swap Alice SK Bob ( PK , J v K , J w K ) J max( v, w ) K ( v < w )

Compare & Swap Alice SK Bob ( PK , J v K , J w K ) Compare ( v < w ) ∅ J max( v, w ) K ( v < w )

Compare & Swap Alice SK Bob ( PK , J v K , J w K ) Compare ( v < w ) ∅ Swap ∅ J max( v, w ) K ( v < w )

Compare & Swap Alice SK Bob ( PK , J v K , J w K ) EncCompare b = ( v < w ) J max( v, w ) K ( v < w )

Compare & Swap Alice SK Bob ( PK , J v K , J w K ) EncCompare b = ( v < w ) ( r, s ) ← M 2 J v 0 K = J v + r K J w 0 K = J w + s K J max( v, w ) K ( v < w )

Compare & Swap Alice SK Bob ( PK , J v K , J w K ) EncCompare b = ( v < w ) ( r, s ) ← M 2 J v 0 K , J w 0 K J v 0 K = J v + r K J w 0 K = J w + s K J max( v, w ) K ( v < w )

Compare & Swap Alice SK Bob ( PK , J v K , J w K ) EncCompare b = ( v < w ) ( r, s ) ← M 2 J v 0 K , J w 0 K J v 0 K = J v + r K J w 0 K = J w + s K ( J w 0 K if b J m 0 K ← J v 0 K o/w. J max( v, w ) K ( v < w )

Compare & Swap Alice SK Bob ( PK , J v K , J w K ) EncCompare b = ( v < w ) ( r, s ) ← M 2 J v 0 K , J w 0 K J v 0 K = J v + r K J w 0 K = J w + s K ( J w 0 K if b J m 0 K ← J v 0 K o/w. ( J b K , J m 0 K ) J max( v, w ) K ( v < w )

Compare & Swap Alice SK Bob ( PK , J v K , J w K ) EncCompare b = ( v < w ) ( r, s ) ← M 2 J v 0 K , J w 0 K J v 0 K = J v + r K J w 0 K = J w + s K ( J w 0 K if b J m 0 K ← J m K ← J m 0 K · ( g � 1 · J b K ) r J v 0 K o/w. ( J b K , J m 0 K ) · J b K � s J max( v, w ) K ( v < w )

Compare & Swap Alice SK Bob ( PK , J v K , J w K ) EncCompare b = ( v < w ) ( r, s ) ← M 2 J v 0 K , J w 0 K J v 0 K = J v + r K J w 0 K = J w + s K ( J w 0 K if b J m 0 K ← J m K ← J m 0 K · ( g � 1 · J b K ) r J v 0 K o/w. ( J b K , J m 0 K ) · J b K � s J m K ← J m 0 − ¯ b.r − b.s K J max( v, w ) K ( v < w )

Argmax • Protocol : n-1 Compare & Swap Alice Bob J m K ← J a 1 K

Argmax • Protocol : n-1 Compare & Swap Alice Bob J m K ← J a 1 K C & S J m K ← J max( m, a 2 ) K ( m < a 2 )

Argmax • Protocol : n-1 Compare & Swap Alice Bob J m K ← J a 1 K C & S J m K ← J max( m, a 2 ) K ( m < a 2 ) C & S J m K ← J max( m, a i ) K ( m < a i )

Argmax • Protocol : n-1 Compare & Swap Alice Bob J m K ← J a 1 K C & S J m K ← J max( m, a 2 ) K ( m < a 2 ) C & S J m K ← J max( m, a i ) K ( m < a i ) C & S J m K ← J max( m, a n ) K ( m < a n )

Argmax • Protocol : n-1 Compare & Swap Alice Bob J m K ← J a 1 K C & S ( m < a 2 ) J m K ← J max( a 1 , a 2 ) K s { C & S ( m < a i ) J m K ← max j ∈ [1 ,i ] a j s { C & S ( m < a n ) J m K ← j ∈ [1 ,n ] a j max

Argmax • Protocol : n-1 Compare & Swap Alice Bob J m K ← J a 1 K C & S ( a 1 < a 2 ) J m K ← J max( a 1 , a 2 ) K s { C & S J m K ← ( m < a i ) ⇒ argmax a j max j ∈ [1 ,i ] a j j ∈ [1 ,i ] s { C & S J m K ← ( m < a n ) ⇒ argmax a j j ∈ [1 ,n ] a j max j ∈ [1 ,n ]

Argmax • Protocol : n-1 Compare & Swap Alice Bob J m K ← J a π (1) K C & S ( a π (1) < a π (1) ) J m K ← J max( a π (1) , a π (2) ) K s { ( m < a π ( i ) ) C & S J m K ← j ∈ [1 ,i ] a π ( j ) max ⇒ argmax a π ( j ) j ∈ [1 ,i ] s { C & S ( m < a π ( n ) ) J m K ← j ∈ [1 ,n ] a π ( j ) max ⇒ argmax a π ( j ) j ∈ [1 ,n ] π (argmax a j ) max a j

Argmax • Protocol : n-1 Compare & Swap

Argmax • Protocol : n-1 Compare & Swap sequentially

Argmax • Protocol : n-1 Compare & Swap sequentially or in parallel

Argmax • Protocol : n-1 Compare & Swap sequentially or in parallel 7000 Party A Party B 6000 Communication Tree 5000 Time (ms) 4000 3000 2000 1000 0 4 5 6 7 8 9 1 1 1 1 1 1 1 1 1 1 2 2 3 3 5 0 1 2 3 4 5 6 7 8 9 0 5 0 5 0 Elements

Decision Trees y x ≥ x 2 x < x 2 B D y 2 y < y 1 y > y 2 y 1 E D B A C x ≥ x 1 x < x 1 E C A x 1 x 2 x

Decision Trees b 1 0 1 b 2 b 3 0 1 0 1 b 4 c 1 c 2 c 3 0 1 c 4 c 5 P ( b 1 , b 2 , b 3 , b 4 , c 1 , . . . , c 5 ) = b 1 · ( b 3 · ( b 4 · c 5 + (1 − b 4 ) · c 4 ) + (1 − b 3 ) · c 3 ) +(1 − b 1 ) · ( b 2 · c 2 + (1 − b 2 ) · c 1 )

Decision Trees P ( b 1 , b 2 , b 3 , b 4 , c 1 , . . . , c 5 ) = b 1 · ( b 3 · ( b 4 · c 5 + (1 − b 4 ) · c 4 ) + (1 − b 3 ) · c 3 ) +(1 − b 1 ) · ( b 2 · c 2 + (1 − b 2 ) · c 1 ) • Polynomial evaluation Leveled Homomorphic Encryption • Binary Variables ) Efficient LHE • Binary Coefficients ! (SIMD)

Classifiers In Practice • Linear Classifier • Naïve Bayes Classifier • Decision Trees

Linear Classifier • Separate two sets of points • Very common classifier • Dot product + Encrypted compare

Linear Classifier Computation Time / protocol Model Total Comm. Inter. Dot Enc. Size Client Server Product Comp. 30 46.4 ms 43.8 ms 194 ms 9.67 ms 204 ms 35.84 kB 7 47 55.5 ms 43.8 ms 194 ms 23.6 ms 217 ms 40.19 kB 7 Evaluation on UC Irvine ML databases 
 40 ms network latency 
 2,66 GHz Intel Core i7

Naïve Bayes Classifier

Naïve Bayes Classifier • Classification argmax p ( C = c i | X = x ) i ∈ [ k ]

Naïve Bayes Classifier • Classification argmax p ( C = c i | X = x ) i ∈ [ k ] p ( C = c i , X = x ) • Bayes Formula argmax p ( X = x ) i ∈ [ k ]

Naïve Bayes Classifier • Classification argmax p ( C = c i | X = x ) i ∈ [ k ] • Bayes Formula argmax p ( C = c i , X = x ) i ∈ [ k ]

Naïve Bayes Classifier • Classification argmax p ( C = c i | X = x ) i ∈ [ k ] • Bayes Formula argmax p ( C = c i , X = x ) i ∈ [ k ] • Naïve Model argmax p ( C = c i , X 1 = x 1 , . . . , X d = x d ) i ∈ [ k ]

Naïve Bayes Classifier • Classification argmax p ( C = c i | X = x ) i ∈ [ k ] • Bayes Formula argmax p ( C = c i , X = x ) i ∈ [ k ] d Y • Naïve Model argmax p ( C = c i ) p ( X j = x j | C = c i ) i ∈ [ k ] j =1

Naïve Bayes Classifier • Classification argmax p ( C = c i | X = x ) i ∈ [ k ] • Bayes Formula argmax p ( C = c i , X = x ) i ∈ [ k ] d Y • Naïve Model argmax p ( C = c i ) p ( X j = x j | C = c i ) i ∈ [ k ] j =1

Naïve Bayes Classifier • Classification argmax p ( C = c i | X = x ) i ∈ [ k ] • Bayes Formula argmax p ( C = c i , X = x ) i ∈ [ k ] d Y • Naïve Model argmax p ( C = c i ) p ( X j = x j | C = c i ) i ∈ [ k ] j =1 d X argmax log p ( C = c i ) log p ( X j = x j | C = c i ) i ∈ [ k ] j =1