[PPT] - I n c r e m e n t a l B a c k u p s ( ! ) G o PowerPoint Presentation

SLIDE 1

I n c r e m e n t a l B a c k u p s

J

h

n S n

w

( y e s , I k n

w

)

S

f

t w a r e E n g i n e e r , R e d H a t 2 1 7

2
5

( G

d

t h i n g s c

m

e i n s m a l l p a c k a g e s ! )

SLIDE 2

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 2

A c k n

w

l e d g m e n t s

( B e c a u s e c

m

p u t e r s a r e a w f u l a n d I n e e d h e l p s

m

e t i m e s )

N

f

e a t u r e i s a n i s l a n d , s

I

' d l i k e t

a

c k n

w

l e d g e :

J

a g a n e S u n d a r

I

n i t i a l f e a t u r e p r

p
s

a l a n d p r i

r

w

r

k ( 2 1 1 )

F

a m Z h e n g

I

n i t i a l d r a f t s f

r

c u r r e n t v e r s i

n

( 2 1 4

2

1 5 )

S

t e f a n H a j n

c

z i & M a x R e i t z

R

e v i e w s a n d p a t i e n c e

SLIDE 3

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 3

A c k n

w

l e d g m e n t s

( B e c a u s e c

m

p u t e r s a r e a w f u l a n d I n e e d h e l p s

m

e t i m e s )

N

f

e a t u r e i s a n i s l a n d , s

I

' d l i k e t

a

c k n

w

l e d g e :

V

l a d i m i r S e m e n t s

v
O

g i e v s k i y , V i r t u

z

z

A

d v a n c e d f e a t u r e s ( P e r s i s t e n c e , M i g r a t i

n

)

P

e r f

r

m a n c e e n h a n c e m e n t s

R

e v i e w s , P a t i e n c e , a n d g e n e r a l e x c e l l e n c e

D

e n i s L u n e v , V i r t u

z

z

D

e d i c a t e d a n d p e r s i s t e n t i n v

l

v e m e n t

SLIDE 4

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 4

O v e r v i e w

( T h i n g s I h

p

e n

t

t

s

t a m m e r t h r

u

g h )

P r

l
g

u e

P

r

b

l e m S t a t e m e n t

A

p p r

a

c h

D

e s i g n G

a

l s A c t I : B u i l d i n g B l

c

k s

B

l

c

k D i r t y B i t m a p s

Q

M P i n t e r f a c e a n d u s a g e

Q

M P t r a n s a c t i

n

s

SLIDE 5

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 5

O v e r v i e w

( T h i n g s I h

p

e n

t

t

s

t a m m e r t h r

u

g h )

A c t I I : L i f e

c

y c l e

I

n c r e m e n t a l b a c k u p l i f e

c

y c l e

E

x a m p l e s A s i d e : T r a n s a c t i

n

s

B

l

c

k J

b

s

T

r a n s a c t i

n

s

M

u l t i

d

r i v e C

h

e r e n c y

E

r r

r

s

SLIDE 6

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 6

O v e r v i e w

( T h i n g s I h

p

e n

t

t

s

t a m m e r t h r

u

g h )

A c t I I I : A d v a n c e d F e a t u r e s

M

i g r a t i

n
P

e r s i s t e n c e

P

u s h v s P u l l m

d

e l b a c k u p s

T

O D O s D é n

u

e m e n t

P

r

j

e c t S t a t u s , Q u e s t i

n

s a n d A n s w e r s

SLIDE 7

P R O L O G U E

( I n w h i c h

u

r h e r

e

s c

m

e t

k

n

w

t h e e n e m y )

SLIDE 8

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 8

G r

s

s . T h e P r

b

l e m

( I j u s t w a n d e r e d i n t

t

h i s t a l k , w h a t ' s i t a b

u

t ? )

W e d n e s d a y 1 2 8 G i B

A

b y s m a l s t

r

a g e e f fi c i e n c y

C

l u n k y , s l

w
B

u t a d m i t t e d l y s i m p l e a n d c

n

v e n i e n t T u e s d a y 1 2 8 G i B M

n

d a y 1 2 8 G i B T h u r s d a y 1 2 8 G i B F r i d a y 1 2 8 G i B

SLIDE 9

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 9

M u c h B e t t e r ! T h e P r

b

l e m

( I j u s t w a n d e r e d i n t

t

h i s t a l k , w h a t ' s i t a b

u

t ? )

M

n

d a y 1 2 8 G i B T u e s d a y 2 G i B W e d n e s d a y 2 . 5 G i B T h u r s d a y 2 . 2 1 G i B F r i d a y 1 G i B

E

f fi c i e n t :

n

l y c

p

i e s m

d

i fi e d d a t a

F

a s t !

M
r

e c

m

p l i c a t e d . . . ?

SLIDE 10

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 1

W e l c

m

e !

( Y

u

’ r e i n m y w

r

l d n

w

)

Q E M U a d d e d p r e l i m i n a r y s u p p

r

t f

r

i n c r e m e n t a l b a c k u p s i n Q E M U 2 . 4 , 2 1 5

8
1

1 .

(

I c a n ’ t c

m

m i t t

e

i t h e r U S

r

E U d a t e s , s

e

n j

y

t h i s I S O

n

e i n s t e a d )

D

e v e l

p

m e n t i s

n

g

i

n g a s

f

2 . 8

N
t

i n c l u d e d a s “ s u p p

r

t e d ” i n a R e d H a t p r

d

u c t y e t

S
,

i t ’ s m

s

t l y f

r

t h e b r a v e .

B

u t w e ’ r e n e a r i n g f e a t u r e c

m

p l e t i

n

.

SLIDE 11

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 1 1

A p p r

a

c h

( W h e r e d i d w e c

m

e f r

m

; w h e r e d i d w e g

)

I n c r e m e n t a l L i v e B a c k u p s h a v e a s t

r

i e d l i n e a g e .

J

a g a n e S u n d a r ' s L i v e B a c k u p ( 2 1 1 )

S

e p a r a t e C L I t

l

s

E

n t i r e l y n e w n e t w

r

k p r

t
c
l
R

a n a s a n i n d e p e n d e n t t h r e a d

U

t i l i z e d t e m p

r

a r y s n a p s h

t

s f

r

a t

m

i c i t y

I

m p l e m e n t e d w i t h i n

m

e m

r

y d i r t y b l

c

k b i t m a p s

W

a s u l t i m a t e l y n

t

m e r g e d

SLIDE 12

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 1 2

A p p r

a

c h

( W h e r e d i d w e c

m

e f r

m

; w h e r e d i d w e g

)

F a m Z h e n g ' s I n c r e m e n t a l B a c k u p ( 2 1 4 )

A

l s

d

i r t y s e c t

r

b i t m a p b a s e d

U

s e s e x i s t i n g H B i t m a p / B d r v D i r t y B i t m a p p r i m i t i v e s

N
n

e w e x t e r n a l t

l

i n g

r

p r

t
c
l

s

M

a n a g e d v i a Q M P

I

m p l e m e n t e d s i m p l y a s a n e w b a c k u p m

d

e

C

a n b e u s e d w i t h a n y i m a g e f

r

m a t

M

a x i m i z e s c

m

p a t i b i l i t y w i t h e x i s t i n g b a c k u p t

l

s

SLIDE 13

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 1 3

D e s i g n G

a

l s

( W h a t d

w

e w a n t ? )

R

e u s e e x i s t i n g p r i m i t i v e s a s m u c h a s p

s

s i b l e

K

e y s t r u c t u r e : ' b l

c

k d r i v e r d i r t y b i t m a p '

A

l r e a d y t r a c k s d i r t y s e c t

r

s

U

s e d f

r

d r i v e m i r r

r

i n g , b l

c

k m i g r a t i

n
C
n

fi g u r a b l e g r a n u l a r i t y

M

a n y b i t m a p s c a n b e u s e d p e r

d

r i v e

SLIDE 14

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 1 4

D e s i g n G

a

l s

( W h a t d

w

e w a n t ? E f fi c i e n t B a c k u p s ! )

R

e u s e e x i s t i n g p r i m i t i v e s

K

e y i n t e r f a c e : d r i v e

b

a c k u p

I

m p l e m e n t e d v i a w e l l

k

n

w

n Q M P p r

t
c
l
U

s e d t

c

r e a t e e . g . f u l l b a c k u p s

A

l r e a d y c a p a b l e

f

p

i

n t

i

n

t

i m e l i v e b a c k u p s

C

a n a l r e a d y e x p

r

t d a t a v i a N B D

W

e m e r e l y a d d a n e w s y n c = i n c r e m e n t a l m

d

e

.

. . A n d a b i t m a p = < n a m e > a r g u m e n t .

SLIDE 15

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 1 5

D e s i g n G

a

l s

( W h e n d

w

e w a n t i t ? )

C
h

e r e n c y

M

u l t i

d

r i v e p

i

n t

i

n

t

i m e b a c k u p a c c u r a c y

U

t i l i z e e x i s t i n g Q M P t r a n s a c t i

n

f e a t u r e

P

e r s i s t e n c e

B

i t m a p s m u s t s u r v i v e s h u t d

w

n s a n d r e b

t

s

M

u s t n

t

d e p e n d

n

d r i v e d a t a f

r

m a t

N
r
n

t h e b a c k u p t a r g e t f

r

m a t

SLIDE 16

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 1 6

D e s i g n G

a

l s

( W h e n d

w

e w a n t i t ? B y 2 . 9 h

p

e f u l l y ! )

M

i g r a t i

n
s

a f e

M

i g r a t i n g m u s t n

t

r e s e t

r

l

s

e b i t m a p d a t a

E

r r

r

H a n d l i n g

B

i t m a p d a t a m u s t n

t

b e l

s

t

n

b a c k u p f a i l u r e

S

t a r t i n g a n e w f u l l b a c k u p i s n

t

s u f fi c i e n t l y r

b

u s t

I

n t e g r i t y

W

e m u s t b e a b l e t

d

e t e c t d e s y n c b e t w e e n p e r s i s t e n c e d a t a a n d b l

c

k d a t a

SLIDE 17

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 1 7

W h y n

t

u s e s n a p s h

t

s ?

( S a v i n g y

u

t i m e d u r i n g t h e Q & A )

“ B

t

h

f

f e r p

i

n t

i

n

t

i m e v i e w s

f

d a t a , w h y n

t

u s e t h e e x i s t i n g m e c h a n i s m ? ”

N
n

e e d t

p

a r s e f

r

m a t

s

p e c i fi c s n a p s h

t

s

n

d i s k

W

e c a n u s e a n y f

r

m a t

I

n c r e m e n t a l b a c k u p s a r e i n e r t a n d d

n
t

g r

w
N
I

O r e q u i r e d t

d

e l e t e i n c r e m e n t a l s

W

e c a n u t i l i z e e x i s t i n g b a c k u p f r a m e w

r

k s

A

c c e s s t

Q

E M U ' s N B D s e r v e r

SLIDE 18

A C T I : B U I L D I N G B L O C K S

( I n w h i c h

u

r h e r

e

s p r e p a r e f

r

b a t t l e )

SLIDE 19

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 1 9

B l

c

k D i r t y B i t m a p s

( N

t

h i n g t

d
w

i t h y

u

r i m a g e s e a r c h s e t t i n g s )

B e f

r

e s h

w

c a s i n g i n c r e m e n t a l s , s

m

e b a c k g r

u

n d :

B

d r v D i r t y B i t m a p i s t h e e x i s t i n g b l

c

k l a y e r s t r u c t u r e u s e d t

t

r a c k w r i t e s

A

l r e a d y u s e d f

r

d r i v e

m

i r r

r

, l i v e b l

c

k m i g r a t i

n
I

m p l e m e n t e d u s i n g h i e r a r c h i c a l b i t m a p

A

n y n u m b e r c a n b e a t t a c h e d t

a

d r i v e

A

l l

w

s f

r

m u l t i p l e i n d e p e n d e n t b a c k u p r e g i m e s

SLIDE 20

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 2

B l

c

k D i r t y B i t m a p s

( N

t

h i n g t

d
w

i t h y

u

r i m a g e s e a r c h s e t t i n g s )

H b i t m a p h i e r a r c h y :

SLIDE 21

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 2 1

B l

c

k D i r t y B i t m a p s

( N

t

h i n g t

d
w

i t h y

u

r i m a g e s e a r c h s e t t i n g s )

B i t m a p p l u r a l i t y : i d = d r i v e b i t ma p i d = d r i v e 1 f

4

b a r 9 F O S D E M

SLIDE 22

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 2 2

B l

c

k D i r t y B i t m a p s

N

a m i n g

( A b i t m a p b y a n y

t

h e r n a m e w

u

l d s m e l l a s s w e e t . . . ? )

B

l

c

k d i r t y b i t m a p s m a y h a v e n a m e s :

E

x i s t i n g i n t e r n a l u s a g e s a r e a n

n

y m

u

s

T

h e n a m e i s u n i q u e t

t

h e d r i v e

B

i t m a p s

n

d i f f e r e n t d r i v e s c a n h a v e t h e s a m e n a m e

T

h e ( n

d

e , n a m e ) p a i r i s t h e b i t m a p I D

U

s e d t

i

s s u e b i t m a p m a n a g e m e n t c

m

m a n d s

SLIDE 23

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 2 3

B l

c

k D i r t y B i t m a p s

N

a m i n g

( A b i t m a p b y a n y

t

h e r n a m e w

u

l d s m e l l a s s w e e t . . . ? )

B i t m a p n a m i n g : i d = d r i v e b i t ma p i d = d r i v e 1 b i t ma p 1 b i t ma p

SLIDE 24

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 2 4

B l

c

k D i r t y B i t m a p s

G

r a n u l a r i t y

( B a c k u p s f r

m

F r e n c h P r e s s t

T

u r k i s h )

B

l

c

k d i r t y b i t m a p s h a v e g r a n u l a r i t i e s :

S

m a l l g r a n u l a r i t y – s m a l l e r b a c k u p s *

U

s e s m

r

e m e m

r

y

1

T i B w / g = 3 2 K i B → 4 M i B

1

T i B w / g = 1 2 8 K i B → 1 M i B

D

e f a u l t : 6 4 K i B * *

A

t t e m p t s t

m

a t c h c l u s t e r s i z e

6

4 K i B c l u s t e r s ( d e f a u l t ) f

r

q c

w

2

SLIDE 25

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 2 5

G r a n u l a r i t i e s – I n D e t a i l

( T u n e d l i k e t h e fi n e s t $ 4 u k u l e l e )

B

i t m a p s t r a c k w r i t e s p e r

s

e c t

r
C
n

fi g u r e g r a n u l a r i t y i n b y t e s

6

4 K → 1 2 8 s e c t

r

s ( 5 1 2 b y t e s / s e c t

r

)

T

h e b a c k u p e n g i n e i t s e l f c

p

i e s

u

t p e r

c

l u s t e r

C

u r r e n t l y : n

n
c
n

fi g u r a b l e , 6 4 K c l u s t e r s

T

h e fi l e f

r

m a t a l s

h

a s a c l u s t e r s i z e

q

c

w

2 d e f a u l t s t

6

4 K .

C
n

c l u s i

n

: 6 4 K i s p r

b

a b l y b e s t ( f

r

n

w

)

SLIDE 26

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 2 6

B l

c

k D i r t y B i t m a p s

M

a n a g e m e n t

( B i t m a p w r a n g l i n g 1 1 )

W e n e e d t

m

a n a g e t h e s e b i t m a p s t

m

a k e b a c k u p s .

M

a n a g e d v i a Q M P

G
d

n e w s i f y

u

' r e a c

m

p u t e r !

F
u

r c

m

m a n d s :

block-dirty-bitmap-add
block-dirty-bitmap-remove
block-dirty-bitmap-clear
query-block

SLIDE 27

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 2 7

B l

c

k D i r t y B i t m a p s

C

r e a t i

n

( L e t t h e r e b e . . . b i t s ! )

B

i t m a p s c a n b e c r e a t e d a t a n y t i m e ,

n

a n y n

d

e

B

i t m a p s b e g i n r e c

r

d i n g w r i t e s i m m e d i a t e l y

G

r a n u l a r i t y i s

p

t i

n

a l

{ "execute": "block-dirty-bitmap-add", "arguments": { "node": "drive0", "name": "bitmap0", "granularity": 131072 } } { "execute": "block-dirty-bitmap-add", "arguments": { "node": "drive0", "name": "bitmap0", "granularity": 131072 } }

SLIDE 28

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 2 8

B l

c

k D i r t y B i t m a p s

D

e l e t i

n

( F

r

d a y s w h e n l e s s i s m

r

e )

C

a n

n

l y b e d e l e t e d w h e n n

t

i n u s e

B

i t m a p s a r e a d d r e s s e d b y t h e i r ( n

d

e , n a m e ) p a i r

H

a s n

e

f f e c t

n

b a c k u p s a l r e a d y m a d e

H

a s n

e

f f e c t

n
t

h e r b i t m a p s

r

n

d

e s

{ "execute": "block-dirty-bitmap-remove", "arguments": { "node": "drive0", "name": "bitmap0" } } { "execute": "block-dirty-bitmap-remove", "arguments": { "node": "drive0", "name": "bitmap0" } }

SLIDE 29

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 2 9

B l

c

k D i r t y B i t m a p s

R

e s e t t i n g

( S

m

e t i m e s w e j u s t w a n t a s e c

n

d c h a n c e )

B

i t m a p s c a n b e c l e a r e d

f

a l l d a t a

P

r i m a r i l y f

r

c

n

v e n i e n c e

B

e g i n s r e c

r

d i n g n e w w r i t e s i m m e d i a t e l y , l i k e a d d

{ "execute": "block-dirty-bitmap-clear", "arguments": { "node": "drive0", "name": "bitmap0" } } { "execute": "block-dirty-bitmap-clear", "arguments": { "node": "drive0", "name": "bitmap0" } }

SLIDE 30

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 3

B l

c

k D i r t y B i t m a p s

Q

u e r y i n g

( W h

a

r e y

u

? W h

w

h

,

w h

w

h

?

)

B i t m a p d a t a c a n b e r e t r i e v e d v i a b l

c

k

q

u e r y .

{"execute": "query-block", "arguments": {}} {"return": [{ … “device”: “drive0”, "dirty-bitmaps": [{ "status": "active", "count": 296704, "name": "bitmap0", "granularity": 65536 }] … }]} {"execute": "query-block", "arguments": {}} {"return": [{ … “device”: “drive0”, "dirty-bitmaps": [{ "status": "active", "count": 296704, "name": "bitmap0", "granularity": 65536 }] … }]}

SLIDE 31

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 3 1

B l

c

k D i r t y B i t m a p s

Q

u e r y i n g

( W h

a

r e y

u

? W h

w

h

,

w h

w

h

?

)

B i t m a p d a t a c a n b e r e t r i e v e d v i a b l

c

k

q

u e r y .

{"execute": "query-block", "arguments": {}} {"return": [{ … “device”: “drive0”, "dirty-bitmaps": [{ "status": "active", (or “frozen”!) "count": 296704, "name": "bitmap0", "granularity": 65536 }] … }]} {"execute": "query-block", "arguments": {}} {"return": [{ … “device”: “drive0”, "dirty-bitmaps": [{ "status": "active", (or “frozen”!) "count": 296704, "name": "bitmap0", "granularity": 65536 }] … }]}

SLIDE 32

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 3 2

B l

c

k D i r t y B i t m a p s

Q

u e r y i n g

( W h

a

r e y

u

? W h

w

h

,

w h

w

h

?

)

B i t m a p d a t a c a n b e r e t r i e v e d v i a b l

c

k

q

u e r y .

{"execute": "query-block", "arguments": {}} {"return": [{ … “device”: “drive0”, "dirty-bitmaps": [{ "status": "active", "count": 296704, (sectors!) "name": "bitmap0", "granularity": 65536 }] (2318 clusters) … }]} {"execute": "query-block", "arguments": {}} {"return": [{ … “device”: “drive0”, "dirty-bitmaps": [{ "status": "active", "count": 296704, (sectors!) "name": "bitmap0", "granularity": 65536 }] (2318 clusters) … }]}

SLIDE 33

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 3 3

B u i l d i n g C

g

n i t i v e D i s s

n

a n c e

( P r

b

l e m S t a t e m e n t 2 : E l e c t r i c B

g

a l

)
Q

M P c

m

m a n d s a r e n

t

p a r t i c u l a r l y u s e f u l a l

n

e

T

h e y a r e n

t

a t

m

i c

O

n l y “ s a f e ” w h e n V M i s

f

fl i n e

N
c

r

s

s

d

r i v e c

h

e r e n c e g u a r a n t e e

SLIDE 34

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 3 4

I n c r e m e n t a l T r a n s a c t i

n

s

( D i s s

n

a n c e a b a t e d ! )

B

i t m a p m a n a g e m e n t t r a n s a c t i

n

s a l l

w

u s t

—
C

r e a t e f u l l b a c k u p s a l

n

g s i d e a b i t m a p r e s e t

C

r e a t e a f u l l b a c k u p a l

n

g s i d e a n e w b i t m a p

R

e s e t b i t m a p s a c r

s

s m u l t i p l e d r i v e s

I

s s u e a n u m b e r

f

i n c r e m e n t a l b a c k u p s a c r

s

s m u l t i p l e d r i v e s

SLIDE 35

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 3 5

I n c r e m e n t a l T r a n s a c t i

n

s

( D i s s

n

a n c e a b a t e d ! )

S

u p p

r

t e d t r a n s a c t i

n

a c t i

n

s :

type:block-dirty-bitmap-add
type:block-dirty-bitmap-clear
N
t

r a n s a c t i

n

n e e d e d f

r

r e m

v

e

W
r

k s i n c

n

j u n c t i

n

w i t h type:drive-backup

F
r

i n c r e m e n t a l s ( m u l t i

d

r i v e c

h

e r e n c y )

F
r

f u l l b a c k u p s

n

e w i n c r e m e n t a l c h a i n s / s y n c p

i

n t s

SLIDE 36

A C T I I : L I F E C Y C L E

( I n w h i c h

u

r h e r

e

s s a v e t i m e a n d m

n

e y )

SLIDE 37

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 3 7

I n c r e m e n t a l s – L i f e C y c l e

1 ) C r e a t e a n e w b a c k u p c h a i n ,

r

2 ) S y n c h r

n

i z e a n e x i s t i n g b a c k u p c h a i n 3 ) C r e a t e t h e fi r s t i n c r e m e n t a l b a c k u p 4 ) C r e a t e s u b s e q u e n t i n c r e m e n t a l b a c k u p s

N e w B i t ma p S y n c P

i

n t ( F u l l B a c k u p ) I n c r e me n t a l

SLIDE 38

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 3 8

L i f e C y c l e – N e w C h a i n

( T h e r e a n d b a c k u p a g a i n )

E x a m p l e 1 : S t a r t a n e w b a c k u p c h a i n a t

m

i c a l l y

{ "execute": "transaction", "arguments": { "actions": [ {"type": "block-dirty-bitmap-add", "data": {"node": "drive0", "name": "bitmap0"} }, {"type": "drive-backup", "data": {"device": "drive0", "target": "/path/to/full.qcow2", "sync": "full", "format": "qcow2"} } ] } } { "execute": "transaction", "arguments": { "actions": [ {"type": "block-dirty-bitmap-add", "data": {"node": "drive0", "name": "bitmap0"} }, {"type": "drive-backup", "data": {"device": "drive0", "target": "/path/to/full.qcow2", "sync": "full", "format": "qcow2"} } ] } }

SLIDE 39

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 3 9

L i f e C y c l e – N e w C h a i n

( T h e r e a n d b a c k u p a g a i n )

i d = d r i v e

SLIDE 40

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 4

L i f e C y c l e – N e w C h a i n

( T h e r e a n d b a c k u p a g a i n )

i d = d r i v e b i t ma p c

u

n t = f u l l . q c

w

2

SLIDE 41

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 4 1

L i f e C y c l e – N e w S y n c P

i

n t

( S u n d a y n i g h t m a i n t e n a n c e b l u e s )

E x a m p l e 2 : T a k e a n e x i s t i n g b i t m a p a n d c r e a t e a n e w f u l l b a c k u p a s a s y n c h r

n

i z a t i

n

p

i

n t .

{ "execute": "transaction", "arguments": { "actions": [ {"type": "block-dirty-bitmap-clear", "data": {"node": "drive0", "name": "bitmap0"} }, {"type": "drive-backup", "data": {"device": "drive0", "target": "/path/to/new_full_backup.qcow2", "sync": "full", "format": "qcow2"} } ] } } { "execute": "transaction", "arguments": { "actions": [ {"type": "block-dirty-bitmap-clear", "data": {"node": "drive0", "name": "bitmap0"} }, {"type": "drive-backup", "data": {"device": "drive0", "target": "/path/to/new_full_backup.qcow2", "sync": "full", "format": "qcow2"} } ] } }

SLIDE 42

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 4 2

L i f e C y c l e – N e w S y n c P

i

n t

( S u n d a y n i g h t m a i n t e n a n c e b l u e s )

i d = d r i v e b i t ma p c

u

n t = 2 9 6 7 4 f u l l . q c

w

2

SLIDE 43

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 4 3

L i f e C y c l e – N e w S y n c P

i

n t

( S u n d a y n i g h t m a i n t e n a n c e b l u e s )

i d = d r i v e b i t ma p c

u

n t = f u l l . q c

w

2 n e w . q c

w

2

SLIDE 44

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 4 4

L i f e C y c l e – F i r s t I n c r e m e n t a l

( T h e fi r s t s t e p

f
u

r j

u

r n e y )

E x a m p l e 3 : C r e a t e a n i n c r e m e n t a l b a c k u p . C a n b e d

n

e v i a t r a n s a c t i

n
r

s i n g l e Q M P c

m

m a n d .

{ "execute": "drive-backup", "arguments": { "device": "drive0", "bitmap": "bitmap0", "target": "inc.0.qcow2", "format": "qcow2", "sync": "incremental", "mode": "existing" } } { "execute": "drive-backup", "arguments": { "device": "drive0", "bitmap": "bitmap0", "target": "inc.0.qcow2", "format": "qcow2", "sync": "incremental", "mode": "existing" } } # qemu-img create -f qcow2 inc.0.qcow2 -b full.qcow2 -F qcow2 # qemu-img create -f qcow2 inc.0.qcow2 -b full.qcow2 -F qcow2

SLIDE 45

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 4 5

L i f e C y c l e – F i r s t I n c r e m e n t a l

( T h e fi r s t s t e p

f
u

r j

u

r n e y )

i d = d r i v e b i t ma p c

u

n t = 2 9 6 7 4 f u l l . q c

w

2

SLIDE 46

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 4 6

L i f e C y c l e – F i r s t I n c r e m e n t a l

( T h e fi r s t s t e p

f
u

r j

u

r n e y )

i d = d r i v e b i t ma p c

u

n t = f u l l . q c

w

2 i n c . . q c

w

2

SLIDE 47

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 4 7

L i f e C y c l e – S u b s e q u e n t B a c k u p s

( T

i

n fi n i t y , a n d b e y

n

d ! )

E x a m p l e s [ 4 , ∞) : C r e a t e s u b s e q u e n t i n c r e m e n t a l s .

{ "execute": "drive-backup", "arguments": { "device": "drive0", "bitmap": "bitmap0", "target": "inc.<n>.qcow2", "format": "qcow2", "sync": "incremental", "mode": "existing" } } { "execute": "drive-backup", "arguments": { "device": "drive0", "bitmap": "bitmap0", "target": "inc.<n>.qcow2", "format": "qcow2", "sync": "incremental", "mode": "existing" } }

# qemu-img create -f qcow2 inc.<n>.qcow2 -b inc.<n-1>.qcow2 -F qcow2 # qemu-img create -f qcow2 inc.<n>.qcow2 -b inc.<n-1>.qcow2 -F qcow2

SLIDE 48

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 4 8

L i f e C y c l e – S u b s e q u e n t B a c k u p s

( T

i

n fi n i t y , a n d b e y

n

d ! )

i d = d r i v e b i t ma p c

u

n t = 6 1 4 4 f u l l . q c

w

2 i n c . . q c

w

2

SLIDE 49

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 4 9

L i f e C y c l e – S u b s e q u e n t B a c k u p s

( T

i

n fi n i t y , a n d b e y

n

d ! )

i d = d r i v e b i t ma p c

u

n t = f u l l . q c

w

2 i n c . . q c

w

2 i n c . 1 . q c

w

2

SLIDE 50

I n t e r l u d e

SLIDE 51

I n t e r l u d e : T r a n s a c t i

n

s

( J u s t k i d d i n g , w e ’ r e g

n

n a t a l k a b

u

t m

r

e s t u f f )

SLIDE 52

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 5 2

E x p l a i n e r : B l

c

k J

b

s

( J

b

s & T h e E c

n
m

y : R e d u x )

W

h a t a r e j

b

s ?

Q

M P c

m

m a n d s a r e s y n c h r

n
u

s

Q

M P s

c

k e t b l

c

k s

n

e a c h c

m

m a n d

S
w

h a t a b

u

t l

n

g

r

u n n i n g c

m

m a n d s ?

B

l

c

k J

b

s : A s y n c h r

n
u

s t a s k A P I

A

l l

w

s m a n a g e m e n t v i a f u r t h e r Q M P c

m

m a n d s

F
r

m

r

e i n f

:

S e e l i t e r a l l y * a n y t a l k f r

m

K V M F

r

u m 2 1 6

( h a h a h a )

* fi g u r a t i v e l y

SLIDE 53

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 5 3

T r a n s a c t i

n

s

d

e t a i l

( I n c a s e y

u

f

r

g

t

? S

r

r y , t h e r e ’ s a l

t
f

s t u f f . )

T r a n s a c t i

n

s :

A

l l

w

b a t c h i n g

f

c e r t a i n Q M P c

m

m a n d s

E

a c h i n d i v i d u a l i t e m i s a n “ a c t i

n

”

T

r a n s a c t i

n

s u c c e e d s

n

l y i f a l l a c t i

n

s d

S
m

e a c t i

n

s / c

m

m a n d s l a u n c h j

b

s

S
m

e d

n
t

.

W
w

, I h

p

e t h a t d

e

s n ’ t c a u s e a n y p r

b

l e m s .

( O f c

u

r s e i t d i d . )

SLIDE 54

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 5 4

T r a n s a c t i

n

s X J

b

s

( T r a n s a c t i

n

I n t e r a c t i

n

I n t e r s e c t i

n

)

H

w

d

j
b
a

c t i

n

s w

r

k ?

B

e f

r

e 2 . 5 :

A

c t i

n

s u c c e e d s i f j

b

i s s t a r t e d

J
b

s f a i l i n g l a t e r h a v e n

e

f f e c t

n
t

h e r j

b

s

S
m

e b a c k u p s s u c c e e d , s

m

e f a i l

completion_mode=individual

SLIDE 55

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 5 5

T r a n s a c t i

n

s X J

b

s

( T r a n s a c t i

n

I n t e r a c t i

n

I n t e r s e c t i

n

)

H

w

d

j
b
a

c t i

n

s w

r

k ?

A

f t e r 2 . 5 , w i t h completion_mode=grouped …

A

c t i

n

s u c c e e d s i f j

b

i s s t a r t e d

N
c

h a n g e f r

m

‘ i n d i v i d u a l ’ m

d

e

J
b

s c a n n

t

c

m

p l e t e u n t i l a l l j

b

s r e a d y t

O

n e j

b

w i l l c a u s e a l l

t

h e r s t

f

a i l

C

l i e n t s c a n a v

i

d k e e p i n g s t a t e

n

p a r t i a l f a i l u r e s

SLIDE 56

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 5 6

M u l t i d r i v e C

h

e r e n c y

( T r a n s a c t i

n

a c t i

n

s i n a c t i

n

( n

t

t

b

e c

n

f u s e d w i t h i n a c t i

n

)

{ "execute": "transaction", "arguments": { "actions": [ { "type": "drive-backup", "data": { "device": "drive0", "bitmap": "bitmap0", "format": "qcow2", "mode": "existing", "sync": "incremental", "target": "inc0.a.qcow2" } }, { "type": "drive-backup", "data": { "device": "drive1", "bitmap": "bitmap1", "format": "qcow2", "mode": "existing", "sync": "incremental", "target": "inc1.a.qcow2" } }, ] } } { "execute": "transaction", "arguments": { "actions": [ { "type": "drive-backup", "data": { "device": "drive0", "bitmap": "bitmap0", "format": "qcow2", "mode": "existing", "sync": "incremental", "target": "inc0.a.qcow2" } }, { "type": "drive-backup", "data": { "device": "drive1", "bitmap": "bitmap1", "format": "qcow2", "mode": "existing", "sync": "incremental", "target": "inc1.a.qcow2" } }, ] } }

SLIDE 57

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 5 7

M u l t i d r i v e C

h

e r e n c y

( T w i c e a s n i c e ! )

i d = d r i v e b i t ma p c

u

n t = 1 5 8 2 f u l l . q c

w

2 i d = d r i v e 1 b i t ma p 1 c

u

n t = 8 2 5 2 f u l l 1 . q c

w

2

SLIDE 58

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 5 8

M u l t i d r i v e C

h

e r e n c y

( T h r i c e a s n i c e ? )

i d = d r i v e b i t ma p c

u

n t = f u l l . q c

w

2 i n c . a . q c

w

2 i n c 1 . a . q c

w

2 i d = d r i v e 1 b i t ma p 1 c

u

n t = f u l l 1 . q c

w

2

SLIDE 59

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 5 9

M u l t i d r i v e C

h

e r e n c y

( . . . f r i c e ? )

i d = d r i v e b i t ma p c

u

n t = f u l l . q c

w

2 i n c . a . q c

w

2 i n c 1 . a . q c

w

2 i d = d r i v e 1 b i t ma p 1 c

u

n t = f u l l 1 . q c

w

2

t

d r i v e

= t

d r i v e 1

SLIDE 60

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 6

P a r t i a l F a i l u r e s , I n d i v i d u a l

( N

t

m y p r

b

l e m )

i d = d r i v e b i t ma p c

u

n t = 1 5 8 2 f u l l . q c

w

2 i n c . a . q c

w

2 i n c 1 . a . q c

w

2 i d = d r i v e 1 b i t ma p 1 c

u

n t = 8 2 5 2 f u l l 1 . q c

w

2

SLIDE 61

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 6 1

P a r t i a l F a i l u r e s , I n d i v i d u a l

( N

t

m y p r

b

l e m )

i d = d r i v e b i t ma p c

u

n t = f u l l . q c

w

2 i n c . a . q c

w

2 i d = d r i v e 1 b i t ma p 1 c

u

n t = 8 2 5 2 f u l l 1 . q c

w

2

SLIDE 62

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 6 2

P a r t i a l F a i l u r e s , G r

u

p e d

( S t r

n

g e r t

g

e t h e r ? )

i d = d r i v e b i t ma p c

u

n t = 1 5 8 2 f u l l . q c

w

2 i n c . a . q c

w

2 i n c 1 . a . q c

w

2 i d = d r i v e 1 b i t ma p 1 c

u

n t = 8 2 5 2 f u l l 1 . q c

w

2

SLIDE 63

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 6 3

P a r t i a l F a i l u r e s , G r

u

p e d

( S t r

n

g e r t

g

e t h e r ? )

i d = d r i v e b i t ma p c

u

n t = 1 5 8 2 f u l l . q c

w

2 i n c . a . q c

w

2 i n c 1 . a . q c

w

2 i d = d r i v e 1 b i t ma p 1 c

u

n t = 8 2 5 2 f u l l 1 . q c

w

2

SLIDE 64

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 6 4

P a r t i a l F a i l u r e s , G r

u

p e d

( S t r

n

g e r t

g

e t h e r ? )

i d = d r i v e b i t ma p c

u

n t = 1 5 8 2 f u l l . q c

w

2 i d = d r i v e 1 b i t ma p 1 c

u

n t = 8 2 5 2 f u l l 1 . q c

w

2

SLIDE 65

A C T I I I : A D V A N C E D F E A T U R E S

( I n w h i c h

u

r h e r

e

s r i s e a b

v

e )

SLIDE 66

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 6 6

B i t m a p M i g r a t i

n
1

s t

a t t e m p t

( P a c k y

u

r d a t a , w e ' r e m

v

i n g t

<

t a r g e t > )

M

e c h a n i s m s i m i l a r t

d

i s k m i g r a t i

n
D

a t a s p l i t i n t

c

h u n k s ( 1 K i B )

B

i t m a p s s e r i a l i z e d p i e c e

b

y

p

i e c e

F
r

s e t s

f

b i t m a p s b e l

w

1 M i B …

S

k i p t h e l i v e p h a s e a n d c

p

y t h e d a t a w h

l

e s a l e .

6

4 G i B d i s k b i t m a p i s

n

l y 1 2 8 K i B

(

+ n

d

e a n d b i t m a p n a m e s , a n d s t r e a m m e t a d a t a )

SLIDE 67

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 6 7

B i t m a p M i g r a t i

n
1

s t

a t t e m p t

( P a c k y

u

r d a t a , w e ' r e m

v

i n g t

<

t a r g e t > )

B

i t m a p s n

t

t r a n s f e r r e d a l

n

g s i d e d a t a

T

r a n s f e r r e d s e p a r a t e l y f

r

fl e x i b i l i t y

“

m e t a b i t m a p s ” ( d i r t y “ d i r t y b i t m a p ” b i t m a p s ! ? )

C

a p t u r e s c h a n g e s d u r i n g l i v e m i g r a t i

n
P

i e c e s c a n b e r e s e n t i f n e e d e d .

U

s e s v e r y l i t t l e m e m

r

y : 6 4 G i B → 1 6 b y t e s

SLIDE 68

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 6 8

B i t m a p M i g r a t i

n
2

n d

a t t e m p t

( W e ’ r e

n

t h e r

a

d a g a i n . . . )

1

s t

a p p r

a

c h w

r

s e n s c

n

v e r g e n c e p r

b

l e m

M

a y n

t

s c a l e w e l l

N

e w a p p r

a

c h u s e s a p

s

t

c
p

y t e c h n i q u e

S

i m p l y s e n d t h e w h

l

e b i t m a p p

s

t

p

i v

t
R

e c

r

d n e w w r i t e s

n

t a r g e t

P

r

h

i b i t b a c k u p s u n t i l d a t a a r r i v e s

R

e

m

e r g e b i t m a p s

n

t a r g e t

SLIDE 69

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 6 9

B i t m a p M i g r a t i

n
F

a i l u r e s

( M i s s i

n

F a i l e d ! W e ’ l l g e t ‘ e m n e x t t i m e . )

W h a t h a p p e n s i f t h e s

u

r c e d i e s p

s

t

p

i v

t

?

C
n

s i d e r e d n

n
c

r i t i c a l l

s

s

B

i t m a p c h a i n s c a n b e r e

s

t a r t e d

F

u t u r e :

R

e c

n

s t r u c t b i t m a p f r

m

t w

i

m a g e s ?

O

t h e r O p t i

n

s :

U

s e s h a r e d

s

t

r

a g e m i g r a t i

n
W

i t h p e r s i s t e n c e < s t a y t u n e d >

SLIDE 70

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 7

B i t m a p P e r s i s t e n c e – C h a n g e

f

P l a n s

( I h a v e a l t e r e d t h e c

d

e . P r a y I d

n
t

a l t e r i t f u r t h e r . )

P

l a n s w e r e f

r

a f

r

m a t

a

g n

s

t i c f

r

m a t

U

s i n g q c

w

2 t

s

t

r

e b i t m a p s f

r

a r b i t r a r y fi l e s

P

l a n s s c r a p p e d …

N
w

, w e ’ r e t a r g e t i n g q c

w

2

M
r

e

n
t

h e r f

r

m a t s i n a b i t …!

SLIDE 71

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 7 1

B i t m a p P e r s i s t e n c e

( O b j e c t p e r m a n e n c e : n

t

j u s t f

r

t

d

d l e r s )

P

e r s i s t e n c e t a r g e t s t h e q c

w

2 f

r

m a t .

M

u l t i p l e b i t m a p s c a n b e s t

r

e d p e r

fi

l e

B

i t m a p s h a v e ‘ t y p e s , ’ w e u s e a ‘ d i r t y ’ b i t m a p

B

i t m a p s c a n ‘ a u t

l
a

d ’ i n Q E M U

S

p e c a m e n d m e n t i s m e r g e d !

P

a t c h e s r e a d y

n
l

i s t f r

m

V i r t u

z

z

SLIDE 72

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 7 2

B i t m a p P e r s i s t e n c e – N

n

q c

w

2

( A K A , “ C a n I p l e a s e u s e t h i s w i t h r a w ? ” )

W

e h a v e s

m

e

p

t i

n

s f

r
t

h e r f

r

m a t s .

S
m

e f

r

m a t s m a y a d d p r i m a r y s u p p

r

t

V

i r t u

z

z

h

a s e x p r e s s e d i n t e r e s t f

r

p a r a l l e l s

Q

c

w

2 w i t h w r i t e

f
r

w a r d i n g b a c k i n g fi l e s ?

I

n s t e a d

f

r e a d

n

l y

O

f f e r t

f
r

w a r d w r i t e s

A

l l

w

f

r

a n y f

r

m a t

O

t h e r b e n e fi t s

SLIDE 73

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 7 3

“ P u s h M

d

e l ” b a c k u p s

( L e t ’ s t a k e a l l

u

r p r

b

l e m s … a n d p u s h t h e m s

m

e w h e r e e l s e ! )

B a c k u p s d e s c r i b e d s

f

a r a r e “ P u s h ” m

d

e l :

Q

E M U “ p u s h e s ” t h e d a t a t

a

t a r g e t

I

t k n

w

s w h a t s e c t

r

s n e e d t

b

e p u s h e d

T

h i s w

r

k s

u

t p r e t t y O K , b u t …

S
m

e v e n d

r

s w a n t e d a d i f f e r e n t m

d

e l

SLIDE 74

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 7 4

“ P u l l M

d

e l ” b a c k u p s

( s

m

e t i m e s i t ’ s n i c e w h e n d

r

s w

r

k b

t

h w a y s )

T h e “ P u l l m

d

e l ” i s d i f f e r e n t :

Q

E M U

f

f e r s a t e m p

r

a r y , l i g h t w e i g h t s n a p s h

t
“

I m a g e F l e e c i n g ”

E

x p

r

t e d v i a N B D

V

i a N B D e x t e n s i

n

s , c l i e n t q u e r i e s f

r

s t a t u s

C

l i e n t c

n

t r

l

s d a t a fl

w
S

n a p s h

t

i s d e l e t e d

n

c l

s

e

SLIDE 75

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 7 5

“ P u l l M

d

e l ” b a c k u p s

( s

m

e t i m e s i t ’ s n i c e w h e n d

r

s w

r

k b

t

h w a y s )

S

n a p s h

t

v i e w i s p

i

n t

i

n

t

i m e

(

l i k e p u s h m

d

e l )

R

e q u i r e s

n
d

i s k c a c h e

O

f f e r s f u l l c

n

t r

l
n

w h a t i s c

p

i e d

H
w

t h e d a t a i s s t

r

e d i s d e c i d e d b y t h e c l i e n t

M
s

t “ Q E M U

a

g n

s

t i c ” m e t h

d
O

n l y w a y t

q

u e r y d i r t y b l

c

k s

SLIDE 76

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 7 6

T O D O s

( < T O D O : i n s e r t c h e e k y j

k

e > )

Q

M P i n t e r f a c e f

r

“ p u l l ” m

d

e l

Q

M P i n t e r f a c e f

r

m

d

i f y i n g p e r s i s t e n c e a t t r i b u t e s

C

L I t

l

s f

r

v e r i fi c a t i

n

, a n a l y s i s

D

e l e t i

n

/ c l e a n i n g t

l

s

“

O f fl i n e ” i n c r e m e n t a l b a c k u p s u p p

r

t ?

“

f s c k s u p p

r

t ”

q

e m u

i

m g c h e c k

r

( ? )

SLIDE 77

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 7 7

T O D O s

( < T O D O : i n s e r t c h e e k y j

k

e > )

D

a t a i n t e g r i t y

P

e r i

d

i c /

p

p

r

t u n i s t i c fl u s h i n g

G

S O C / O u t r e a c h y 2 1 7 :

R

e f e r e n c e i m p l e m e n t a t i

n
C

L I b a c k u p t

l
P

y t h

n

?

K

e e p y

u

r e y e s p e e l e d :

h

t t p : / / w i k i . q e m u .

r

g / G

g

l e _ S u m m e r _

f

_ C

d

e _ 2 1 7

SLIDE 78

D é n

u

e me n t

( I n w h i c h

u

r h e r

e

s l i v e i n c r e m e n t a l l y e v e r a f t e r )

SLIDE 79

I n c r e m e n t a l B a c k u p s : J

h

n S n

w

; F O S D E M 2 1 7 7 9

P r

j

e c t S t a t u s

( W h e n d

w

e g e t t

u

s e i t ! ? )

b

l

c

k

d

i r t y

b

i t m a p Q M P i n t e r f a c e

s

y n c = i n c r e m e n t a l m

d

e ( p u s h )

T

r a n s a c t i

n

s

Q

c

w

2 P e r s i s t e n c e ( S p e c )

G

r

u

p e d T r a n s a c t i

n

s

M

i g r a t i

n
P

e r s i s t e n c e

P

u l l m

d

e l

M

e r g e d ! ( 2 . 4 )

M

e r g e d ! ( 2 . 4 )

M

e r g e d ! ( 2 . 5 )

M

e r g e d ! ( 2 . 6 )

M

e r g e d ! ( 2 . 8 )

R

e v i e w , ( 2 . 9 )

R

e v i e w , ( 2 . 9 )

S

p e c s , ( 2 . 1 + )

SLIDE 80

Q u e s t i

n

s ?

SLIDE 81

F u r t h e r R e a d i n g :

Q E M U p r

j

e c t w i k i : h t t p : / / q e mu

p

r

j

e c t .

r

g / M a i n _ P a g e B i t ma p s D

c

u me n t a t i

n

:

…/ q e mu / d

c

s / b i t ma p s . md

Q E M U i

t

e s t s : …/ q e mu / t e s t s / q e mu

i
t

e s t s / 1 2 4 P r

j

e c t s t a t u s w h i t e p a p e r ( P D F ) : h t t p : / / g

.

g l / t T 6 n 8 S K V M F

r

u m 2 1 6 ‘ j

b

s ’ t a l k :

h t t p : / / e v e n t s . l i n u x f

u

n d a t i

n

.

r

g / s i t e s / e v e n t s / fi l e s / s l i d e s / k v m2 1 6 _ v 1 6 . p d f

SLIDE 82

T H A N K Y O U ! T H A N K Y O U !

M

r

e q u e s t i

n

s ? j s n

w

@ r e d h a t . c

m

c c : q e mu

d

e v e l @ n

n

g n u .

r