[PPT] - T h e C e p h F S G a t e w a y s S a m b a a PowerPoint Presentation

SLIDE 1

D a v i d D i s s e l d

r

p

d d i s s @s a m b a .

r

g

S u p r i t i S i n g h

s u p r i t i . s i n g h @s u s e . c

m

T h e C e p h F S G a t e w a y s S a m b a a n d N F S

G

a n e s h a

SLIDE 2

A g e n d a

W

h y

– E

x p

r

t i n g C e p h F S

v

e r S a m b a a n d N F S

G

a n e s h a

W

h a t

– A

r c h i t e c t u r e & F e a t u r e s

S

a m b a

N

F S

G

a n e s h a

H
w

– I

n t e r

p

e r a b i l i t y

f

S a m b a a n d N F S

G

a n e s h a

SLIDE 3

C e p h A r c h i t e c u t r e

Image Source: http://docs.ceph.com/docs/giant/architecture/

SLIDE 4

C e p h F S C l i e n t s : S a m b a a n d N F S

G

a n e s h a

SLIDE 5

C e p h F S C l i e n t s : S a m b a a n d N F S

G

a n e s h a

K-Client Samba NFS- Ganesha

SLIDE 6

N F S

G

a n e s h a

SLIDE 7

N F S

G

a n e s h a

O

p e n s

u

r c e

U

s e r s p a c e N F S s e r v e r

S

u p p

r

t s m u l t i p l e F i l e s y s t e m B a c k e n d s :

– C

e p h F S

– R

G W ( R A D O S G a t e w a y )

– G

l u s t e r

– G

P F S

SLIDE 8

N F S

G

a n e s h a

O

p e n s

u

r c e

U

s e r s p a c e N F S s e r v e r

S

u p p

r

t s m u l t i p l e F i l S y s t e m B a c k e n d s :

– CephFS – R

G W ( R A D O S G a t e w a y )

– G

l u s t e r

– G

P F S

SLIDE 9

N F S

G

a n e s h a a n d C e p h F S

librgw

NFS Mount Point Ganesha RADOS Cluster

libcephfs

NFS Mount Point NFS-Ganesha Kernel CephFS Client

SLIDE 10

N F S

G

a n e s h a A r c h i t e c t u r e

Duplicate Request Layer

RPCSEC_GSS RPC Dispatcher Network Channel NFSv3, NFSv4 FileSystem Abstraction Layer(FSAL) MDCACHE CephFS

Admin DBus

Network Channel

Log

SLIDE 11

N F S

G

a n e s h a M

d

u l a r A r c h i t e c t u r e

R

P C l a y e r :

– u

s e s l i b n t i r p c

F

i l e s y s t e m A b s t r a c t i

n

l a y e r ( F S A L )

– P

r

v

i d e s A P I f

r

e x p

r

t e d n a m e s p a c e

M

e t a d a t a c a c h e ( M D C A C H E )

– C

h u n k e d d i r e n t c a c h e ( v e r s i

n

2 . 6 )

D

b u s I n t e r f a c e

– S

y s t e m m a n a g e m e n t a n d c

m

m u n i c a t i

n
L
g

M a n a g e m e n t

– S

u p p

r

t f

r

i n t e r n a l l

g

g i n g

SLIDE 12

N F S

G

a n e s h a k e y f e a t u r e s

S

i n g l e n f s

g

a n e s h a i n s t a n c e c a n s u p p

r

t :

– M

u l t i p l e e x p

r

t s

– M

u l t i p l e f i l e s y s t e m b a c k e n d

– M

u l t i p l e P r

t
c
l

s

R

P C S E C _ G S S w i t h k r b 5 a u t h e n t i c a t i

n
D

y n a m i c a l l y e x p

r

t / u n e x p

r

t e n t r i e s u s i n g D B u s

SLIDE 13

N F S

G

a n e s h a C e p h F S f e a t u r e s

C

e p h x a u t h

r

i z a t i

n
R

e a d d e l e g a t i

n

s

E

x p

r

t s u b d i r e c t

r

i e s

– L

a

d b a l a n c i n g

SLIDE 14

N F S

G

a n e s h a H i g h A v a i l a b i l i t y

P

r

b

l e m : S i n g l e N F S

G

a n e s h a s e r v e r

– S

i n g l e p

i

n t

f

f a i l u r e

– B

t

t l e n e c k

– C

a n n

t

s c a l e w i t h b a c k e n d f i l e s y s t e m .

S
l

u t i

n

: C l u s t e r i n g

– H

i g h a v a i l a b i l i t y

– L

a

d b a l a n c i n g

SLIDE 15

N F S

G

a n e s h a H A : A c t i v e

P

a s s i v e

Pacemaker/Corosync

Virtual IP NFS-Client

mount -t nfs <Virtual_IP> /mnt

SLIDE 16

N F S

G

a n e s h a H A : A c t i v e

P

a s s i v e

Pacemaker/Corosync

Virtual IP NFS-Client

mount -t nfs <Virtual_IP> /mnt

SLIDE 17

N F S

G

a n e s h a H A : A c t i v e

A

c t i v e

Pacemaker/Corosync

NFS-Client

mount -t nfs <Virtual_IP> /mnt

SLIDE 18

S a m b a

SLIDE 19

S a m b a

F

i l e a n d p r i n t s h a r i n g

– S

M B / C I F S , S M B 2 a n d S M B 3 + d i a l e c t s

A

u t h e n t i c a t i

n

– N

T L M v 2 a n d K e r b e r

s
I

d e n t i t y m a p p i n g

– W

i n d

w

s SIDs t

uids

a n d gids

– A

c t i v e D i r e c t

r

y d

m

a i n m e m b e r

r

d

m

a i n c

n

t r

l

l e r

SLIDE 20

P r

t
c
l
S

M B / C I F S

– L

e g a c y d i a l e c t

– H

u n d r e d s

f

c

m

m a n d s a n d s u b c

m

m a n d s

– U

N I X e x t e n s i

n

s

S

M B 2

– C

l e a n b r e a k f r

m
l

d d i a l e c t s

– M

d

e r n , s i m p l i f i e d p r

t
c
l

w i t h i m p r

v

e d p e r f

r

m a n c e

SLIDE 21

P r

t
c
l

( c

n

t i n u e d )

S

M B 2 . 1 → S M B 3 . 1 . 1

– M

s

t r e c e n t p r

t
c
l

r e v i s i

n

s

– L

e a s e i m p r

v

e m e n t s

– R

D M A e x t e n s i

n

s

– M

u l t i c h a n n e l

– W

i t n e s s P r

t
c
l

– E

n d

t
e

n d e n c r y p t i

n

SLIDE 22

C l i e n t s

W

i n d

w

s

– R

e f e r e n c e c l i e n t

– P

r

t
c
l

s p e c i f i c a t i

n

p u b l i s h e r

m

a c O S

– A

F P r e p l a c e d b y S M B 2 a s d e f a u l t f

r

M a v e r i c k s ( 2 1 3 )

L

i n u x

– C

I F S k e r n e l m

d

u l e a n d S a m b a s m b c l i e n t

SLIDE 23

C l u s t e r e d T r i v i a l D a t a b a s e ( C T D B )

S

a m b a p e r s i s t e n t s t a t e s t

r

e d i n T D B k e y

v

a l u e s t

r

e

S

h a r e s

m

e

f

t h i s s t a t e a c r

s

s m u l t i p l e n

d

e s

– C

l u s t e r c

n

s i s t e n t d a t a b a s e

– R

e l i a b l e m e s s a g i n g

H

A f e a t u r e s b

l

t e d

n

– M

n

i t

r

i n g a n d f a i l

v

e r

SLIDE 24

C T D B

N
d

e s p a r t i c i p a t e i n e l e c t i

n
f

r e c

v

e r y m a s t e r

R

e c

v

e r y m a s t e r m

n

i t

r

s s t a t e

f

c l u s t e r

P

e r f

r

m s d a t a b a s e r e c

v

e r y i f n e c e s s a r y

– C

l u s t e r

w

i d e m u t e x u s e d t

p

r e v e n t s p l i t b r a i n

“

T i c k l e ” c l i e n t s

n

I P f a i l

v

e r

SLIDE 25

Samba

Ceph VFS module libcephfs TDB

SLIDE 26

S a m b a C e p h I n t e g r a t i

n
C

e p h F S m

d

u l e f

r

S a m b a : v fs_ceph

– M

a p s S M B f i l e a n d d i r e c t

r

y I / O t

l

i b c e p h f s A P I c a l l s

S

t a t i c c e p h x c r e d e n t i a l s

– R

e g a r d l e s s

f

S a m b a a u t h e n t i c a t e d u s e r

P

O S I X A C L s

SLIDE 27

S a m b a C e p h I n t e g r a t i

n
C

e p h R A D O S c l u s t e r e d m u t e x h e l p e r f

r

C T D B

C

e p h l i b r a d

s

s e r v i c e i n t e g r a t i

n

(coming soon)

SLIDE 28

T e s t i n g

S

a m b a s m b t

r

t u r e

c

i f s . k

f

s t e s t s

I

n t e r

p

e r a b i l i t y

– M

a c O S , W i n d

w

s , H y p e r

V

, e t c .

SLIDE 29

P e r f

r

m a n c e

SLIDE 30

B e n c h m a r k S e t u p

C

e p h s e t u p

n

8 n

d

e s

– 5

O S D n

d

e s – 2 4 c

r

e s – 1 2 8 G B R A M

– 3

M O N / M D S n

d

e s – 2 4 c

r

e s – 1 2 8 G B R A M

– 6

O S D d a e m

n

s p e r n

d

e – B l u e s t

r

e – S S D / N V M E j

u

r n a l s

1

c l i e n t n

d

e s

– 1

6 c

r

e s – 1 6 G B R A M

N

e t w

r

k i n t e r c

n

n e c t

– P

u b l i c n e t w

r

k 1 G b i t / s

– C

l u s t e r n e t w

r

k 1 G b i t / s

SLIDE 31

F I O J

b
R

e a d / w r i t e d a t a t

#

S I Z E f i l e f

r

# T I M E

1

c l i e n t n

d

e s . O n e a c h c l i e n t n

d

e , j

b

s a r e e x e c u t e d

A

s i n g l e j

b

i s

f

t y p e :

– {

n u m b e r _

f

_ w

r

k e r s } r w _ { b l

c

k _ s i z e } _ {

p

} , w h e r e :

N

u m b e r

f

w

r

k e r t h r e a d s :

– 1

, 4 , 8 , 1 6

– E

a c h w

r

k e r t h r e a d c r e a t e s a f i l e

f

1 g s i z e

B

l

c

k S i z e s :

– 4

k , 6 4 k , 1 m , 4 m , 8 m

O

p :

– r

w

SLIDE 32

P e r f

r

m a n c e : N F S

G

a n e s h a v s C e p h F S

B

e n c h m a r k i n g w a s p e r f

r

m e d f

r

:

– N

F S

G

a n e s h a v 2 . 5 . 2

– C

e p h V e r s i

n

1 2 . 2 . 1

– S

i n g l e N F S

G

a n e s h a s e r v e r

– N

F S v e r s i

n

4 .

– R

e a d / W r i t e d a t a 1 G B f i l e f

r

2 m i n s

SLIDE 33

N F S

G

a n e s h a v s C e p h F S K e r n e l c l i e n t : A g g r e g a t e d B / W

v

e r 1 c l i e n t s

SLIDE 34

N F S

G

a n e s h a v s C e p h F S K e r n e l c l i e n t : R e a d / W r i t e r a t i

B

/ W c

m

p a r i s

n

( i n % )

SLIDE 35

O b e s e r v a t i

n

s

F
r

s i n g l e t h r e a d , n f s b / w i s 8 %

f

c e p h f s b / w .

P

e r f

r

m a n c e d e g r a d e s a s n

.
f

t h r e a d s i n c r e a s e

S

i n g l e n f s

g

a n e s h a s e r v e r i s b

t

t l e n e c k .

SLIDE 36

N F S

G

a n e s h a v s C e p h F S : 1 6 t h r e a d s l a t e n c y

NFS-Ganesha CephFS

SLIDE 37

P e r f

r

m a n c e : S a m b a v s C e p h F S

P

r e l i m i n a r y r e s u l t s !

E

n v i r

n

m e n t :

– C

e p h V e r s i

n

1 2 . 2 . 2

– S

a m b a 4 . 6 . 9

T

h r e e S a m b a g a t e w a y s

vfs_ceph

w i t h

p

l

c

k s / l e a s e s d i s a b l e d

N
n
v

e r l a p p i n g s h a r e p a t h s

– L

i n u x c i f s . k

c

l i e n t

4

. 4 k e r n e l w i t h m a n y b a c k p

r

t s

S

M B 3 . m

u

n t

SLIDE 38

SLIDE 39

SLIDE 40

C h a l l e n g e s a n d F u t u r e

SLIDE 41

C h a l l e n g e s

C

r

s

s

p

r

t
c
l

c l i e n t s u p p

r

t

– S

h a r e d ( N F S , C e p h F S ) A C L m

d

e l

R

i c h A C L s

r

P O S I X d r a fu A C L s

– C

h

e r e n t c l i e n t c a c h i n g

M

a p l e a s e s t

C

e p h F S FILE a n d AUTH c a p a b i l i t i e s

U

n i f i e d a u t h e n t i c a t i

n

a n d u s e r m a p p i n g

– U

s e K e r b e r

s

/ A D f

r

S a m b a g a t e w a y a n d c e p h x

SLIDE 42

C h a l l e n g e s

l

i b c e p h f s a s y n c h r

n
u

s I / O

M

u l t i c h a n n e l s u p p

r

t

– E

x p e r i m e n t a l i n u p s t r e a m S a m b a

– N

t

i n t e g r a t e d w i t h C T D B

A

u t

m

a t e d d e p l

y

m e n t

SLIDE 43

C h a l l e n g e s

W

i t n e s s p r

t
c
l

– C

n

t i n u

u

s a v a i l a b i l i t y

f

S M B s h a r e s

– A

d v e r t i s e S a m b a c l u s t e r s t a t e t

c

l i e n t s

– T

r a n s p a r e n t c l i e n t f a i l

v

e r

– L

a

d b a l a n c i n g

SLIDE 44

S a m b a : F u t u r e

C

e p h b a c k e d k e y

v

a l u e s t

r

e f

r

S a m b a

R

e p l a c e

r

m

d

i f y C T D B

– R

c

k s d b ?

– S

a m b a d a t a b a s e A P I d e m a n d i n g

M

u l t i p l e p r

c

e s s e s a n d w r i t e r s

R

e c

r

d l

c

k i n g a n d t r a n s a c t i

n

s

SLIDE 45

N F S

G

a n e s h a : F u t u r e

c

l u s t e r i n g w i t h

u

t L i n u x H A

N

F S v 4 . 1 s u p p

r

t

L

i b r a d

s

s e r v i c e i n t e g r a t i

n

SLIDE 46

R e f e r e n c e s

S

a m b a : h t t p s : / / s a m b a .

r

g /

C

T D B : h t t p s : / / c t d b . s a m b a .

r

g /

S

M B 3 . 1 . 1 e n c r y p t i

n

: h t t p s : / / t e c h n e t . m i c r

s
fu

. c

m

/ e n

u

s / l i b r a r y / d n 5 5 1 3 6 3 ( v = w s . 1 1 ) . a s p x

M

u l t i c h a n n e l d e p l

y

m e n t : h t t p s : / / t e c h n e t . m i c r

s
fu

. c

m

/ e n

u

s / l i b r a r y / d n 6 1 9 8 ( v = w s . 1 1 ) . a s p x

W

i t n e s s P r

t
c
l

: h t t p : / / w w w . s a m b a x p .

r

g / a r c h i v e _ d a t a / S a m b a X P 2 1 5

S

L I D E S / w e d / t r a c k 1 / s a m b a x p 2 1 5

w

e d

t

r a c k 1

G

u e n t h e r _ D e s c h n e r

I

m p l e m e n t i n g T h e W i t n e s s P r

t
c
l

I n S a m b a . p d f

S

a m b a M u l t i c h a n n e l B l

c

k e r B u g : h t t p s : / / b u g z i l l a . s a m b a .

r

g / s h

w

_ b u g . c g i ? i d = 1 1 8 9 7

C

e p h F S c a c h e f l a g s : h t t p s : / / j t l a y t

n

. w

r

d p r e s s . c

m

/ 2 1 6 / 9 / 1 / c e p h f s

a

n d

t

h e

n

f s v 4

c

h a n g e

a

t t r i b u t e /

SLIDE 47

J

i

n U s a t w w w .

p

e n s u s e .

r

g

SLIDE 48

L i c e n s e

T h i s s l i d e d e c k i s l i c e n s e d u n d e r t h e C r e a t i v e C

m

m

n

s A t t r i b u t i

n
S

h a r e A l i k e 4 . I n t e r n a t i

n

a l l i c e n s e . I t c a n b e s h a r e d a n d a d a p t e d f

r

a n y p u r p

s

e ( e v e n c

m

m e r c i a l l y ) a s l

n

g a s A t t r i b u t i

n

i s g i v e n a n d a n y d e r i v a t i v e w

r

k i s d i s t r i b u t e d u n d e r t h e s a m e l i c e n s e . D e t a i l s c a n b e f

u

n d a t h t t p s : / / c r e a t i v e c

m

m

n

s .

r

g / l i c e n s e s / b y

s

a / 4 . /

G e n e r a l D i s c l a i m e r

T h i s d

c

u m e n t i s n

t

t

b

e c

n

s t r u e d a s a p r

m

i s e b y a n y p a r t i c i p a t i n g

r

g a n i s a t i

n

t

d

e v e l

p

, d e l i v e r ,

r

m a r k e t a p r

d

u c t . I t i s n

t

a c

m

m i t m e n t t

d

e l i v e r a n y m a t e r i a l , c

d

e ,

r

f u n c t i

n

a l i t y , a n d s h

u

l d n

t

b e r e l i e d u p

n

i n m a k i n g p u r c h a s i n g d e c i s i

n

s .

p

e n S U S E m a k e s n

r

e p r e s e n t a t i

n

s

r

w a r r a n t i e s w i t h r e s p e c t t

t

h e c

n

t e n t s

f

t h i s d

c

u m e n t , a n d s p e c i f i c a l l y d i s c l a i m s a n y e x p r e s s

r

i m p l i e d w a r r a n t i e s

f

m e r c h a n t a b i l i t y

r

f i t n e s s f

r

a n y p a r t i c u l a r p u r p

s

e . T h e d e v e l

p

m e n t , r e l e a s e , a n d t i m i n g

f

f e a t u r e s

r

f u n c t i

n

a l i t y d e s c r i b e d f

r
p

e n S U S E p r

d

u c t s r e m a i n s a t t h e s

l

e d i s c r e t i

n
f
p

e n S U S E . F u r t h e r ,

p

e n S U S E r e s e r v e s t h e r i g h t t

r

e v i s e t h i s d

c

u m e n t a n d t

m

a k e c h a n g e s t

i

t s c

n

t e n t , a t a n y t i m e , w i t h

u

t

b

l i g a t i

n

t

n
t

i f y a n y p e r s

n
r

e n t i t y

f

s u c h r e v i s i

n

s

r

c h a n g e s . A l l

p

e n S U S E m a r k s r e f e r e n c e d i n t h i s p r e s e n t a t i

n

a r e t r a d e m a r k s

r

r e g i s t e r e d t r a d e m a r k s

f

S U S E L L C , i n t h e U n i t e d S t a t e s a n d

t

h e r c

u

n t r i e s . A l l t h i r d

p

a r t y t r a d e m a r k s a r e t h e p r

p

e r t y

f

t h e i r r e s p e c t i v e

w

n e r s .

C r e d i t s

T e mp l a t e R i c h a r d B r

w

n r b r

w

n @

p

e n s u s e .

r

g D e s i g n & I n s p i r a t i

n
p

e n S U S E D e s i g n T e a m h t t p : / /

p

e n s u s e . g i t h u b . i

/

b r a n d i n g

g

u i d e l i n e s /