T e s t i n g T e c h n i q u e s A p p l i e - - PowerPoint PPT Presentation

SLIDE 1

T e s t i n g T e c h n i q u e s A p p l i e d t

• V

i r t D e v e l

• p

m e n t

C l e b e r R

• s

a S r . S

• f

t w a r e E n g i n e e r O c t . 2 6

T h

, 2 1 7

SLIDE 2

2

A G E N D A

• S
• f

t w a r e T e s t i n g B a s i c s

• E

q u i v a l e n c e P a r t i t i

• n

i n g

• B
• u

n d a r y V a l u e A n a l y s i s

• C
• m

b i n a t

• r

i a l T e s t i n g

• A

p p l y i n g t h e T h e

• r

y

SLIDE 3

3

G l e n f

• r

d J . M y e r s ’ T r i a n g l e C h e c k

• T

a k e s a s i n p u t : l e n g t h s

• f

a t r i a n g l e ’ s s i d e s

• O

u t p u t s t h e t r i a n g l e c l a s s i fi c a t i

• n

:

• E

q u i l a t e r a l

• I

s

• s

c e l e s

• S

c a l e n e

• H
• w

h a r d c a n i t b e t

• w

r i t e a c

• m

p r e h e n s i v e s e t

• f

t e s t c a s e s ?

SLIDE 4

4

T r i a n g l e C h e c k – F i r s t ( N a i v e ) V e r s i

• n

def triangle_check(a, b, c): if a == b == c: return "equilateral" elif a != b != c: return "scalene" else: return "isoceles" def triangle_check(a, b, c): if a == b == c: return "equilateral" elif a != b != c: return "scalene" else: return "isoceles"

SLIDE 5

5

T r i a n g l e C h e c k B a s i c T e s t C a s e s

I N P U T E X P E C T E D O U T C O M E 1 , 1 , 1 E q u i l a t e r a l 2 , 2 , 3 I s

• s

c e l e s 3 , 4 , 5 S c a l e n e

SLIDE 6

6

T r i a n g l e C h e c k – T e s t f

• r

F i r s t ( N a i v e ) V e r s i

• n

class Triangle(Test): def test_equilateral(self): self.assertEqual(triangle_check(1, 1, 1), "equilateral") def test_isosceles(self): self.assertEqual(triangle_check(2, 2, 3), "isosceles") def test_scalene(self): self.assertEqual(triangle_check(3, 4, 5), "scalene") class Triangle(Test): def test_equilateral(self): self.assertEqual(triangle_check(1, 1, 1), "equilateral") def test_isosceles(self): self.assertEqual(triangle_check(2, 2, 3), "isosceles") def test_scalene(self): self.assertEqual(triangle_check(3, 4, 5), "scalene")

SLIDE 7

7

T r i a n g l e C h e c k

• A

n

• t

h e r B a s i c T e s t C a s e

I N P U T E X P E C T E D O U T C O M E 1 , 1 , 1 E q u i l a t e r a l 2 , 2 , 3 I s

• s

c e l e s 2 , 3 , 2 I s

• s

c e l e s 3 , 4 , 5 S c a l e n e

SLIDE 8

8

T r i a n g l e C h e c k – E x t r a T e s t f

• r

F i r s t V e r s i

• n

class Triangle(Test): def test_equilateral(self): self.assertEqual(triangle_check(1, 1, 1), "equilateral") def test_isosceles(self): self.assertEqual(triangle_check(2, 2, 3), "isosceles") self.assertEqual(triangle_check(3, 2, 3), "isosceles") def test_scalene(self): self.assertEqual(triangle_check(3, 4, 5), "scalene") class Triangle(Test): def test_equilateral(self): self.assertEqual(triangle_check(1, 1, 1), "equilateral") def test_isosceles(self): self.assertEqual(triangle_check(2, 2, 3), "isosceles") self.assertEqual(triangle_check(3, 2, 3), "isosceles") def test_scalene(self): self.assertEqual(triangle_check(3, 4, 5), "scalene")

SLIDE 9

9

T r i a n g l e C h e c k E r r

• r

T e s t C a s e s

I N P U T E X P E C T E D O U T C O M E , 1 , 1 E r r

• r
• 1

, 1 , 1 E r r

• r

1 , 1 , 2 E r r

• r

( n

• t

i s

• s

c e l e s ) 1 , 2 , 3 E r r

• r

( n

• t

s c a l e n e )

SLIDE 10

1

T r i a n g l e C h e c k – I n v a l i d I n d i v i d u a l L e n g t h s

class Triangle(Test): ... def test_no_length(self): self.assertEqual(triangle_check(0, 1, 1), "error") self.assertEqual(triangle_check(-1, 1, 1), "error") class Triangle(Test): ... def test_no_length(self): self.assertEqual(triangle_check(0, 1, 1), "error") self.assertEqual(triangle_check(-1, 1, 1), "error")

SLIDE 11

1 1

T r i a n g l e C h e c k – I n v a l i d L e n g t h s f

• r

a T r i a n g l e

class Triangle(Test): ... def test_sum_2_sides_larger_3rd(self): self.assertEqual(triangle_check(1, 1, 2), "error") self.assertEqual(triangle_check(1, 2, 3), "error") class Triangle(Test): ... def test_sum_2_sides_larger_3rd(self): self.assertEqual(triangle_check(1, 1, 2), "error") self.assertEqual(triangle_check(1, 2, 3), "error")

SLIDE 12

1 2

T r i a n g l e C h e c k E x t e n d e d T e s t C a s e s

• P

e r m u t a t i

• n

s

• f

l e n g t h s

• r

d e r

• “

( A + B ) < = C ” . v s . “ ( C + B ) < = A ”

• I

n p u t i s n

• t

a n u m b e r

• G

i v e m e a s i d e w i t h l e n g t h “ π ”

• M
• r

e

• r

l e s s t h a n 3 i n p u t v a l u e s

• A

K A “ w h a t d

• y
• u

m e a n b y t r i a n g l e s m u s t h a v e t h r e e s i d e s ? ”

SLIDE 13

1 3

L e s s

• n

s f r

• m

a s i m p l e e x a m p l e

• E

v e n e x p e r i e n c e d d e v e l

• p

e r s w i l l

• n

l y t h i n k

• f

a s u b s e t

• f

t h

• s

e t e s t c a s e s

• I

f t h e s

• f

t w a r e w e w r i t e i s n

• t

t h i s s i m p l e , w e s h

• u

l d d

• b

e t t e r t h a n i n t h i s e x a m p l e

• C

h

• s

i n g g

• d

i n p u t d a t a i s k e y

• S
• me

i n p u t c a n b e n

• b

e t t e r t h a n

• t

h e r i n p u t a l r e a d y b e i n g u s e d

• N
• t

a l l i n p u t a r e c r e a t e d e q u a l , s

• m

e w i l l h a v e a b e t t e r s h

• t

a t fi n d i n g i s s u e s

SLIDE 14

1 4

E q u i v a l e n c e P a r t i t i

• n

i n g

• D
• n

’ t l e t t h e n a m e s c a r e y

• u
• T

h i n k

• f

g r

• u

p s

• f

i n p u t t h a t s h

• u

l d g e n e r a t e s i m i l a r

• u

t c

• m

e

• A

g

• d

p i c k i s w

• r

t h a t l e a s t

• t

h e r t w

• i

n d i v i d u a l i n p u t s

• I

t u s u a l l y t e l l s u s a b

• u

t w h a t w

• u

l d h a p p e n ( e r r

• r

s ? ) w h e n v a l u e s a b

• v

e

• r

b e y

• n

d i t s e l f w

• u

l d b e u s e d

SLIDE 15

1 5

I d e n t i f y i n g I n p u t T y p e s a n d E q u i v a l e n t C l a s s e s

// snippets from qemu/hw/acpi/cpu_hotplug.c /* The current AML generator can cover the APIC ID range [0..255], * inclusive, for VCPU hotplug. */ QEMU_BUILD_BUG_ON(ACPI_CPU_HOTPLUG_ID_LIMIT > 256); ... if (pcms->apic_id_limit > ACPI_CPU_HOTPLUG_ID_LIMIT) { error_report("max_cpus is too large. APIC ID of last CPU is %u", pcms->apic_id_limit - 1); exit(1); } // snippets from qemu/hw/acpi/cpu_hotplug.c /* The current AML generator can cover the APIC ID range [0..255], * inclusive, for VCPU hotplug. */ QEMU_BUILD_BUG_ON(ACPI_CPU_HOTPLUG_ID_LIMIT > 256); ... if (pcms->apic_id_limit > ACPI_CPU_HOTPLUG_ID_LIMIT) { error_report("max_cpus is too large. APIC ID of last CPU is %u", pcms->apic_id_limit - 1); exit(1); }

SLIDE 16

1 6

I n p u t T y p e s a n d C l a s s e s

N u m b e r

• f

C P U s

I N V A L I D V A L I D I N V A L I D < = 1 . . 2 5 6 > = 2 5 7

C P U I D s

I N V A L I D V A L I D I N V A L I D < =

• 1

. . 2 5 5 > = 2 5 6

SLIDE 17

1 7

B

• u

n d a r y A n a l y s i s

• A

l s

• n
• t

s c a r y

• W

h e n i n p u t c l a s s e s a r e

• r

d e r e d , y

• u

c a n e a s i l y s p

• t

t h e m

• T

h e s e v a l u e s a r e u s u a l l y v e r y g

• d

b e t s f

• r

t e s t s

SLIDE 18

1 8

B

• u

n d a r y V a l u e s

N u m b e r

• f

C P U s

I N V A L I D V A L I D I N V A L I D 1 2 5 6 2 5 7

C P U I D s

I N V A L I D V A L I D I N V A L I D

• 1

2 5 5 2 5 6

SLIDE 19

1 9

B

• u

n d a r y V a l u e s i n T e s t s

// snippets from tp-qemu/qemu/tests/cfg/cpu_add.cfg smp = 4 vcpu_maxcpus = 255 variants:

• cpuid_outof_range:

cpuid_hotplug_vcpu0 = 255 qmp_error_recheck = Unable to add CPU:.*, max allowed:.*

• invalid_vcpuid:

cpuid_hotplug_vcpu0 = -1 qmp_error_recheck = Invalid parameter type.*, expected:.*

• cpuid_already_exist:

cpuid_hotplug_vcpu0 = 1 qmp_error_recheck = Unable to add CPU:.*, it already exists // snippets from tp-qemu/qemu/tests/cfg/cpu_add.cfg smp = 4 vcpu_maxcpus = 255 variants:

• cpuid_outof_range:

cpuid_hotplug_vcpu0 = 255 qmp_error_recheck = Unable to add CPU:.*, max allowed:.*

• invalid_vcpuid:

cpuid_hotplug_vcpu0 = -1 qmp_error_recheck = Invalid parameter type.*, expected:.*

• cpuid_already_exist:

cpuid_hotplug_vcpu0 = 1 qmp_error_recheck = Unable to add CPU:.*, it already exists

SLIDE 20

2

B

• u

n d a r y A n a l y s i s i n “ q e m u

• i

m g b e n c h ”

• “

R u n a s i m p l e s e q u e n t i a l I / O b e n c h m a r k

• n

t h e s p e c i fi e d i m a g e . ”

• “

A t

• t

a l n u m b e r

• f

c

• u

n t I / O r e q u e s t s i s p e r f

• r

m e d ”

SLIDE 21

2 1

q e m u

• i

m g b e n c h – n u m b e r

• f

I / O r e q u e s t s

A c t u a l

I N V A L I D V A L I D

I N V A L I D

• 1

1 I N T _ M A X I N T _ M A X + 1

P e r c e i v e d

I N V A L I D V A L I D

I N V A L I D

1 U I N T _ M A X U I N T _ M A X + 1

SLIDE 22

2 2

C

• m

b i n a t

• r

i a l T e s t i n g

• A

l s

• k

n

• w

n a s “ p a i r

• w

i s e ”

• B

a s i c p r i n c i p l e i s t h a t t h e g e n e r a t e d t e s t c a s e s w i l l h a v e a t l e a s t a p a i r

• f

u n i q u e v a l u e s

• G
• d

v a l u e s c a n u s e E q u i v a l e n t C l a s s e s a n d B

• u

n d a r y A n a l y s i s

• C
• m

b i n a t

• r

i a l w i l l

• p

t i ma l l y t e s t a l l u n i q u e v a l u e s

• n

a s i n g l e t e s t p l a n e x e c u t i

• n

SLIDE 23

2 3

T h e m

• r

e k n

• b

s y

• u

h a v e …

Source: https://en.wikipedia.org/wiki/Cockpit#/media/File:Airbus_A380_cockpit.jpg

SLIDE 24

2 4

C

• m

b i n a t

• r

i a l T e s t i n g R e s u l t s i n t h e I n d u s t r y

Source: https://csrc.nist.gov/Projects/Automated-Combinatorial-Testing-for-Software

SLIDE 25

2 5

q e m u

• i

m g c

• n

v e r t k n

• b

s

[--object objectdef] [--image-opts] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-o options] [-s snapshot_id_or_name] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] filename [filename2 [...]] output_filename [--object objectdef] [--image-opts] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-o options] [-s snapshot_id_or_name] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] filename [filename2 [...]] output_filename

SLIDE 26

2 6

q e m u

• i

m g c

• n

v e r t k n

• b

s – l e t ’ s p i c k s

• m

e

[--object objectdef] [--image-opts] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-o options] [-s snapshot_id_or_name] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] filename [filename2 [...]] output_filename [--object objectdef] [--image-opts] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-o options] [-s snapshot_id_or_name] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] filename [filename2 [...]] output_filename

SLIDE 27

2 7

P i c k i n g a t

• l
• p

i c t

• p

i c t , a s i n “ P a i r w i s e I n d e p e n d e n t C

• m

b i n a t

• r

i a l T e s t i n g ”

• “

P I C T g e n e r a t e s t e s t c a s e s a n d t e s t c

• n

fi g u r a t i

• n

s ”

• “

W i t h P I C T , y

• u

c a n g e n e r a t e t e s t s t h a t a r e m

• r

e e f f e c t i v e t h a n m a n u a l l y g e n e r a t e d t e s t s a n d i n a f r a c t i

• n
• f

t h e t i m e r e q u i r e d b y h a n d s

• n

t e s t c a s e d e s i g n . ”

• h

t t p s : / / g i t h u b . c

• m

/ m i c r

• s
• f

t / p i c t

SLIDE 28

2 8

q e m u

• i

m g c

• n

v e r t p a r a m e t e r fi l e

# Interesting values (at boundaries): -1, 0b, 9223372036854775808b (== 8EiB exbibytes), 8EiB+1b # Values that shows some issues: 0b # Safe values: 2b, 1M, 1G input_size: 2b, 1M, 1G # qemu-img convert parameters fmt: parallels, qcow, qcow2, qed, raw, vdi, vhdx, vmdk, vpc

• utput_fmt: parallels, qcow, qcow2, qed, raw, vdi, vhdx, vmdk, vpc

src_cache: off, writeback, unsafe, writethrough cache: off, writeback, unsafe, writethrough # Interesting values (at boundaries): -1, 0b, 9223372036854775808b (== 8EiB exbibytes), 8EiB+1b # Values that shows some issues: 0b # Safe values: 2b, 1M, 1G input_size: 2b, 1M, 1G # qemu-img convert parameters fmt: parallels, qcow, qcow2, qed, raw, vdi, vhdx, vmdk, vpc

• utput_fmt: parallels, qcow, qcow2, qed, raw, vdi, vhdx, vmdk, vpc

src_cache: off, writeback, unsafe, writethrough cache: off, writeback, unsafe, writethrough

SLIDE 29

2 9

P i c k i n g a t e s t r u n n e r

• a

v

• c

a d

• V

e r s a t i l e t e s t r u n n e r

• L
• n

g t i m e h i s t

• r

y ( h e r i t a g e ) i n v i r t u a l i z a t i

• n

t e s t i n g

• K

V M A u t

• t

e s t , v i r t

• t

e s t , A v

• c

a d

• V

T

• B

u i l t

• i

n s u p p

• r

t f

• r

“ t e s t v a r i a n t s ”

• P

r

• f
• f

c

• n

c e p t i n t e g r a t i

• n

w i t h p i c t :

• h

t t p s : / / g i t h u b . c

• m

/ a v

• c

a d

• f

r a m e w

• r

k / a v

• c

a d

• /

p u l l / 2 5

• M
• r

e i n f

• :
• h

t t p : / / a v

• c

a d

• f

r a m e w

• r

k . g i t h u b . i

• /
• h

t t p s : / / g i t h u b . c

• m

/ a v

• c

a d

• f

r a m e w

• r

k / a v

• c

a d

• h

t t p s : / / a v

• c

a d

• f

r a m e w

• r

k . r e a d t h e d

• c

s . i

SLIDE 30

3

W h a t ’ s N e x t ?

• M

a k e C

• m

b i n a t

• r

i a l T e s t i n g ( a n d

• t

h e r t

• l

s a n d t e c h n i q u e s ) e a s i l y a v a i l a b l e t

• V

i r t d e v e l

• p

e r s

• P

a c k a g i n g p i c t i n f

• r

F e d

• r

a a n d E P E L

• U

p s t r e a m w

• r

k i n Q E M U

• C
• m

b i n a t

• r

i a l T e s t i n g I m p l e m e n t a t i

• n
• M

e r g e t h e p i c t i n t e g r a t i

• n

i n t

• A

v

• c

a d

• ’

s n e x t r e l e a s e

• A

n

• t

h e r I m p l e m e n t a t i

• n

t

• b

e m a d e n a t i v e t

• A

v

• c

a d

• C

u r r e n t l y i n d e v e l

• p

m e n t

• C
• l

l a b

• r

a t i

• n

w i t h t h e C z e c h t e c h n i c a l u n i v e r s i t y i n P r a g u e

• L

e t t h e b e s t s

• l

u t i

• n

w i n !

• S

p e c i a l I n t e r e s t G r

• u

p ?

SLIDE 31

T H A N K Y O U