the many flavors Keith Stobie Doyenz (while at Microsoft) Copies - - PowerPoint PPT Presentation

▶
the many flavors
SMART_READER_LITE
LIVE PREVIEW

the many flavors Keith Stobie Doyenz (while at Microsoft) Copies - - PowerPoint PPT Presentation

Nov 19, 2022 305 likes •544 views

Testing In Production the many flavors Keith Stobie Doyenz (while at Microsoft) Copies may not be made or distributed for commercial use Pre-Requisites (risk reduction) Redundancy state quickly recovered if catastrophic failure,

slide-1
SLIDE 1

Testing In Production the many flavors

Keith Stobie Doyenz (while at Microsoft)

Copies may not be made or distributed for commercial use

slide-2
SLIDE 2

Pre-Requisites (risk reduction)

  • Redundancy –

state quickly recovered if catastrophic failure,

  • Data corruption or state-related failures

extremely unlikely (functional testing must first be passing),

  • Errors must be detected and the engineering

team (rather than operations) must be monitoring system health of the code in test

  • Can quickly roll back all changes

Rollback tested before going into production

Testing in Production -- Keith Stobie 2 2011-10-10 PNSQC Copies may not be made or distributed for commercial use

slide-3
SLIDE 3

Isolate Test Data

  • Don’t let test data affect Production metrics.

Can’t bill for test data!

  • Number of users requests must be distinct

from test requests.

  • Is test data visible? Fake products?

Image from : http://blogs.msdn.com/b/seliot/archive/2011/06/07/testing-in-production-tip-it-really-happens-and-that-s-a-good-thing.aspx

Testing in Production -- Keith Stobie 3 2011-10-10 PNSQC Copies may not be made or distributed for commercial use

slide-4
SLIDE 4

Exposure Control

Growing visibility (exposure / risk )

  • Opt-in: Preview, Beta, . . .
  • Google: Internal, Friends&Family, Beta, . . .
  • % of Machines/DCs : 1%, 5%, 10%, 25% 50%
  • Markets: US, Europe,

Asia, America

Testing in Production -- Keith Stobie 4 2011-10-10 PNSQC Copies may not be made or distributed for commercial use

slide-5
SLIDE 5

Asserts == Testing

  • Leave asserts that don’t significantly

impact performance. Assert(inputRef != null) (if not in frequent path)

  • Self-test, like hardware, some software

has low-pri thread for idle times to walk and verify data structures, etc.

Testing in Production -- Keith Stobie 5 2011-10-10 PNSQC Copies may not be made or distributed for commercial use

slide-6
SLIDE 6

Monitoring == Testing

  • Example: In Testbed,

is the response time < threshold(s)?

  • Monitor same thresholds in production:

– External “synthetic” transactions – Real time data collection

  • Transactional availability
  • Feature availability
  • “pure” monitor is a test Oracle

% of requests Threshold 50% 100ms 90% 200ms 99% 300ms

Testing in Production -- Keith Stobie 6 2011-10-10 PNSQC Copies may not be made or distributed for commercial use

slide-7
SLIDE 7

Delta Assertion (Meszaros)

Testing in Production -- Keith Stobie 7 2011-10-10 PNSQC

Image from : http://xunitpatterns.com/Delta%20Assertion.html

Copies may not be made or distributed for commercial use

slide-8
SLIDE 8

Model of a Service

User Requests Service Backend service Frontend service Data Data Feed

Testing in Production -- Keith Stobie 8 2011-10-10 PNSQC Copies may not be made or distributed for commercial use

slide-9
SLIDE 9

Production Environment (Identical Hardware, operations, etc.)

(Offline) Test in Production Environment

Test User Requests Service Backend service Frontend service Data Data Feed

Testing in Production -- Keith Stobie 9 2011-10-10 PNSQC Copies may not be made or distributed for commercial use

slide-10
SLIDE 10

Testing in Production -- Keith Stobie

Fork(copy) Traffic (drop test return)

User Requests Service Backend service Frontend service Data Data Feed Service Backend service Data Data Feed Productio n Test Production Test

10 2011-10-10 PNSQC Copies may not be made or distributed for commercial use

slide-11
SLIDE 11

Experiment with Graphics/Color

  • Which one converts (to search) better?

A B

Source: Marketing Experiments http://www.marketingexperiments.com Testing in Production -- Keith Stobie 11 2011-10-10 PNSQC Copies may not be made or distributed for commercial use

slide-12
SLIDE 12

A/B Test

Visitors Randomly Distributed

50% C T R

Version B is better than Version A

Version A (Control)

1.2% of users with a Page View clicked on Signup 2.8% of users with a Page View clicked on Signup

Version B (Treatment)

50% Is the observed difference statistically significant? C T R YES User interactions instrumented, analyzed and compared

Page Title Signup Here Title of Page Signup Here

Figure courtesy of Seth Elliot from his SASQAG (Seattle Area Software Quality Assurance Group) April 2011 Talk Testing in Production - Your Key to Engaging Customers

Click Thru Rate

Testing in Production -- Keith Stobie 13 2011-10-10 PNSQC Copies may not be made or distributed for commercial use

slide-13
SLIDE 13

Experiments (alternate return)

User Requests Service Backend service Frontend service Data Data Feed Service Production Experiment

Separate experimental service providing the 2nd alternate

  • utput

Testing in Production -- Keith Stobie 14 2011-10-10 PNSQC Copies may not be made or distributed for commercial use

slide-14
SLIDE 14

Experiments (alternate code)

User Requests Service Backend service Frontend service Data Data Feed Service Production Experiment

Service architected internally to have code paths for different alternate

  • utput

Testing in Production -- Keith Stobie 15 2011-10-10 PNSQC Copies may not be made or distributed for commercial use

slide-15
SLIDE 15

Experiments (alternate data)

User Requests Service Backend service Frontend service Data Data Feed Production Experiment Data

Sometimes it is just the data, e.g. different ranking values that are being tried.

Testing in Production -- Keith Stobie 16 2011-10-10 PNSQC Copies may not be made or distributed for commercial use

slide-16
SLIDE 16

Shadowing (compare returns)

Future Environment Production Environment User Requests Service Backend service Frontend service Data Data Feed Service Backend service Data Data Feed

Equally good?

Test Production Next Environ

Testing in Production -- Keith Stobie 17 2011-10-10 PNSQC Copies may not be made or distributed for commercial use

slide-17
SLIDE 17

Fake Traffic (capacity test)

User Requests Service Backend service Frontend service Data Data Feed Production Test replayed Requests

Capacity test a single service, or entire offering (via the frontend)

Testing in Production -- Keith Stobie 18 2011-10-10 PNSQC Copies may not be made or distributed for commercial use

slide-18
SLIDE 18

Modern Cap Generation Toolset

Controller

DC1

Monitoring system Capacity clients Bing Entry point

VI P

DC3

Monitoring system Capacity clients Bing entry point

VI P

DC2

Monitoring system Capacity clients Bing Entry point

VI P

DC…n

Monitoring system Capacity clients Bing entry point

VI P

From: Greg Veith – Workshop on Performance & Reliability 2010 (WOPR14)

Testing in Production -- Keith Stobie 19

  • 1. Controller drives each of the clients across any/all DCs
  • 2. Capacity is generated within each datacenter to avoid cross-DC network

utilization

  • 3. Controller pulls data from monitoring system auto-throttling based on SLA
  • 4. 2 -3 engineers required for a typical full scale test

2011-10-10 PNSQC Copies may not be made or distributed for commercial use

slide-19
SLIDE 19

Tracers (data flow)

User Requests Service Backend service Frontend service Data Data Feed Production Test Data Feed Test Requests

Observe test data as it flows across components and out of the system.

Testing in Production -- Keith Stobie 20 2011-10-10 PNSQC Copies may not be made or distributed for commercial use

slide-20
SLIDE 20

Testing in Production -- Keith Stobie

Product Catalog

Shopping Trace

  • 1. Product search page displays

product when users queries for it?

  • 2. Review search page displays

reviews when users queries for it?

  • 3. Product details page for a given

product contains matched offer and review information? Product Authority Product Data Merchant Offers Self Service Portal Product Matching Aggregator User Serving Crawl Pipeline Test Test Review Test Offer Test Product Test Review in Feed? Offers & Reviews matched to Products? Offers & Reviews counts correct? Test Product in Feed? Production Test Offer in Feed?

21 2011-10-10 PNSQC Copies may not be made or distributed for commercial use

slide-21
SLIDE 21

Data corruption Network latency

Testing in Production -- Keith Stobie

Fault Injection

User Requests Service Backend service Frontend service Data Data Feed Productio n Test Production Test

2011-10-10 PNSQC

Service Loss of Service (or machine or … ) Loss of Service (or machine or … )

Image from : http://behance.vo.llnwd.net/profiles/78972/projects/114549/789721221731305.jpg

Copies may not be made or distributed for commercial use

slide-22
SLIDE 22

Summary

Approach Risk of Causing issues Exposure Control Data Control Asserts Medium Optional

  • Monitoring

Low

  • Offline

Low

  • Controlled

Experiment Medium Required Useful Live Capacity High Useful Required Tracers Medium Useful Useful Fault Injection High Optional Useful

Testing in Production -- Keith Stobie 23 2011-10-10 PNSQC Copies may not be made or distributed for commercial use

slide-23
SLIDE 23

Testing in Production

Testing in Production -- Keith Stobie 24 2011-10-10 PNSQC Copies may not be made or distributed for commercial use