SLIDE 1
Microservices From Winning the War …To Keeping the Peace
Andrew McVeigh QConSF October 2018
Background
A bit about Andrew’s work...
◎ Many different domains ◎ Trading & risk systems ◎ PhD on software components ◎ Riot Games ◎ Hulu
2
Microservices From Winning the War To Keeping the Peace Andrew - - PowerPoint PPT Presentation
Microservices From Winning the War To Keeping the Peace Andrew - - PowerPoint PPT Presentation
Microservices From Winning the War To Keeping the Peace Andrew McVeigh QConSF October 2018 Background A bit about Andrews work... Many different domains Trading & risk systems PhD on software components Riot Games
SLIDE 2
SLIDE 3
Microservices Work!
But make sure you prepare for the challenges...
3
SLIDE 4
- 1. Microservices in Gaming
- 2. Microservices for Video
- 3. Believe the Hype
- 4. Believe the Challenges
4
SLIDE 5
1.
Microservices in Gaming
A microservice architecture @ scale
5
SLIDE 6
League of Legends
◎ Here you have a list of items ◎ And some text ◎ But remember not to overload your slides with content Your audience will listen to you or read the content, but won’t do both.
6
SLIDE 7
@ Scale
7
SLIDE 8
Gaming Particulars
◎ Low latency ◎ Need to match up players = shared state ◎ Rapid development cycles ◎ Lots of engineers working on 1 game
8
SLIDE 9
1A.
Winning the War
Evolving from a monolith
9
SLIDE 10
Evolving the Architecture
◎ 2009: Large service monolith
○ Matchmaking, game selection, inventory etc
◎ 2012: Started evolving to microservices
○ http://bit.ly/evolving-league
10
SLIDE 11
Front End Box
League of Legends
11
platform.war
Distributed in-memory cache MySQL Matchmaking Game Server Mgr
platform.war platform.war platform.war platform.war platform.war
MySQL
SLIDE 12
Microservices for New Features
12
Platform
platform.war
Distributed in-memory cache MySQL
Service Proxy / API
MySQL Match Making
tb.jar
Cache
SLIDE 13
REST SWAGGER
Dashboards Service Discovery Config Service
Standard Infrastructure
13
Client of Service
Metrics Client REST library S/W Load Balancer
Service
Metrics Server REST library HTTP Server
SLIDE 14
1B.
Keeping the Peace
Held back by remains of the monolith
14
SLIDE 15
Not Quite Free of the Monolith...
15
Platform
- Inventory
- Stats
- Summoner details
Distributed in-memory cache MySQL MySQL Microservice Cache
SLIDE 16
Ouch - Hitting the Wall
16
SLIDE 17
If We Could Redo?
◎ Decouple state completely
○ Inventory service ○ Catalog service ○ Runes service
◎ Socialize to get prioritization ◎ Simplify infrastructure
○ Config System too “clever” ○ Too much “smarts” in fat libraries
17
SLIDE 18
2.
Microservices for Internet Video
Hundreds of tiny pieces...
18
SLIDE 19
Hulu
◎ 2016: Full microservices architecture
○ Evolved / replaced existing VOD architecture ○ Live TV ○ 20m+ total subs ○ 1m+ live subs
◎ 15 month development!
○ 800+ microservices
http://bit.ly/hulu-landscape
19
SLIDE 20
Video System Particulars
◎ Lots of caching to support browsing ◎ TV show metadata needed everywhere ◎ Real-time playback to support live TV ◎ Lots of integration (billing, ads etc)
20
SLIDE 21
Infrastructure for Microservices
21
Donki PaaS
Apps Github Jenkins Apps Apps
Redis Cluster MySQL Cluster Provisioning System
Apps Apps VMs Load Balancers DNS Statsd Graphite
SLIDE 22
Microservice Ownership
22
Playback team Browse team Recording team Etc...
SLIDE 23
Data Distribution Pattern
23
Master Cache Cache Cache Cache
Ingestion + Admin Reading Pub-sub / read-through
SLIDE 24
Scaling for Launch Day
24
Browse Queries Per Second
College kids falling asleep with XBOX on!
SLIDE 25
2B.
Keeping the Peace
Every issue gets magnified...
25
SLIDE 26
Arch / Ops Dashboard
26
Login Browse & Search Playback Ads DVR … etc Playback
SLIDE 27
http://bit.ly/hulu-scaling
Scaling For Growth
27
SLIDE 28
Circuit Breakers avoid Firestorms
28
A B C X Y X A B C Z
SLIDE 29
Cross-Cutting Requirements
29
SLIDE 30
Cloud versus DC
◎ Could target cloud or DC
○ But no elasticity, must overprovision
◎ Abstracting cloud & DC == Lowest common denominator ◎ Hard to do proper blue-green in DC
30
SLIDE 31
3.
Believe the Hype
The many benefits of microservices
31
SLIDE 32
They Actually Work!
◎ Ownership & Independence ◎ Development velocity ◎ Operational & Development scaling
32
SLIDE 33
They Actually Work!
◎ Granular deployment ◎ Evolution ◎ Organizational alignment
33
SLIDE 34
4.
Believe the Challenges
Standing on the shoulders of constantly improving infrastructure...
34
SLIDE 35
Common CICD + Operating Env
Runtime Infra
35
Build + CICD Operational Infra
Deploy & Rollback Jenkins Spinnaker / Harness Terraform K8s, istio Stackdriver Cloud!
SLIDE 36
Cloud Approach
Preferred approach ◎ Pick one cloud provider (per workflow?) ◎ Consider costs early ◎ Multi-region, multi-account on day 1 Cloud has so many advantages over DC ◎ Elasticity ◎ Easy environments (blue-green) ◎ Better shared services (db, queues etc) ◎
36
SLIDE 37
Circuit Breakers by Default!
37
SLIDE 38
API Gateways
38
Gateway
Service Service
Envoy
API Docs
SLIDE 39
Platforms
For things that need holistic treatment … but still allow self-service ◎ Load testing ◎ Billing ◎ Browse caching ◎ A / B testing ◎ UI layout
39
SLIDE 40
Takeaways
Microservices offer many benefits ◎ Isolation & independence ◎ Granular deployment, scaling & evolution Use infra to protect against common issues ◎ Full CICD ◎ Infrastructure as code ◎ Circuit breakers to prevent firestorms ◎ Istio to help with monitoring + more ◎ Cloud elasticity FTW
40
SLIDE 41
Thanks for Listening!
Any questions?
andrew.mcveigh@gmail.com
41
AMA
Decomposing the Monolith Tues 2:55pm Boardroom C
SLIDE 42
42
SLIDE 43
Scaling For Growth
43