SLIDE 1 Monitoring Modern Architectures with Data Science
QCon 2017 Dave Casper, CTO
SLIDE 2 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
Much has changed since simple distributed client/server architectures and so-too have the technologies and industry practices around monitoring. Cloud-Native, DevOps, blue/green deployments, server-less, edge/fog, IoT all fit into a world much better handled by the emerging Artificial Intelligence for IT Operations domain more-so than traditional ITIL/SDLC approaches.
Abstract
SLIDE 3 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
Software continues to eat the world. Software automates, defines. The world is "going digital" and it's quite exciting -- but this always-connected from-everything-to-everywhere world adds complexity to software systems and this talk will dive in to some of that complexity and how modern data science and algorithms are being applied to "fight machines with machines," so to speak.
Abstract
SLIDE 4 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
623992118 1343963318
SLIDE 5 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
SLIDE 6 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
moogsoft
SLIDE 7 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
SLIDE 8 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
discovery monitoring
(observing)
analytics
SLIDE 9 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
fluid infrastructure
containers dc/os server-less software defined/dynamic
SLIDE 10 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
anything anywhere anytime
data/tx from
mobile IoT bots/RUM
SLIDE 11 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
SLIDE 12 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
"if/else" rules algorithms ML millions millions noise filt. clustering prc deja vu
SLIDE 13 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
AIOps AI for IT Ops
SLIDE 14 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
customer/ business perspective
SLIDE 15 COURAGE INSIGHT CONTEXT VELOCITY
ARE YOU READY TO GO DIGITAL ?
This slide courtesy Andy Brown, Sandhill East https://www.linkedin.com/in/andybrown63/
SLIDE 16 “Silicon Valley is coming. There are hundreds of startups with a lot
- f brains and money working on various
alternatives to traditional banking. They are very good at reducing the ‘pain points’ …”
JAMIE DIMON
JPMorgan Chase & Co. Chairman & Chief Executive Officer April 2015
This slide courtesy Andy Brown, Sandhill East https://www.linkedin.com/in/andybrown63/
SLIDE 17 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
go digital
SLIDE 18 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
gs wants to become "google
SLIDE 19 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
stanford PhD CIO CFO marquee data analytics data api api monitor
analyze
SLIDE 20 THE REALLY BIG PICTURE
2020 2021 2022 2023 2024
In 5 – 10 years, every company will be a Digital Software Business Security, service assurance and consumer centricity become THE BOARD LEVEL PRIORITY Enterprises going DIGITAL ADOPT HYBRID IT
This slide courtesy Andy Brown, Sandhill East https://www.linkedin.com/in/andybrown63/
SLIDE 21 40% Change 60% Run Infrastructure Led Owns Facilities, Data Centers, Hardware, Networks et al Has Refresh Cycles caused by Capital Depreciation Still using Waterfall for App Dev Thinking led by Inf Technologists (hardware, DB, OS et al)\ Traditional Procurement Less Agile, Change resistant 60% Change 40% Run AppDev starting to lead Owns less Facilities, Data Centers, Hardware, Networks, et al Still Has Refresh Cycles caused by CapitalDepreciation Combination Waterfall & Agilefor App Dev Thinking led by CIO “Move to Cloud” Traditional Procurement weakening More Agile, Less Change resistant 80% Change 20% Run AppDev leads decisioning Doesn’t own hardware Refresh doesn’t exist All Agile for App Dev Thinking led by CIO “Move to Cloud” Cloud Centric “Marketplace” Procurement Embraces Change, Very Agile
traditional hybrid digital
This slide courtesy Andy Brown, Sandhill East https://www.linkedin.com/in/andybrown63/
SLIDE 22 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
2045 ?
SLIDE 23 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
SNMP / traps
Daylight Savings
SLIDE 24 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
AMRS EMEA APAC EPS every ip interface globally
AIOps EdgeOps EdgeOps EdgeOps
SLIDE 25 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
algorithms we use
SLIDE 26 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
This slide courtesy our Chief Scientist Dr. Rob Harper -- Do check out his great 3-part blog on Machine Learning in Moogsoft AIOps: https://www.moogsoft.com/author/robharper/
SLIDE 27 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
By Hui Li on Subconscious Musings April 12, 2017
SLIDE 28 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
This slide courtesy our Chief Scientist Dr. Rob Harper -- Do check out his great 3-part blog on Machine Learning in Moogsoft AIOps: https://www.moogsoft.com/author/robharper/
SLIDE 29 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
This slide courtesy our Chief Scientist Dr. Rob Harper -- Do check out his great 3-part blog on Machine Learning in Moogsoft AIOps: https://www.moogsoft.com/author/robharper/
SLIDE 30 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
regression classification clustering
This slide courtesy our Chief Scientist Dr. Rob Harper -- Do check out his great 3-part blog on Machine Learning in Moogsoft AIOps: https://www.moogsoft.com/author/robharper/
SLIDE 31 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
classification supervised
“learn by example” approach. Supervised learning systems need to be given examples of what is “good” and what is “bad”
This slide courtesy our Chief Scientist Dr. Rob Harper -- Do check out his great 3-part blog on Machine Learning in Moogsoft AIOps: https://www.moogsoft.com/author/robharper/
SLIDE 32 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
classification
SLIDE 33 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
clustering unsupervised
Patterns that you didn’t know existed prior. Recommender systems rely heavily on these techniques.
SLIDE 34 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
supervised machine learning "hot dog?" "not hot dog?"
SLIDE 35 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
SLIDE 36 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
algorithms we use
SLIDE 37 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
lua code: https://pastebin.com/ZZmSNaHX
SethBling mar i/o neural nets
https://www.youtube.com/watch?v=qv6UVOQ0F44
SLIDE 38 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
k-means clustering
SLIDE 39 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
matrix factorization
SLIDE 40 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
shannon entropy
SLIDE 41 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
typical entropy distribution
SLIDE 42 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
algorithmic workflow
SLIDE 43 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
millions events de-duplication cluster analysis algorithms situation room teams-centric thousands of alerts algorithmic noise filtering [shannon entropy] tens of alert clusters (situations) "today's warnings are tomorrows outages" "all about the MTTR" algorithmic probable root cause AIOps
Algorithmic IT Operations
knowledge capture auto-recurrance detect
entropy_threshold
non-noisy alerts
what you're likely doing today
L1 "Catch & Dispatch" (automated) ignore situation next steps
SLIDE 44 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
...speaking of classification
SLIDE 45 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
fault vs audit
fix → optimize
SLIDE 46 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
monitoring fail-around
analytics analytics fail-around fail-around monitor
SLIDE 47 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
weld the datacenter doors shut
SLIDE 48 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
<lofty_tangent>
non-technical
SLIDE 49
THE INDUSTRIAL REVOLUTION 1.0
SLIDE 50
SLIDE 51
SLIDE 52
SLIDE 53
WE FACE EXISTENTIAL THREATS TO OUR PROGRESS
SLIDE 54
SLIDE 55
THE INDUSTRIAL REVOLUTION 2.0 CAN HELP SAVE US
SLIDE 56
v1.0 v2.0
SLIDE 57
COMPLEXITY IS THE PRINCIPAL THREAT TO THE REVOLUTION
SLIDE 58 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
</lofty_tangent>
non-technical
SLIDE 59 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
SLIDE 60 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
SLIDE 61 NOT FOR GENERAL DISTRIBUTION ONLY INTENDED FOR REGISTERED ATTENDEES OF QCON SF 2017
theory applied sharing | giving back
SLIDE 62
