CIRUS : A Cloud Infrastructure for Real-time Ubilytics (aka - - PowerPoint PPT Presentation
CIRUS : A Cloud Infrastructure for Real-time Ubilytics (aka ubiquitous big data analytics) Didier Donsez Universit de Grenoble LIG / ERODS P.N@imag.fr 1 21/06/14 D. Donsez, CIRUS, EclipseCon 2014 From Processing.org Thanks to Manfred
21/06/14
1
CIRUS : A Cloud Infrastructure for Real-time Ubilytics (aka ubiquitous big data analytics)
Didier Donsez
Université de Grenoble LIG / ERODS P.N@imag.fr
From Processing.org 21/06/14
3
Thanks to Manfred for the introduction of my talk
21/06/14
4
Web X.0 (X > 3) Emerging ICT domains
21/06/14
5
Internet of Things (IoT)
Instrumentation Communication Mediation Decision Action Mining
phones robot RFID / NFC SCADA sensor nodes Industrial IoT (IIoT)
21/06/14
6
21/06/2014 6
Big Picture of Internet of (Every)Things, Data and Services
Home Automation SOHO Smart Public Space Smart Building Industry 4.0 Smart Cities Urban Spaces Geographic Scale Network cell size
WAN MAN LAN WLAN WSN PAN BAN
21/06/14
7
Big Picture of Internet of (Every)Things, data and services
21/06/14
8
Internet(s) of Data, Things and Services
21/06/14
9
What is Cloud Computing ?
– IaaS, PaaS, SaaS – Public, Private, Hybrid, Community, User-Centric,
Souverain
– Virtualization, TCO, Resilience, Elasticity, Energy
efficency, Big Data Analytics …
– Confidentiality (Privacy ,Industrial properties, …) – Souverainety
10
Cloud Services Models
Virtual/Physical Infrastructure (FaaS) Infrastructure as a Service Amazon EC2, ... Platform as a Service Google App Engine, Amazon Hadoop ... Software as a Service Saleforce, Steam, ...
Smart Green Grid
H2
IT cooling
21/06/14
11
Cloud Computing : UbiCloud, Cloud of Things, ...
– Clouds with/for Ubi-terminals (smartphones, tablet
cars, IDS, ...)
– Cloud for Things (data collection and long-term
storage ...)
– Things are facilities in the FaaS
21/06/14
12
Cloud of Things
Virtual/Physical Infrastructure (FaaS) Inrastructure as a Service (IaaS)
(on-demand cpu/storage/nw infrastructures)
Platform as a Service (PaaS)
(on-demand platform of delivering your own application)
Software as a Service (SaaS)
(on-demand access to any applications)
Smart Green Grid
H2
IT cooling
Deltadrone, Cloud Robotics Xively, Axeda, Eurotech
21/06/14
13
What is Big Data (Analytics) ?
21/06/14
14
Big data is like teenage sex
about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it…”
Duke University, TED speaker
21/06/14
15
The Data Deluge and the next IoT Data Deluge
http://www.snia.org/sites/default/files2/ABDS2012/Tutorials/RobPeglar_Introduction_Analytics%20_Big%20Data_Hadoop.pdf1ZB=10^12 GB
21/06/14
16
The 4+1V of Big Data
Volume Velocity Variety Veracity + Value
21/06/14
17
IoT Big Data 5V
Volume Velocity Variety Veracity Value
21/06/14
18
Big Data and IoT
21/06/14
19
How analyze the Iot Data Deluge ?
→ Business Intelligence tools (OLTP, OLAP, …) can't ! Now computing models are avalaible
rethink
21/06/14
20
Computing Models for Big Data
– TBs / PBs of stored data
→ Map Reduce
–
Hadoop, SciDB, Spark, Giraph, ...
– 100 MB/s of live data
Decision Support → Event Stream Processing
– Storm, S4, Samza, Millwheel, ..
Processing
21
Event Stream Processing
flows of events (sensors data, ...)
– Low-latency (few millisec after)
mutable state node 1 input records node 2 input records mutable state node 3 node 5 Node 4
records
22
Example : Event Stream Processing
Trending Topics
VoIPSTREAM (VS) Twitter Sentiment Analysis (SA)
From Maycon Bordin's ms thesis
Lambda Architecture
Nathan Marz (Twitter, Backtype)
Batch layer can compute the analytics model
A n a l y t i c a l M
e l
21/06/14
24
Ubilytics Ubiquitous Big Data Analytics
– PaaS for « Ubilytics »
– End-to-End
– « Simple as Possible »
– Huge variety of needs – Huge variety of technologies
New trendy Job : IoT Data Scientist
http://nirvacana.com/thoughts/becoming-a-data-scientist/
Gartner says big data creates big jobs: 4.4 million IT jobs globally to support big d a t a b y 2 0 1 5 . http://www.gartner.com/newsroom/id/22 07915 The U.S. could face a shortage by 2018
analytical talent" and of 1.5 million people capable of analyzing data in ways that enable business decisions. (McKinsey & Co) Big Data industry is worth more than $100 billion growing at almost 10% a year (roughly twice as fast as the software business)
How can this guy deal with this deluge of technologies ? How to make this guy productive ?
21/06/14
26
Who is able to develop/ deploy Ublilytics infrastructures ? IoT Data Scientist
Cloud Computing Big Data Analytics Internet
21/06/14
27
Towards "Ubilytics" PaaSs
Message Broker
Mosquitto, RabbitMQ, … Protocols : MQTT, AMQP, STOMP, XMPP, CoAP, WebRTC, Motwin ...
Message Broker
Mosquitto, RabbitMQ, … Protocols : MQTT, AMQP, STOMP, XMPP, CoAP, WebRTC, Motwin ...
SmartPhone
@ Car, City, …M2M Gateway
@ Home, Office, City, Warehouse (OpenHAB, IoTSys)Sensors data messages ie energy Consumption, temperature, images , ... Send selected sensors meausrements
Embedded boards smartphones by millions
In elastic Cloud
Realtime ESP Storm, Samza, S4, Spark Streaming, … topologies MQTT, AMQP, STOMP ... Monitoring Placement (static,dynamic)
Deployment & (Re)Configuration (Roboconf) MapReduce
Hadoop
NoSQL Store MongoDB, Cassandra, HDFS
In Elastic Hybrid Cloud
Computed prediction model
Predictions Trends, ...
Mashup, Reporting, Dashboard, . (history charts, ...)
Storing agregates
28
Ubilytics Example Energy Consumption Forecast
Domain : Smart Grid
Sensor Dataset*:
Challenge* : Forecasts loads at 1min, 5min, 15min, 60min and 120min Goal : anticipate electricity demand for ajusting the production (ie. save energy and avoid blackout)
* DEBS GC 2014 http://www.cse.iitb.ac.in/debs2014/?page_id=42#
29
Ubilytics Example Energy Consumption Forecast
instant load (W) and cumulative load (kW)
MQTT Broker + Storm Topology + Cassandra DB
21/06/14
30
Smart Campus (ie Small Smart City)
OpenHAB, Galileo, MQTT, Storm, Azure, ...
21/06/14
31
Smart Campus
21/06/14
32
Smart Campus
21/06/14
33
SmartCampus
21/06/14
34
Smart Campus
21/06/14
35