INDUSTRY 4.0 OPEN ARCHITECTURE FOR DATA COLLECTION AND TRANSPORT - - PowerPoint PPT Presentation

INDUSTRY 4.0 – OPEN ARCHITECTURE FOR DATA COLLECTION AND TRANSPORT

Bologna - 17/05/2019

Agenda

› Industry 4.0 – definition and principles › Architecture for data collection and transport for Industry 4.0 › Enabling technologies

▪ Apache Kafka ▪ Docker ▪ Kubernetes

2

Industry 4.0 – definition and principles

3

Industry 4.0 - Definition

“Industry 4.0 is a term applied to a group of rapid transformations in the design, manufacture, operation and service of manufacturing systems and products […] everything in and around a manufacturing operation (suppliers, the plant, distributors, even the product itself) is digitally connected, providing a highly integrated value chain”

http://www.europarl.europa.eu/RegData/etudes/BRIE/2015/568337/EPRS_BRI(2015)568337_EN.pdf

4

Industry 4.0 - Definition

1st Industrial revolution

1784 – mid 19° century

Water and steam-powered mechanical manufacturing

2nd Industrial revolution Late 19° century -1970s

Electric-powered mass production based on the division of labour

3rd Industrial revolution 1970s - Today

Electronics and information technology drives new levels of automation of complex tasks

4th Industrial revolution – Industry 4.0 Today -

Electronics and information technology drives new levels of automation of complex tasks

5 http://www.europarl.europa.eu/RegData/etudes/BRIE/2015/568337/EPRS_BRI(2015)568337_EN.pdf

I4.0 - Cornerstones

The application of information and communication technology (ICT) to digitise information and integrate systems

Cyber-physical systems that use ICTs to monitor and control physical processes and systems. These may involve embedded sensors, intelligent robots that can configure themselves to suit the immediate product to be created, or additive manufacturing (3D printing) devices

Network communications including wireless and internet technologies that serve to link machines, work products, systems and people, both within the manufacturing plant, and with suppliers and distributors Simulation, modelling and virtualisation in the design of products and the establishment of manufacturing processes Collection of vast quantities of data, and their analysis and exploitation, either immediately on the factory floor, or through big data analysis and cloud computing Greater ICT-based support for human workers, including robots, augmented reality and intelligent tools

6

I4.0 - Cornerstones

The application of information and communication technology (ICT) to digitise information and integrate systems

Cyber-physical systems that use ICTs to monitor and control physical processes and systems. These may involve embedded sensors, intelligent robots that can configure themselves to suit the immediate product to be created, or additive manufacturing (3D printing) devices

Network communications including wireless and internet technologies that serve to link machines, work products, systems and people, both within the manufacturing plant, and with suppliers and distributors Simulation, modelling and virtualisation in the design of products and the establishment of manufacturing processes Collection of vast quantities of data, and their analysis and exploitation, either immediately on the factory floor, or through big data analysis and cloud computing Greater ICT-based support for human workers, including robots, augmented reality and intelligent tools

7

Industry 4.0 – Key factors and challenges ▪ Business practices ▪ Corporate standards ▪ Security/safety policies and procedures ▪ Business and Application requirements ▪ Regulatory compliance ▪ Risk management

8

Cornerstones of a joint collaboration

▪ Industry (SACMI)

▪ Business domain expertise (regulations, value stream, etc…) ▪ Cyber-physical systems expertise

▪ Academia (UniBO)

▪ Private/public infrastructure interoperability ▪ Protocol interoperability, efficiency, and performance tuning/evaluation ▪ Digital Twins

▪ Enterprise Architecture (Imola)

▪ Large-scale system design and integration on private/public infrastructures ▪ Software development/delivery process governance ▪ Data Governance and Cybersecurity

9

Architecture for data collection and transport for Industry 4.0

10

Project goal Design and implementation of a data collection and transport platform/architecture for Industry 4.0 scenarios

11

Project requirements ▪ OT principles and regulations should be guaranteed; (Physical) Safety AND Cybersecurity ▪ Heterogeneity: need to accomodate a large number of (mainly legacy) equipments and data formats ▪ Scaling: need to accomodate ever-increasing volumes of connected equipments (new plants) and message rates ▪ Near real-time processing requirements: business/safety-critical actions happen at the equipment level

12

Data collection and transport architecture – logical layers

13

• Pluggable output towards multiple data stores
• Multiple data analysis and reporting platforms

Storage/analysis layer

• Data format modeling and definition
• Data format validation and transformation

Interoperability layer

• Asynchronous, publish-subscribe model
• Scalable message transport

Transport/messaging layer

• HW/SW integration via possibly legacy and/or proprietary

protocols

• Different equipment models and HW/firmware versions

Data collection layer

Data collection and transport architecture – architecture

14

Enabling technologies

Apache Kafka

15

Transport layer – Apache Kafka

Apache Kafka is a streaming platform with three key capabilities

▪ Publish and subscribe to streams of records, similar to a message queue or enterprise messaging system ▪ Store streams of records in a fault-tolerant durable way ▪ Process streams of records as they occur

Apache Kafka typical use cases

▪ Real-time streaming data pipelines used for data aggregation, processing, and transport; ▪ Event-reactive streaming applications used for fraud detection, data validation, email sending confirmation; ▪ Applications for real-time data analytics, stream processing, log aggregation, messaging, audit trail, sync for cooperative nodes.

16

Publish-subscribe model

17

Publisher Subscriber Topic Subscriber Publisher Publisher Subscriber Topic Subscriber Publisher Publisher Subscriber Subscriber

Apache Kafka Topics and Partitions

18

Publisher Consumer group 1 Topic – Football Teams Publisher Publisher

Partition 1 - A-M Team names Partition 2 - P-Z Team names

• ffset:0

Name: Boca Juniors

• ffset:1

Name: Arsenal

• ffset:0

Name: Real Madrid

• ffset:1

Name: Borussia Dortmund

Consumer group 2 Consumer process Consumer process Consumer process

• Partitions and consumer groups allow for

scalability

• Partitions are spread across different

machines

• Partitions split commit log by given criteria
• Partitions preserve message order

Apache Kafka – Brokers and Cluster

19

Publisher Consumer group 1 Publisher Publisher

Partition 2 - P-Z Team names

• ffset:0

Name: Real Madrid

Partition 1 - A-M Team names

• ffset:0

Name: Boca Juniors

• ffset:1

Name: Arsenal

• ffset:1

Name: Borussia Dortmund

Consumer group 2 Consumer process Consumer process Consumer process

BROKER 2 BROKER 1

KAFKA CLUSTER

Apache Kafka – Apache Zookeeper coordination

20

Publisher Consumer group 1 Publisher Publisher

Partition 2 - P-Z Team names

• ffset:0

Name: Real Madrid

Partition 1 - A-M Team names

• ffset:0

Name: Boca Juniors

• ffset:1

Name: Arsenal

• ffset:1

Name: Borussia Dortmund

Consumer group 2 Consumer process Consumer process Consumer process

BROKER 2 BROKER 1

KAFKA CLUSTER

APACHE ZOOKEEPER

Apache Kafka commit log

21

Interoperability layer – Kafka Connect APACHE Kafka Connect – framework to define connectors to stream data into and out of Kafka ▪ A common framework for Kafka connectors ▪ REST interface ▪ Automatic offset management ▪ Distributed and scalable

22

Interoperability layer – Kafka Connect

23

Elasticsearch Sink Connector Kafka Topic PostgreSQL DB Source Connector Amazon S3 Sink Connector

Amazon S3

Enabling technologies

Containers & Docker

24

Infrastructure - Containers

25 PHYSICAL INFRASTRUCTURE HOST OS HYPERVISOR

GUEST OS LIBS/BIN APP GUEST OS LIBS/BIN APP GUEST OS LIBS/BIN APP

PHYSICAL INFRASTRUCTURE HOST OS DOCKER ENGINE

LIBS/BIN APP LIBS/BIN APP LIBS/BIN APP VIRTUAL MACHINE CONTAINER

Infrastructure - Containers

▪ containers include an application/service together with its

dependencies

▪ containers share kernel with other containers ▪ containers run as isolated processes ▪ higher efficiency w/r to virtualization ▪ images are the cornerstone in crafting declarative/automated, easily

repeatable, and scalable services and applications

26

Infrastructure - Docker An open platform for distributed applications for developers and sysadmins Docker allows you to package an application with all of its dependencies into a standardized unit for software development. https://docs.docker.com/engine/

27

What is Docker? ▪Docker consists of:

▪ The Docker Engine - our lightweight and powerful open

source containerization technology combined with a work flow for building and containerizing your applications.

▪ Docker Hub - our SaaS service for sharing and managing

your application stacks.

28

Docker inception

29

▪ 2013: Docker comes to life as an open-source project at dotCloud Inc. ▪ 2014: company changed name to “Docker Inc.” and joined the Linux Foundation ▪ 2015: tremendous increase in popularity ▪ Today: https://blog.docker.com/2018/03/5-years-later-docker-journey/

Docker - Under the hood

▪

Standard Bodies: Open Container Initiative (OCI), Cloud Native Computing Foundation (CNCF)

▪

OCI Image specification

▪

OCI Runtime Specification

▪ runc runtime (formerly libcontainer) ▪ an abstraction/unification layer to decouple Docker from kernel-specific container features (e.g. LXC, libvirt, ...) ▪ The Docker Images: ▪ copy-on-write filesystems (e.g. AUFS) ▪ The Go programming language ▪ a statically typed programming language developed by Google with syntax loosely based on C

30

Docker Architecture

▪ Docker daemon – The Docker daemon listens for Docker API requests and manages Docker objects such as images, containers, networks, and volumes. ▪ Docker client – The Docker client (docker) is the primary way that many Docker users interact with Docker. When you use commands such as docker run, the client sends these commands to the docker daemon, which carries them out. ▪ Docker registries – A Docker registry stores Docker images. Docker Hub and Docker Cloud are public registries that anyone can use, and Docker is configured to look for images on Docker Hub by default. You can even run your own private registry. Docker registries are the distribution component of Docker.

Docker objects

Docker images A Docker image is a read-only template. For example, an image could contain an Ubuntu operating system with Apache and your web application installed. Images are used to create Docker containers. Docker provides a simple way to build new images or update existing images, or you can download Docker images that other people have already created. Docker images are the build component of Docker. Docker containers Docker containers are similar to a directory. A Docker container holds everything that is needed for an application to run. Each container is created from a Docker

• image. Docker containers can be run, started, stopped, moved, and deleted. Each

container is an isolated and secure application platform. Docker containers are the run component of Docker.

Enabling technologies

Kubernetes Container Orchestration Platform

33

Infrastructure orchestration ( Kubernetes, Swarm, Mesos) ▪ Manage the lifecycle of execution environments (containers) in a cluster ▪ Check the state of the containers in the worker nodes ▪ Simplify the implementation of : ▪ High availability (HA) with load balancers ▪ advanced deployment strategies: ▪blue/green deployment, canary release, ... ▪rollback in case of problems ▪ health checks

34

Kubernetes container orchestration platform

Kubernetes is an open source system for managing containerized applications across multiple hosts; providing basic mechanisms for deployment, maintenance, and scaling

• f applications.

Kubernetes builds upon a decade and a half of experience at Google running production workloads at scale using a system called Borg, combined with best-of- breed ideas and practices from the community. Kubernetes is hosted by the Cloud Native Computing Foundation (CNCF) https://github.com/kubernetes/kubernetes

35

Infrastructure - Kubernetes

36

Physical Infrastructure

VM VM VM VM VM VM VM VM Cluster 1 Cluster 2 Container Orchestration Engine

APP1 APP1 Docker runtime Docker runtime Docker runtime Docker runtime Docker runtime APP1 APP2 APP3 APP3

Live demo!

37

Data collection and transport architecture screenshots

38

▪ Più di 20 anni di esperienza nell’Enterprise IT ▪ Consulenza e Skill Transfer su Architetture, Integrazione e Processo ▪ OMG Influence Member, JSR 312 Expert Group, CSI, WWISA, OpenESB Key Partner, NetBeans Strategic Partner

▪ La comunita’ italiana dedicata a Java ▪ 10 anni di articoli, pubblicazioni, libri, eventi, training ▪ Dai programmatori agli architetti ▪ Piu’ di 1.000.000 pagine lette al mese ▪ Business partner in progetti con alto grado di innovazione ▪ Padroni in tecnologie e architetture mobile ▪ Competenti in architetture dell‘informazione, UX e Design

39