Distributed Logging Architecture in Container Era LinuxCon Japan - - PowerPoint PPT Presentation

Distributed Logging Architecture in Container Era

LinuxCon Japan 2016 at Jun 13 2016 Satoshi "Moris" Tagomori (@tagomoris)

Satoshi "Moris" Tagomori (@tagomoris) Fluentd, MessagePack-Ruby, Norikra, ... Treasure Data, Inc.

Topics

• Microservices and logging in various industries
• Difficulties of logging with containers
• Distributed logging architecture
• Patterns of distributed logging architecture
• Case Study: Docket and Fluentd
• Why OSS are important for logging

Logging

Logging in Various Industries

• Web access logs
• Views/visitors on media
• Views/clicks on Ads
• Commercial transactions (EC, Game, ...)
• Data from devices
• Operation logs on Apps of phones
• Various sensor data

Microservices and Logging

• Monolithic service
• a service produces all data

about an users access

• Microservices
• many services produce data

about an users access

• it's needed to collect logs

from many services to know what is happening

Users Service (Application) Logs Users Logs

Logging and Containers

Containers: "a must" for microservices

• Dividing a service into services
• a service requires less computing resources


(VM -> containers)

• Making services independent from each other
• but it is very difficult :(
• some dependency must be solved even in

development environment
 (containers on desktop)

Redesign Logging: Why?

• No permanent storages
• No fixed physical/network address
• No fixed mapping between servers and roles

Containers: immutable & disposable

• No permanent storages
• Where to write logs?
• file in container


→ be gone w/ container instance 😟

• dir shared from host


→ hosts are shared by many services ☹

• TODO: ship logs from container to anywhere ASAP

Containers: unfixed addresses

• No fixed physical / network address
• Where should we go to fetch logs?
• Service discovery (e.g., consul)


→ one more component 😟

• rsync? ssh+tail? or ..? Is it installed in container?


→ one more tool to depend on ☹

• TODO: push logs to anywhere from containers

Containers: instances per roles

• No fixed mapping between servers and roles
• How can we parse / store these logs?
• Central repository about log syntax


→ very hard to maintain 😟

• Label logs by source address


→ many containers/roles in a host ☹

• TODO: label & parse logs at source of logs

Distributed Logging Architecture

Core Architecture

• Collector nodes
• Aggregator nodes
• Destination

Collector nodes (Docker containers + agent) Destination
 (Storage, Database, ...) Aggregator nodes

• Parse (collector)
• Raw logs are not good for processing
• Convert logs to structured data (key-value pairs)
• Sort/Shuffle (aggregator)
• Mixed logs are not good for scanning
• Split whole data stream into streams
• Store (destination)
• Format logs(records) as destination expects

Collecting and Storing Data

Scaling Logging

• Network traffic
• CPU load to parse / format
• Parse logs on each collector (distributed)
• Format logs on aggregator (to be distributed)
• Capability
• Make aggregators redundant
• Controlling delay

Patterns

source aggregation NO source aggregation YES destination aggregation NO destination aggregation YES

Aggregation Patterns

Source Side Aggregation Patterns

w/o source aggregation

w/ source aggregation

collector aggregator aggregate container

Without Source Aggregation

• Pros:
• Simple configuration
• Cons:
• fixed aggregator (endpoint) address
• many network connections
• high load in aggregator

collector aggregator

With Source Aggregation

• Pros:
• less connections
• lower load in aggregator
• less configuration in containers


(by specifying localhost)

• highly flexible configuration


(by deployment only for aggregate containers)

• Cons:
• a bit much resource (+1 container per host)

aggregate container

Destination Side Aggregation Patterns

w/o destination aggregation

w/ destination aggregation

aggregator collector destination

Without Destination Aggregation

• Pros:
• Less nodes
• Simpler configuration
• Cons:
• Storage side change affects collector side
• Worse performance: many small write requests
• n storage

With Destination Aggregation

• Pros:
• Collector side configuration is


free from storage side changes

• Better performance with fine tune

• n destination side aggregator
• Cons:
• More nodes
• A bit complex configuration

aggregator

Scaling Patterns

Scaling Up Endpoints HTTP/TCP load balancer Huge queue + workers Scaling Out Endpoints Round-robin clients

Load balancer Backend nodes Collector nodes Aggregator nodes

Scaling Up Endpoints

• Pros:
• Simple configuration


in collector nodes

• Cons:
• Scaling up limit

Load balancer Backend nodes

Scaling Out Endpoints

• Pros:
• Unlimited scaling


by adding aggregator nodes

• Cons:
• Complex configuration
• Client features for round-robin

Without
 Destination Aggregation With
 Destination Aggregation Scaling Up Endpoints Systems in early stages Collecting logs over Internet

• r

Using queues Scaling Out Endpoints Impossible :( Collector nodes must know all endpoints ↓ Uncontrollable Collecting logs in datacenter

Case Studies

Case Study: Docker+Fluentd

• Destination aggregation + scaling up
• Fluent logger + Fluentd
• Source aggregation + scaling up
• Docker json logger + Fluentd + Elasticsearch
• Docker fluentd logger + Fluentd + Kafka
• Source/Destination aggregation + scaling out
• Docker fluentd logger + Fluentd

Why Fluentd?

• Docker Fluentd logging driver
• Docker container can send logs into Fluentd

directly - less overhead

• Pluggable architecture
• Various destination systems
• Small memory footprint
• Source aggregation requires +1 container per host
• Less additional resource usage ( < 100MB )

Destination aggregation + scaling up

• Sending logs directly over TCP by Fluentd logger


in application code

• Same with patterns of New Relic

Application code

Source aggregation + scaling up

• Kubernetes: Json logger + Fluentd + Elasticsearch
• Applications write logs to STDOUT
• Docker writes logs as JSON in files
• Fluentd


reads logs from file
 parse JSON objects
 writes logs to Elasticsearch

http://kubernetes.io/docs/getting-started-guides/logging-elasticsearch/ Elasticsearch Application code

Files (JSON)

Source aggregation + scaling up

• Docker fluentd logging driver + Fluentd + Kafka
• Applications write logs to STDOUT
• Docker sends logs


to localhost Fluentd

• Fluentd


gets logs over TCP
 pushes logs into Kafka

Kafka Application code

Source/Destination aggregation + scaling out

• Docker fluentd logging driver + Fluentd
• Applications write logs to STDOUT
• Docker sends logs


to localhost Fluentd

• Fluentd


gets logs over TCP
 sends logs into Aggregator Fluentd
 w/ round-robin load balance

Application code

What's the Best?

• Writing logs from containers: Some way to do it
• Docker logging driver
• Write logs on files + read/parse it
• Send logs from apps directly
• Keep it scalable!
• Source aggregation: Fluentd on localhost
• Scalable storage: (Kafka, external services, ...)
• No destination aggregation + Scaling up
• Non-scalable storage: (Filesystems, RDBMSs, ...)
• Destination aggregation + Scaling out

Why OSS Are Important For Logging?

Why OSS?

• Logging layer is interface
• transparency
• interoperability
• Keep it scalable
• number of nodes
• number of types of source/destination

Use OSS, Make Logging Scalable