The Best of Apache Kafka Architecture Ranganathan Balashanmugam - - PowerPoint PPT Presentation

▶

Oct 10, 2023 173 likes •521 views

Apache: Big Data 2015 The Best of Apache Kafka Architecture Ranganathan Balashanmugam @ran_than Hell Budapest About Me Graduated as Civil Engineer. <dev> 10+ years </dev> <Thoughtworker from=India/>

SLIDE 1

The Best of Apache Kafka Architecture

Ranganathan Balashanmugam @ran_than Apache: Big Data 2015

SLIDE 2

Helló Budapest

SLIDE 3

About Me

❏ Graduated as Civil Engineer. ❏ <dev> 10+ years </dev> ❏ <Thoughtworker from=”India”/> ❏ Organizer of Hyderabad Scalability Meetup with 2000+ members.

SLIDE 4

“Form follows function.”

Louis Sullivan

SLIDE 5

Gravity Dam

Indirasagar Dam, India

img src: http://www.montanhydraulik.in

SLIDE 6

Forces on a gravity dam

Dam weight Head Water Tail Water Uplift

SLIDE 7

❏ publish-subscribe messaging service ❏ distributed commit/write-ahead log “producers produce, consumers consume, in large distributed reliable way -- real time”

SLIDE 8

❏ DBs ❏ Logs ❏ Brokers ❏ HDFS “For highly distributed messages, Kafka stands out.”

Why Kafka?

SLIDE 9

Kafka Vs ________

src: https://softwaremill.com/mqperf/

SLIDE 10

Timeline

2011 2012 2013 2014 2015

Open sourced by LinkedIn, as version 0.6 Graduated from Apache Latest stable - 0.8.2.1 Several Engineers who built Kakfa create Confluent

SLIDE 11

A Kafka Message

CRC attributes key length key message message length message content kafka.message.Message magic

Change requested:KAFKA-2511

SLIDE 12

Producers - push

Kafka Broker

rg.apache.kafka.clients.producer.KafkaProducer

Response => [TopicName [Partition ErrorCode Offset]] Request => RequiredAcks Timeout [TopicName [Partition MessageSetSize MessageSet]]

SLIDE 13

Topic

number of messages time size Remove messages based on kafka.common.Topic

SLIDE 14

Partitions

kafka.cluster.Partition

Serves: Horizontal scaling, Parallel consumer reads

SLIDE 15

Consumers - pull

kafka.consumer.ConsumerConnector, kafka.consumer.SimpleConsumer Consumer 1 Consumer 2

SLIDE 16

Consumer offsets

committing and fetching consumer offsets

img src: http://www.reynanprinting.com/photos/undefined/impresion-offset1.jpg

SLIDE 17

kafka:// - protocol

Metadata
Send
Fetch
Offsets
Offset commit
Offset fetch

“Binary protocol over TCP”

SLIDE 18

Mechanical Sympathy

"The most amazing achievement of the computer software industry is its continuing cancellation of the steady and staggering gains made by the computer hardware industry." - Henry Peteroski

Image source: http://www.theguide2surrey.com

SLIDE 19

Persistence

“Everything is faster till the disk IO.”

SLIDE 20

Disk faster than RAM

src: http://queue.acm.org/detail.cfm?id=1563874

SLIDE 21

Linear Read & Writes

On high level there are only two operations: Append to end of log fetch messages from a partition beginning from a particular message id sequential file I/O

SLIDE 22

“Let us play pictionary”

SLIDE 23

Linux Page Cache

“Kafka ate my RAM”

SLIDE 24

ZeroCopy

src: http://www.ibm.com/developerworks/library/j-zerocopy/

SLIDE 25

Batching

small latency to improve throughput

img src: https://prashanthpanduranga.files.wordpress.com/2015/05/tirupati.jpg

SLIDE 26

Compression

bandwidth is more expensive per-byte to scale than disk I/O, CPU,

r network bandwidth capacity within a facility

kafka.message.CompressionCodec

SLIDE 27

Log compaction

img src: http://kafka.apache.org/083/documentation.html

kafka.log.LogCleaner, LogCleanerManager

SLIDE 28

Message Delivery

Atleast once Atmost once Exactly once

SLIDE 29

Replication

un-replicated = replication factor of one

SLIDE 30

Quorum based

Better latency
To tolerate “f” failures, need “2f+1” replicas

SLIDE 31

Primary-backup replication

Broker 1 Broker 2 Broker 3 Broker 4 Topic 1 Topic 1 Topic 1 Topic 2 Topic 2 Topic 2 Topic 3 Topic 3 Topic 3

SLIDE 32

ZooKeeper

cluster coordinator

SLIDE 33

THANK YOU

For questions or suggestions: Ran.ga.na.than B ranganab@thoughtworks.com @ran_than