libTSDB Goutham Veeramachaneni Student @ IIT Hyderabad, India - - PowerPoint PPT Presentation

▶
libtsdb
SMART_READER_LITE
LIVE PREVIEW

libTSDB Goutham Veeramachaneni Student @ IIT Hyderabad, India - - PowerPoint PPT Presentation

Nov 12, 2023 711 likes •1.55k views

libTSDB Goutham Veeramachaneni Student @ IIT Hyderabad, India ex-intern @ CoreOS putadent gouthamve TSDB: github.com/prometheus/tsdb Prometheus 2.0s storage engine A lib. vendored by Prometheus. Why?

slide-1
SLIDE 1

libTSDB

Goutham Veeramachaneni Student @ IIT Hyderabad, India ex-intern @ CoreOS putadent gouthamve

slide-2
SLIDE 2

TSDB: github.com/prometheus/tsdb

  • Prometheus 2.0’s storage engine
  • A lib. vendored by Prometheus.
slide-3
SLIDE 3

Why?

  • Time-series is everywhere!
  • A nice API for large datasets
  • Awesome compression: 1 Billion points in ~1.2GB
slide-4
SLIDE 4

Simple use-case: Prometheus with PUSH!

  • Lots of requests.
  • Several people (including me) built “aggregators” which expose

push data to Prometheus.

  • Let’s build a native Prometheus server with push functionality!
slide-5
SLIDE 5

Introducing PromFlux

  • Ingest using Influx line protocol - pre-built client libs!
  • Query using PromQL <3
slide-6
SLIDE 6

Introducing PromFlux

  • Ingest using Influx line protocol - pre-built client libs!
  • Query using PromQL <3
slide-7
SLIDE 7

Umm, NO

  • This is not how Prometheus works.
  • These stunts are performed by an amateur, don't try this in

production.

slide-8
SLIDE 8

Umm, NO

  • This is not how Prometheus works.
  • These stunts are performed by an amateur, don't try this in

production.

slide-9
SLIDE 9

Umm, NO

  • This is not how Prometheus works.
  • These stunts are performed by an amateur, don't try this in

production.

slide-10
SLIDE 10

YOLO!

slide-11
SLIDE 11

Some basics

  • A time series:

(t0, v0), (t1, v1), (t2, v2), (t3, v3), ....

slide-12
SLIDE 12

Some basics

time series

slide-13
SLIDE 13

Some basics

requests_total{path="/status", method="GET", instance="10.0.0.1:80"} requests_total{path="/status", method="POST", instance="10.0.0.3:80"} requests_total{path="/", method="GET", instance="10.0.0.2:80"} ...

slide-14
SLIDE 14

Some basics

{ __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, }

slide-15
SLIDE 15

Some basics

{ __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, }

{ name=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, }

slide-16
SLIDE 16

Some basics

{ __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, }

{ pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, }

slide-17
SLIDE 17

Some basics

requests_total{path="/status", method="GET", instance="10.0.0.1:80"}

slide-18
SLIDE 18

Some basics

{name="requests_total", path="/status", method="GET", instance="10.0.0.1:80"} requests_total{path="/status", method="GET", instance="10.0.0.1:80"}

slide-19
SLIDE 19

Some basics

requests_total{path="/status", instance="10.0.0.1:80"} requests_total{path="/status", instance="10.0.0.3:80"} requests_total{path="/", instance="10.0.0.2:80"}

Select: requests_total

slide-20
SLIDE 20

{name="requests_total", path="/status", instance="10.0.0.1:80"} {name="requests_total", path="/status", instance="10.0.0.3:80"} {name="requests_total", path="/", instance="10.0.0.2:80"}

Select: {name="requests_total"}

Some basics

slide-21
SLIDE 21

Some basics

requests_total{path="/status", instance="10.0.0.1:80"} requests_total{path="/status", instance="10.0.0.3:80"} requests_total{path="/", instance="10.0.0.2:80"}

Select: requests_total{path="/status"}

slide-22
SLIDE 22

Some basics

{name="requests_total", path="/status", instance="10.0.0.1:80"} {name="requests_total", path="/status", instance="10.0.0.3:80"} {name="requests_total", path="/", instance="10.0.0.2:80"}

Select: {name="requests_total", path="/status”}

slide-23
SLIDE 23

Line Protocol (simplified)

cpu,host=server01,region=uswest value=1 cpu,host=server02,region=uswest value=3 {name=”cpu”, host=”server01”, region=”uswest”} 1 {name=”cpu”, host=”server02”, region=”uswest”} 3

slide-24
SLIDE 24

Code

slide-25
SLIDE 25

Creation

slide-26
SLIDE 26

Creation

func Open(dir string, l log.Logger, r prometheus.Registerer, opts *Options) (*DB, error)

type Options struct { // The interval at which the write ahead log is flushed to disc. WALFlushInterval time.Duration // Duration of persisted data to keep in milliseconds. RetentionDuration uint64 // The sizes of the Blocks in milliseconds. BlockRanges []int64 }

slide-27
SLIDE 27

Code

slide-28
SLIDE 28

Insertion

slide-29
SLIDE 29

Insertion

func (db *DB) Appender() Appender type Appender interface { Add(series labels.Labels, t int64, v float64) (ref string, err error) // Add adds a sample pair for the referenced series. It is generally faster // than adding a sample by providing its full label set. AddFast(ref string, t int64, v float64) error // Commit submits the collected samples and purges the batch. Commit() error // Rollback rolls back all modifications made in the appender so far. Rollback() error }

slide-30
SLIDE 30

Insertion

func (db *DB) Appender() Appender type Appender interface { Add(series labels.Labels, t int64, v float64) (ref string, err error) // Add adds a sample pair for the referenced series. It is generally faster // than adding a sample by providing its full label set. AddFast(ref string, t int64, v float64) error // Commit submits the collected samples and purges the batch. Commit() error // Rollback rolls back all modifications made in the appender so far. Rollback() error }

slide-31
SLIDE 31

The samples of each series need to be ordered. Add(ser1, 10, 4) → ✔ Add(ser1, 15, 7) → ✔ Add(ser2, 10, 7) → ✔ Add(ser1, 12, 7) → ✘

Appender: Ordering

slide-32
SLIDE 32

Appender

time series

slide-33
SLIDE 33

Appender

time series

slide-34
SLIDE 34

Appender

time series

slide-35
SLIDE 35

Appender

time series

slide-36
SLIDE 36

Appender

time series

slide-37
SLIDE 37

Appender

time series

slide-38
SLIDE 38

Appender

time series

slide-39
SLIDE 39

Appender

time series

slide-40
SLIDE 40

Appender

time series

slide-41
SLIDE 41

Appender

time series

slide-42
SLIDE 42

Appender

time series

slide-43
SLIDE 43

Appender

time series

slide-44
SLIDE 44

Util

func LineToMetrics(buf []byte) ([]Metric, error) type Metric struct { Series labels.Labels Timestamp int64 Value float64 }

slide-45
SLIDE 45

Code

slide-46
SLIDE 46

Querying

slide-47
SLIDE 47

Querying

time series

slide-48
SLIDE 48

Querying

time series

{name=~”prom.*”, host=”host1”}

slide-49
SLIDE 49

// {name=~”prom.*”, host=”host1”} type Matcher interface { // Name returns the label name the matcher should apply to. Name() string // Matches checks whether a value fulfills the constraints. Matches(v string) bool }

Querying: Matcher

slide-50
SLIDE 50

Querying: Matcher

em := labels.NewEqualMatcher(“name”, “prometheus”) // {name=”prometheus”} em.Matches(“prometheus”) // → true em.Matches(“influx”) // → false // Check if a series has a label m.Name(), if yes, then call m.Matches() on label value. If it matches then the series is Matched. // So em matches all series that have {name=”prometheus”} as a label.

slide-51
SLIDE 51

Querying: Matcher

regM := labels.NewRegExpMatcher(“name”, “prom.*”) // {name=~”prom.*”} regM.Matches(“prometheus”) // → true regM.Matches(“promflux”) // → true regM.Matches(“influx”) // → false Select([]labels.Matcher) SeriesSet // The set of series that match all the given Matchers

slide-52
SLIDE 52

Querying

time series

slide-53
SLIDE 53

Querying

time series

slide-54
SLIDE 54

Querying

time series

slide-55
SLIDE 55

Querying

time series

slide-56
SLIDE 56

Querying

time series

slide-57
SLIDE 57

Querying

time series

slide-58
SLIDE 58

Querying

time series

slide-59
SLIDE 59

Querying

time series

slide-60
SLIDE 60

Querying

time series

slide-61
SLIDE 61

Querying

time series

slide-62
SLIDE 62

Querying

time series

slide-63
SLIDE 63

Querying

time series

slide-64
SLIDE 64

Querying

time series

slide-65
SLIDE 65

Querying

time series

slide-66
SLIDE 66

Querying

time series

slide-67
SLIDE 67

Querying

time series

slide-68
SLIDE 68

Querying

time series

slide-69
SLIDE 69

Querying

func (s *DB) Querier(mint, maxt int64) Querier type Querier interface { // Select returns a set of series that matches the given label matchers. Select(...labels.Matcher) SeriesSet // LabelValues returns all potential values for a label name. LabelValues(string) ([]string, error) // Close releases the resources of the Querier. Close() error }

slide-70
SLIDE 70

Querying

Select(...labels.Matcher) SeriesSet type SeriesSet interface { Next() bool At() Series Err() error }

slide-71
SLIDE 71

Querying

time series

slide-72
SLIDE 72

Querying

Select(...labels.Matcher) SeriesSet type SeriesSet interface { Next() bool At() Series Err() error }

slide-73
SLIDE 73

Querying

time series

slide-74
SLIDE 74

Querying

At() Series type Series interface { // Labels returns the complete set of labels identifying the series. Labels() labels.Labels // Iterator returns a new iterator of the data of the series. Iterator() SeriesIterator }

slide-75
SLIDE 75

Querying

type SeriesIterator interface { // Seek advances the iterator forward to the given timestamp. // If there's no value exactly at t, it advances to the first value // after t. Seek(t int64) bool // At returns the current timestamp/value pair. At() (t int64, v float64) // Next advances the iterator by one. Next() bool // Err returns the current error. Err() error }

slide-76
SLIDE 76

Querying

time series

slide-77
SLIDE 77

Querying

type SeriesIterator interface { // Seek advances the iterator forward to the given timestamp. // If there's no value exactly at t, it advances to the first value // after t. Seek(t int64) bool // At returns the current timestamp/value pair. At() (t int64, v float64) // Next advances the iterator by one. Next() bool // Err returns the current error. Err() error }

slide-78
SLIDE 78

Querying

time series

slide-79
SLIDE 79

Util

func PromQLToMatchers(buf []byte) ([]labels.Matcher, error) PromQLToMatchers({name=~”prom.*”, host=”123”}) // → []Matcher type response struct { Series []series } type series struct { Labels labels.Labels Points []point // point → struct{ t v } }

slide-80
SLIDE 80

Code

https://github.com/gouthamve/promflux

slide-81
SLIDE 81

Demo

slide-82
SLIDE 82

Questions?

Goutham Veeramachaneni Student @ IIT Hyderabad, India ex-intern @ CoreOS putadent gouthamve