[PPT] - a framework for historical analysis and real-4me monitoring of BGP PowerPoint Presentation

SLIDE 1

Chiara Orsini, Alistair King, Danilo Giordano, Vasileios Giotsas, Alberto Dainotti alistair@caida.org CAIDA, UC San Diego

a framework for historical analysis and real-4me monitoring of BGP data

SLIDE 2

2

BGPSTREAM

BGP data analysis for the masses

Open source libraries, APIs and tools for live and historical BGP data analysis
Simple API
Versa?le
Facilitates reproducibility and repeatability
Real?me monitoring
Stable: h"ps:/

/bgpstream.caida.org

SLIDE 3

3

MOTIVATION

Why BGPStream?

BGP research and monitoring is important
Lots of exis?ng BGP measurement data
Route Views and RIPE RIS have >15 years of data (16TB)
BUT, dis?nct lack of good tooling for processing/analyzing BGP data
State of the art?

SLIDE 4

4

MOTIVATION

Why BGPStream?

BGP research and monitoring is important
Lots of exis?ng BGP measurement data
Route Views and RIPE RIS have >15 years of data (16TB)
BUT, dis?nct lack of good tooling for processing/analyzing BGP data
State of the art?

wget http://archive.org/xyz/abc/file.mrt 

SLIDE 5

5

MOTIVATION

Why BGPStream?

BGP research and monitoring is important
Lots of exis?ng BGP measurement data
Route Views and RIPE RIS have >15 years of data (16TB)
BUT, dis?nct lack of good tooling for processing/analyzing BGP data
State of the art?

wget http://archive.org/xyz/abc/file.mrt  bgpdump -m file.mrt | my_parser.py

SLIDE 6

6

MOTIVATION

Why BGPStream?

BGP research and monitoring is important
Lots of exis?ng BGP measurement data
Route Views and RIPE RIS have >15 years of data (16TB)
BUT, dis?nct lack of good tooling for processing/analyzing BGP data
State of the art?

wget http://archive.org/xyz/abc/file.mrt  bgpdump -m file.mrt | my_parser.py

SLIDE 7

7

THE BGPSTREAM FRAMEWORK

An overview

SLIDE 8

8

THE BGPSTREAM FRAMEWORK

An overview

Metadata Broker

SLIDE 9

9

THE BGPSTREAM FRAMEWORK

An overview

Metadata Broker User Libraries

SLIDE 10

10

THE BGPSTREAM FRAMEWORK

An overview

…

metadata crawler Public HTTP Data Archives

Metadata Broker User Libraries

SLIDE 11

11

THE BGPSTREAM FRAMEWORK

An overview

…

metadata crawler Public HTTP Data Archives

Metadata Broker User Libraries

metadata query

SLIDE 12

12

THE BGPSTREAM FRAMEWORK

An overview

…

metadata crawler Public HTTP Data Archives

Metadata Broker User Libraries

metadata query MRT data (via HTTP)

SLIDE 13

13

THE BGPSTREAM FRAMEWORK

An overview

…

metadata crawler Public HTTP Data Archives

Metadata Broker User Libraries

metadata query MRT data (via HTTP)

libBGPStream Python API User Code

SLIDE 14

14

THE BGPSTREAM FRAMEWORK

Stacked view

SLIDE 15

15

THE BGPSTREAM FRAMEWORK

Stacked view 1

SLIDE 16

16

THE BGPSTREAM FRAMEWORK

Stacked view 1 2

SLIDE 17

17

THE BGPSTREAM FRAMEWORK

Stacked view 1 2 3

SLIDE 18

18

BGPSTREAM USER LIBRARY

libBGPStream

Issues queries to metadata broker
Retrieves data directly from Data Providers
Currently supports MRT (RFC 6396)
De-mul?plexes data from many sources

into a single stream

Provides ?me-ordered sor?ng

SLIDE 19

19

RECORDS & ELEMS

ExtracAng informaAon from MRT

BGPStream Record:
Encapsulates an MRT record
Adds metadata (e.g. collector)
MRT records (may) contain info for mul?ple

peers/prefixes

E.g. routes to a single prefix from mulOple peers

in a RIB dump

Records are decomposed into BGPStream Elems:
E.g. prefix announcement from a single peer

Field Type Function project string project name (e.g., Route Views) collector string collector name (e.g., rrc00) type enum RIB or Updates dump time long time the containing dump was begun position enum first, middle, or last record of a dump time long timestamp of the MRT record status enum record validity flag MRT record struct de-serialized MRT record

BGPStream Record

Table 1: BGPStream elem fields. Field Type Function type enum route from a RIB dump, an- nouncement, withdrawal, or state message time long timestamp of MRT record peer address struct IP address of the VP peer ASN long AS number of the VP prefix* struct IP prefix next hop* struct IP address of the next hop AS path* struct AS path community* struct community attribute

ld state*

enum FSM state (before the change) new state* enum FSM state (after the change) * denotes a field conditionally populated based on

BGPStream Elem

SLIDE 20

20

C API

Specifying a stream

SLIDE 21

21

C API

Specifying a stream

SLIDE 22

22

C API

Specifying a stream

SLIDE 23

23

C API

Specifying a stream

SLIDE 24

24

C API

Consuming the stream

SLIDE 25

25

C API

Consuming the stream

SLIDE 26

26

C API

Consuming the stream

SLIDE 27

27

PYTHON BINDINGS - CASE STUDY

Studying AS path inflaAon using PyBGPStream

from _pybgpstream import BGPStream, BGPRecord, BGPElem 1 from collections import defaultdict 2 from itertools import groupby 3 import networkx as nx 4 5 stream = BGPStream() 6 as_graph = nx.Graph() 7 rec = BGPRecord() 8 bgp_lens = defaultdict(lambda: defaultdict(lambda: None)) 9 stream.add_filter(’record-type’,’ribs’) 10 stream.add_interval_filter(1438415400,1438416600) 11 stream.start() 12 13 while(stream.get_next_record(rec)): 14 elem = rec.get_next_elem() 15 while(elem): 16 monitor = str(elem.peer_asn) 17 hops = [k for k, g in groupby(elem.fields[’as-path’].split(" "))] 18 if len(hops) > 1 and hops[0] == monitor: 19

rigin = hops[-1]

20 for i in range(0,len(hops)-1): 21 as_graph.add_edge(hops[i],hops[i+1]) 22 bgp_lens[monitor][origin] = \ 23 min(filter(bool,[bgp_lens[monitor][origin],len(hops)])) 24 elem = rec.get_next_elem() 25 for monitor in bgp_lens: 26 for origin in bgp_lens[monitor]: 27 nxlen = len(nx.shortest_path(as_graph, monitor, origin)) 28 print monitor, origin, bgp_lens[monitor][origin], nxlen 29

AS path length discrepancy PMF

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 lin 10-7 10-6 10-5 10-4 10-3 10-2 0.1 1 2 3 4 5 6 7 8 9 10 11 log AS path length difference[d]

30 LINES OF PYTHON CODE

How many AS paths are longer than the shortest path between two ASes?

SLIDE 28

28

PY BGPSTREAM

Python bindings

Single script includes data specifica?on and analysis logic:
Enhances reproducibility/repeatability
All of the power of the C API, available in Python

SLIDE 29

29

PYTHON BINDINGS - CASE STUDY

Timely reacAve measurements

We monitor community-based black-holing
Vic?m of DoS aWack announces prefix with special

community aPribute to request neighbors drop traffic

We trigger traceroutes to characterize the black-holing

event (using 50-100 probes per event)

probed 253 vic?ms (90-95% of black-holing events)

while black-holing in effect

Combined passive control-plane and ac2ve data-plane

measurements to capture and inves2gate transient rou2ng policies

SLIDE 30

30

BGP CORSARO

ConAnuous realAme monitoring

Plugin-based tool for processing live

BGP data

Con?nuously extracts derived data from

BGPStream in regular 2me bins

Incl. “prefix-monitor” sample plugin
Monitor your own address space
How many prefixes/origin ASes?

Hijacking of AS137 (GARR) - Jan 2015*

*originally described by Dyn Research: http://research.dyn.com/2015/01/vast-world-of-fraudulent-routing/

SLIDE 31

31

BIG DATA

BGP data analysis for the 1%

“Students can write scripts to analyze BGP data, but I need to do REAL analysis…”
We conducted a proof-of-concept study using PyBGPStream with Apache Spark:
Analyzed 15 years of data:
one RIB per month
all Route Views and RIPE RIS collectors
> 3000 RIBs, ~44 billion BGPStream Elems
See the paper for more details about lessons learned
PyBGPStream/Spark template script: hWps:/

/github.com/CAIDA/bgpstream

SLIDE 32

32

BIG DATA - CASE STUDIES

RouAng table size over Ame

2002 2004 2006 2008 2010 2012 2014 2016 100k 200k 300k 400k 500k # IPv4 prefjxes

SLIDE 33

33