HILTI: An Abstract Execution Environment for Deep, Stateful Network - - PowerPoint PPT Presentation

HILTI: An Abstract Execution Environment for Deep, Stateful Network Traffic Analysis

Robin Sommer

International Computer Science Institute, & Lawrence Berkeley National Laboratory

robin@icsi.berkeley.edu http://www.icir.org/robin

HILTI — A High-Level Intermediary Language for Traffic Inspection.

A Tale of Three Open-Source IDS

2

Suricata

Shared functionality?

Same for packet filters, firewalls, proxies, routers, switches, OS stack …

Essentially none.

HILTI — A High-Level Intermediary Language for Traffic Inspection.

Deep Packet Inspection

3

Tap

Internet Internal Network

IDS

• 1. Find and parse all Web traffic.
• 2. Find and extract binaries.
• 3. Compute hash and compare with database.
• 4. Report, and potentially kill, if found.

Example: Finding downloads of known malware.


HILTI — A High-Level Intermediary Language for Traffic Inspection.

DPI Architecture

4

Network Traffic

Application … and that even though this stuff is hard. Common primitives & idioms — but hardly any reuse … Why? Different low-level structure & data flows.

No “common language”.

Flow table, DFAs, request/reply correlation

State Management

Pattern matching, packet classification, event correlation, multiplexing

Analysis Primitives

Signature matching, policy enforcement

Analysis Logic

IP, TCP, HTTP, DNS

Protocol Parsing

Configuration, logs, alarms

User Interface

HILTI — A High-Level Intermediary Language for Traffic Inspection.

Analysis Compiler

HILTI Abstract Machine

Firewall rules, IDS signatures, forwarding rules, …

Intermediary language Execution Model LLVM-based compiler Runtime library Reusable components

A High-Level Intermediary Language for Traffic Inspection

5

Network Traffic

Application

State Management Analysis Primitives Protocol Parsing Analysis Logic User Interface

Host Application

Library of Reusable Functionality

HILTI — A High-Level Intermediary Language for Traffic Inspection.

Example: BPF Filters

6

host 192.168.1.1 or src net 10.0.5.0/24

bool filter(ref<bytes> packet) { local addr a1, a2 local bool b1, b2, b3 a1 = overlay.get IP::Header src packet b1 = equal a1 192.168.1.1 a1 = overlay.get IP::Header dst packet b2 = equal a2 192.168.1.1 b1 = or b1 b2 b2 = equal 10.0.5.0/24 a1 b3 = or b1 b2 return b3 } type IP::Header = overlay { hdr_len: int<8> at 0 unpack UInt8InBigEndian (0, 3), version: int<8> at 0 unpack UInt8InBigEndian (4, 7), [...] src: addr at 12 unpack IPv4InNetworkOrder, dst: addr at 16 unpack IPv4InNetworkOrder }

HILTI — A High-Level Intermediary Language for Traffic Inspection.

Instruction Set

7

Bitsets Packet input Booleans Packet classification CIDR masks Packet dissection Callbacks Ports Closures Profiling Channels Raw data Debug support References Doubles Regular expressions Enumerations Strings Exceptions Structs File i/o Unions Flow control Time intervals Hashmaps Timer management Hashsets Timers IP addresses Times Integers Tuples Lists Vectors/arrays

HILTI — A High-Level Intermediary Language for Traffic Inspection.

HILTI Machine Model

8

Rich Domain-specific Data Types High-level Optimization Concurrent Analysis Real-time Performance Robust & Secure Execution Comprehensive Host Interface Flexible Control Flow Debugging & Profiling Support

Focus Areas

HILTI — A High-Level Intermediary Language for Traffic Inspection.

Implementation: The HILTI Toolchain

9

Host Application

App Core Analysis Compiler Analysis Specification LLVM Bitcode Compiler/ Linker HILTI Compiler

HILTI Environment LLVM Toolchain

Runtime Library

C Interface Stubs

Native Executable

HILTI Machine Code

Just-in time via C++ API

HILTI — A High-Level Intermediary Language for Traffic Inspection.

Hello, World!

10

module Main import Hilti void run() { call Hilti::print("Hello, World!") } # hilti-build hello.hlt -o a.out && ./a.out Hello, World!

hello.hlt

# hiltic -j hello.hlt Hello, World!

HILTI — A High-Level Intermediary Language for Traffic Inspection.

Can HILTI support complex applications?

11

HILTI — A High-Level Intermediary Language for Traffic Inspection.

Application Case Studies

BPF Filter Stateful Firewall Protocol Parsing Bro Script Execution

12

HILTI — A High-Level Intermediary Language for Traffic Inspection.

BinPAC - A Yacc for Network Protocols

13

Bro plugin integrates parsers at startup. BinPAC compiles grammar into HILTI parser. HILTI compiles parser into executable code just-in-time.

type SSH::Banner = unit { magic : /SSH-/; version : /[^-]*/; dash : /-/; software: /[^\r\n]*/; }

Grammar example: Parsing SSH banners. SSH-2.0-OpenSSH_3.8.1p1

HILTI — A High-Level Intermediary Language for Traffic Inspection.

Hello, World!

14

grammar ssh.pac2; protocol analyzer SSH over TCP:
 parse with SSH::Banner, port 22/tcp;

• n SSH::Banner
• > event ssh_banner(self.version, self.software);

ssh.evt

# bro -r ssh.trace ssh.evt ssh.bro OpenSSH_3.9p1, 1.99 OpenSSH_3.8.1p1, 2.0 type SSH::Banner = unit { magic : /SSH-/; version : /[^-]*/; dash : /-/; software: /[^\r\n]*/; }

ssh.pac2

event ssh_banner(version: string, software: string) { { print software, version; }

ssh.bro

HILTI — A High-Level Intermediary Language for Traffic Inspection.

Application Case Studies

BPF Filter Stateful Firewall Protocol Parsing Bro Script Execution

15

HILTI — A High-Level Intermediary Language for Traffic Inspection.

Bro Scripts

16

global attempts: table[addr] of count &default=0; event connection_rejected(c: connection) { local orig = c$id$orig_h; # Get originator address. local n = ++attempts[orig]; # Increase counter. if ( n == SOME_THRESHOLD ) # Check for threshold. NOTICE(...); # Alarm. }

Script example: A simple scan detector. Bro plugin compiles scripts into HILTI code. HILTI compiles that into executable code just-in-time.

HILTI — A High-Level Intermediary Language for Traffic Inspection.

Evaluation

17

DNS: Full Berkeley port 53 traffic. 1GB trace, 10min, 65M messages. HTTP: 1/25 of Berkeley port 80 traffic. 30GB trace, 52min, 340k messages.

Correctness

HILTI captures semantics correctly.

Use HILTI plugin for Bro to compare parsing & script execution with a native Bro.

Let’s see.

Performance

Traces:

HILTI — A High-Level Intermediary Language for Traffic Inspection.

Protocol Parsing

18

0.0B 0.2B 0.4B 0.6B 0.8B 1.0B 1.2B 1.4B 1.6B 1.8B C38 cycOHs 6tanGaUG HIL7I 6tanGaUG HIL7I

1567G 683G 643G 241G 1580G 852G 450G 21G 258G 712G 177G 356G 180G 1173G 469G 405G 81G 217G

HTTP DNS

Protocol PDrsing 6cULSt ExHcutLon HIL7I-to-BUo GOuH 2thHU

1.25x 2.65x

HILTI — A High-Level Intermediary Language for Traffic Inspection.

Bro Scripts

19

0.0B 0.5B 1.0B 1.5B 2.0B C38 cycOHs 6tanGaUG HIL7I 6tanGaUG HIL7I

1562G 683G 635G 244G 1810G 698G 781G 76G 254G 709G 175G 358G 176G 694G 175G 243G 139G136G

HTTP DNS

3UotocoO 3aUsLng ScriSt ExHcution HIL7I-to-BUo GOuH 2thHU

1.23x 0.68x

HILTI — A High-Level Intermediary Language for Traffic Inspection.

Summary

HILTI: A new platform for network traffic analysis.

A compiler-target for host applications to leverage. Provides common data structures and control flow primitives.

Case studies demonstrate aptness of design.

Packet filter, stateful firewall, protocol parsing, Bro scripts.

Initial performance experiments encouraging.

Not too different from native applications. It’s still a prototype, with lots of potential.

20

Sommer/Vallentin/De Carli/Paxson: “HILTI: An Abstract Execution Environment for Deep, Stateful Network Traffic Analysis”. ACM IMC 2014.

The HILTI Vision

Performance via Abstraction

Transparent improvement under the hood. Integration of non-standard hardware. High-level compiler optimizations. Automatic parallelization.❊

❊ De Carli/Sommer/Jha: “Beyond Pattern Matching: A Concurrency Model for Stateful Deep Packet Inspection”. ACM CCS 2014.

Means and glue to share functionality. HILTI library of common high-level components.

Facilitate Reuse

http://www.icir.org/hilti

HILTI is available under BSD license at