Cassandra Offline Analytics Dongqian Liu, Yi Liu 2017/05/02 Agenda - - PowerPoint PPT Presentation

▶
cassandra offline analytics
SMART_READER_LITE
LIVE PREVIEW

Cassandra Offline Analytics Dongqian Liu, Yi Liu 2017/05/02 Agenda - - PowerPoint PPT Presentation

Mar 21, 2023 50 likes •332 views

Cassandra Offline Analytics Dongqian Liu, Yi Liu 2017/05/02 Agenda Introduction Use Case Problem & Solution Suitable User Scenario Cassandra Internals Implementation Details Performance Similar Projects

slide-1
SLIDE 1

Cassandra Offline Analytics

Dongqian Liu, Yi Liu

2017/05/02

slide-2
SLIDE 2

Agenda

  • Introduction
  • Use Case
  • Problem & Solution
  • Suitable User Scenario
  • Cassandra Internals
  • Implementation Details
  • Performance
  • Similar Projects
  • Quick Start
  • Q&A

2

slide-3
SLIDE 3

Cassandra

  • Cassandra is low latency, high throughput big table. It’s suitable for real-time query.
  • Good read/write performance
  • Bad performance for some operations like scan, count, groupBy, etc.

3

slide-4
SLIDE 4

Our Use Case

  • Our team supports Feeds system for eBay Paid Internet Marketing
  • We use Cassandra to build our live item snapshots and other use cases
  • We need analytics on the data to better boost our business
  • Cluster

–30 nodes –r/w throughput: avg. 40k/s, peak 100k/s –Data size: 8~10T

4

slide-5
SLIDE 5

Problem

  • Operations like full scan take long time. Ex. simple groupBy and count on a single table takes 10+ hrs
  • Cause cluster overload
  • Pollute in-memory cache that makes read request performance much worse

5

slide-6
SLIDE 6

Solution

  • Cassandra internal structure is similar with HBase from high level.
  • We can do similar thing as what HBase MapReduce does. There are two gaps:

–Cassandra sstables are on local disks of cluster nodes, the HBase HFiles are on HDFS. –There is no good way to read raw SSTables in MapReduce or Spark job efficiently.

  • To fill in above gaps, we can upload the sstables to HDFS in parallel and write some code to let MR and

Spark job be able to read the raw SSTables efficiently and finally transform it to Hadoop Table.

6

slide-7
SLIDE 7

Suitable User Scenario

  • Require offline analytics on big table, besides low latency and high throughput read/write performance.
  • Find HBase can not satisfy the requirements of random read/write performance.

7

slide-8
SLIDE 8

High level Overview

8

Cassandra

SSTables on Cassandra cluster

Hadoop

Table A Table A SSTables on Hadoop snapshot Downstream offline analytics

slide-9
SLIDE 9

Table Transformation

  • Compaction
  • Deduplication
  • Consistency

9

slide-10
SLIDE 10

JIRA in Community

  • https://issues.apache.org/jira/browse/CASSANDRA-2527

10

slide-11
SLIDE 11

11

Cassandra Internals

slide-12
SLIDE 12

Cassandra Write

12

slide-13
SLIDE 13

Storing Data on Disk

  • Data (Data.db): SSTable data
  • Primary Index (Index.db): Index of the row keys with pointers to their positions in the data file
  • Bloom filter (Filter.db)
  • Compression Information (CompressionInfo.db): A file holding information about uncompressed data

length, chunk offsets and other compression information

  • Statistics (Statistics.db)
  • Digest (Digest.crc32 …)
  • CRC (CRC.db)
  • SSTable Index Summary (SUMMARY.db)
  • SSTable Table of Contents (TOC.txt)
  • Secondary Index (SI_.*.db)

13

slide-14
SLIDE 14

Cassandra Read

14

slide-15
SLIDE 15

Cassandra Read

15

slide-16
SLIDE 16

16

Implementation Details

slide-17
SLIDE 17

Build Split index for SSTables

17

slide-18
SLIDE 18

Compaction, Deduplication and Consistency

  • Compaction works on a collection of SSTables. From these SSTables, compaction collects all versions of

each unique row and assembles one complete row, using the most up-to-date version (by timestamp) of each of the row's columns.

  • We use reducer to handle the compaction, deduplication and consistency, which is the same logic as C*.

18

slide-19
SLIDE 19

Core Classes

  • SSTableIndexInputFormat
  • SSTableIndexRecordReader
  • SSTableSplitIndexOutputFormat
  • SSTableSplitIndexRecordWriter
  • SSTableInputFormat
  • SSTableRecordReader
  • SSTableReducer
  • SSTableRowWritable

19

slide-20
SLIDE 20

SSTableIndexInputFormat

  • protected boolean isSplitable(JobContext context, Path filename) {

return false; }

  • protected List<FileStatus> listStatus(JobContext job) throws IOException {

List<FileStatus> files = super.listStatus(job); List<FileStatus> indexFiles = new ArrayList<FileStatus>(); for (FileStatus file : files) { if (file.getLen() > 0 && IS_SSTABLE_INDEX.apply(file.getPath())) { indexFiles.add(file); } } return indexFiles; }

20

slide-21
SLIDE 21

Performance

  • A SSTable of size 100G takes ~10 mins to complete building a snapshot table on HDFS

21

slide-22
SLIDE 22

Similar Projects in Industry

  • https://github.com/fullcontact/hadoop-sstable
  • https://github.com/Knewton/KassandraMRHelper
  • https://github.com/Netflix/aegisthus
  • https://github.com/Netflix/Priam

22

slide-23
SLIDE 23

23

Quick Start

slide-24
SLIDE 24

Configuration

  • hadoop.sstable.cql="CREATE TABLE ..."
  • hadoop.sstable.keyspace="<KEYSPACE>"
  • mapreduce.job.reduces=<REDUCE_NUM>

24

slide-25
SLIDE 25

Upload SSTables to Hadoop

  • sstable-backup/bin/backup.sh
  • Example:
  • pssh -i -h conf/hosts -p 7 -t 0 /data/applications/sstable-backup/bin/backup.sh -k <keyspace> -cf

<column_family> -s <snapshot> -o <output_dir>

25

slide-26
SLIDE 26

Build index

  • $HADOOP_HOME/bin/hadoop jar hadoop-cassandra-1.0-SNAPSHOT-allInOne.jar

com.ebay.traffic.cassandra.sstable.index.hadoop.mapreduce.BuildIndex <input> <output>

26

slide-27
SLIDE 27

SSTable to Hadoop file format

  • $HADOOP_HOME/bin/hadoop jar hadoop-cassandra-1.0-SNAPSHOT-allInOne.jar

com.ebay.traffic.cassandra.sstable.hadoop.mapreduce.SSTable2Hadoop -D mapreduce.job.reduces=<REDUCE_NUM> <input> <output>

27

slide-28
SLIDE 28

28

Q & A Thanks