[S PARK ] Shrideep Pallickara Computer Science Colorado State - - PDF document

▶
s park
SMART_READER_LITE
LIVE PREVIEW

[S PARK ] Shrideep Pallickara Computer Science Colorado State - - PDF document

Oct 10, 2023 663 likes •956 views

CS555: Distributed Systems [Fall 2019] Dept. Of Computer Science , Colorado State University CS 555: D ISTRIBUTED S YSTEMS [S PARK ] Shrideep Pallickara Computer Science Colorado State University CS555: Distributed Systems [Fall 2019] October 3,

slide-1
SLIDE 1

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.1

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS 555: DISTRIBUTED SYSTEMS

[SPARK]

Shrideep Pallickara Computer Science Colorado State University

October 3, 2019

L12.1 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.2 Professor: SHRIDEEP PALLICKARA

Frequently asked questions from the previous class survey

¨ Custom Partitioners ¤ How often, example? ¨ How does Hadoop decide whether or not to call the combiner?

October 3, 2019

slide-2
SLIDE 2

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.2

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.3 Professor: SHRIDEEP PALLICKARA

Topics covered in this lecture

¨ Spark ¤ Software stack ¤ Interactive shells in Spark ¤ Core Spark concepts

October 3, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

APACHE SPARK

October 3, 2019

L12.4

slide-3
SLIDE 3

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.3

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.5 Professor: SHRIDEEP PALLICKARA

Spark: What is it?

¨ Cluster computing platform ¤ Designed to be fast and general purpose ¨ Speed ¤ Often considered to be a design alternative for Apache MapReduce ¤ Extends MapReduce to support more types of computations n Interactive queries, iterative tasks, and stream processing ¨ Why is speed important? ¤ Difference between waiting for hours versus exploring data interactively

October 3, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.6 Professor: SHRIDEEP PALLICKARA

Spark: Influences and Innovations

¨ Spark has inherited parts of its API, design, and supported formats

from other existing computational frameworks

¤ Particularly DryadLINQ ¨ Spark’s internals, especially how it handles failures, differ from many

traditional systems

¨ Spark’s ability to leverage lazy evaluation within memory

computations makes it particularly unique

October 3, 2019

slide-4
SLIDE 4

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.4

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.7 Professor: SHRIDEEP PALLICKARA

Where does Spark fit in the Analytics Ecosystem?

¨ Spark provides methods to process data in parallel that are

generalizable

¨ On its own, Spark is not a data storage solution ¤ Performs computations on Spark JVMs that last only for the duration of a

Spark application

¨ Spark is used in tandem with: ¤ A distributed storage system (e.g., HDFS, Cassandra, or S3) n To house the data processed with Spark ¤ A cluster manager — to orchestrate the distribution of Spark applications

across the cluster

October 3, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.8 Professor: SHRIDEEP PALLICKARA

Key enabling idea in Spark

¨ Memory-resident data ¨ Spark loads data into the memory of worker nodes ¤ Processing is performed on memory-resident data

October 3, 2019

slide-5
SLIDE 5

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.5

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.9 Professor: SHRIDEEP PALLICKARA

A look at the memory hierarchy

Item time Scaled time in human terms (2 billion times slower)

Processor cycle Cache access Memory access Context switch Disk access Quantum

0.5 ns (2 GHz) 1 second 1 ns (1 GHz) 2 seconds 70 ns 140 seconds 5,000 ns (5 μs) 167 minutes 7,000,000 ns (7 ms) 162 days 100,000,000 ns (100 ms) 6.3 years Source: Kay Robbins & Steve Robbins. Unix Systems Programming, 2nd edition, Prentice Hall.

October 3, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.10 Professor: SHRIDEEP PALLICKARA

Spark covers a wide range of workloads

¨ Batch applications ¨ Iterative algorithms ¨ Queries ¨ Stream processing ¨ This has previously required multiple, independent tools

October 3, 2019

slide-6
SLIDE 6

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.6

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.11 Professor: SHRIDEEP PALLICKARA

Running Spark

October 3, 2019

¨ You can use Spark from Python, Java, Scala, R, or SQL ¨ Spark itself is written in Scala, and runs on the Java Virtual Machine

(JVM)

¤ You can Spark either on your laptop or a cluster, all you need is an

installation of Java

¨ If you want to use the Python API, you will also need a Python

interpreter (version 2.7 or later)

¨ If you want to use R, you will need a version of R on your machine.

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.12 Professor: SHRIDEEP PALLICKARA

Spark integrates well with other tools

¨ Can run in Hadoop clusters ¨ Access Hadoop data sources, including Cassandra

October 3, 2019

slide-7
SLIDE 7

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.7

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.13 Professor: SHRIDEEP PALLICKARA

At its core, Spark is a computational engine

¨ Spark is responsible for several aspects of applications that comprise ¤ Many tasks across many machines (compute clusters) ¨ Responsibilities include:

① Scheduling ② Distributions ③ Monitoring

October 3, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.14 Professor: SHRIDEEP PALLICKARA

Spark execution

October 3, 2019

¨ The cluster of machines that Spark will use to execute tasks is

managed by a cluster manager

¤ Spark’s standalone cluster manager, YARN, or Mesos ¨ We submit Spark Applications to these cluster managers, which will

grant resources to the application to complete the work

slide-8
SLIDE 8

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.8

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.15 Professor: SHRIDEEP PALLICKARA

Spark applications

October 3, 2019

¨ Spark Applications consist of ¤ A driver process n The driver process is absolutely essential n The heart of a Spark Application and maintains all relevant information during the

lifetime of the application

¤ A set of executor processes

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.16 Professor: SHRIDEEP PALLICKARA

The Driver

October 3, 2019

¨ The driver process runs your main() function, sits on a node in the

cluster

¨ Driver is responsible for three things: ¤ Maintaining information about the Spark Application ¤ Responding to a user’s program or input ¤ Analyzing, distributing, and scheduling work across the executors

slide-9
SLIDE 9

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.9

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.17 Professor: SHRIDEEP PALLICKARA

The executors

October 3, 2019

¨ The executors are responsible for actually carrying out the work that

the driver assigns them

¨ Each executor is responsible for only two things: ¤ Executing code assigned to it by the driver, and ¤ Reporting the state of the computation on that executor back to the driver

node

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.18 Professor: SHRIDEEP PALLICKARA

Architecture of a Spark Application

October 3, 2019

Cluster Manager

Spark Session User Code

Executors

Driver Process

slide-10
SLIDE 10

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.10

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.19 Professor: SHRIDEEP PALLICKARA

How Spark runs Python or R

October 3, 2019

¨ You write Python and R code that Spark translates into code that it

then can run on the executor JVM

Spark Session Python Process R Process JVM To executors

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.20 Professor: SHRIDEEP PALLICKARA

SparkSession

October 3, 2019

¨ We need a way to send user commands and data to a Spark

Application

¤ We do that by first creating a SparkSession ¨ The SparkSession instance is the way Spark executes user-defined

manipulations across the cluster.

¤ There is a one-to-one correspondence between a SparkSession and a Spark

Application

¤ In Scala and Python, the variable is available as spark when you start the

console.

slide-11
SLIDE 11

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.11

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.21 Professor: SHRIDEEP PALLICKARA

Partitions

October 3, 2019

¨ To allow every executor to perform work in parallel, Spark breaks up

the data into chunks called partitions

¤ For e.g., a partition may be a collection of rows that sit on one physical

machine in your cluster

¨ If you have one partition, Spark will have a parallelism of only one,

even if you have thousands of executors

¨ If you have many partitions but only one executor? ¤ Spark will still have a parallelism of only one because there is only one

computation resource

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.22 Professor: SHRIDEEP PALLICKARA

Manipulation of Partitions

October 3, 2019

¨ An important thing to note is that you do not (for the most part)

manipulate partitions manually or individually

¤ You simply specify high-level transformations of data in the physical

partitions

n Spark determines how this work will actually execute on the cluster.

slide-12
SLIDE 12

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.12

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.23 Professor: SHRIDEEP PALLICKARA

Transformations

October 3, 2019

¨ In Spark, the core data structures are immutable ¤ They cannot be changed after they’re created ¨ If you cannot change it, how are you supposed to use it? ¤ You need to instruct Spark how you would like to modify it ¤ These instructions are called transformations

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.24 Professor: SHRIDEEP PALLICKARA

More about transformations

October 3, 2019

¨ Transformations do not return an output ¤ Spark will not act on transformations until we call an action

slide-13
SLIDE 13

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.13

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.25 Professor: SHRIDEEP PALLICKARA

Lazy evaluation

October 3, 2019

¨ Spark will wait until the very last moment to execute the graph of

computation instructions

¨ When you express some operation? ¤ You do not modify the data immediately ¤ Rather, you build up a plan of transformations that you would like to apply

to your source data.

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.26 Professor: SHRIDEEP PALLICKARA

Why lazy evaluation works …

October 3, 2019

¨ By waiting until the last minute to execute the code ¤ Spark compiles this plan from your transformations to a streamlined physical

plan that will run as efficiently as possible across the cluster

¨ This provides immense benefits because Spark can optimize the entire

data flow from end to end

¤ E.g. predicate pushdowns

slide-14
SLIDE 14

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.14

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.27 Professor: SHRIDEEP PALLICKARA

Spark in a nutshell

October 3, 2019

¨ Spark is a distributed programming model in which the user specifies

transformations

¤ Multiple transformations build up a directed acyclic graph of instructions ¨ An action begins the process of executing that graph of instructions ¤ As a single job ¤ By breaking it down into stages and tasks to execute across the cluster

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.28 Professor: SHRIDEEP PALLICKARA

Spark APIs

October 3, 2019

¨ Spark has two fundamental sets of APIs: ¤ The low-level “unstructured” APIs, and ¤ The higher-level structured APIs

slide-15
SLIDE 15

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.15

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.29 Professor: SHRIDEEP PALLICKARA

Structured APIs

October 3, 2019

¨ Structured APIs are a tool for manipulating all sorts of data ¤ From unstructured log files to semi-structured CSV files and highly structured

Parquet files

¨ Refers to three core types of distributed collection APIs: ¤ Datasets ¤ DataFrames ¤ SQL tables and views ¨ Majority of the Structured APIs apply to both batch and streaming

computation

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.30 Professor: SHRIDEEP PALLICKARA

Spark’s Toolset

October 3, 2019

Low Level APIs RDDs Distributed variables Structured APIs Datasets DataFrames SQLs Structured Streaming Advanced Analytics Libraries & Ecosystem

slide-16
SLIDE 16

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.16

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.31 Professor: SHRIDEEP PALLICKARA

Spark has two notions of structured collections: DataFrames and Datasets

October 3, 2019

¨ DataFrames and Datasets are (distributed) table-like collections with

well-defined rows and columns

¨ Each column: ¤ Must have the same number of rows as all the other columns (although you

can use null to specify the absence of a value)

¤ Has type information that must be consistent for every row in the collection.

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.32 Professor: SHRIDEEP PALLICKARA

DataFrames versus Datasets

October 3, 2019

¨ DataFrames are considered “untyped” ¨ Datasets are considered “typed”

slide-17
SLIDE 17

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.17

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.33 Professor: SHRIDEEP PALLICKARA

How does Spark view DataFrames and Datasets?

October 3, 2019

¨ To Spark, DataFrames and Datasets represent immutable, lazily

evaluated plans that specify what operations to apply to data residing at a location to generate some output

¨ When we perform an action on a DataFrame, we instruct Spark to

perform the actual transformations and return the result

¨ These represent plans of how to manipulate rows and columns to

compute the user’s desired result.

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.34 Professor: SHRIDEEP PALLICKARA

The DataFrame is the most common Structured API

October 3, 2019

¨ Simply represents a table of data with rows and columns ¨ The list that defines the columns and the types within those columns is

called the schema

slide-18
SLIDE 18

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.18

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.35 Professor: SHRIDEEP PALLICKARA

The DataFrame concept is not unique to Spark

October 3, 2019

¨ R and Python both have similar concepts. ¤ However, Python/R DataFrames (with some exceptions) exist on one machine

rather than multiple machines

¤ This limits what you can do with a given DataFrame to the resources that

exist on that specific machine

¨ A Spark DataFrame can span thousands of computers.

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

THE SPARK SOFTWARE STACK

October 3, 2019

L12.36

slide-19
SLIDE 19

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.19

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.37 Professor: SHRIDEEP PALLICKARA

The Spark stack

Spark Core YARN Mesos Standalone Scheduler

Spark SQL (semi-) structured data Spark Streaming real-time MLib & ML machine learning GraphX Graph processing

October 3, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.38 Professor: SHRIDEEP PALLICKARA

Spark Core

¨ Basic functionality of Spark ¨ Task scheduling, memory management, fault recovery, and interacting

with storage systems

¨ Also, the API that defines Resilient Distributed Datasets (RDDs) ¤ Spark’s core programming abstraction ¤ Represents collection of data items dispersed across many compute nodes n Can be manipulated concurrently (parallel)

October 3, 2019

slide-20
SLIDE 20

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.20

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.39 Professor: SHRIDEEP PALLICKARA

Spark SQL

¨ Package for working with structured data ¨ Allows querying data using SQL and HQL (Hive Query Language) ¤ Data sources: Hive tables, Parquet, and JSON ¨ Allows intermixing queries with programmatic data manipulations

support by RDDs

¤ Using Scala, Java, and Python

October 3, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.40 Professor: SHRIDEEP PALLICKARA

Semi-structured data and Spark SQL

¨ Spark SQL defines an interface for a semi-structured data type,

called DataFrames

¤ And as of Spark 1.6, a semi-structured, typed version of RDDs called

Datasets

¨ Spark SQL is a very important component for Spark performance ¨ Much of what can be accomplished with Spark Core can be done by

leveraging Spark SQL.

October 3, 2019

slide-21
SLIDE 21

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.21

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.41 Professor: SHRIDEEP PALLICKARA

Spark Streaming

¨ Enables processing of live streams of data from sources such as: ¤ Logfiles generated by production webservers ¤ Messages containing web service status updates ¨ Uses the scheduling of the Spark Core for streaming analytics on

minibatches of data

¨ Has a number of unique considerations, such as the window sizes used

for batches

October 3, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.42 Professor: SHRIDEEP PALLICKARA

MLib

¨ MLlib is a package of machine learning and statistics algorithms

written with Spark

¨ Algorithms include: ¤ Clustering, classification, regression, clustering, and collaborative filtering ¨ Low-level primitives ¤ Generic gradient descent optimization algorithm ¨ Alternatives? ¤ Mahout, sci–kit learn, VW, WEKA, and R among others

October 3, 2019

slide-22
SLIDE 22

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.22

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.43 Professor: SHRIDEEP PALLICKARA

What about Spark ML?

¨ Still in the early stages, and has only existed since Spark 1.2 ¨ Spark ML provides a higher-level API than MLlib ¤ Goal is to allow users to more easily create practical machine learning

pipelines

¤ Spark MLlib is primarily built on top of RDDs and uses functions from Spark

Core, while ML is built on top of Spark SQL DataFrames

¨ Eventually the Spark community plans to move over to ML and

deprecate MLlib

October 3, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.44 Professor: SHRIDEEP PALLICKARA

Graph X

¨ Library for manipulating graphs ¨ Graph-parallel computations ¨ Extends Spark RDD API ¤ Create a directed graph, with arbitrary properties attached to each vertex

and edge

October 3, 2019

slide-23
SLIDE 23

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.23

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.45 Professor: SHRIDEEP PALLICKARA

Cluster Managers

¨ Spark runs over a variety of cluster managers ¨ These include: ¤ Hadoop YARN ¤ Apache Mesos ¤ Standalone Scheduler n Included within Spark

October 3, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.46 Professor: SHRIDEEP PALLICKARA

Storage Layers for Spark

¨ Spark can create distributed datasets from any file stored in HDFS ¨ Plus, other storage systems supported by the Hadoop API ¤ Amazon S3, Cassandra, Hive, HBase, etc.

October 3, 2019

slide-24
SLIDE 24

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.24

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

INTERACTIVE SHELLS IN SPARK

October 3, 2019

L12.47 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.48 Professor: SHRIDEEP PALLICKARA

Spark Shells

¨ Interactive [Python and Scala] ¤ Similar to shells like Bash or Windows command prompt ¨ Ad hoc data analysis ¨ Traditional shells manipulate data using disk and memory on a single

machine

¤ Spark shells allow interaction with data that is distributed across many

machines

¤ Spark manages complexity of distributing processing

October 3, 2019

slide-25
SLIDE 25

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.25

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.49 Professor: SHRIDEEP PALLICKARA

Several software were designed to run on the Java Virtual Machine

¨ Languages that compile to run on the JVM and can interact with Java

software packages but are not actually Java

¨ There are a number of non-Java JVM languages ¤ The two most popular ones used in real-time application development: Scala

and Clojure

October 3, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.50 Professor: SHRIDEEP PALLICKARA

Scala

¨ Has spent most of its life as an academic language ¤ Still largely developed at universities ¤ Has a rich standard library that has made it appealing to developers of

high-performance server applications

¨ Like Java, Scala is a strongly typed object-oriented language ¤ Includes many features from functional programming languages that are not

in standard Java

¤ Interestingly, Java 8 incorporate several of the more useful features of

Scala and other functional languages.

October 3, 2019

slide-26
SLIDE 26

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.26

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.51 Professor: SHRIDEEP PALLICKARA

What is functional programming?

¨ When a method is compiled by Java, it is converted to instructions

called byte code and …

¤ Then largely disappears from the Java environment n Except when it is called by other methods ¨ In a functional language, functions are treated the same way as data ¤ Can be stored in objects similar to integers or strings, returned from

functions, and passed to other functions

October 3, 2019 CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.52 Professor: SHRIDEEP PALLICKARA

What about Clojure?

¨ Based on Lisp ¨ Javascript? ¤ Name was a marketing gimmick ¤ Closer to Clojure and Scala than it is to Java

October 3, 2019

slide-27
SLIDE 27

SLIDES CREATED BY: SHRIDEEP PALLICKARA L12.27

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

CS555: Distributed Systems [Fall 2019]

  • Dept. Of Computer Science, Colorado State University

L12.53 Professor: SHRIDEEP PALLICKARA

The contents of this slide-set are based on the following references

¨ Learning Spark: Lightning-Fast Big Data Analysis. 1st Edition. Holden Karau, Andy

Konwinski, Patrick Wendell, and Matei Zaharia. O'Reilly. 2015. ISBN-13: 978-

  • 1449358624. [Chapters 1-4]

¨ Karau, Holden; Warren, Rachel. High Performance Spark: Best Practices for Scaling

and Optimizing Apache Spark. O'Reilly Media. 2017. ISBN-13: 978-1491943205. [Chapter 2]

¨ Chambers, Bill,Zaharia, Matei. Spark: The Definitive Guide: Big Data Processing

Made Simple. O'Reilly Media. ISBN-13: 978-1491912218. 2018. [Chapters 1, 2, and 3].

¨ Real-Time Analytics: Techniques to Analyze and Visualize Streaming Data. Byron Ellis.

  • Wiley. [Chapter 2]

October 3, 2019