Lecture 14: Parallel Algorithms Abhinav Bhatele, Department of - - PowerPoint PPT Presentation

▶
lecture 14 parallel algorithms
SMART_READER_LITE
LIVE PREVIEW

Lecture 14: Parallel Algorithms Abhinav Bhatele, Department of - - PowerPoint PPT Presentation

Mar 27, 2024 102 likes •237 views

Introduction to Parallel Computing (CMSC498X / CMSC818X) Lecture 14: Parallel Algorithms Abhinav Bhatele, Department of Computer Science Communication algorithms Reduction All-to-all Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING 2

slide-1
SLIDE 1

Lecture 14: Parallel Algorithms

Abhinav Bhatele, Department of Computer Science

Introduction to Parallel Computing (CMSC498X / CMSC818X)

slide-2
SLIDE 2

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Communication algorithms

  • Reduction
  • All-to-all

2

slide-3
SLIDE 3

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Types of reduction

  • Scalar reduction: every process contributes one number
  • Perform some commutative associate operation
  • Vector reduction: every process contributes an array of numbers

3

slide-4
SLIDE 4

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Parallelizing reduction

4

slide-5
SLIDE 5

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Parallelizing reduction

  • Naive algorithm: every process sends to the root

4

slide-6
SLIDE 6

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Parallelizing reduction

  • Naive algorithm: every process sends to the root
  • Spanning tree: organize processes in a k-ary tree

4

slide-7
SLIDE 7

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Parallelizing reduction

  • Naive algorithm: every process sends to the root
  • Spanning tree: organize processes in a k-ary tree
  • Start at leaves and send to parents
  • Intermediate nodes wait to receive data from all their children

4

slide-8
SLIDE 8

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Parallelizing reduction

  • Naive algorithm: every process sends to the root
  • Spanning tree: organize processes in a k-ary tree
  • Start at leaves and send to parents
  • Intermediate nodes wait to receive data from all their children
  • Number of phases: logkp

4

slide-9
SLIDE 9

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

All-to-all

  • Each process sends a distinct message to every other process
  • Naive algorithm: every process sends the data pair-wise to all other processes

5

https://www.codeproject.com/Articles/896437/A-Gentle-Introduction-to-the-Message-Passing-Inter

slide-10
SLIDE 10

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Virtual topology: 2D mesh

  • Phase 1: every process sends to its row neighbors
  • Phase 2: every process sends to column neighbors

6

slide-11
SLIDE 11

Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING

Virtual topology: hypercube

  • Hypercube is an n-dimensional analog of a square (n=2) and cube (n=3)
  • Special case of k-ary d-dimensional mesh

7

https://en.wikipedia.org/wiki/Hypercube

slide-12
SLIDE 12

Abhinav Bhatele 5218 Brendan Iribe Center (IRB) / College Park, MD 20742 phone: 301.405.4507 / e-mail: bhatele@cs.umd.edu