Porting the Plasma Simulation PIConGPU to Heterogeneous - - PowerPoint PPT Presentation

• Prof. Peter Mustermann I Institut xxxxx I www.hzdr.de

Porting the Plasma Simulation PIConGPU to Heterogeneous Architectures with Alpaka

René Widera1, Erik Zenker1,2, Guido Juckeland1, Benjamin Worpitz1,2, Axel Huebl1,2, Andreas Knüpfer2, Wolfgang E. Nagel2, Michael Bussmann1

1 Helmholtz-Zentrum Dresden – Rossendorf 2 Technische Universität Dresden

Mitglied der Helmholtz-Gemeinschaft

2

Electron Acceleration with Lasers

• Compact X-Ray sources

Ion Acceleration with Lasers

• Tumor Therapy

Plasma Instabilities

• Astrophysics

PICon GPU

Mitglied der Helmholtz-Gemeinschaft

3

+ + + + + + + + + + + + + + + + + +

─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─

Domain Decomposition ─ Field and Particle Domain

Mitglied der Helmholtz-Gemeinschaft

4

+ + + + + + + + + + + + + + + + + +

─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─

• Moving Particles create Fields
• Fields act back on Particles
• Particles change Cells

Domain Decomposition ─ Field and Particle Domain

Mitglied der Helmholtz-Gemeinschaft

5

Field Domain Particle Domain 1 2 3 4

+ + + +

Creating Vectorized Data Structures for Particles and Fields

Mitglied der Helmholtz-Gemeinschaft

6

Field Domain Particle Domain

Cell 1 Cell 1 Cell 2 Cell 4

+

1 2 3 4

+ + +

Creating Vectorized Data Structures for Particles and Fields

Mitglied der Helmholtz-Gemeinschaft

7

Field Domain Particle Domain

Cell 1 Cell 1 Cell 2 Cell 4

+

1 2 3 4

+ + +

• chunked in supercells
• line wise aligned

Creating Vectorized Data Structures for Particles and Fields

Mitglied der Helmholtz-Gemeinschaft

8

Field Domain Particle Domain

Cell 1 Cell 1 Cell 2 Cell 4

+

1 2 3 4

+ + +

• fixed size frames
• struct of aligned arrays
• chunked in supercells
• line wise aligned

Creating Vectorized Data Structures for Particles and Fields

Mitglied der Helmholtz-Gemeinschaft

9

Field Domain Particle Domain

Cell 1 Cell 1 Cell 2 Cell 4

+

1 2 3 4

+ + +

• fixed size frames
• struct of aligned arrays
• chunked in supercells
• line wise aligned

Creating Vectorized Data Structures for Particles and Fields

Mitglied der Helmholtz-Gemeinschaft

10

Cell 1 Cell 1 Cell 2 Cell 4

+

1 2 3 4

+ + +

Algorithm Driven Cache Strategy

Mitglied der Helmholtz-Gemeinschaft

11

Global Memory Cell 1 Cell 1 Cell 2 Cell 4

+

1 2 3 4

+ + +

Algorithm Driven Cache Strategy

Mitglied der Helmholtz-Gemeinschaft

12

Global Memory Shared Memory Cell 1 Cell 1 Cell 2 Cell 4

+

1 2 3 4

+ + +

Algorithm Driven Cache Strategy

Mitglied der Helmholtz-Gemeinschaft

13

Shared Memory Global Memory Cell 1 Cell 1 Cell 2 Cell 4

+

1 2 3 4

+ + +

Algorithm Driven Cache Strategy

Mitglied der Helmholtz-Gemeinschaft

14

Global Memory Shared Memory Cell 1 Cell 1 Cell 2 Cell 4

+

1 2 3 4

+ + +

THREAD BLOCK THREAD 1 THREAD 2 THREAD 4 THREAD 3

High Utilization of Threads

Mitglied der Helmholtz-Gemeinschaft

15

Task-Parallel Execution of Kernels + Asynchronous Communication

Mitglied der Helmholtz-Gemeinschaft

16

strong scaling

PIConGPU ─ Scales up to 16,384 GPUs

1 10 100 1000 10000 1 10 100 1000 10000 speedup number of GPUs ideal 1 to 32 8 to 256 64 to 2048 512 to 16384 4096 to 16384

Mitglied der Helmholtz-Gemeinschaft

17

weak scaling efficiency strong scaling

PIConGPU ─ Scales up to 16,384 GPUs

80 85 90 95 100 105 1 10 100 1000 10000 efficiency [%] number of GPUs ideal PIConGPU

1 10 100 1000 10000 1 10 100 1000 10000 speedup number of GPUs ideal 1 to 32 8 to 256 64 to 2048 512 to 16384 4096 to 16384

Mitglied der Helmholtz-Gemeinschaft

18

weak scaling efficiency strong scaling

Efficiency >95% PIConGPU ─ Scales up to 16,384 GPUs

80 85 90 95 100 105 1 10 100 1000 10000 efficiency [%] number of GPUs ideal PIConGPU

1 10 100 1000 10000 1 10 100 1000 10000 speedup number of GPUs ideal 1 to 32 8 to 256 64 to 2048 512 to 16384 4096 to 16384

Mitglied der Helmholtz-Gemeinschaft

19

weak scaling efficiency strong scaling

Efficiency >95% PIConGPU ─ Scales up to 16,384 GPUs

80 85 90 95 100 105 1 10 100 1000 10000 efficiency [%] number of GPUs ideal PIConGPU

6.9 PFlop/s (SP)

1 10 100 1000 10000 1 10 100 1000 10000 speedup number of GPUs ideal 1 to 32 8 to 256 64 to 2048 512 to 16384 4096 to 16384

Mitglied der Helmholtz-Gemeinschaft

20

─

+

─

More Physics, More Computations, More Power!

s1 s2

Mitglied der Helmholtz-Gemeinschaft

21

─

+

─

More Physics, More Computations, More Power!

s1 s2 s1,1 s1,2 s1,3 ... sn,m Old atom state

Mitglied der Helmholtz-Gemeinschaft

22

─

+

─

More Physics, More Computations, More Power!

s1 s2 s1,1 s1,2 s1,3 ... sn,m Old atom state t1,1 t1,2 t1,3 … t1,n t2,1 . t3,1 . … . tn,1 tn,n Atom-physical effects

Mitglied der Helmholtz-Gemeinschaft

23

─

+

─

More Physics, More Computations, More Power!

s1 s2 s1,1 s1,2 s1,3 ... sn,m Old atom state = s1,1 s1,2 s1,3 ... sn,m New atom state t1,1 t1,2 t1,3 … t1,n t2,1 . t3,1 . … . tn,1 tn,n Atom-physical effects

Mitglied der Helmholtz-Gemeinschaft

24

─

+

─

More Physics, More Computations, More Power!

s1 s2 s1,1 s1,2 s1,3 ... sn,m Old atom state = s1,1 s1,2 s1,3 ... sn,m New atom state t1,1 t1,2 t1,3 … t1,n t2,1 . t3,1 . … . tn,1 tn,n Atom-physical effects

Really Big Data Task

◾ Random access on big amounts of data > 100 GB ◾ Good job for powerful CPUs ◾ Efficient CPU/GPU cooperation

Mitglied der Helmholtz-Gemeinschaft

25

Small Open Source Communities need Maintainable Codes Single Source

Heterogeneity Write once, execute everywhere Testability Validate once, get correct results everywhere Sustainability Porting implies minimal code changes Optimizability Tune for good performance at minimum coding effort Openness Open source and

• pen standards

Mitglied der Helmholtz-Gemeinschaft

26

Alpaka

Mitglied der Helmholtz-Gemeinschaft

27

Fibers

C++

Threads

Next parallelism model

Good News: there are Alpakas on the Compute Meadow

• Single zero overhead interface to existing parallelism models
• Single source C++11 kernels
• Data-agnostic memory model

Mitglied der Helmholtz-Gemeinschaft

28

Abstract Hierarchical Redundant Parallelism Model

Grid Synchronize Parallel Sequential

Mitglied der Helmholtz-Gemeinschaft

29

Abstract Hierarchical Redundant Parallelism Model

Grid Synchronize Block Parallel Sequential

Mitglied der Helmholtz-Gemeinschaft

30

Abstract Hierarchical Redundant Parallelism Model

Grid Synchronize Block Thread Parallel Sequential

Mitglied der Helmholtz-Gemeinschaft

31

Abstract Hierarchical Redundant Parallelism Model

Grid Synchronize Block Thread Element Parallel Sequential

• Element level is an

explicit sequential layer

Mitglied der Helmholtz-Gemeinschaft

32

Device Global Memory Grid Explicit deep copy Host Memory

Data Structure Agnostic Memory Model

Mitglied der Helmholtz-Gemeinschaft

33

Device Global Memory Grid Explicit deep copy Shared Memory Block Host Memory

Data Structure Agnostic Memory Model

Mitglied der Helmholtz-Gemeinschaft

34

Device Global Memory Grid Explicit deep copy Shared Memory Block Register Memory Register Memory Thread Host Memory

Data Structure Agnostic Memory Model

Mitglied der Helmholtz-Gemeinschaft

35

Map the Abstraction Model to your Desired Acceleration Back-End

• Explicit mapping of parallelization levels to hardware

CPU RAM L3 L3

Core L1/2 R Core L1/2 R Core L1/2 R Core L1/2 R Package Package

AVX AVX AVX AVX

Mitglied der Helmholtz-Gemeinschaft

36

Map the Abstraction Model to your Desired Acceleration Back-End

• Explicit mapping of parallelization levels to hardware

CPU RAM L3 L3

Core L1/2 R Core L1/2 R Core L1/2 R Core L1/2 R Package Package

AVX AVX AVX AVX

Grid Global Memory

Mitglied der Helmholtz-Gemeinschaft

37

Map the Abstraction Model to your Desired Acceleration Back-End

• Explicit mapping of parallelization levels to hardware

Core L1/2 R Core L1/2 R Core L1/2 R Core L1/2 R

CPU RAM L3 L3

Core L1/2 R Core L1/2 R Core L1/2 R Core L1/2 R Package Package

AVX AVX AVX AVX

Grid Block Shared Memory Global Memory

Mitglied der Helmholtz-Gemeinschaft

38

Map the Abstraction Model to your Desired Acceleration Back-End

• Explicit mapping of parallelization levels to hardware

Core L1/2 R Core L1/2 R Core L1/2 R Core L1/2 R

CPU RAM L3 L3

Core L1/2 R Core L1/2 R Core L1/2 R Core L1/2 R Package Package

AVX AVX AVX AVX

Grid Block Thread Register Memory Shared Memory Global Memory

Mitglied der Helmholtz-Gemeinschaft

39

Map the Abstraction Model to your Desired Acceleration Back-End

• Explicit mapping of parallelization levels to hardware

Core L1/2 R Core L1/2 R Core L1/2 R Core L1/2 R

CPU RAM L3 L3

Core L1/2 R Core L1/2 R Core L1/2 R Core L1/2 R Package Package

AVX AVX AVX AVX

Grid Block Thread Element Register Memory Shared Memory Global Memory

Mitglied der Helmholtz-Gemeinschaft

40

Map the Abstraction Model to your Desired Acceleration Back-End

• Specific unsupported levels of the model can be ignored
• Abstract interface allows to extend the set of mappings

Mitglied der Helmholtz-Gemeinschaft

41

Alpaka : Vector Addition Kernel

struct VectorAdd { template<typename TAcc, typename TElem, typename TSize> ALPAKA_FN_ACC auto operator()( TAcc const & acc, TSize const & numElements, TElem const * const X, TElem * const Y) const -> void { } };

Mitglied der Helmholtz-Gemeinschaft

42

Alpaka : Vector Addition Kernel

struct VectorAdd { template<typename TAcc, typename TElem, typename TSize> ALPAKA_FN_ACC auto operator()( TAcc const & acc, TSize const & numElements, TElem const * const X, TElem * const Y) const -> void { } }; using alp = alpaka; auto globalIdx = alp::idx::getIdx<alp::Grid, alp::Threads>(acc)[0u]; auto elemsPerThread = alp::workdiv::getWorkDiv<alp::Thread, alp::Elems>(acc)[0u];

Mitglied der Helmholtz-Gemeinschaft

43

Alpaka : Vector Addition Kernel

struct VectorAdd { template<typename TAcc, typename TElem, typename TSize> ALPAKA_FN_ACC auto operator()( TAcc const & acc, TSize const & numElements, TElem const * const X, TElem * const Y) const -> void { } }; using alp = alpaka; auto globalIdx = alp::idx::getIdx<alp::Grid, alp::Threads>(acc)[0u]; auto elemsPerThread = alp::workdiv::getWorkDiv<alp::Thread, alp::Elems>(acc)[0u]; auto begin = globalIdx * elemsPerThread; auto end = min(begin + elemsPerThread, numElements); for(TSize i = begin; i < end; ++i){ Y[i] = X[i] + Y[i]; }

Mitglied der Helmholtz-Gemeinschaft

44

Alpaka : Initialization

// Configure Alpaka using Dim = alpaka::dim::DimInt<3u> using Size = std::size_t using Acc = alpaka::acc::AccCpuSerial<Dim, Size>; using Host = alpaka::acc::AccCpuSerial<Dim, Size>; using Stream = alpaka::stream::StreamCpuSync; using WorkDiv = alpaka::workdiv::WorkDivMembers<Dim, Size>; using Elem = float;

Mitglied der Helmholtz-Gemeinschaft

45

Alpaka : Initialization

// Configure Alpaka using Dim = alpaka::dim::DimInt<3u> using Size = std::size_t using Acc = alpaka::acc::AccCpuSerial<Dim, Size>; using Host = alpaka::acc::AccCpuSerial<Dim, Size>; using Stream = alpaka::stream::StreamCpuSync; using WorkDiv = alpaka::workdiv::WorkDivMembers<Dim, Size>; using Elem = float; // Retrieve devices and stream DevHost devHost ( alpaka::dev::DevMan<Host>::getDevByIdx(0) ); DevAcc devAcc ( alpaka::dev::DevMan<Acc>::getDevByIdx(0) ); Stream stream ( devAcc);

Mitglied der Helmholtz-Gemeinschaft

46

Alpaka : Initialization

// Configure Alpaka using Dim = alpaka::dim::DimInt<3u> using Size = std::size_t using Acc = alpaka::acc::AccCpuSerial<Dim, Size>; using Host = alpaka::acc::AccCpuSerial<Dim, Size>; using Stream = alpaka::stream::StreamCpuSync; using WorkDiv = alpaka::workdiv::WorkDivMembers<Dim, Size>; using Elem = float; // Retrieve devices and stream DevHost devHost ( alpaka::dev::DevMan<Host>::getDevByIdx(0) ); DevAcc devAcc ( alpaka::dev::DevMan<Acc>::getDevByIdx(0) ); Stream stream ( devAcc); // Specify work division auto elementsPerThread ( alpaka::Vec<Dim, Size>::ones() ); auto threadsPerBlock ( alpaka::Vec<Dim, Size>::all(2u) ); auto blocksPerGrid ( alpaka::Vec<Dim, Size>(4u, 8u, 16u) ); WorkDiv workdiv(alpaka::workdiv::WorkDivMembers<Dim, Size>(blocksPerGrid, threadsPerBlock, elementsPerThread));

Mitglied der Helmholtz-Gemeinschaft

47

Alpaka : Call the Kernel

// Memory allocation and host to device memory copy auto X_h = alpaka::mem::buf::alloc<int, int>(devHost, extent); auto Y_h = alpaka::mem::buf::alloc<int, int>(devHost, extent); auto X_d = alpaka::mem::buf::alloc<Val, Size>(devAcc, extent); auto Y_d = alpaka::mem::buf::alloc<Val, Size>(devAcc, extent); alpaka::mem::view::copy(stream, X_d, X_h, extent); alpaka::mem::view::copy(stream, Y_d, Y_h, extent);

Mitglied der Helmholtz-Gemeinschaft

48

Alpaka : Call the Kernel

// Memory allocation and host to device memory copy auto X_h = alpaka::mem::buf::alloc<int, int>(devHost, extent); auto Y_h = alpaka::mem::buf::alloc<int, int>(devHost, extent); auto X_d = alpaka::mem::buf::alloc<Val, Size>(devAcc, extent); auto Y_d = alpaka::mem::buf::alloc<Val, Size>(devAcc, extent); alpaka::mem::view::copy(stream, X_d, X_h, extent); alpaka::mem::view::copy(stream, Y_d, Y_h, extent); // Kernel creation and execution VectorAdd kernel; auto const exec(alpaka::exec::create<Acc>( workDiv, kernel, numElements alpaka::mem::view::getPtrNative(X_d), alpaka::mem::view::getPtrNative(Y_d))); alpaka::stream::enqueue(stream, exec);

Mitglied der Helmholtz-Gemeinschaft

49

Alpaka : Call the Kernel

// Memory allocation and host to device memory copy auto X_h = alpaka::mem::buf::alloc<int, int>(devHost, extent); auto Y_h = alpaka::mem::buf::alloc<int, int>(devHost, extent); auto X_d = alpaka::mem::buf::alloc<Val, Size>(devAcc, extent); auto Y_d = alpaka::mem::buf::alloc<Val, Size>(devAcc, extent); alpaka::mem::view::copy(stream, X_d, X_h, extent); alpaka::mem::view::copy(stream, Y_d, Y_h, extent); // Kernel creation and execution VectorAdd kernel; auto const exec(alpaka::exec::create<Acc>( workDiv, kernel, numElements alpaka::mem::view::getPtrNative(X_d), alpaka::mem::view::getPtrNative(Y_d))); alpaka::stream::enqueue(stream, exec); // Copy memory back to host alpaka::mem::view::copy(stream, Y_h, Y_d, extent);

Mitglied der Helmholtz-Gemeinschaft

50

Alpaka : Call the Kernel

// Memory allocation and host to device memory copy auto X_h = alpaka::mem::buf::alloc<int, int>(devHost, extent); auto Y_h = alpaka::mem::buf::alloc<int, int>(devHost, extent); auto X_d = alpaka::mem::buf::alloc<Val, Size>(devAcc, extent); auto Y_d = alpaka::mem::buf::alloc<Val, Size>(devAcc, extent); alpaka::mem::view::copy(stream, X_d, X_h, extent); alpaka::mem::view::copy(stream, Y_d, Y_h, extent); // Kernel creation and execution VectorAdd kernel; auto const exec(alpaka::exec::create<Acc>( workDiv, kernel, numElements alpaka::mem::view::getPtrNative(X_d), alpaka::mem::view::getPtrNative(Y_d))); alpaka::stream::enqueue(stream, exec); // Copy memory back to host alpaka::mem::view::copy(stream, Y_h, Y_d, extent);

Mitglied der Helmholtz-Gemeinschaft

51

Cupla for the rescue : very fast porting !

Alpaka

Mitglied der Helmholtz-Gemeinschaft

52

Cupla for the rescue : very fast porting !

Alpaka

Fast enough for

live

hack session !

Mitglied der Helmholtz-Gemeinschaft

53

Live Cupla Hack Session

Tiling matrix-matrix multiplication algorithm

• Live port of CUDA to Cupla

application

• Starting point: matrix multiplication

algorithm of CUDA samples

• Aim: single source code executed on

GPU and CPU hardware

Mitglied der Helmholtz-Gemeinschaft

54

1450 GFLOPS

Single Source Alpaka DGEMM Kernel on Various Architectures

150 GFLOPS 540 GFLOPS 480 GFLOPS 560 GFLOPS

Theoretical Peak Performance DGEMM: C ← αAB + βC

283 121 74 117 35

Mitglied der Helmholtz-Gemeinschaft

55

What happend so far...

Mitglied der Helmholtz-Gemeinschaft

56

What happend so far...

Mitglied der Helmholtz-Gemeinschaft

57

What happend so far...

Mitglied der Helmholtz-Gemeinschaft

58

What happend so far...

Mitglied der Helmholtz-Gemeinschaft

59

What happend so far...

Mitglied der Helmholtz-Gemeinschaft

60

PIConGPU Runtime on Various Architectures

Mitglied der Helmholtz-Gemeinschaft

61

PIConGPU Efficiency on Various Architectures

Mitglied der Helmholtz-Gemeinschaft

62

Clone us from GitHub

https://github.com/ComputationalRadiationPhysics

git clone https://github.com/ComputationalRadiationPhysics/picongpu git clone https://github.com/ComputationalRadiationPhysics/alpaka git clone https://github.com/ComputationalRadiationPhysics/cupla Alpaka paper pre-print: http://arxiv.org/abs/1602.08477

Mitglied der Helmholtz-Gemeinschaft

63