Particle-in-cell simulations of mirror mode structures in the - - PowerPoint PPT Presentation

▶
particle in cell simulations of mirror mode structures in
SMART_READER_LITE
LIVE PREVIEW

Particle-in-cell simulations of mirror mode structures in the - - PowerPoint PPT Presentation

Jan 01, 2023 370 likes •626 views

PSC Mirror modes Summary Particle-in-cell simulations of mirror mode structures in the magnetosphere K. Germaschewski, N. Ahmadi, and J. Raeder Space Science Center / Dept. of Physics University of New Hampshire July 2, 2013 K.

slide-1
SLIDE 1

PSC Mirror modes Summary

Particle-in-cell simulations of mirror mode structures in the magnetosphere

  • K. Germaschewski, N. Ahmadi, and J. Raeder

Space Science Center / Dept. of Physics University of New Hampshire

July 2, 2013

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-2
SLIDE 2

PSC Mirror modes Summary

Outline

1

Plasma Simulation Code Introduction Numerical Heating PSC on GPUs

2

Mirror Modes in Geospace Introduction Simulation results

3

Summary / Outlook

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-3
SLIDE 3

PSC Mirror modes Summary Introduction Numerical Heating PSC on GPUs

Plasma Simulation Code (PSC)

1D, 2D, 3D configuration space relativistic, electromagnetic boost frame, moving window, PMLs, collisions, ionization... modular architecture: switching from legacy Fortran particle pusher to GPU pusher can be done on the command line. support for modern hardware (GPUs, Intel MIC)

Color indicates the MPI process responsible for local domain.

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-4
SLIDE 4

PSC Mirror modes Summary Introduction Numerical Heating PSC on GPUs

Kinetic Kelvin-Helmholtz Instability

(Loading khc025_2.mp4) 15360 × 7680 cells, 100 particles per cell performed on 900 GPUs (M2090, TitanDev) in ≈ 24 h wallclock

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-5
SLIDE 5

PSC Mirror modes Summary Introduction Numerical Heating PSC on GPUs

Particle-in-cell: Numerical Heating

Numerical Heating

Finite Grid Instability. Aliasing of unresolved grid modes gives rise to a numerical instability if the Debye length is not resolved. Stochastic heating. Particle noise leads to errors in the electromagnetic fields that heat the plasma linearly (∝ 1/N).

1000 2000 3000 4000 5000

time

50 100 150 200 250 300 350 400

total energy

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-6
SLIDE 6

PSC Mirror modes Summary Introduction Numerical Heating PSC on GPUs

Numerical Heating: dependence on particle shape

Remedies: Use more particles, or use higher order particles. Heating rate

101 102 103

particles per cell

10-7 10-6 10-5 10-4 10-3 10-2

heating rate

  • rder 1
  • rder .5
  • rder 2
  • rder 1.5

Performance

(16-core AMD Opteron / Nvidia K20X) pusher performance

  • rder 2/1.5

23 M/sec

  • rder 1

59 M/sec

  • rder 1 (single)

78 M/sec

  • rder 1 (SSE2)

94 M/sec

  • rder 1 (CUDA)

824 M/sec

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-7
SLIDE 7

PSC Mirror modes Summary Introduction Numerical Heating PSC on GPUs

PSC on GPUs

Multi-level decomposition of the problem, expose parallelism At the top-level, decompose spatial domain into patches. Each MPI process gets assigned one or more patches. Patches communicate via ghost cells / particle exchange. (Hybrid level can be introduced here: Each MPI process will distribute patches onto a set of cores or GPUs using OpenMP / threads) GPU: Each patch gets further divided into blocks (a.k.a. supercells) of multiple cells. These blocks are handled (in parallel) by threadblocks. Particles in a block are processed in parallel by threads in the threadblock (GPU) / by SIMD instructions (CPU/MIC).

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-8
SLIDE 8

PSC Mirror modes Summary Introduction Numerical Heating PSC on GPUs

PSC on GPUs

Particle-in-cell algorithm

for timestep n = 0,1,2,...: for each particle m: advance momentum: pn

m →

pn+1

m

(using interpolated En+1/2, Bn+1/2) advance position: xn+1/2

m

→ xn+3/2

m

deposit current density contribution jn+1

m

  • nto mesh.

advance fields: En+1/2, Bn+1/2 → En+3/2, Bn+3/2 using jn+1.

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-9
SLIDE 9

PSC Mirror modes Summary Introduction Numerical Heating PSC on GPUs

PSC on GPUs

Current deposition – 1st order Villasenor-Buneman

PSC used a 2nd order particle shape function and a corresponding charge-conserving current depo- sition scheme [Esirkepov 2001]. It calculates 40 contributions to the current density. Using 1st order particle shape functions and the charge-conserving scheme proposed by [Vil- lasenor, Buneman 1992], at most 10 points con- tribute.

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

Kernel Performance [particles/sec] 1st V-B, sorted by cell 250 × 106 1st V-B, randomized by block 500 × 106

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-10
SLIDE 10

PSC Mirror modes Summary Introduction Numerical Heating PSC on GPUs

PSC on GPUs – TitanDev/BlueWaters Performance

16-core AMD 6274 CPU, Nvidia Tesla M2090 / Tesla K20X Kernel Performance [particles/sec] 2D push & V-B current, CPU (AMD) 130 × 106 2D push & V-B current, GPU (M2090) 565 × 106 2D push & V-B current, GPU (K20X) 710 × 106

For best performance, need to use GPU and CPU simultaneously. Patch-based load balancing enables us to do that: On each node, we have 1 MPI-process that has ≈ 45 patches that are processed on the GPU, and 15 MPI-processes that have 1 patch each that are processed on the remaining CPU cores.

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-11
SLIDE 11

PSC Mirror modes Summary Introduction Numerical Heating PSC on GPUs

PSC on GPUs – Sorting

Particles need to be kept sorted in order to make use of shared memory for caching fields and updating currents. GPUs do best when“streaming” through particles, complicated data structures are often detrimental to highly threaded parallel performance. Our approach: Keep particle data in a streaming-friendly

  • ne-large-array layout (or SOA), sorted by block (a.k.a.

“supercell”, a block of, e.g. 4 × 4 cells.) Rearranging particles

1

Find block indices

2

Sort (block index, index) pairs

3

Use sort result to move each particle into its new position.

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-12
SLIDE 12

PSC Mirror modes Summary Introduction Numerical Heating PSC on GPUs

PSC on GPUs – Parallel Performance

Weak scaling study on Craxy XK7 “Titan” at ORNL.

20 320 80 1280 320 5120 1280 20480 5120 81920 top: number of Cray XK nodes (16-core AMD CPU + 1 Tesla 20X GPU) bottom: number of Cray XK/XE AMD CPU cores 100 200 300 400 500 600 million particles per second / node

GPU accelerated AMD CPUs

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-13
SLIDE 13

PSC Mirror modes Summary Introduction Simulation results

Temperatur anisotropy instabilities

Ion temperature T⊥ > T anistropy in a plasma with background magnetic field can drive kinetic instabilities: ion cyclotron instability (propagating) mirror instability (standing) Mirror mode instability condition T⊥ T > 1 + 1/β⊥ Mirror modes are observed in the solar wind, magnetosheath, and magnetosphere (near tail).

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-14
SLIDE 14

PSC Mirror modes Summary Introduction Simulation results

Mirror mode structures in the magnetosheath

Soucek et al., 2008

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-15
SLIDE 15

PSC Mirror modes Summary Introduction Simulation results

Mirror mode structures in the near tail

Ge et al., 2011

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-16
SLIDE 16

PSC Mirror modes Summary Introduction Simulation results

Mirror mode questions

Why is the mirror mode observed, rather than ion cyclotron waves? (depends on β, Helium stabilizes IC, IC propagate away, 2-d vs 3-d) How does the mirror mode evolve nonlinearly? What determines peaks vs dips? (mirror mode stable vs unstable regions) What role do electrons play? (isotropic in mirror structures) What is the spatial extent of mirror mode structures? (observations: smaller than ion gyroradius, tens of electron radii)

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-17
SLIDE 17

PSC Mirror modes Summary Introduction Simulation results

Linear growth rates in the presence of Helium

Maximum growth rates for βi = 4., βe = 0.4, βh = 4.,T⊥i/Ti = 1.5, T⊥e/Te = 1, T⊥h/Th = 1.5, mh/mi = 4.,

0.005 0.01 0.015 0.02 0.025 0.03 0.02 0.04 0.06 0.08 0.1 γm/Ωp nh Ion Cyclotron Mirror

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-18
SLIDE 18

PSC Mirror modes Summary Introduction Simulation results

2-d PIC simulation

Bi-Maxwellian ions and helium are uniformly distributed in the simulation space with T⊥i/Ti = 1.5. A constant background magnetic field B0 = 0.1 is assumed in the z direction. Other parameters are: βi = 4., βe = 2., βh = 4., T⊥e/Te = 1, T⊥h/Th = 1.5, mh/mi = 4., mi/me = 25, nh = 0.1

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-19
SLIDE 19

PSC Mirror modes Summary Introduction Simulation results

2-d PIC simulation

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-20
SLIDE 20

PSC Mirror modes Summary Introduction Simulation results

Anti-correlation of ni and δBz

0.84 0.85 0.86 0.87 0.88 0.89 0.9 0.91 0.92 0.93 0.94 0.95 10 20 30 40 50 60

  • 0.008
  • 0.006
  • 0.004
  • 0.002

0.002 0.004 0.006 0.008 0.01 0.012 0.014 ni δBz yωpi/c z = 32 c/ωpi ni δBz 1.35 1.4 1.45 1.5 1.55 1.6 1.65 1.7 10 20 30 40 50 60

  • 0.008
  • 0.006
  • 0.004
  • 0.002

0.002 0.004 0.006 0.008 0.01 0.012 0.014 Ai δBz yωpi/c z = 32 c/ωpi Ai δBz 0.84 0.86 0.88 0.9 0.92 0.94 0.96 10 20 30 40 50 60

  • 0.01
  • 0.005

0.005 0.01 0.015 ni δBz yωpi/c z = 64 c/ωpi ni δBz 1.35 1.4 1.45 1.5 1.55 1.6 1.65 1.7 10 20 30 40 50 60

  • 0.01
  • 0.005

0.005 0.01 0.015 Ai δBz yωpi/c z = 64 c/ωpi Ai δBz

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-21
SLIDE 21

PSC Mirror modes Summary Introduction Simulation results

Energy Balance, Evolution of Anisotropy

Energy Balance Anisotropies

  • 80
  • 60
  • 40
  • 20

20 40 60 80 200 400 600 800 1000 1200 ∆E Ωpt Ions Heliums Electrons fields 1.465 1.47 1.475 1.48 1.485 1.49 1.495 1.5 200 400 600 800 1000 1200 0.98 0.982 0.984 0.986 0.988 0.99 0.992 0.994 0.996 0.998 1 1.002 Tperp/Tpar Tperp e/Tpar e Ωpt Ions Heliums Electrons

T⊥i = 0.03, Ti = 0.02, T⊥h = 0.03, Th = 0.02, T⊥e = Te = 0.01, mi/me = 25

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-22
SLIDE 22

PSC Mirror modes Summary Introduction Simulation results

Skewness

The skewness is a statistical value to measure an asymmetry

  • f a distribution of samples. We use the skewness (Soucek et

al., 2008) to identify the magnetic structures as magnetic peaks (positive) or dips (negative). For a sample of n values the sample skewness is S =

1 n n

  • i=1

(xi−¯ x)3 ( 1

n n

  • i=1

(xi−¯ x)2)3/2

where ¯ x is sample mean.

Figure: http://en.wikipedia.org/wiki/Skewness

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-23
SLIDE 23

PSC Mirror modes Summary Introduction Simulation results

Peaks or Dips

In mirror unstable region (R = β⊥(T⊥/T − 1) > 0), we see peaks.

2 2.05 2.1 2.15 2.2 2.25 2.3 2.35 2.4 2.45 2.5 200 400 600 800 1000 1200 R Ωpt

  • 0.02

0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 200 400 600 800 1000 1200 skewness Ωpt Bz Skewness 0.84 0.85 0.86 0.87 0.88 0.89 0.9 0.91 0.92 0.93 0.94 0.95 10 20 30 40 50 60

  • 0.008
  • 0.006
  • 0.004
  • 0.002

0.002 0.004 0.006 0.008 0.01 0.012 0.014 ni δBz yωpi/c z = 32 c/ωpi ni δBz 0.84 0.86 0.88 0.9 0.92 0.94 0.96 10 20 30 40 50 60

  • 0.01
  • 0.005

0.005 0.01 0.015 ni δBz yωpi/c z = 64 c/ωpi ni δBz

  • K. Germaschewski et al.

PIC simulations of mirror modes

slide-24
SLIDE 24

PSC Mirror modes Summary

Summary / Outlook

GPU can accelerate PIC calculations substantially (> 4× on Titan) Particle shape order is very important for numerical heating, more work is needed to support 2nd order Esirkepov charge deposition on GPUs. Mirror instability has been benchmarked with nonlinear theory, nonlinear results show peaks as expected, but there are still many open questions.

Can we find dips in mirror stable regime? (R < 0) What is the energy exchange process between ion, electron thermal energy? What role does the simulation mass ratio play? Analyze the evolution of electron anisotropy.

  • K. Germaschewski et al.

PIC simulations of mirror modes