Flexible, Scalable Mesh and Data Management using PETSc DMPlex M. - - PowerPoint PPT Presentation
Flexible, Scalable Mesh and Data Management using PETSc DMPlex M. Lange 1 M. Knepley 2 L. Mitchell 3 G. Gorman 1 1 AMCG, Imperial College London 2 Computational and Applied Mathematics, Rice University 3 Computing, Imperial College London June 24,
1AMCG, Imperial College London 2Computational and Applied Mathematics, Rice University 3Computing, Imperial College London
DMPlex Mesh Management
DMPlex Mesh Management
◮ Many tasks are common across applications:
Mesh input, partitioning, checkpointing, . . .
◮ File I/O can become severe bottleneck!
◮ Range of mesh generators and formats
Gmsh, Cubit, Triangle, ExodusII, Fluent, CGNS, . . .
◮ No universally accepted format ◮ Applications often “roll their own” ◮ No interoperability between codes
DMPlex Mesh Management
◮ Abstract mesh topology interface ◮ Provided by a widely used library1 ◮ Extensible support for multiple formats ◮ Single point for extension and optimisation ◮ Many applications inherit capabilities ◮ Mesh management optimisations ◮ Scalable read/write routines ◮ Parallel partitioning and load-balancing ◮ Mesh renumbering techniques ◮ Unstructured mesh adaptivity
Engineering, 2015
DMPlex Mesh Management
DMPlex Mesh Management
◮ Abstract mesh connectivity ◮ Directed Acyclic Graph (DAG)2 ◮ Dimensionless access ◮ Topology separate from discretisation ◮ Multigrid preconditioners ◮ PetscSection ◮ Describes irregular data arrays (CSR) ◮ Mapping DAG points to DoFs ◮ PetscSF: Star Forest3 ◮ One-sided description of shared data ◮ Performs sparse data communication 2 3 4 1 9 14 12 11 10 13 5 6 7 8 9 10 11 12 13 14 1 2 3 4
3Jed Brown. Star forests as a parallel communication model, 2011
DMPlex Mesh Management
2 3 4 1 9 14 12 11 10 13 5 6 7 8 9 10 11 12 13 14 1 2 3 4
cone(5) = {9, 10, 11}
5 6 7 8 9 10 11 12 13 14 1 2 3 4
support(4) = {12, 13, 14}
5 6 7 8 9 10 11 12 13 14 1 2 3 4
stratumheight(1) = {5, 6, 7, 8}
5 6 7 8 9 10 11 12 13 14 1 2 3 4
closure(1) = {1, 2, 3, 5, 9, 10, 11}
5 6 7 8 9 10 11 12 13 14 1 2 3 4
star(14) = {0, 4, 6, 7, 8, 12, 13, 14}
DMPlex Mesh Management
◮ Input: ExodusII, Gmsh, CGNS,
◮ Output: HDF5 + Xdmf ◮ Visualizable checkpoints ◮ Parallel distribution ◮ Partitioners: Chaco, Metis/ParMetis ◮ Automated halo exchange via PetscSF ◮ Mesh renumbering ◮ Reverse Cuthill-McGee (RCM) 2 3 4 1 9 14 12 11 10 13 5 6 7 8 9 10 11 12 13 14 1 2 3 4
DMPlex Mesh Management
DMPlex Mesh Management
◮ Mesh partitioning ◮ Topology-based partitioning ◮ Metis/ParMetis, Chaco ◮ Mesh and data migration ◮ One-to-all and all-to-all ◮ Data migration via SF ◮ Parallel overlap computation ◮ Generic N-level point overlap ◮ FVM and FEM adjacency
Partition 0 Partition 1
DMPlex Mesh Management
DMPlex Mesh Management
DMPlex Mesh Management
DMPlex Mesh Management
◮ Cray XE30 with 4920 nodes; 2 × 12-core E5-2697 @2.7GHz1 ◮ 1283 unit cube with ≈ 12 mio. cells
One-to-all distribution
2 6 12 24 48 96 Number of processors 100 101 Time [sec]
Distribute Overlap Distribute: Partition Distribute: Migration Overlap: Partition Overlap: Migration
All-to-all redistribution
2 6 12 24 48 96 Number of processors 100 101 Time [sec]
Redistribute Redistribute: Partition Redistribute: Migration
◮ Remaining bottleneck: Sequential partitioning 1ARCHER: www.archer.ac.uk
DMPlex Mesh Management
◮ Distribute coarse mesh and refine in parallel ◮ Generate overlap on resulting fine mesh
2 6 12 24 48 96 Number of processors 10-1 100 101 102 time [sec]
Overlap, m 128, refine 0 Distribute, m 128, refine 0 Generate, m 128, refine 0 Refine, m 128, refine 0 Overlap, m 64, refine 1 Distribute, m 64, refine 1 Generate, m 64, refine 1 Refine, m 64, refine 1 Overlap, m 32, refine 2 Distribute, m 32, refine 2 Generate, m 32, refine 2 Refine, m 32, refine 2
DMPlex Mesh Management
DMPlex Mesh Management
◮ Unstructured finite element code ◮ Anisotropic mesh adaptivity ◮ Uses PETSc as linear solver engine ◮ Applications: ◮ CFD, geophysical flows, ocean
modelling, reservoir modelling, mining, nuclear safety, renewable energies, etc.
2 4 6 8 1 2 1 2
DMPlex Mesh Management
Zoltan
DMPlexDistribute
Load Balance
DMPlex Mesh Management
◮ Delegate mesh input to DMPlex ◮ More formats and improved interoperability ◮ Potential performance improvements ◮ Maintained by third-party library ◮ Domain decomposition at run-time ◮ Remove pre-processing step ◮ Avoid unnecessary I/O cycle ◮ Topology partitioning reduces communication ◮ Replaces existing mesh readers ◮ Build and distribute DMPlex object ◮ Fluidity initialisation in parallel ◮ Only one format-agnostic reader method
DMPlex Mesh Management
◮ Fluidity Halos ◮ Separate L1/L2 regions ◮ “Trailing receives” ◮ Requires permutation ◮ DMPlex provides RCM ◮ Locally generated as a
permutation
◮ Combine with halo reordering ◮ Fields inherit reordering ◮ Better cache coherency
DMPlex Mesh Management
◮ Cray XC30 ◮ 4920 nodes (118,080 cores) ◮ 12-core E5-2697 (Ivy Bridge)
◮ Flow past a square cylinder ◮ 3D mesh, generated with Gmsh
DMPlex Mesh Management
◮ Runtime distribution wins ◮ Fast topology distribution ◮ No clear I/O gains ◮ Gmsh does not scale 8615 1183692 1942842 2944992 Mesh Size [elements] 5 10 15 20 25 30 time [sec]
Fluidity Startup - File I/O
Preprocessor-Read Preprocessor-Write Fluidity-Read Fluidity-Total DMPlex-DAG
8615 1183692 1942842 2944992 Mesh Size [elements] 20 40 60 80 100 120 time [sec]
Fluidity Startup
Preprocessor Fluidity (preprocessed) Fluidity Total Fluidity-DMPlex
8615 1183692 1942842 2944992 Mesh Size [elements] 10 20 30 40 50 60 70 time [sec]
Fluidity Startup - Distribute
Zoltan + Callbacks DMPlexDistribute
DMPlex Mesh Management
◮ Mesh with ∼2 mio elements ◮ Preprocessor + 10 timesteps ◮ RCM brings improvements ◮ Pressure solve ◮ Velocity assembly 2 6 12 24 48 96 Number of Processes 102 time [sec]
Pressure Solve
Fluidity-DMPlex: RCM Fluidity-DMPlex: native Fluidity-Preprocessor
2 6 12 24 48 96 Number of Processes 103 time [sec]
Full Simulation
Fluidity-DMPlex: RCM Fluidity-DMPlex: native Fluidity-Preprocessor
2 6 12 24 48 96 Number of Processes 102 time [sec]
Velocity Assembly
Fluidity-DMPlex: RCM Fluidity-DMPlex: native Fluidity-Preprocessor
DMPlex Mesh Management
DMPlex Mesh Management
◮ Re-envision FEniCS2 φn+1/2 = φn ´ ∆t 2 pn pn+1 = pn + ż
Ω
∇φn+1/2 ¨ ∇v dx ż
Ω
v dx @v P V φn+1 = φn+1/2 ´ ∆t 2 pn+1 where ∇φ ¨ n = 0 on ΓN p = sin(10πt) on ΓD from firedrake import * mesh = Mesh("wave_tank.msh") V = FunctionSpace(mesh , ’Lagrange ’, 1) p = Function(V, name="p") phi = Function(V, name="phi") u = TrialFunction(V) v = TestFunction(V) p_in = Constant (0.0) bc = DirichletBC(V, p_in , 1) T = 10. dt = 0.001 t = 0 while t <= T: p_in.assign(sin (2*pi*5*t)) phi
p += assemble(dt * inner(grad(v), grad(phi))*dx) \ / assemble(v*dx) bc.apply(p) phi
t += dt
finite element method by composing abstractions. Submitted to ACM TOMS, 2015
DMPlex Mesh Management
◮ Implements UFL1 ◮ Outer framework in Python ◮ Run-time C code generation ◮ PyOP2: Assembly kernel
execution framework
◮ Domain topology from DMPlex ◮ Mesh generation and file I/O ◮ Derive discretisation-specific
mappings at run-time
◮ Geometric Multigrid Unified Form Language
PyOP2 Interface
modified FFC Parallel scheduling, code generation CPU (OpenMP/ OpenCL) GPU (PyCUDA / PyOpenCL) Future arch.
FEM problem (weak form PDE) Local assembly kernels (AST) Parallel loops: kernels executed over mesh Explicitly parallel hardware- specific implemen- tation Meshes, matrices, vectors
PETSc4py (KSP, SNES, DMPlex)
Firedrake/FEniCS language
MPI
Geometry, (non)linear solves assembly, compiled expressions
FIAT parallel loop parallel loop COFFEE AST optimiser data structures (Set, Map, Dat)
Domain specialist: mathematical model using FEM Numerical analyst: generation of FEM kernels Domain specialist: mathematical model on un- structured grid Parallel programming expert: hardware architectures,
Expert for each layer
DMPlex Mesh Management
◮ DMPlex encodes topology ◮ Parallel distribution ◮ Application ordering ◮ Section encodes discretisation ◮ Maps DAG to solution DoFs ◮ Generated via FIAT element1 ◮ Derives PyOP2 indirection
maps for assembly
◮ SF performs halo exchange ◮ DMPlex derives SF from
section and overlap
Mesh
DMPlex
IS
Permutation
FunctionSpace FIAT.Element PetscSection DAG→DoF PyOP2.Map Cell→DoF Halo PetscSF
Function Vec Local Remote
30(4):502–516, 2004
DMPlex Mesh Management
◮ Run-time code generation ◮ Intermediate representation ◮ Kernel optimisation via AST1 ◮ Overlapping communication ◮ Core: Execute immediately ◮ Non-core: Halo-dependent ◮ Halo: Communicate while
computing over core
◮ Imposes ordering constraint
Intensity for Finite Element Local Assembly. Accepted for publication, ACM Transactions on Architecture and Code Optimization, 2015
DMPlex Mesh Management
◮ Mesh renumbering ◮ Improves cache coherency ◮ Reverse Cuhill-McKee (RCM) ◮ Combine RCM with PyOP2
◮ Filter cell reordering ◮ Apply within PyOP2 classes ◮ Add DoFs per cell (closure)
SISC Special Issue, 2015
DMPlex Mesh Management
◮ Cell integral:
◮ Facet integral: L = u(’+’)* dS
1 2 6 12 24 48 96 Number of processors 10-2 10-1 100 time [sec]
Cell integral, RCM Facet integral, RCM Cell integral, Native Facet integral, Native
1 2 6 12 24 48 96 Number of processors 10-2 10-1 100 time [sec]
Cell integral, RCM Facet integral, RCM Cell integral, Native Facet integral, Native
DMPlex Mesh Management
◮ Eq:
∂c ∂t + ▽· (
◮ L-shaped mesh with ≈ 3.1 mio. cells
1 2 6 12 24 48 96 Number of processors 10-2 10-1 100 101 time [sec]
Advection, RCM Diffusion, RCM Advection, Native Diffusion, Native
1 2 6 12 24 48 96 Number of processors 10-1 100 101 102 time [sec]
Advection, RCM Diffusion, RCM Advection, Native Diffusion, Native
DMPlex Mesh Management
◮ Eq:
∂c ∂t + ▽· (
◮ L-shaped mesh with ≈ 3.1 mio. cells
1 2 6 12 24 48 96 Number of processors 10-2 10-1 100 101 time [sec]
Advection, RCM Diffusion, RCM Advection, Native Diffusion, Native
1 2 6 12 24 48 96 Number of processors 10-1 100 101 102 time [sec]
Advection, RCM Diffusion, RCM Advection, Native Diffusion, Native
DMPlex Mesh Management
◮ Advection solver: CG + Jacobi ◮ Diffusion solver:
1 2 6 12 24 48 96 Number of processors 10-2 10-1 100 101 102 103 time [sec]
Advection, RCM Diffusion, RCM Advection, Native Diffusion, Native
1 2 6 12 24 48 96 Number of processors 100 101 102 103 104 time [sec]
Advection, RCM Diffusion, RCM Advection, Native Diffusion, Native
DMPlex Mesh Management
DMPlex Mesh Management
◮ Unified mesh reader/writer interface ◮ Improves compatability and interoperability ◮ Delegate I/O and it’s optimisation ◮ Improved DMPlexDistribute ◮ Run-time domain decomposition ◮ Scalable overlap generation ◮ All-to-all load balancing ◮ Firedrake FE environment ◮ DMPlex as topology abstraction ◮ Derive indirection maps for assembly ◮ Compact RCM renumbering
◮ Parallel mesh file reads ◮ Anisotropic mesh adaptvity
DMPlex Mesh Management
https://fluidityproject.github.io www.firedrakeproject.org http://www.archer.ac.uk/ Intel PCC
DMPlex Mesh Management