An Error Correction Solver for Linear Systems Evaluation of Mixed - - PowerPoint PPT Presentation

Karlsruhe Institute of Technology KIT – University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz Association

www.kit.edu

An Error Correction Solver for Linear Systems

Evaluation of Mixed Precision Implementations

Hartwig Anzt Berkeley, June 23rd 2010

Engineering Mathematics and Computing Lab (EMCL)

Karlsruhe Institute of Technology

Numerical Simulation, Optimization and HPC

Engineering Mathematics and Computing Lab

2/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Numerical Simulation, Optimization and HPC

Engineering Mathematics and Computing Lab

Topics Meteorology and Environment Medicine and Biotechnology Energy Tools Mathematical Modeling FEM-Software Scientific Computing

3/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Numerical Simulation, Optimization and HPC

Engineering Mathematics and Computing Lab

Topics Meteorology and Environment Medicine and Biotechnology Energy Tools Mathematical Modeling FEM-Software Scientific Computing

www.united-airways.eu www.emcl.kit.edu Simulation of the Human Respiratory System Simulation of Tropical Cyclones

4/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Motivation Hybrid Mixed Precision Solvers Iterative Refinement Method Numerical Experiments Conclusions

Computational Fluid Dynamics

Linear Systems in Fluid Dynamics

Partial differential equations model Fluid Flow Problems Discretization methods lead to nonlinear equations systems Linearization methods lead to linear system Ax = b

Solving the linear system is often the most time-consuming step in the simulation process.

5/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Motivation Hybrid Mixed Precision Solvers Iterative Refinement Method Numerical Experiments Conclusions

Acceleration Scenarios

Computational Effort

Characteristics of the linear problem Characteristics of the applied solver Used floating point format

Acceleration Concepts

Use of different precision formats Use of Coprocessor Technology Parallelization of the linear solvers

Fundamental Idea of Mixed Precision Solvers:

Acceleration without loss of accuracy High precision only in relevant parts Low precision for most of the algorithm Outsource low precision computations on parallel low precision Coprocessor Use different precision formats within the Iterative Refinement Method

6/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Motivation Hybrid Mixed Precision Solvers Iterative Refinement Method Numerical Experiments Conclusions

Iterative Refinement Method

Newton’s Method: xi+1 = xi − (∇f(xi))−1f(xi) Idea: Apply Newton’s Method to the function f(x) = b − Ax

xi+1 = xi − (∇f(xi))−1f(xi) = xi − (−A−1)(b − Axi) = xi + A−1(b − Axi) = xi + A−1ri

=:ci

1: initial guess as starting vector: x0 2: compute initial residual: r0 = b − Ax0 3: while ( Axi − b 2> ε r0 ) do 4:

ri = b − Axi

5:

solve: Aci = ri

6:

update solution: xi+1 = xi + ci

7: end while

7/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Motivation Hybrid Mixed Precision Solvers Iterative Refinement Method Numerical Experiments Conclusions

Iterative Refinement Method

Newton’s Method: xi+1 = xi − (∇f(xi))−1f(xi) Idea: Apply Newton’s Method to the function f(x) = b − Ax

xi+1 = xi − (∇f(xi))−1f(xi) = xi − (−A−1)(b − Axi) = xi + A−1(b − Axi) = xi + A−1ri

=:ci

1: initial guess as starting vector: x0 2: compute initial residual: r0 = b − Ax0 3: while ( Axi − b 2> ε r0 ) do 4:

ri = b − Axi

5:

solve: Aci = ri

6:

update solution: xi+1 = xi + ci

7: end while

8/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Motivation Hybrid Mixed Precision Solvers Iterative Refinement Method Numerical Experiments Conclusions

Iterative Refinement Method

Newton’s Method: xi+1 = xi − (∇f(xi))−1f(xi) Idea: Apply Newton’s Method to the function f(x) = b − Ax

xi+1 = xi − (∇f(xi))−1f(xi) = xi − (−A−1)(b − Axi) = xi + A−1(b − Axi) = xi + A−1ri

=:ci

In-Exact Newton Method

Apply iterative method to solve Aci = ri Residual stopping criterion εinner >> ε E.g. Krylov Subspace Solver

9/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Motivation Hybrid Mixed Precision Solvers Iterative Refinement Method Numerical Experiments Conclusions

Mixed Precision Iterative Refinement

1: initial guess as starting vector: x0 2: compute initial residual: r0 = b − Ax0 3: while ( Axi − b 2> ε r0 ) do 4:

ri = b − Axi

5:

solve: Aci = ri

6:

update solution: xi+1 = xi + ci

7: end while

Any error correction solver can be used i.e. Krylov Subspace Methods (CG, GMRES ...)

Compute residual

Anzt Anzt Anzt Anzt

Set initial Vector Update Solution Check Stopping Criterion Solution Solve error correction equation

10/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Motivation Hybrid Mixed Precision Solvers Iterative Refinement Method Numerical Experiments Conclusions

Mixed Precision Iterative Refinement

1: initial guess as starting vector: x0 2: compute initial residual: r0 = b − Ax0 3: while ( Axi − b 2> ε r0 ) do 4:

ri = b − Axi

5:

solve: Aci = ri

6:

update solution: xi+1 = xi + ci

7: end while

Any error correction solver can be used i.e. Krylov Subspace Methods (CG, GMRES ...)

Compute residual

Low Precision Format High Precision Format

Anzt Anzt Anzt Anzt

Set initial Vector Update Solution Check Stopping Criterion Solution T ypecast residual into low precision T ypecast solution update into high precision Solve error correction equation in low precision

11/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Motivation Hybrid Mixed Precision Solvers Iterative Refinement Method Numerical Experiments Conclusions

Numerical Experiments

Implementation Issues

Double precision GMRES-(10)

• n 1 core

Mixed precision GMRES-(10)

• n 1 core

Double precision GMRES-(10)

• n 4 cores

Mixed precision GMRES-(10)

• n 4 cores

Mixed precision GMRES-(10)

• n NVIDIA TESLA S1070

Test Matrices

Artificially created Test Matrices Increasing dimension Fixed Condition Number sparse/dense Real Application Test Matrices Discretization of Fluid Flow in Venturi Nozzle different dimension, condition number, sparsity

12/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Motivation Hybrid Mixed Precision Solvers Iterative Refinement Method Numerical Experiments Conclusions

Test-Case: Artificial Test Matrices

M1 M2 M3

          10 · n ∗ · · · · · · ∗ ∗ 10 · n ... . . . . . . ... 10 · n ... . . . . . . ... ... ∗ ∗ · · · · · · ∗ 10 · n                     W V ∗ · · · ∗ V W V ... . . . ∗ V W ... ∗ . . . ... ... ... V ∗ · · · ∗ V W          

                           H −1 · · · −1 · · · −1 H −1 ... ... ... ... . . . ... H ... ... ... ... . . . ... ... ... ... −1 −1 ... . . . ... ... ... . . . ... ... ... ... H −1 · · · −1 · · · −1 H                           

problem: artificial matrix problem size: variable sparsity: nnz = n2

• cond. number: κ < 3

storage format: MAS problem: artificial matrix problem size: variable sparsity: nnz = n2

• cond. number: κ ≈ 8 · 103

storage format: MAS problem: artificial matrix problem size: variable sparsity: nnz = 5n

• cond. number: κ ≈ 8 · 103

storage format: CRS 13/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Motivation Hybrid Mixed Precision Solvers Iterative Refinement Method Numerical Experiments Conclusions

Mixed Precision Iterative Refinement GMRES

problem size [n] computation time [s]

1e+ 04 2e+ 04 2 4 6 8 10 12 14

mixed precision GMRES-(10) solver plain double precision GMRES-(10) solver

problem size [n] time factor f

1e+ 04 2e+ 04 1 2

Artificial Test-Matrix M1: κ < 3, nnz = n2 Constant factor f Plain GMRES performs faster due to low condition number

14/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Motivation Hybrid Mixed Precision Solvers Iterative Refinement Method Numerical Experiments Conclusions

Mixed Precision Iterative Refinement GMRES

problem size [n] computation time [s]

1e+04 2e+04 10 20

plain double precision GMRES-(10) on 1 core mixed precision GMRES-(10) on 1 core plain double precision GMRES-(10) on 4 cores mixed precision GMRES-(10) on 4 cores mixed precision GMRES-(10) on TESLA S1070 plain double precision GMRES-(10) on 1 core mixed precision GMRES-(10) on 1 core plain double precision GMRES-(10) on 4 cores mixed precision GMRES-(10) on 4 cores mixed precision GMRES-(10) on TESLA S1070

Artificial Test-Matrix M1: κ < 3, nnz = n2

15/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Motivation Hybrid Mixed Precision Solvers Iterative Refinement Method Numerical Experiments Conclusions

Mixed Precision Iterative Refinement GMRES

problem size [n] computation time [s]

1e+ 04 2e+ 04 1e+ 03 2e+ 03 3e+ 03

mixed precision GMRES-(10) solver plain double precision GMRES-(10) solver

problem size [n] time factor f

1e+ 04 2e+ 04 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1

Artificial Test-Matrix M2: κ ≈ 8000, nnz = n2 Constant factor f Mixed Precision Iterative Refinement GMRES performs faster due to high condition number

16/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Motivation Hybrid Mixed Precision Solvers Iterative Refinement Method Numerical Experiments Conclusions

Mixed Precision Iterative Refinement GMRES

problem size [n] computation time [s]

• ✁ ✂

• ✁ ✂

• ✁

• ✁

• ✁

plain

plain

plain

plain

Artificial Test-Matrix M2: κ ≈ 8000, nnz = n2

17/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Motivation Hybrid Mixed Precision Solvers Iterative Refinement Method Numerical Experiments Conclusions

Mixed Precision Iterative Refinement GMRES

problem size [n] computation time [s]

1e+ 06 2e+ 06 200 400 600 800 1e+ 03 1.2e+ 03 1.4e+ 03

problem size [n] time factor f

1e+ 06 2e+ 06 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1 1.2

mixed precision GMRES-(10) solver plain double precision GMRES-(10) solver

Artificial Test-Matrix M3: κ ≈ 8000, nnz = 5n Constant factor f Mixed Precision Iterative Refinement GMRES performs faster due to high condition number

18/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Motivation Hybrid Mixed Precision Solvers Iterative Refinement Method Numerical Experiments Conclusions

Mixed Precision Iterative Refinement GMRES

problem size [n] computation time [s]

plain

plain

Artificial Test-Matrix M3: κ ≈ 8000, nnz = 3n

19/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Motivation Hybrid Mixed Precision Solvers Iterative Refinement Method Numerical Experiments Conclusions

Result Interpretation

Numerical Experiments

Plain solver is superior for low condition numbers Mixed precision mode superior for high condition numbers Quotient between plain- and mixed-precision- mode independent of problem size Acceleration by Coprocessor Technology

dimension n of the Matrix Plain Solver Superiority Mixed Precision Iterative Refinement Superiority Condition Number

• f the Matrix

20/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Motivation Hybrid Mixed Precision Solvers Iterative Refinement Method Numerical Experiments Conclusions

Test-Case: Fluid Flow in Venturi Nozzle

CFD1 CFD2 CFD3

problem: 2D fluid flow problem size: n=395009 sparsity: nnz=3544321 storage format: CRS problem: 2D fluid flow problem size: n=634453 sparsity: nnz=5700633 storage format: CRS problem: 2D fluid flow problem size: n=1019967 sparsity: nnz=9182401 storage format: CRS

21/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Motivation Hybrid Mixed Precision Solvers Iterative Refinement Method Numerical Experiments Conclusions

Test-Case: Fluid Flow in Venturi Nozzle

discretization: n=395009 nnz=3544321 discretization: n=634453 nnz=5

discretization: n=101996

nnz=9

1 2 3 5 6

4

speedup

plain double precision GMRES-(10) on 1 core mixed precision GMRES-(10) on 1 core plain double precision GMRES-(10) on 4 cores mixed precision GMRES-(10) on 4 cores mixed precision GMRES-(10) on NVIDIA TESLA S1070

22/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems

Karlsruhe Institute of Technology

Motivation Hybrid Mixed Precision Solvers Iterative Refinement Method Numerical Experiments Conclusions

Conclusions

Mixed Precision Solvers

Exploiting the excellent low precision performance of Accelerator Technology (e.g. GPU) High potential in case of computationally expensive problems (e.g. our CFD-Problems) FPGA-Technology offers free choice of Floating Point Formats

www.conveycomputer.com

23/23 Berkeley, June 23rd 2010 Hartwig Anzt - An Error Correction Solver for Linear Systems