mpiFileUtils Parallel File Utilities for HPC November 13, 2017 - PowerPoint PPT Presentation

mpiFileUtils Parallel File Utilities for HPC November 13, 2017 Danielle Sikich LLNL-PRES-740981 This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC

Motivation § What is mpiFileUtils? — Suite of MPI-based suite of file utilities built on a common library for handling large HPC datasets — Similar to current unix file commands such as cp, rm, cmp, etc. § Why mpiFileUtils? — Applications that produce and consume data are highly- parallel and frequently optimized for HPC file systems — UNIX file management utilities are single-process solutions 2 LLNL-PRES-740981

Tools § Currently available for use: dchmod – change permissions & group access 1. dcp – copy files 2. dcmp – compare files 3. drm – remove files 4. dstripe – restripe files 5. dwalk – list files 6. 3 LLNL-PRES-740981

1 Million File Experiment Tool 1 Proc 64 Procs 256 Procs cp 12 hours dcp 1x (12hr) 37x (19m) 51x (14m) rsync 6 hours dcmp 0.5x (12hr) 51x (7m) 60x (6m) rm 15 minutes drm 0.5x (28m) 3x (5m) 5x (3m) 4 LLNL-PRES-740981

Common Library § Easy for contributors to add new parallel file utilities based on a common API Your tool here! 5 LLNL-PRES-740981

Acknowledgements & Download § Result of multi-institutional collaboration: LANL, LLNL, ORNL, DataDirect Networks, Red Hat, and Australian National University § We are in Spack — spack install mpifileutils § Link to GitHub site: — https://github.com/hpc/mpifileutils 6 LLNL-PRES-740981