Parallelizing Quickref Didier Verna EPITA / LRDE - - PowerPoint PPT Presentation

Conclusion Introduction Toolchain Experiments Solution Perspectives

Parallelizing Quickref

e ELS 2019 E Didier Verna

EPITA / LRDE

didier@lrde.epita.fr

lrde/~didier @didierverna didier.verna in/didierverna

Conclusion Introduction Toolchain Experiments Solution Perspectives

Conclusion

◮ Quickref parallelized ◮ Performance improvement: 4x Thank you! Any questions?

Parallelizing Quickref / ELS 2019 – Didier Verna 2/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Introduction

◮ Reference manuals for CL (Quicklisp) libraries ◮ quickref.common-lisp.net

• r local builds

◮ Originally: sequential loop

• ver 1700+ libraries

◮ 1h30 – 7h depending on the conditions ◮ Parallelism worth investigating

Parallelizing Quickref / ELS 2019 – Didier Verna 3/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Plan

Toolchain Experiments Parallel Solution Discussion and Perspectives

Parallelizing Quickref / ELS 2019 – Didier Verna 4/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Plan

Toolchain Experiments Parallel Solution Discussion and Perspectives

Parallelizing Quickref / ELS 2019 – Didier Verna 5/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Toolchain

Quicklisp foo/ Declt foo.texi Makeinfo foo.html

Important remarks: ◮ Declt works by introspection

◮ Compilation / loading (of dependencies) may be required ◮ Avoid loading 1700+ libraries in the same Lisp image! ◮ Run Declt in external processes

◮ Makeinfo is a Perl/C script

◮ Ditto

Parallelizing Quickref / ELS 2019 – Didier Verna 6/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Plan

Toolchain Experiments Parallel Solution Discussion and Perspectives

Parallelizing Quickref / ELS 2019 – Didier Verna 7/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Scenarios

◮ Quickref Options

◮ Libraries and Update

◮ full / installed-only ◮ download / compilation may occur

◮ Compilation cache Policy

◮ global / local ◮ global cache may cause problems

◮ Scenarios

• 1. All libraries already compiled (1h 27m)
• 2. Nothing compiled, global compilation cache (1h 51m)
• 3. Nothing compiled, local compilation cache (7h 01m)

Parallelizing Quickref / ELS 2019 – Didier Verna 8/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Time Distribution

Makeinfo 48 % Declt 48 % Loading 4 %

1

Makeinfo 40 % Declt 41 % Loading 3 % Compilation 16 %

2

Makeinfo 10 % Declt 10 % Loading 1 % Compilation 79 %

3 Scenario Texinfo Html 1 52% 48% 2 60% 40% 3 90% 10%

Parallelizing Quickref / ELS 2019 – Didier Verna 9/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Plan

Toolchain Experiments Parallel Solution Discussion and Perspectives

Parallelizing Quickref / ELS 2019 – Didier Verna 10/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Parallel Solution

Library Pool Declt Declt Declt Texinfo Files Makeinfo Makeinfo Makeinfo HTML Files

◮ Usable in all scenarios ◮ Dependency graph management cheap ◮ Scenario 2

◮ Best results: 4 Declt threads / 4 Makeinfo threads ◮ Total time: 28m 17s (25% of sequential time)

Parallelizing Quickref / ELS 2019 – Didier Verna 11/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Library Pool Management

Dependency Graph

5 6 1 2 4 3

Library Pool Done

2 1 3 4 5 6

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Library Pool Management

Dependency Graph

5 6 1 2 4 3

Library Pool

1 2

Done

2 1 3 4 5 6

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Library Pool Management

Dependency Graph

5 6 1 2 4 3

Library Pool

1 3

Done

2 1 3 4 5 6

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Library Pool Management

Dependency Graph

5 6 1 2 4 3

Library Pool

4

Done

2 1 3 4 5 6

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Library Pool Management

Dependency Graph

5 6 1 2 4 3

Library Pool

5

Done

2 1 3 4 5 6

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Library Pool Management

Dependency Graph

5 6 1 2 4 3

Library Pool

6

Done

2 1 3 4 5 6

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Library Pool Management

Dependency Graph

5 6 1 2 4 3

Library Pool Done

2 1 3 4 5 6

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Plan

Toolchain Experiments Parallel Solution Discussion and Perspectives

Parallelizing Quickref / ELS 2019 – Didier Verna 14/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Discussion and Perspectives

◮ Alternative Algorithms ◮ Dependency Management Issues

◮ Based on static information provided by Quicklisp ◮ Fragile, not always correct

◮ CPU vs. I/O Consumption

◮ 4x is a bit disappointing ◮ Open the Declt and Makeinfo black boxes

◮ SJF-like Scheduling

◮ Very diffjcult to fjgure out where complexity comes from ◮ Collect timings and use them in next run

◮ SSD!

Parallelizing Quickref / ELS 2019 – Didier Verna 15/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Acknowledgments

◮ Initial code base: Antoine Martin ◮ Author index and parallel algorithm #4: Antoine Hacquard ◮ Hosting (code & website): CLF / common-lisp.net

Parallelizing Quickref / ELS 2019 – Didier Verna 16/16