clang-scan-deps Fast Dependency Scanning For Explicit Modules - - PowerPoint PPT Presentation

▶
clang scan deps
SMART_READER_LITE
LIVE PREVIEW

clang-scan-deps Fast Dependency Scanning For Explicit Modules - - PowerPoint PPT Presentation

Oct 10, 2023 414 likes •982 views

clang-scan-deps Fast Dependency Scanning For Explicit Modules Alex Lorenz, Michael Spencer, Apple LLVM Developers Meeting, Brussels, Belgium, April 2019 1 Clang Modules Dependency Scanning Fast Dependency Scanning

slide-1
SLIDE 1

Alex Lorenz, Michael Spencer, Apple LLVM Developers’ Meeting, Brussels, Belgium, April 2019

  • clang-scan-deps
  • Fast Dependency Scanning For Explicit Modules
  • 1
slide-2
SLIDE 2
  • Clang Modules
  • Dependency Scanning
  • Fast Dependency Scanning
  • Dependency Extraction
  • Future Work
  • 2
slide-3
SLIDE 3

Clang Modules

  • Replace the textual preprocessor inclusions with an import of an AST
  • Widely used in SDKs shipped with Xcode
  • Implicit modules: Clang builds modules as they’re included
  • Users don’t have to specify modular dependencies
  • Requires a build system in the compiler

😬

  • 3
slide-4
SLIDE 4

Implicit Modules

A.cpp B.cpp C.cpp D.cpp

Compiler Discovered Build System Known

  • 4
slide-5
SLIDE 5

Implicit Modules

A.cpp B.cpp C.cpp D.cpp Transforms

Compiler Discovered Build System Known

  • 5
slide-6
SLIDE 6

Module Maps Implicit Modules

module LLVM_Transforms { requires cplusplus umbrella "Transforms" module * { export * } }

  • 6
slide-7
SLIDE 7

Implicit Modules

A.cpp B.cpp C.cpp D.cpp Transforms

Compiler Discovered Build System Known

  • 7
slide-8
SLIDE 8

Implicit Modules

A.cpp B.cpp C.cpp D.cpp Transforms

Compiler Discovered Build System Known

  • 8
slide-9
SLIDE 9

Implicit Modules

A.cpp B.cpp C.cpp D.cpp Transforms Analysis Analysis

Compiler Discovered Build System Known

  • 9
slide-10
SLIDE 10

Implicit Modules

A.cpp B.cpp C.cpp D.cpp Transforms Analysis

Compiler Discovered Build System Known

  • 10
slide-11
SLIDE 11

Implicit Modules

A.cpp B.cpp C.cpp D.cpp Transforms Analysis IR

Compiler Discovered Build System Known

  • 11
slide-12
SLIDE 12

Implicit Modules

A.cpp B.cpp C.cpp D.cpp Transforms Analysis IR

Compiler Discovered Build System Known

  • 12
slide-13
SLIDE 13

Implicit Modules

A.cpp B.cpp C.cpp D.cpp Transforms Analysis IR E.cpp -DFOO

Compiler Discovered Build System Known

  • 13
slide-14
SLIDE 14

Implicit Modules

A.cpp B.cpp C.cpp D.cpp Transforms Analysis IR E.cpp -DFOO Transforms

Compiler Discovered Build System Known

  • 14
slide-15
SLIDE 15

Implicit Modules

A.cpp B.cpp C.cpp D.cpp Transforms Analysis IR E.cpp -DFOO Transforms Analysis

Compiler Discovered Build System Known

  • 15
slide-16
SLIDE 16

Implicit Modules

A.cpp B.cpp C.cpp D.cpp Transforms Analysis IR E.cpp -DFOO Transforms Analysis IR

☹

Compiler Discovered Build System Known

  • 16
slide-17
SLIDE 17

Explicit Modules

A.cpp B.cpp C.cpp D.cpp Transforms Analysis IR E.cpp -DFOO Transforms Analysis IR

Compiler Discovered Build System Known

  • 17

Transforms IR

slide-18
SLIDE 18

Explicit Clang Modules

  • Better model: knowing modular dependencies before compiling
  • Allow more robust and reproducible builds
  • Faster builds

🏏

  • Constraint: users shouldn’t have to specify modular dependencies
  • Problem: which modules are needed?
  • Solution: dependency discovery build phase for a build target
  • 18
slide-19
SLIDE 19
  • Clang Modules
  • Dependency Scanning
  • Fast Dependency Scanning
  • Dependency Extraction
  • Future Work
  • 19
slide-20
SLIDE 20

Canonical Dependency Scanning Phase

  • Preprocess all translation units of a build target
  • Write out included files into a .d
  • clang -cc1 -Eonly -MT —dependency-file foo.d foo.c
  • How fast is the preprocessor?
  • 20
slide-21
SLIDE 21

Clang and LLVM sources: preprocessing time on an 18-Core iMac Pro

  • 21

Time (seconds)

15 30 45 60

Number Of Workers

6 12 18 24 30 36

Preprocessing Time

slide-22
SLIDE 22

Time (seconds)

15 30 45 60

Number Of Workers

6 12 18 24 30 36

Preprocessing Time

Clang and LLVM sources: the 12 workers scenario

  • 22
slide-23
SLIDE 23

31 seconds to preprocess

Time (seconds)

15 30 45 60

Number Of Workers

6 12 18 24 30 36

Preprocessing Time

Clang and LLVM sources: the 12 workers scenario

  • 23

31 seconds to preprocess 👏 Not fast enough for every build! 31 seconds to preprocess

slide-24
SLIDE 24
  • Clang Modules
  • Dependency Scanning
  • Fast Dependency Scanning
  • Dependency Extraction
  • Future Work
  • 24
slide-25
SLIDE 25

What Does The Preprocessor Do?

#ifndef HEADER_FILE #define HEADER_FILE #include “Compiler.h” // Clang is an awesome tool! class Clang: public Compiler { public: void buildAllCode(); #ifndef NDEBUG void dump(); #endif }; #endif

Lex tokens… Evaluate #ifndef & #define Lex more tokens… Include “Compiler.h” Lex more tokens… Lex even more tokens 😬

  • 25
slide-26
SLIDE 26

Reducing Preprocessor Workload

#ifndef HEADER_FILE #define HEADER_FILE #include “Compiler.h” // Clang is an awesome tool! class Clang: public Compiler { public: void buildAllCode(); #ifndef NDEBUG void dump(); #endif }; #endif

  • 26

31 seconds to preprocess 31 seconds to preprocess Dependencies aren’t affected by these tokens

slide-27
SLIDE 27

Reducing Preprocessor Workload

#ifndef HEADER_FILE #define HEADER_FILE #include “Compiler.h” // Clang is an awesome tool! class Clang: public Compiler { public: void buildAllCode(); #ifndefdNDEBUG void dump(); #endif }; #endif

  • 27

31 seconds to preprocess 31 seconds to preprocess Dependencies aren’t affected by these tokens

slide-28
SLIDE 28

Source Minimization

#ifndef HEADER_FILE #define HEADER_FILE #include “Compiler.h” #endif

  • 28
slide-29
SLIDE 29

Source Minimization

#ifndef HEADER_FILE #define HEADER_FILE #include “Compiler.h” #endif

💢 Keep directives that may affect dependency list Strip everything else Context free: source reused in any compilation

  • 29
slide-30
SLIDE 30

Clang and LLVM sources: 30% faster preprocessing

  • 30

Time (seconds)

15 30 45 60

Number Of Workers

6 12 18 24 30 36

Preprocessing Time Minimized Source Preprocessing Time

slide-31
SLIDE 31

Problem: Clang Invocations

Preprocess Preprocess Minimize Source Minimize Source #include “Test.h” Preprocess #include “Test.h” Preprocess

Parallel invocations do redundant work Read the same file twice Minimize the same file twice

  • 31
slide-32
SLIDE 32

Introducing clang-scan-deps

  • Library and command line tool for dependency scanning
  • Tool currently accepts compilation database and emits dependencies
  • Runs preprocessor invocations in parallel
  • Efficient: Reads and minimizes a source file only once
  • one shared FileSystem with shared minimized file cache
  • one shared FileManager
  • 32
slide-33
SLIDE 33

Minimized File Cache

  • Maps from file name to cache entry
  • Shared by worker threads: lock required access the StringMap
  • High lock contention for many threads
  • 33

Time (seconds)

1.25 2.5 3.75 5

Number Of Workers

18 24 30 36

One StringMap

slide-34
SLIDE 34

Optimizing Minimized File Cache

  • Solution: Array of StringMap addressed by hash of file name
  • 34

Time (seconds)

1.25 2.5 3.75 5

Number Of Workers

18 24 30 36

One StringMap Nine StringMaps

slide-35
SLIDE 35

Preprocessor Block Skipping

#ifdef NOT_TAKEN // Important comment #include “LexMeNot.h” #elif #include “IAmLexed.h” #endif

When this #ifdef is not taken…

  • 35

The tokens inside it are lexed… Until the #elif is found Took up to 10-15% of time in our profiles

slide-36
SLIDE 36

Optimizing Preprocessor Block Skipping

#ifdef NOT_TAKEN // Important comment #include “LexMeNot.h” #elif #include “IAmLexed.h” #endif

When this #ifdef is not taken…

  • 36

Skip to #elif: add offset to Lexer’s pointer Offset computed when minimizing file

slide-37
SLIDE 37

Clang and LLVM sources: 5-10x faster dependency scanning

  • 37

Time (seconds)

10 20 30 40

Number Of Workers

6 12 18 24 30 36

Preprocessing Time scan-deps Time

slide-38
SLIDE 38

#define AT_IMPORT @import AT_IMPORT Foundation;

Things We Aren’t Going To Support

➡ We want to disallow this behavior in Clang

  • 38

#define WHY(X) _##X ("clang module import X") WHY(Pragma);

slide-39
SLIDE 39

Modular Dependencies

  • clang-scan-deps builds implicit modules with minimized files
  • For now still uses old implicit module build machinery
  • Dependencies are extracted from the fast implicit build
  • 39
slide-40
SLIDE 40
  • Clang Modules
  • Dependency Scanning
  • Fast Dependency Scanning
  • Dependency Extraction
  • Future Work
  • 40
slide-41
SLIDE 41

Dependency Extraction

A.cpp.min B.cpp.min C.cpp.min D.cpp.min Transforms.min Analysis.min IR.min

  • 41
slide-42
SLIDE 42

Dependency Extraction

A.cpp.min B.cpp.min C.cpp.min D.cpp.min Transforms.min Analysis.min IR.min

  • 42
slide-43
SLIDE 43

Dependency Extraction

A.cpp.min B.cpp.min C.cpp.min D.cpp.min Transforms.min Analysis.min IR.min

  • 43
slide-44
SLIDE 44

Dependency Extraction

build LLVM_Transforms.pcm: cxx_explicit_module llvm/include/llvm/module.modulemap | LLVM_IR.pcm LLVM_Analysis.pcm std.pcm module_id = LLVM_Transforms moduledeps = -fmodule-file=LLVM_Config_Config.pcm -fmodule-file=std.pcm

  • fmodule-file=LLVM_IR.pcm -fmodule-file=LLVM_Analysis.pcm

args = builds/release/bin/clang++ -cc1 -fmodules ...

slide-45
SLIDE 45

Initial Results - Scanning

  • Faster than modules -Eonly and -Eonly, but slower than scan-deps
  • Building modules takes time

☹

Scan Time (seconds)

50 100 150 200

Number Of Workers

6 12 18 24 30 36

Modules -Eonly

  • Eonly

Modules scan-deps scan-deps

  • 45
slide-46
SLIDE 46

Initial Results - Scanning

  • Faster than modules -Eonly and -Eonly, but slower than scan-deps

Scan Time (seconds)

15 30 45 60

Number Of Workers

6 12 18 24 30 36

Modules -Eonly

  • Eonly

Modules scan-deps scan-deps

  • 46
slide-47
SLIDE 47

Initial Results - Scanning

Scan Time (seconds)

5.5 11 16.5 22

Number Of Workers

6 12 18 24 30 36

Modules -Eonly

  • Eonly

Modules scan-deps scan-deps

  • 47

The cost of building modules during scanning

slide-48
SLIDE 48

Initial Results - Building

  • About 5-15% speedup on an 18-Core iMac Pro

Build Time (seconds)

325 650 975 1300

Number Of Workers

6 12 18 24 30 36

Implicit Scan + Explicit Explicit

  • 48
slide-49
SLIDE 49

Bugs 🐟

  • Implicit and Explicit modules behave differently
  • Different ideas about textual headers vs. modular headers
  • Changes dependencies
  • Implicit creation of module maps for frameworks
  • Different code paths
slide-50
SLIDE 50
  • Clang Modules
  • Dependency Scanning
  • Fast Dependency Scanning
  • Dependency Extraction
  • Future Work
  • 50
slide-51
SLIDE 51

Future Work

  • Optimize
  • Don’t build modules, just need deps
  • Cache results, don’t write to disk
  • Incremental
  • Merge the build graph for compatible modules
  • 51
slide-52
SLIDE 52

Merging Modules

A.cpp B.cpp C.cpp D.cpp Transforms Analysis IR E.cpp -DFOO Transforms Analysis IR

Compiler Discovered Build System Known

  • 52

Transforms IR

slide-53
SLIDE 53

Merging Modules

A.cpp B.cpp C.cpp D.cpp Transforms Analysis IR E.cpp

Compiler Discovered Build System Known

😁

Future Optimization

  • 53
slide-54
SLIDE 54

Future Work

  • Optimize
  • Don’t build modules, just need deps
  • Cache results, don’t write to disk
  • Incremental
  • Merge the build graph for compatible modules
  • C++20 Modules
  • Support for import module and import <header>
  • Upstream
  • Patches: https://reviews.llvm.org/D55463, D60233
  • 54
slide-55
SLIDE 55

clang-scan-deps – Fast Dependency Scanning For Explicit Modules
 Alex Lorenz, Michael Spencer, LLVM Developers’ Meeting, Brussels, Belgium, April 2019

  • Questions?
  • 55