Recursive inventory management Martin F. Krafft madduck@debian.org - - PowerPoint PPT Presentation

▶
recursive inventory management
SMART_READER_LITE
LIVE PREVIEW

Recursive inventory management Martin F. Krafft madduck@debian.org - - PowerPoint PPT Presentation

May 30, 2023 600 likes •1.7k views

Recursive inventory management Martin F. Krafft madduck@debian.org 13 Aug 2013 @ DebConf 13, Vaumarcus, Switzerland Configuration management (system administration) Configuration management CFEngine bcfg2 Puppet Chef Salt Ansible

slide-1
SLIDE 1

Recursive inventory management

Martin F. Krafft madduck@debian.org 13 Aug 2013 @ DebConf 13, Vaumarcus, Switzerland

slide-2
SLIDE 2

Configuration management

(system administration)

slide-3
SLIDE 3

Configuration management

CFEngine bcfg2 Puppet Chef Salt Ansible

slide-4
SLIDE 4

Configuration management

CFEngine bcfg2 Puppet Chef Salt Ansible

slide-5
SLIDE 5

Two approaches

Cloud provisioning System administration (classical)

slide-6
SLIDE 6

Configuring the cloud

Unprecedented ease! Scalability ”Orchestration“ Ad-hoc provisioning Homogeneous

slide-7
SLIDE 7

Classical system administration

slide-8
SLIDE 8

Classical system administration

Longevity

slide-9
SLIDE 9

Classical system administration

Longevity Heterogeneous

slide-10
SLIDE 10

Classical system administration

Longevity Heterogeneous Themed hostnames, not canonical names and numbers

slide-11
SLIDE 11

Classical system administration

Longevity Heterogeneous Themed hostnames, not canonical names and numbers Laziness (vs. unprecedented ease)

slide-12
SLIDE 12

Classical system administration

Longevity Heterogeneous Themed hostnames, not canonical names and numbers Laziness (vs. unprecedented ease) Orchestration

slide-13
SLIDE 13

Configuration management with reclass

reclass comes from classical system administration

slide-14
SLIDE 14

Configuration management with reclass

reclass comes from classical system administration It might well suit your cloud needs

slide-15
SLIDE 15

Configuration management with reclass

reclass comes from classical system administration It might well suit your cloud needs (might require rethinking)

slide-16
SLIDE 16

Targeting nodes vs. classifying hosts

slide-17
SLIDE 17

Targeting nodes

Mailservers

slide-18
SLIDE 18

Targeting nodes

Mailservers Rack somewhere

slide-19
SLIDE 19

Targeting nodes

Mailservers Rack somewhere 192.0.2.64/28

slide-20
SLIDE 20

Targeting nodes

Mailservers Rack somewhere 192.0.2.64/28 Debian has debian fact

slide-21
SLIDE 21

Targeting nodes

Mailservers Rack somewhere 192.0.2.64/28 Debian has debian fact hosted @ Zurich *.zurich.corp

slide-22
SLIDE 22

Targeting nodes

Mailservers Rack somewhere 192.0.2.64/28 Debian has debian fact hosted @ Zurich *.zurich.corp

Backwards!

slide-23
SLIDE 23

Targeting nodes

Mailservers Rack somewhere 192.0.2.64/28 Debian has debian fact hosted @ Zurich *.zurich.corp

Backwards!

To what classes does a specific node belong?

slide-24
SLIDE 24

Targeting nodes

Mailservers Rack somewhere 192.0.2.64/28 Debian has debian fact hosted @ Zurich *.zurich.corp

Backwards!

To what classes does a specific node belong? Behaviour is partially reactive, may depend on data on host

slide-25
SLIDE 25

Classifying hosts

blue.example.org

slide-26
SLIDE 26

Classifying hosts

blue.example.org mailserver ntpclient hosted@zurich

slide-27
SLIDE 27

Classifying hosts

blue.example.org mailserver ntpclient hosted@zurich server = ’0.pool.ntp.org’

slide-28
SLIDE 28

Classifying hosts

blue.example.org mailserver ntpclient hosted@zurich server = ’0.pool.ntp.org’ server = ’0.ch.pool.ntp.org’

slide-29
SLIDE 29

Classifying hosts

white.example.org

slide-30
SLIDE 30

Classifying hosts

white.example.org webserver ntpclient hosted@munich server = ’0.pool.ntp.org’

slide-31
SLIDE 31

Classifying hosts

white.example.org webserver ntpclient hosted@munich server = ’0.pool.ntp.org’ server = ’0.de.pool.ntp.org’

slide-32
SLIDE 32

Puppet

node ’blue.example.org’ { $ntpserver = ’red.example.org’ include common include ntp }

slide-33
SLIDE 33

Puppet

node ’common’ { $ntpserver = ’0.pool.ntp.org’ include common include ntp } node ’blue.example.org ’ inherits ’common ’ { }

slide-34
SLIDE 34

Puppet

node ’common’ { $ntpserver = ’0.pool.ntp.org’ include common include ntp } node ’blue.example.org ’ inherits ’common ’ { $ntpserver = ’red.example.org’ }

slide-35
SLIDE 35

Puppet

node ’common’ { $ntpserver = ’0.pool.ntp.org’ include common include ntp } node ’blue.example.org ’ inherits ’common ’ { $ntpserver = ’red.example.org’ }

Anger!

slide-36
SLIDE 36

Puppet

node ’common’ { $ntpserver = ’0.pool.ntp.org’ include common include ntp } node ’blue.example.org ’ inherits ’common ’ { $ntpserver = ’red.example.org’ }

Anger!

No multiple inheritance

slide-37
SLIDE 37

Puppet

node ’common’ { $ntpserver = ’0.pool.ntp.org’ include common include ntp } node ’blue.example.org ’ inherits ’common ’ { $ntpserver = ’red.example.org’ }

Anger!

No multiple inheritance Inheritance generally discouraged (cf. docs)

slide-38
SLIDE 38

Puppet

node ’common’ { $ntpserver = ’0.pool.ntp.org’ include common include ntp } node ’blue.example.org ’ inherits ’common ’ { $ntpserver = ’red.example.org’ }

Anger!

No multiple inheritance Inheritance generally discouraged (cf. docs) Parametrisation unnecessarily difficult

slide-39
SLIDE 39

System administration automation principles (abridged)

slide-40
SLIDE 40

System administration automation principles (abridged)

Central control, versioned

slide-41
SLIDE 41

System administration automation principles (abridged)

Central control, versioned Parametrisation (no special casing in code)

slide-42
SLIDE 42

System administration automation principles (abridged)

Central control, versioned Parametrisation (no special casing in code) No redundancy

slide-43
SLIDE 43

System administration automation principles (abridged)

Central control, versioned Parametrisation (no special casing in code) No redundancy Use node information as parameters, don’t rely on them for behaviour selection

slide-44
SLIDE 44

recursive external (node) classifier

slide-45
SLIDE 45

Data use by the CMS What applications should a role have? (aka. modules, states, playbooks)

slide-46
SLIDE 46

Data use by the CMS What applications should a role have? (aka. modules, states, playbooks) How does this node differ from all other nodes that have the same application? (aka. parameters, pillar, variables)

slide-47
SLIDE 47

Data use by the CMS What applications should a role have? (aka. modules, states, playbooks) How does this node differ from all other nodes that have the same application? (aka. parameters, pillar, variables) Which nodes belong to a group? (aka. nodegroups, clusters)

slide-48
SLIDE 48

Usage by the system administrator

slide-49
SLIDE 49

Usage by the system administrator Deploy and manage site-wide configuration changes

slide-50
SLIDE 50

Usage by the system administrator Deploy and manage site-wide configuration changes Upgrade all nodes tagged debian@stable

slide-51
SLIDE 51

Usage by the system administrator Deploy and manage site-wide configuration changes Upgrade all nodes tagged debian@stable Update /etc/motd on all hosts hosted@zurich

slide-52
SLIDE 52

Usage by the system administrator Deploy and manage site-wide configuration changes Upgrade all nodes tagged debian@stable Update /etc/motd on all hosts hosted@zurich Fetch logs of all hosts tagged mailserver

slide-53
SLIDE 53

Usage by the system administrator Deploy and manage site-wide configuration changes Upgrade all nodes tagged debian@stable Update /etc/motd on all hosts hosted@zurich Fetch logs of all hosts tagged mailserver

reclass just assembles/provides the data

slide-54
SLIDE 54

Configuration management vs. remote execution ?

slide-55
SLIDE 55

Configuration management & remote execution !

slide-56
SLIDE 56

Single data source

slide-57
SLIDE 57

reclass adapters

Adapters interface between configuration management system and reclass

slide-58
SLIDE 58

reclass adapters

Adapters interface between configuration management system and reclass: Mode of invocation (module, command-line switches, etc.)

slide-59
SLIDE 59

reclass adapters

Adapters interface between configuration management system and reclass: Mode of invocation (module, command-line switches, etc.) Output (YAML, JSON, . . . )

slide-60
SLIDE 60

reclass adapters

Adapters interface between configuration management system and reclass: Mode of invocation (module, command-line switches, etc.) Output (YAML, JSON, . . . ) Adapters provided: Puppet (not yet re-implemented)

slide-61
SLIDE 61

reclass adapters

Adapters interface between configuration management system and reclass: Mode of invocation (module, command-line switches, etc.) Output (YAML, JSON, . . . ) Adapters provided: Puppet (not yet re-implemented) Salt (module)

slide-62
SLIDE 62

reclass adapters

Adapters interface between configuration management system and reclass: Mode of invocation (module, command-line switches, etc.) Output (YAML, JSON, . . . ) Adapters provided: Puppet (not yet re-implemented) Salt (module) Ansible (exec required)

slide-63
SLIDE 63

reclass and Puppet Original reclass written for Puppet

slide-64
SLIDE 64

reclass and Puppet Original reclass written for Puppet . . . out of frustration

slide-65
SLIDE 65

reclass and Puppet Original reclass written for Puppet . . . out of frustration Rage-quit Puppet two years ago

slide-66
SLIDE 66

reclass and Puppet Original reclass written for Puppet . . . out of frustration Rage-quit Puppet two years ago Rewritten reclass from scratch since then

slide-67
SLIDE 67

reclass and Puppet Original reclass written for Puppet . . . out of frustration Rage-quit Puppet two years ago Rewritten reclass from scratch since then Could not be bothered to reimplement

slide-68
SLIDE 68

reclass and Puppet Original reclass written for Puppet . . . out of frustration Rage-quit Puppet two years ago Rewritten reclass from scratch since then Could not be bothered to reimplement Trivially done through adapter plugin

slide-69
SLIDE 69

reclass and Salt Provides top and pillar data

slide-70
SLIDE 70

reclass and Salt Provides top and pillar data Adapter is a Salt module, since 0.16.0

slide-71
SLIDE 71

reclass and Salt Provides top and pillar data Adapter is a Salt module, since 0.16.0 nodegroups not yet implemented (Salt issue #5787)

slide-72
SLIDE 72

reclass and Ansible Provides inventory and node information

slide-73
SLIDE 73

reclass and Ansible Provides inventory and node information Implemented as external script

slide-74
SLIDE 74

reclass and Ansible Provides inventory and node information Implemented as external script Does not yet support batched calls (recent Ansible feature)

slide-75
SLIDE 75

Enough of this boring stuff!

slide-76
SLIDE 76

Parametrisation is key Parametrise modules as much as possible

slide-77
SLIDE 77

Parametrisation is key Parametrise modules as much as sensible

slide-78
SLIDE 78

Parametrisation is key Parametrise modules as much as sensible At all cost, avoid special-casing in module code

slide-79
SLIDE 79

Parametrisation is key Parametrise modules as much as sensible At all cost, avoid special-casing in module code Reclass allows you to keep your parameters modular

slide-80
SLIDE 80

Parametrisation is key Parametrise modules as much as sensible At all cost, avoid special-casing in module code Reclass allows you to keep your parameters modular Define your data in one place only (no redundancy)

slide-81
SLIDE 81

reclass node definition

blue.example.org.yaml:

  • applications:
  • postfix
  • ntp

parameters: ntp: server: 0. pool.ntp.org

slide-82
SLIDE 82

reclass node definition

blue.example.org.yaml:

  • applications:
  • postfix
  • ntp

parameters: ntp: server: 0. pool.ntp.org

slide-83
SLIDE 83

But wait! You promised recursion!

slide-84
SLIDE 84

yaml fs YAML files for nodes in $inventory base uri/nodes

slide-85
SLIDE 85

yaml fs YAML files for nodes in $inventory base uri/nodes YAML files defining classes in $inventory base uri/classes

slide-86
SLIDE 86

yaml fs YAML files for nodes in $inventory base uri/nodes YAML files defining classes in $inventory base uri/classes Nodes and classes files may specify classes to inherit

slide-87
SLIDE 87

yaml fs YAML files for nodes in $inventory base uri/nodes YAML files defining classes in $inventory base uri/classes Nodes and classes files may specify classes to inherit You can think of classes as tags, too!

slide-88
SLIDE 88

yaml fs YAML files for nodes in $inventory base uri/nodes YAML files defining classes in $inventory base uri/classes Nodes and classes files may specify classes to inherit You can think of classes as tags, too! Smart (deep) merging on return from recursive descent walk

slide-89
SLIDE 89

reclass node definition

nodes/blue.example.org.yaml: classes:

  • common
  • mailserver

parameters: ntp: server: 0.ch.pool.ntp.org classes/common.yaml: applications:

  • ntp

parameters: ntp: server: 0. pool.ntp.org

slide-90
SLIDE 90

reclass node definition

nodes/blue.example.org.yaml: classes:

  • common
  • mailserver

parameters: ntp: server: 0.ch.pool.ntp.org classes/common.yaml: applications:

  • ntp

parameters: ntp: server: 0. pool.ntp.org

slide-91
SLIDE 91

reclass node definition

nodes/blue.example.org.yaml: classes:

  • common
  • mailserver

parameters: ntp: server: 0.ch.pool.ntp.org classes/common.yaml: applications:

  • ntp

parameters: ntp: server: 0. pool.ntp.org

slide-92
SLIDE 92

More specific classes

  • verride data defined in

less specific classes

slide-93
SLIDE 93

reclass node definition

nodes/blue.example.org.yaml: classes:

  • common
  • mailserver
  • hosted@zurich

classes/hosted@zurich.yaml: parameters: ntp: server: 0.ch.pool.ntp.org

slide-94
SLIDE 94

reclass node definition

nodes/blue.example.org.yaml: classes:

  • common
  • mailserver
  • hosted@zurich

classes/hosted@zurich.yaml: parameters: ntp: server: 0.ch.pool.ntp.org

slide-95
SLIDE 95

Multiple inheritance well-defined order

slide-96
SLIDE 96

reclass node definition

nodes/blue.example.org.yaml: classes:

  • ssh-server

classes/ssh-server.yaml parameters: permit root login: no

slide-97
SLIDE 97

reclass node definition

nodes/blue.example.org.yaml: classes:

  • ssh-server
  • backup-client

classes/ssh-server.yaml parameters: permit root login: no classes/backup-client.yaml parameters: permit root login: without-password

slide-98
SLIDE 98

reclass node definition

nodes/blue.example.org.yaml: classes:

  • ssh-server
  • backup-client

classes/ssh-server.yaml parameters: permit root login: no classes/backup-client.yaml parameters: permit root login: without-password classes:

  • ssh-server
slide-99
SLIDE 99

Parameter interpolation

nodes/diamond.example.org.yaml: classes:

  • motd

classes/motd.yaml parameters: motd: message: ${floyd reference}

slide-100
SLIDE 100

Parameter interpolation

nodes/diamond.example.org.yaml: classes:

  • motd

parameters:

  • floyd reference: Shine on, you crazy diamond

classes/motd.yaml parameters: motd: message: ${floyd reference}

slide-101
SLIDE 101

reclass future work Package it.

slide-102
SLIDE 102

reclass future work Package it. Doh!

slide-103
SLIDE 103

reclass future work Package it. Doh! preseed.cfg/d-i adapter?

slide-104
SLIDE 104

reclass future work Package it. Doh! preseed.cfg/d-i adapter? Policy classification (regexp → class mappings)

slide-105
SLIDE 105

reclass future work Package it. Doh! preseed.cfg/d-i adapter? Policy classification (regexp → class mappings) Membership lists

slide-106
SLIDE 106

reclass future work Package it. Doh! preseed.cfg/d-i adapter? Policy classification (regexp → class mappings) Membership lists Other data sources?

slide-107
SLIDE 107

reclass future work Package it. Doh! preseed.cfg/d-i adapter? Policy classification (regexp → class mappings) Membership lists Other data sources? Better unit testing (without philosophical debates)

slide-108
SLIDE 108

reclass future work Package it. Doh! preseed.cfg/d-i adapter? Policy classification (regexp → class mappings) Membership lists Other data sources? Better unit testing (without philosophical debates) Your idea here!