Puppet Presentation – BBVA – Puppet Architecture (II)

View Github Repository Open presentation in a new window

karmab

See all presentation from karmab

Puppet Presentation – BBVA – Puppet Architecture (II)

0 2

puppetworkshop

On Github karmab / puppetworkshop

Puppet Presentation

BBVA

By Karim Boumedhel

EMEA Senior Consultant, Red Hat / karim@redhat.com

Agenda

  • Puppet Introduction
  • Puppet master configuration
  • Puppet Language (Basic)
  • Puppet Language (Advanced)
  • Advanced Puppet
  • Open Demo Lab Q&A

Puppet Introduction

What is Puppet?

(Basically a State Machine)

  • Automation software
  • Written in Ruby(...)
  • Designer to manage multiple systems at once
  • Master-client architecture or Masterless...
  • Excels at Configuration Management

Puppet open source vs commercial

  • Several available versions:
    • Open Source (Apache License 2.0)
    • Commercial
    • Current Version 3.1.1, Release Notes
  • Main differences/advantages for commercial version
    • Easy installation, including dependencies
    • Integrated Dashboard
    • Some closed-source components
    • Support from PuppetLabs
  • Satellite6 incorporates puppet(and rpms!)

Puppet Architecture with Version Control Software

(Small view)

All master-agent communication use SSL

Puppet Architecture (II)

  • Puppet makes a clear distinction between:
    • The desired state (what)
    • The procedure to move into that state (how)
  • This distinction is shown by the separation between resource types(which defines what we want) and providers( silently realizes them)

Example

Puppet Architecture

No Master Server (masterless)

  • An alternative option is to run Puppet client, with no master
  • Create files and run them as "scripts"
$ puppet apply manifests.pp

Main features of Puppet

  • Idempotency
    • Run n times, get the same result n times (get back to known state using the same steps)
  • Cross-platform
    • All flavors of Linux, BSD, Solaris, HP-UX, Mac OS X...
    • ... and even Windows (Better support of Win in 3.x.x)
    • Same manifest everywhere
  • Script Agnostic
    • Declarative language
    • No needs to rely on a particular scripting languages(perl,python,ruby)

Facter

And What it does for Puppet

  • Facter provides facts about the system it is running on
  • Puppet uses Facter to gather specific details about the system
  • Then, it can use those details to make decisions
class hardware {
 case $manufacturer {
  'VMware, Inc.': {include hardware::vmware}
  'HP': {include hardware::hp}
  'IBM': {include hardware::ibm}
 }
}

Puppet Master Configuration

Puppet Installation requirements

  • Puppet will be available from Red Hat tools channel when satellite6 is out
  • Available option today:
  • The Puppetlabs repository has a more up to date version than epel , and dependencies

Installing the Puppet Master

  • Ensure Repositories are setup
$ yum install puppet-server
A Puppet Master service is created and available, so you can start it 
$ service puppetmaster start
But don't do it yet! There is some stuff to configure first.

Configuring the puppet master

  • Puppet configuration file is /etc/puppet/puppet.conf
  • The simplest configuration will just specify the module path and the certificate names:
[main]
    modulepath = /etc/puppet/modules
[master]
    certname = puppetmaster.example.com
    dns_alt_names = puppetmaster,puppetmaster.example.com
Now, we can start the Puppet master and have it create the CA Certificates Modify additional parameters carefully, ensure DNS (FQDN) is configured accordingly !

Puppet File server

  • Puppet can serve files to clients, statically or dinamically( using variables like the facts)
    • Using the file resource (more on that later)

Puppet Client - Puppet master configuration

  • Configure Puppet client to point to Puppet Master and enable reporting, at /etc/puppet/puppet.conf 
[main]
    logdir = /var/log/puppet
    rundir = /var/run/puppet
    ssldir = $vardir/ssl
    modulepath = /etc/puppet/modules
    report  = true
[agent]
    server = puppetmaster.example.com
    report = true
    classfile = $vardir/classes.txt
    localconfig = $vardir/localconfig
[master]
    certname = puppetmaster.example.com

Puppet Client - Puppet master configuration (II)

  • Run agent for the first time, it will request a certificate signed by the Puppet master:
$ puppet agent -t
On the Puppet master, check that the certificate is requested and sign it: 
$ puppet cert list
$ puppet cert sign # On Client Validate it works!
$ puppet agent --test
Connection is established. Now, we can enable the puppet service, or run it anytime we want

Automated client certificate hamdling

  • Signing every client cert request is...difficult to handle
  • There is an option to automatically sign certificates according the domain name of the machine:
cat /etc/puppet/autosign.conf
*.example.com

Centralized manifest and module handling

  • When a puppet agent contacts the master, it will first fetch the site.pp manifest from /etc/puppet/manifests/site.pp
  • This manifest defines:
    • The node definitions
    • The classes and resources applying to each system
  • Any included class/module/file will then be fetched from the Puppet Master, as needed.

Example site.pp file

  ## site.pp ##

  # Define filebucket 'main':
  filebucket { 'main':
    server => '.example.com',
    path   => false,
  }

  # Make filebucket 'main' the default backup location
  # for all File resources:
  File { backup => 'main' }

  node default {
  notify { 'example':
      message => 'Hello World!'
    }
  }

Node groups

  • Nodes are defined using the node resource
  • Node inheritance allows us to create node groups
    • Node inheritance is limited to one parent node
node prod {
  notify { 'prod':
    message => 'This is a production machine'
  }
}

node web inherits prod {
  notify { 'web':
    message => 'This is a production web server'
  }
}

node 'webserver1.example.com' inherits web {}

External Node Classifiers (ENC)

  • Managing node groups with the configuration files is exponentially complex of order O(n²) upon number of hosts
    • Manually update files each time a new client is added
    • Growing wildcards/regexps
  • Another option is to use ans ENC
    • LDAP
    • LDAP
    • Arbitrary script as in satellite6
[master]
 node_terminus = exec
 external_nodes = /usr/local/bin/puppet_node_classifier

Puppet Language

(basic)

Resources

  • A resource is the abstraction used by Puppet to define a “state”
  • It defines what we want, not how
  • A resource has:
    • A title
    • 1-n attributes, each with value

A resource, explained

user { 'karim':
  ensure     => present,
  uid        => '8000',
  gid        => 'admin',
  shell      => '/bin/bash',
  home       => '/rhome/karim',
  managehome => true,
}

Core resource types

  • There are many predefined resource types we can just use:
file    package     user    group   service     augeas
notify  host        exec    mount   cron        ssh_authorized_key
Plus some others. The full list will always be available at

The cheatsheet

Augeas

the almighty config file editor

  • Augeas is a config file editor in a key=value
    • Translates a config file into a tree-like structure
    • Changing items in this structure reflects into configuration lines
    • Where to learn more about Augeas?
    • http://augeas.net

Simple augeas rules

augeas { "sshd_config":
  require => File['/etc/ssh/sshd_config'],
  context => "/files/etc/ssh/sshd_config",
  changes => [
    "set Port 22",
    "set LogLevel VERBOSE",
    "set PermitRootLogin yes",
    "set PermitEmptyPasswords no"
  ]
}
  • Basic Actions:
    • set
    • rm
    • ins

augtool

  • This is an interactive augeas tool
    • Allows us to see the tree-like structure
    • Test changes before creating the Puppet manifest
    • Interactive help
# augtool
augtool> ls /files/etc/hosts/1
ipaddr = 127.0.0.1
canonical = localhost.localdomain
alias = localhost

augtool> set /files/etc/hosts/1/ipaddr  172.0.0.2

augtool> ls /files/etc/hosts/1
ipaddr = 172.0.0.2
canonical = localhost.localdomain
alias = localhost

Creating simple manifests

  • Let's start by writing a simple manifest, with only one resource
package { 'httpd':
  ensure => installed,
}
Execute interactively on the client, or place it on the puppet master

Inter-resources requirements

  • We can make a resource depend on another resource
package { 'httpd':
  ensure => installed,
}

service { 'httpd':
  ensure  => running,
  enable  => true,
  require => Package ['httpd'],
}
The httpd service will only be started once the httpd package is installed

Resource conditions

  • We can make a resource be applied only if a certain condition is met
exec { '/bin/touch /tmp/test':
  onlyif => '/usr/bin/test ! -f /tmp/test',
}
/tmp/test is only created if it does not exist as a file

Subscribing to other resources

And notifying other resources

  • We can subscribe to changes in other resources, or notify other resources of changes
exec { 'sysctl -p':
  alias       => 'sysctl',
  path        => '/sbin',
  refreshonly => true,
  subscribe   => Augeas['sysctl'];
}

augeas { 'sysctl':
  context => '/files/etc/sysctl.conf',
  changes => [
    # Enforcing ExecShield
    "set kernel.exec-shield 1",
    "set kernel.randomize_va_space 1"
  ]
}

Subscribing to other resources

And notifying other resources (II)

  • An alternative way of doing this...
exec { 'sysctl -p':
  alias       => 'sysctl',
  path        => '/sbin',
  refreshonly => true,
}

augeas { 'sysctl':
  context => '/files/etc/sysctl.conf',
  changes => [
    # Enforcing ExecShield
    "set kernel.exec-shield 1",
    "set kernel.randomize_va_space 1"
  ],
  notify  => Exec['sysctl -p']
}

Variables

  • Variables can be defined, then used later in a manifest
  • Simple assignment
$serverXaddr = '172.0.0.45'

augeas { 'test1':
  context => '/files/etc/postfix/main.cf',
  changes => [
    "set relayhost ${serverXaddr}"
    ]
}

Conditionals and facts: if

  • We are creating a variable depending on a fact, and then creating the file using contents from the variable
  • A fact is just a variable with a predefined value
if $::is_virtual == 'true' {
  $message = 'This is a virtual system'
} else {
  $message = 'This is a physical system'
}

file { '/root/isvirt.txt':
  content => “Hey man, ${message}”
}

Conditionals and facts: case

case $::operatingsystem {
  centos: { $apache = 'httpd'}
  # Note that these matches are case-insensitive.
  redhat: { $apache = 'httpd' }
  debian: { $apache = 'apache2' }
  ubuntu: { $apache = 'apache2' }
}

package { 'apache':
  ensure => installed,
  name   => $apache,
}

Puppet Language (Advanced)

Classes

  • Classes aggregate several resources into a reusable container
  • We can later include those classes as part of a manifest, which will apply them to the machine
  • Use them to define services, or common system properties handled as one
node puppetossclient2.example.com
{
  include production
  include vsftpd
}

A Sample class

class { 'vsftpd':
  package { 'vsftpd':
    ensure => installed,
  }

  augeas {'vsftpd.conf':
    require => Package ['vsftpd'],
    context => '/files/etc/vsftpd/vsftpd.conf',
    changes => [
      "set anonymous_enable NO",
    ]
  }

  service { 'vsftpd':
    ensure  => running,
    enable  => true,
    require => Augeas ['vsftpd.conf'];
  }
}

Modules

  • Modules are re-usable bundles of code and data
  • By default, Puppet loads all modules in the default path: /etc/puppet/modules
    • So we can declare a class in a module, and it will automatically be available
  • Modules use a standard directory structure
-> {module}/
    -> files/
    -> lib/
    -> manifests/
        -> init.pp
        -> {class}.pp
        -> namespace/
            -> {class}.pp

Example module

-> modules/
  -> hardware/
    -> manifests/
      -> init.pp
      -> vmware.pp
      -> hp.pp
      -> ibm.pp
  • Then, the contents of init.pp are:
class hardware {
  case $manufacturer {
    'VMware, Inc.': { include hardware::vmware }
    'ibm': { include hardware::ibm }
    'HP': { include hardware::hp }
  }
}

File Serving

  • We can serve configuration files to clients
  • Much like RHN Satellite config files
  • Options for storage:
    • General file store 
file/filename.txt
From within a module 
module/files/filename.txt

FIle Usage

  • Create file at /etc/puppet/files/issue
file { '/etc/issue':
  ensure => present,
  source => 'puppet:///files/issue',
}
If the file is part of a module
  • Create under /etc/puppet/modules//files/issue
file { '/etc/issue':
  ensure => present,
  source => 'puppet:///modules/testmodule/issue',
}

Templates

  • We can use templates to deploy files
  • Templates use the ERB templating language
    • Easy to learn... if you know Ruby
  • Templates can use defined variables
    • Global variables
    • Facts
    • Variables defined in the current scope
  • Can also include Ruby code

Simple template usage

  • Create file modulename/templates/testfile.txt.erb
<%= fqdn %> is the name of this server.
Define file resource using template 
file { '/root/testfile.txt':
 ensure  => present,
 content => template('testmodule/testfile.txt.erb'),
}

ERB Language cheat sheet

<% Ruby code -- inline with output %>
   
<%= Ruby expression -- replace with result %>
    
<%# comment -- ignored -- useful in testing %>
    
% a line of Ruby code -- treated as <% line %> (optional)
    
%% replaced with % if first thing on a line and % processing is used
    
<%% or %%> -- replace with <% or %> respectively

<%- and -%> suppress leading and trailing whitespace,
including the trailing newline, and can be used interchangeably with <% and %>.

Custom facts in modules

  • Create them under modulename/lib/facter/mycustom.rb
  • Example:
Facter.add(:iptables_version) do
  confine :kernel => :linux
  setcode do
    version = Facter::Util::Resolution.exec('iptables --version')
    if version
      version.match(/\d+\.\d+\.\d+/).to_s
    else
      nil
    end
  end
end

Advanced Puppet

Auditing

  • Puppet can notify changes in resources
    • Useful for auditing purposes
  • These notifications can be centrally stored
    • At the Puppet master
  • Satellite6(foreman) does a good job of showing these in the GUI

Setting up audit

  • The following lines must be present at the puppet.conf [main] section
archive_files = true
archive_file_server = puppetmaster.example.com
To ensure files are uploaded to the puppet master, add a cron job running puppet inspect every day Files are archived under /var/lib/puppet/bucket in the puppet master Before that, files also archived under /var/lib/puppet/clientbucket in the client
  • Watch out for tampering!

Auditing in manifests

  • Simply add the audit metaparameter
file { '/etc/hosts':
  audit => content,
}
Can audit any resource parameter:
  • owner, group
  • permissions
  • contents
  • ...

Checking audit logs

  • Check individual logs on each client
  • Use the satellite6 dashboard

Scalability

  • The default puppetmaster web server does not scale
    • Anything > 10 is a BAD IDEA !!!!
  • Need to search for alternative options

Extending resources

Extending resources: Example

Final words

Some questions ?

Thanks a lot !