Tuesday 20 December 2011

Why Puppet Enterprise?

Puppet Enterprise is the commercially supported, packaged release of Puppet, the leading open source solution for enterprise systems management, including data centre automation and configuration management. Puppet automates the provisioning, patching, and configuration of operating system and application components across enterprise and cloud infrastructure. Puppet Enterprise packages the underlying components required, and bundles the requisite support and services necessary to support enterprise deployments.

Puppet Open Source vs Puppet Enterprise

(Reference http://puppetlabs.com/puppet/compare/)

1) Simplified Integration & Installation: Get up to speed immediately with a pre-built integration of Puppet, Puppet Master, Dashboard, Facter and all Puppet dependencies such as Ruby, Passenger, and Apache.

2) Scalable: Pre-configured to offer out-of-the-box scalability and the performance levels required in large installations.

3) Tested & Approved: Packaged, tested and QA’d by Puppet Labs engineers

4) Support: Enterprise-class support is included, Issue resolution, feature enhancement priority requests, and best practices advice.

5) Always Current: Maintenance is included in your license. You always have access to the latest and greatest releases of the integrated platform.

6) Security updates: Security is prime importance to any organization for smooth operation. Using Puppet enterprise inbuilt modules it’s really easy to manage security updates, security patches, SSH/SUDO management and testing them before push them to production environment.

7) Patch management: Tracking software, testing news patches and patching is extremely easy with Puppet enterprise solution.

8) Puppet related software security updates: One of the primary advantages of using Puppet Enterprise is the fact that you no longer need to package and maintain the various versions of Apache, Ruby, Passenger, etc. in order to run Puppet effectively. Ensuring that our customers receive timely security updates was carefully considered when we decided to bundle the software packages in Puppet Enterprise. As a response Puppet Labs has integrated a security alert monitoring and update process into our Puppet Enterprise build process.

9) Time: Using Puppet enterprise it improves the time taken to deploy servers and clone processes, packages and systems on the fly.

10) Live management: Using Puppet enterprise Live management feature its really easy to manage services, packages, users attributes, Puppet module management, and advanced tasks like replication.

 11) Discover Resources and Their Relationships: Through the new GUI, right out-of-the-box PE 2.0 automatically discovers all resources – packages, hosts, groups, and users. With Puppet Enterprise you define the desired-state of your infrastructure as relationships between resources. Immediately start seeing results without having to learn CLI commands or the Puppet DSL.

12) Visually Clone Resources To Scale Quickly, Efficiently, and Reliably: Because you need to scale quickly without sacrificing service levels, PE 2.0’s new GUI allows you to visually browse, select, and clone resources across nodes, saving you time and ensuring consistency across your infrastructure.

13) Quickly Deploy Critical Updates with a Single Command: Critical updates, such as security vulnerabilities, require fast yet coordinated action. PE 2.0’s new orchestration capability allows you to query your infrastructure to discover the vulnerable nodes and then, with a single command, schedule simultaneous updates. Upon completion, PE 2.0 gives you a single aggregated report of the results.

14) Monitor Changes, Enforce Compliance Policies: How is your infrastructure changing over time? With PE 2.0’s new compliance capability, you now can visually monitor for any unauthorized changes against your desired-state baseline. Quickly and efficiently get the insights you need to create and support your change management and auditing policies.

15) Rapidly Add Virtual Machines and Cloud Capacity: PE 2.0’s new provisioning capability allows you to quickly and easily create new instances of VMware and Amazon EC2. Then, you can configure and manage these instances using the same manifests you created for your physical infrastructure. Swiftly respond to changing capacity demands while future-proofing your investment in infrastructure configuration and management.

Wednesday 14 December 2011

Automated Puppet Enterprise Master Server Installation

Automated Puppet Enterprise Master server Installation

You can download the package from puppet (You have to sign up with them) They have a free version which support upto 10 nodes - Download Puppet Enterprise

Prerequisites

- This article meant for RedHat and Centos latest versions (I have tested following in Redhat Enterprise version 5.7 and Centos 5.6) - Make sure to have working yum repository setup in order to fix dependancies - Java package needed to be installed (# yum install java) (This is not a mandatory requirement)

Create answer file for automated puppet master installation

Create a puppet enterprise master file,
$ vi /tmp/pe-master.answers
q_puppet_cloud_install=y
q_puppet_enterpriseconsole_auth_password=puppet
q_puppet_enterpriseconsole_auth_user=console
q_puppet_enterpriseconsole_database_install=y
q_puppet_enterpriseconsole_database_name=console
q_puppet_enterpriseconsole_database_password=puppet
q_puppet_enterpriseconsole_database_root_password=puppet
q_puppet_enterpriseconsole_database_user=console
q_puppet_enterpriseconsole_httpd_port=443
q_puppet_enterpriseconsole_install=y
q_puppet_enterpriseconsole_inventory_hostname=puppet
q_puppet_enterpriseconsole_inventory_port=8140
q_puppet_enterpriseconsole_master_hostname=puppet
q_puppet_symlinks_install=y
q_puppetagent_certname=puppet
q_puppetagent_install=y
q_puppetagent_server=puppet
q_puppetmaster_certname=puppet
q_puppetmaster_dnsaltnames=puppet
q_puppetmaster_enterpriseconsole_hostname=localhost
q_puppetmaster_enterpriseconsole_port=443
q_puppetmaster_forward_facts=n
q_puppetmaster_install=y
q_vendor_packages_install=y
q_install=y
Go to folder where you have downloaded the puppet enterprise gunzip file,
$ sudo tar –zxvf puppet-enterprise-2.0-el-5-i386.tar.gz
$ sudo mv puppet-enterprise-2.0-el-5-i386 puppet-enterprise
$ sudo cd puppet-enterprise && ./puppet-enterprise-installer -a /tmp/pe-master.answers
$ sudo cd puppet-enterprise && ./puppet-enterprise-upgrader -a /tmp/pe-master.answers (if you try to upgrade Puppet)
Note: that following password needs to be changed accordingly,
q_puppet_enterpriseconsole_auth_password=puppet 
q_puppet_enterpriseconsole_database_password=puppet 
q_puppet_enterpriseconsole_database_root_password=puppet
To install puppet enterprise with the answer file do following,
§  Un-achieve the installer tarball
§  Navigate to the resulting directory in your shell
§  Run the puppet-enterprise-installer script with root privileges
(Also note that Installing with the -a option will fail if any required variables are not set. Installing with the -A option will prompt the user for any missing answers.) After the successful installation you can check Puppet version,
# puppet -V
2.7.6 (Puppet Enterprise 2.0.0)
Check whether the daemons start after system reboot,
# chkconfig --list |grep -i pe-
pe-activemq                      0:off   1:off   2:on    3:on    4:on    5:on    6:off
pe-httpd                         0:off   1:off   2:on    3:on    4:on    5:on    6:off
pe-mcollective                   0:off   1:off   2:on    3:on    4:on    5:on    6:off
pe-memcached                     0:off   1:off   2:on    3:on    4:on    5:on    6:off
pe-puppet                        0:off   1:off   2:on    3:on    4:on    5:on    6:off
pe-puppet-dashboard-workers      0:off   1:off   2:on    3:on    4:on    5:on    6:off
In order to make them persistent use following command (Note that by default all pe services are turned on),
$ sudo chkconfig pe-activemq on
$ sudo chkconfig pe-httpd on
$ sudo chkconfig pe-mcollective on
$ sudo chkconfig pe-memcached on
$ sudo chkconfig pe-puppet on
$ sudo chkconfig pe-puppet-dashboard-workers on

What Gets Installed Where?

- License File PE license file (which was emailed to us by PuppetLabs when we purchased Puppet Enterprise) should be placed at /etc/puppetlabs/license.key. Puppet Enterprise can be evaluated with a complementary ten-node license; beyond that, a commercial per-node license is required for use. A license key file will have been emailed to us after we purchase, and the puppet master will look for this key at /etc/puppetlabs/license.key. Puppet will log warnings if the license is expired or exceeded, and we can view the status of our license by running following in Puppet master, You can check the current license by executing,
# puppet license
- Configuration Files Puppet Enterprise’s configuration files all live under /etc/puppetlabs, with subdirectories for each of PE’s components. Puppet’s confdir is in /etc/puppetlabs/puppet. This directory contains the puppet.conf file, the site manifest (manifests/site.pp), and the modules directory. MCollective’s config files are in /etc/puppetlabs/mcollective. The console’s config files are in /etc/puppetlabs/puppet-dashboard. - Firewall Configuration Configure the firewalls to accommodate Puppet Enterprise’s network traffic. The short version is that we should open up following ports, 8140 61613 443

How to access puppet dashboard

In your browser point to to following
https://puppet/ (accept the security certificate)
§ Then enter the console username and password,

How to setup Puppet file server

$ Vim /etc/puppetlabs/puppet/fileserver.conf
# This file consists of arbitrarily named sections/modules
# defining where files are served from and to whom
# Define a section 'files'
# Adapt the allow/deny settings to your needs. Order
# for allow/deny does not matter, allow always takes precedence
# over deny
# [files]
  path /var/lib/puppet/files
  allow *

How to restart PE master

$ sudo /etc/init.d/pe-httpd restart

How to get help and more docs (Direct extract from PuppetLabs website)

Man pages for the Puppet subcommands are generated on the fly. To view them, run puppet man . The pe-man command from previous versions of Puppet Enterprise is still functional, but it is deprecated and is slated for removal in a future release.

Installed directories and paths (Direct extract from PuppetLabs website)

All PE software is installed under /opt/puppet. Executable binaries are in /opt/puppet/bin and /opt/puppet/sbin Optionally, you can choose at install time to symlink the most common binaries into /usr/local/bin. The Puppet modules included with PE are installed in /opt/puppet/share/puppet/modules. Don’t edit this directory to add modules of your own; instead, install them in /etc/puppetlabs/puppet/modules. MCollective plugins are installed in /opt/puppet/libexec/mcollective/. If you are adding new plugins to your PE agent nodes, you should distribute them via Puppet.

What Services PE uses (Direct extract from PuppetLabs website)

pe-puppet (on EL platforms) and pe-puppet-agent (on Debian-based platforms) — The puppet agent daemon. Runs on every agent node. pe-httpd — Apache 2, which manages and serves puppet master and the console on servers with those roles. (Note that PE uses Passenger to run puppet master, instead of running it as a standalone daemon.) pe-mcollective — The MCollective server. Runs on every agent node. pe-puppet-dashboard-workers — A supervisor that manages the console’s background processes. Runs on servers with the console role. pe-activemq — The ActiveMQ message server, which passes messages to the MCollective servers on agent nodes. Runs on servers with the puppet master role.

What user Accounts

peadmin — An administrative account which can issue MCollective client commands. This is the only PE user account intended for use in a login shell. See the chapter on orchestration for more about this user. This user exists on servers with the puppet master role, and replaces the mco user that was present in PE 1.2. pe-puppet — A system user which runs the puppet master processes spawned by Passenger. pe-apache — A system user which runs Apache (pe-httpd). pe-activemq — A system user which runs the ActiveMQ message bus used by MCollective. puppet-dashboard — A system user which runs the console processes spawned by Passenger.

Housekeeping commands - crontab

Add following cron jobs into root crontab to optimize the puppet-dashboard MySQL database (It seems that there is a bug in Puppet Enterprise version 2.0, my observation is it's eating up lot of space too quickly, even after using below optimization I have seen that it has eaten 4GB worth of /var (Including /var/lib/mysql/ibdata1: 930M) with one week (Mind you this is for just 5 puppet nodes). Does anyone of you experiance similar problems? Is there a known solution to this? Please leave a comment below. Note: Please don't use below for Production envionmnet, please check PuppletLabs documents for more information.
0,15,30,45 * * * * /opt/puppet/sbin/refresh-mcollective-metadata
00 18 * * * /opt/puppet/bin/rake -f /opt/puppet/share/puppet-dashboard/Rakefile RAILS_ENV=production db:raw:optimize
30 18 * * * /opt/puppet/bin/rake -f /opt/puppet/share/puppet-dashboard/Rakefile RAILS_ENV=production reports:prune upto=1 unit=mon