Sunday 15 January 2012

Pending Background Tasks - Puppet Enterprise Dashboard

Our Puppet enterprise server /var folder ran out of space during the weekend. There were 17123 pending tasks to be waiting in the queue. Refer following,
After increasing the LUN size we have rebooted the server but couldn't get Puppet dashboard to function normally and pending tasks message continue to appear. Did some researching and found this Console Maintenance Tasks but with no avail. The reason is it wont allow to restart the "pe-puppet-dashboard-workers" and giving me following error,
Stopping pe-dashboard-workers: Terminated
If you are like me here is the quick solution, Go to following location, # cd /var/opt/cache/pe-puppet-dashboard/pids Then delete (Keep a copy if you want to run them or investigate it further) all the pids, # rm -fr * Restart Puppet Enterprise and worker modules, # service pe-httpd restart # service pe-puppet-dashboard-workers restart Then should restart normally. That's it.

Tuesday 20 December 2011

Why Puppet Enterprise?

Puppet Enterprise is the commercially supported, packaged release of Puppet, the leading open source solution for enterprise systems management, including data centre automation and configuration management. Puppet automates the provisioning, patching, and configuration of operating system and application components across enterprise and cloud infrastructure. Puppet Enterprise packages the underlying components required, and bundles the requisite support and services necessary to support enterprise deployments.

Puppet Open Source vs Puppet Enterprise

(Reference http://puppetlabs.com/puppet/compare/)

1) Simplified Integration & Installation: Get up to speed immediately with a pre-built integration of Puppet, Puppet Master, Dashboard, Facter and all Puppet dependencies such as Ruby, Passenger, and Apache.

2) Scalable: Pre-configured to offer out-of-the-box scalability and the performance levels required in large installations.

3) Tested & Approved: Packaged, tested and QA’d by Puppet Labs engineers

4) Support: Enterprise-class support is included, Issue resolution, feature enhancement priority requests, and best practices advice.

5) Always Current: Maintenance is included in your license. You always have access to the latest and greatest releases of the integrated platform.

6) Security updates: Security is prime importance to any organization for smooth operation. Using Puppet enterprise inbuilt modules it’s really easy to manage security updates, security patches, SSH/SUDO management and testing them before push them to production environment.

7) Patch management: Tracking software, testing news patches and patching is extremely easy with Puppet enterprise solution.

8) Puppet related software security updates: One of the primary advantages of using Puppet Enterprise is the fact that you no longer need to package and maintain the various versions of Apache, Ruby, Passenger, etc. in order to run Puppet effectively. Ensuring that our customers receive timely security updates was carefully considered when we decided to bundle the software packages in Puppet Enterprise. As a response Puppet Labs has integrated a security alert monitoring and update process into our Puppet Enterprise build process.

9) Time: Using Puppet enterprise it improves the time taken to deploy servers and clone processes, packages and systems on the fly.

10) Live management: Using Puppet enterprise Live management feature its really easy to manage services, packages, users attributes, Puppet module management, and advanced tasks like replication.

 11) Discover Resources and Their Relationships: Through the new GUI, right out-of-the-box PE 2.0 automatically discovers all resources – packages, hosts, groups, and users. With Puppet Enterprise you define the desired-state of your infrastructure as relationships between resources. Immediately start seeing results without having to learn CLI commands or the Puppet DSL.

12) Visually Clone Resources To Scale Quickly, Efficiently, and Reliably: Because you need to scale quickly without sacrificing service levels, PE 2.0’s new GUI allows you to visually browse, select, and clone resources across nodes, saving you time and ensuring consistency across your infrastructure.

13) Quickly Deploy Critical Updates with a Single Command: Critical updates, such as security vulnerabilities, require fast yet coordinated action. PE 2.0’s new orchestration capability allows you to query your infrastructure to discover the vulnerable nodes and then, with a single command, schedule simultaneous updates. Upon completion, PE 2.0 gives you a single aggregated report of the results.

14) Monitor Changes, Enforce Compliance Policies: How is your infrastructure changing over time? With PE 2.0’s new compliance capability, you now can visually monitor for any unauthorized changes against your desired-state baseline. Quickly and efficiently get the insights you need to create and support your change management and auditing policies.

15) Rapidly Add Virtual Machines and Cloud Capacity: PE 2.0’s new provisioning capability allows you to quickly and easily create new instances of VMware and Amazon EC2. Then, you can configure and manage these instances using the same manifests you created for your physical infrastructure. Swiftly respond to changing capacity demands while future-proofing your investment in infrastructure configuration and management.

Wednesday 14 December 2011

Automated Puppet Enterprise Master Server Installation

Automated Puppet Enterprise Master server Installation

You can download the package from puppet (You have to sign up with them) They have a free version which support upto 10 nodes - Download Puppet Enterprise

Prerequisites

- This article meant for RedHat and Centos latest versions (I have tested following in Redhat Enterprise version 5.7 and Centos 5.6) - Make sure to have working yum repository setup in order to fix dependancies - Java package needed to be installed (# yum install java) (This is not a mandatory requirement)

Create answer file for automated puppet master installation

Create a puppet enterprise master file,
$ vi /tmp/pe-master.answers
q_puppet_cloud_install=y
q_puppet_enterpriseconsole_auth_password=puppet
q_puppet_enterpriseconsole_auth_user=console
q_puppet_enterpriseconsole_database_install=y
q_puppet_enterpriseconsole_database_name=console
q_puppet_enterpriseconsole_database_password=puppet
q_puppet_enterpriseconsole_database_root_password=puppet
q_puppet_enterpriseconsole_database_user=console
q_puppet_enterpriseconsole_httpd_port=443
q_puppet_enterpriseconsole_install=y
q_puppet_enterpriseconsole_inventory_hostname=puppet
q_puppet_enterpriseconsole_inventory_port=8140
q_puppet_enterpriseconsole_master_hostname=puppet
q_puppet_symlinks_install=y
q_puppetagent_certname=puppet
q_puppetagent_install=y
q_puppetagent_server=puppet
q_puppetmaster_certname=puppet
q_puppetmaster_dnsaltnames=puppet
q_puppetmaster_enterpriseconsole_hostname=localhost
q_puppetmaster_enterpriseconsole_port=443
q_puppetmaster_forward_facts=n
q_puppetmaster_install=y
q_vendor_packages_install=y
q_install=y
Go to folder where you have downloaded the puppet enterprise gunzip file,
$ sudo tar –zxvf puppet-enterprise-2.0-el-5-i386.tar.gz
$ sudo mv puppet-enterprise-2.0-el-5-i386 puppet-enterprise
$ sudo cd puppet-enterprise && ./puppet-enterprise-installer -a /tmp/pe-master.answers
$ sudo cd puppet-enterprise && ./puppet-enterprise-upgrader -a /tmp/pe-master.answers (if you try to upgrade Puppet)
Note: that following password needs to be changed accordingly,
q_puppet_enterpriseconsole_auth_password=puppet 
q_puppet_enterpriseconsole_database_password=puppet 
q_puppet_enterpriseconsole_database_root_password=puppet
To install puppet enterprise with the answer file do following,
§  Un-achieve the installer tarball
§  Navigate to the resulting directory in your shell
§  Run the puppet-enterprise-installer script with root privileges
(Also note that Installing with the -a option will fail if any required variables are not set. Installing with the -A option will prompt the user for any missing answers.) After the successful installation you can check Puppet version,
# puppet -V
2.7.6 (Puppet Enterprise 2.0.0)
Check whether the daemons start after system reboot,
# chkconfig --list |grep -i pe-
pe-activemq                      0:off   1:off   2:on    3:on    4:on    5:on    6:off
pe-httpd                         0:off   1:off   2:on    3:on    4:on    5:on    6:off
pe-mcollective                   0:off   1:off   2:on    3:on    4:on    5:on    6:off
pe-memcached                     0:off   1:off   2:on    3:on    4:on    5:on    6:off
pe-puppet                        0:off   1:off   2:on    3:on    4:on    5:on    6:off
pe-puppet-dashboard-workers      0:off   1:off   2:on    3:on    4:on    5:on    6:off
In order to make them persistent use following command (Note that by default all pe services are turned on),
$ sudo chkconfig pe-activemq on
$ sudo chkconfig pe-httpd on
$ sudo chkconfig pe-mcollective on
$ sudo chkconfig pe-memcached on
$ sudo chkconfig pe-puppet on
$ sudo chkconfig pe-puppet-dashboard-workers on

What Gets Installed Where?

- License File PE license file (which was emailed to us by PuppetLabs when we purchased Puppet Enterprise) should be placed at /etc/puppetlabs/license.key. Puppet Enterprise can be evaluated with a complementary ten-node license; beyond that, a commercial per-node license is required for use. A license key file will have been emailed to us after we purchase, and the puppet master will look for this key at /etc/puppetlabs/license.key. Puppet will log warnings if the license is expired or exceeded, and we can view the status of our license by running following in Puppet master, You can check the current license by executing,
# puppet license
- Configuration Files Puppet Enterprise’s configuration files all live under /etc/puppetlabs, with subdirectories for each of PE’s components. Puppet’s confdir is in /etc/puppetlabs/puppet. This directory contains the puppet.conf file, the site manifest (manifests/site.pp), and the modules directory. MCollective’s config files are in /etc/puppetlabs/mcollective. The console’s config files are in /etc/puppetlabs/puppet-dashboard. - Firewall Configuration Configure the firewalls to accommodate Puppet Enterprise’s network traffic. The short version is that we should open up following ports, 8140 61613 443

How to access puppet dashboard

In your browser point to to following
https://puppet/ (accept the security certificate)
§ Then enter the console username and password,

How to setup Puppet file server

$ Vim /etc/puppetlabs/puppet/fileserver.conf
# This file consists of arbitrarily named sections/modules
# defining where files are served from and to whom
# Define a section 'files'
# Adapt the allow/deny settings to your needs. Order
# for allow/deny does not matter, allow always takes precedence
# over deny
# [files]
  path /var/lib/puppet/files
  allow *

How to restart PE master

$ sudo /etc/init.d/pe-httpd restart

How to get help and more docs (Direct extract from PuppetLabs website)

Man pages for the Puppet subcommands are generated on the fly. To view them, run puppet man . The pe-man command from previous versions of Puppet Enterprise is still functional, but it is deprecated and is slated for removal in a future release.

Installed directories and paths (Direct extract from PuppetLabs website)

All PE software is installed under /opt/puppet. Executable binaries are in /opt/puppet/bin and /opt/puppet/sbin Optionally, you can choose at install time to symlink the most common binaries into /usr/local/bin. The Puppet modules included with PE are installed in /opt/puppet/share/puppet/modules. Don’t edit this directory to add modules of your own; instead, install them in /etc/puppetlabs/puppet/modules. MCollective plugins are installed in /opt/puppet/libexec/mcollective/. If you are adding new plugins to your PE agent nodes, you should distribute them via Puppet.

What Services PE uses (Direct extract from PuppetLabs website)

pe-puppet (on EL platforms) and pe-puppet-agent (on Debian-based platforms) — The puppet agent daemon. Runs on every agent node. pe-httpd — Apache 2, which manages and serves puppet master and the console on servers with those roles. (Note that PE uses Passenger to run puppet master, instead of running it as a standalone daemon.) pe-mcollective — The MCollective server. Runs on every agent node. pe-puppet-dashboard-workers — A supervisor that manages the console’s background processes. Runs on servers with the console role. pe-activemq — The ActiveMQ message server, which passes messages to the MCollective servers on agent nodes. Runs on servers with the puppet master role.

What user Accounts

peadmin — An administrative account which can issue MCollective client commands. This is the only PE user account intended for use in a login shell. See the chapter on orchestration for more about this user. This user exists on servers with the puppet master role, and replaces the mco user that was present in PE 1.2. pe-puppet — A system user which runs the puppet master processes spawned by Passenger. pe-apache — A system user which runs Apache (pe-httpd). pe-activemq — A system user which runs the ActiveMQ message bus used by MCollective. puppet-dashboard — A system user which runs the console processes spawned by Passenger.

Housekeeping commands - crontab

Add following cron jobs into root crontab to optimize the puppet-dashboard MySQL database (It seems that there is a bug in Puppet Enterprise version 2.0, my observation is it's eating up lot of space too quickly, even after using below optimization I have seen that it has eaten 4GB worth of /var (Including /var/lib/mysql/ibdata1: 930M) with one week (Mind you this is for just 5 puppet nodes). Does anyone of you experiance similar problems? Is there a known solution to this? Please leave a comment below. Note: Please don't use below for Production envionmnet, please check PuppletLabs documents for more information.
0,15,30,45 * * * * /opt/puppet/sbin/refresh-mcollective-metadata
00 18 * * * /opt/puppet/bin/rake -f /opt/puppet/share/puppet-dashboard/Rakefile RAILS_ENV=production db:raw:optimize
30 18 * * * /opt/puppet/bin/rake -f /opt/puppet/share/puppet-dashboard/Rakefile RAILS_ENV=production reports:prune upto=1 unit=mon

Tuesday 8 November 2011

Puppet Enterprise Server Installation and Configuration

What is Puppet Enterprise?
Puppet Enterprise is the commercially supported, packaged release of Puppet, the leading open source solution for enterprise systems management, including data centre automation and configuration management. Puppet automates the provisioning, patching, and configuration of operating system and application components across enterprise and cloud infrastructure. Puppet Enterprise packages the underlying components required, and bundles the requisite support and services necessary to support enterprise deployments. Puppet Enterprise pricing is based on number of nodes it manages and you can find detail information here - Puppet Enterprise Pricing

How is it different than the open source distribution of Puppet?
Simplified Integration & Installation: Get up to speed immediately with a pre-built integration of Puppet, Puppet Master, Dashboard, Facter and all Puppet dependencies such as Ruby, Passenger, and Apache.

Scalable: Pre-configured to offer out-of-the-box scalability and the performance levels required in large installations.
Tested & Approved: Packaged, tested and QA’d by Puppet Labs engineers
Support: Enterprise-class support is included: issue resolution, feature enhancement priority requests, and best practices advice.
Always Current: Maintenance is included in your license. You always have access to the latest and greatest releases of the integrated platform.

What are the specific components and dependencies included with Puppet Enterprise?
Puppet Enterprise includes the following in a single installer:

·   PUPPET COMPONENTS:
-  Puppet Master
-  Puppet Agent
-  Puppet Dashboard
-  Puppet Module Tool
-  Puppet Compliance
-  User Account Management
-  Facter

·   THIRD-PARTY PRODUCTS:
-  Ruby
-  Apache
-  Phusion Passenger

.   THIRD-PARTY LIBRARIES:
-  Ruby on Rails
-  RubyGems
-  Rack
-  Rake
-  Haml
-  jQuery
-  ActiveMQ

Proposed Puppet architecture for our environment

Puppet Enterprise Installation
Login as root and then run following,

#mkdir /usr/local/download && cd /usr/local/download
#export http_proxy=http://proxy:3128 (Only if you are using proxy server)
#wget --proxy-user={username} --proxy-password={password} http://pm.puppetlabs.com/puppet-enterprise/1.2.3/puppet-enterprise-1.2.3-all.tar.gz
#tar zxvf puppet-enterprise-1.2.3-all.tar.gz (This was the latest as 8 November 2011)
#cd puppet-enterprise-1.2.3-all
#./puppet-enterprise-installer

By default all Puppet clients search for Puppet master as Puppet, therefore it’s required to update
the DNS with CANME, do following (You may update /etc/hosts in both server and the node but it’s recommended to use CNAME),

puppet   IN   CNAME  bakingcake.cake.com.

Start Puppet master using following command and make it to start when system reboots,
service pe-puppet start
chkconfig pe-puppet on

Also you can start Puppet enterprise Dashboard by,
service pe-puppet-dashboard-workers restart
chkconfig pe-puppet-dashboard-workers on

Here are the lists of ports used by Puppet and Puppet related applications,
Puppet Master – 8140 (Eg: puppet:8140)
Puppet Dashboard – 3000 (Eg: http://puppet:3000)
MCollective - 61613

Add following entry in to iptables in Puppet Master server where client nodes can connect,
-A INPUT -p tcp -m state --state NEW --dport 8140 -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 3000 -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 61613 -j ACCEPT

Puppet Client Installation
Login as root and then run following,

#export http_proxy=http://proxy:3128
#rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm
#yum install ruby ruby-libs ruby-shadow -y
#yum install puppet puppet-server facter -y

Start Puppet using following commands,
chkconfig puppet on
service puppet start

Send first message to Puppet Master,
#puppetd --server puppet --waitforcert 60 –test

Once above message send to Puppet Master, Administrator should authorised the Public key for the node to be able to initiate connection. It is preventive mechanism used by Puppet to restrict unauthorised communication with Puppet Master. Please do following,

puppet cert –list (This gives list of nodes waiting to be authenticated)
puppet cert --sign “name of the client”
Puppet Configuration

Puppet Master Configuration
Below visio diagram shows the file structure in puppet master,

Following scenario has been selected in order to explain the end to end configuration to be done on puppet master and the client.
Scenario – We want to create a central ntp.conf file and it needed to be pushed to puppet client. Also we want to make sure once file get copied to the client successfully reboot the ntp daemon. Assume puppet client already authenticated with the puppet master.
 Below servers have selected to do the testing,
-       Puppet master - node_name: puppet client (FQDN - puppet.mycake.org.au)
-       Puppet client – node name: client (FQDN - client.mycake.org.au)
Do the following as root,
[root@puppet ]# cd /etc/puppetlabs/puppet
[root@puppet /etc/puppetlabs/puppet]#

Allow puppet client to access file server on puppet master,
[root@puppet /etc/puppetlabs/puppet]# more fileserver.conf
# This file consists of arbitrarily named sections/modules
# defining where files are served from and to whom

# Define a section 'files'
# Adapt the allow/deny settings to your needs. Order
# for allow/deny does not matter, allow always takes precedence
# over deny
[files]
  path /etc/puppetlabs/puppet/files
  allow node_name
  allow *.example.com

Check whether puppet client is allowed to connect to puppet master,
[root@puppet /etc/puppetlabs/puppet/manifests]# pwd
/etc/puppetlabs/puppet/manifests
[root@puppet manifests]# ls
 nodes.pp
 site.pp

[root@puppet manifests]# more nodes.pp
node 'client.my.bakary.org.au' {
include sudo
}

Specify remote file bucket,
[root@puppet /etc/puppetlabs/puppet/manifests]# more site.pp
# specify remote filebucket
filebucket { 'main':
  server => 'puppet.mycake.org.au ',
  path => false,
}

Create sample ntp directory and sample ntp.config file -> /etc/puppet/files
[root@puppet /etc/puppetlabs/files]# mkdir ntp && cd ntp
[root@puppet /etc/puppetlabs/files/ntp]# cp /etc/ntp.conf ntp.conf

Add following to site.pp,
-       Create class and specify the file to be updated or created and the location to be inserted in puppet node
-       Ensure the file is resides
-       Set file permissions
-       Set users and groups it belongs
-       Source location on puppet master
-       Tell puppet which client it should be run

[root@puppet /etc/puppetlabs/puppet/manifests]# vi site.pp
class ntp_class {
        # create the ntp file
        file { "/etc/ntp.conf":
                ensure => present,
                owner => root,
                group => root,
                mode => 0440,
                source => "puppet:///files/ntp/ntp.conf"
        }
}

# tell puppet on which client to run the class
node client.mycake.org.au {
    include ntp_class
}

File { backup => 'main' }

Puppet Dashboard Configuration
Edit /etc/puppetlabs/puppet/puppet.conf and include following in the [master] section,
[master]
reports = puppet_dashboard,store
reportdir = /var/lib/puppet/reports
reporturl = http://puppet:3000/reports

Restart puppet master and start puppet-dashboard
[root@puppet ]# service pe-puppet restart
Stopping puppetmaster:                                     [  OK  ]
Starting puppetmaster:                                     [  OK  ]
[root@puppet ]#  service pe-puppet-dashboard-workers start
Starting puppet-dashboard:                                 [  OK  ]

In order to test web GUI go to the following link in your browser
http://puppet:3000/

In order to add puppet clients to dashboard do following in the client /etc/puppet/puppet.conf
 [agent]
    report = true

Run puppet in noop mode on the client
[root@client]# puppetd --noop --test

What Next
·         Identify basic modules to be managed by Puppet
·         Select at least 3 Puppet clients
·         Based on above define re-usable component classes on Puppet
·         Setup svn repository to manage all editable files used by Puppet
·         Manage above clients via Puppet
·         Extend the master puppet configuration
·         Setup Cobbler to manage Kickstart
·         Configure to manage Nagios NRPE, LDAP and SUDO via Puppet

I will post most of the very soon.

Please let me know what you think and any obvious mistakes and improvements. Love to know what you guys think. Thanks

References
[1]. http://www.engineyard.com/blog/2011/why-puppet-should-manage-your-infrastructure/
[2]. http://bhuga.net/rapid-and-incremental-infrastructure-development-puppet
[3]. http://docs.puppetlabs.com/guides/from_source.html
[4]. http://docs.puppetlabs.com/mcollective/reference/basic/configuration.html
[5]. http://projects.puppetlabs.com/projects/puppet/wiki/Yum_Server_Build_Patterns
[6]. http://projects.puppetlabs.com/projects/puppet/wiki/Sudo_Patterns
[7]. http://www.howtoforge.com/installing_puppet_on_ubuntu
[8]. http://docs.puppetlabs.com/learning/
[9]. http://itservices.stanford.edu/strategy/sysadmin/automation
[10]. http://docs.puppetlabs.com/guides/language_guide.html
[11]. http://projects.puppetlabs.com/projects/1/wiki/Ldap_Nodes
[12]. http://blog.gurski.org/index.php/2010/01/28/automatic-monitoring-with-puppet-and-nagios/
[13]. http://docs.puppetlabs.com/guides/exported_resources.html
[14]. http://docs.puppetlabs.com/guides/setting_up.html
[15]. http://projects.puppetlabs.com/projects/1/wiki/Puppet_Scalability
[16]. http://projects.puppetlabs.com/projects/puppet/wiki/Big_Picture
[17]. http://www.craigdunn.org/2010/08/part-3-installing-puppet-dashboard-on-centos-puppet-2-6-1/

Friday 10 June 2011

My First Blog Post :)

This is my first blog post. Not that I wasn't writing but never thought about writing a blog until recently. I would  have to say that I was a taker more than a giver. The knowledge that I have gathered is very much thanks to many people's hard work and their generosity. I can confidently say that without the help and generosity of all the people who took their time and effort to document online, I 'd be no one. That is the simple truth. Therefore firstly a BIG THANK YOU all of them. Also on the other hand I have never been comfortable with my writing (as you can see). But I think time has come to move and give it a go.

I am the CoFounder/CTO of Mudboard.com - If I tell you about mudboard a bit more, Mudboard allows everybody to share all their creativity. That’s anything from images and writing to videos, music, narration and animation – and all in one place. That means whether you're a hobbyist, student, amateur or professional there's now a place where like minded creative people can inspire each other, collaborate, and have fun. And within this trusted community, people can buy and sell creativity to a global audience, which allows artists to generate revenue through Mudboard.

The purpose of the blog. Mostly I am going to share my personal experience behind Mudboard architecture building and many interesting topics related to the business as a whole. Hope I might be able to provide some information where someone can use it for their benefit. If that happens I will be most honored. As you can imagine, the journey is going to be long one, but we have to start from somewhere.

I am concluding the post with a saying I admire and love from Albert Einstein - A person who never made a mistake never tried anything new

Happy blogging!