Puppet Master and agent configuraton

Puppet is a configuration management tool which works on master- agent configuration and communication . It is centralised tool  for configuration and automaton purpose in multi OS environments.

Puppet has two layers: a configuration language to describe how the hosts and services should look, and an abstraction layer that allows the administrator to implement the configuration on a variety of platforms, including Unix, linux, Windows  Administrators can encode the configuration of a service as a policy, which Puppet then monitors and enforces.

Puppet s written in ruby language 

How does the puppet works 

Puppet agent is the daemon which runs in the client servers ( servers where we need the configuration changes) and the main server which is act as a master server will have puppet server installed. Puppet agent will communicate with the puppet server for the configuration changes through the SSL channel and fetch the changes ( default time interval for  the puppet agent for communication with the server for fetching the changes is 30 minutes )

 

As per the above picture there are 3 nodes with puppet agent configured and the puppet master is configured at the centralised server.Now there are multiple steps involved whenever a puppet agent of any node connects to a puppet master server for fetching data. These steps are mentioned below.

Step 1: Whenever a client node connects to the master, the master server analyzes the configuration to be applied to the node, and how to apply that configs on the node.

Step 2:Puppet master server Takes and collects all the resources and configurations to be applied to the node, and compiles it and make it a catalogue. This catalogue is given to the puppet agent of the node.

Step 3: Puppet agent will apply the configuration on the node, according to the catalogue, and then reply back, and submit the report of the configuration applied to the puppet master server.

As i pointed earlier puppet will work on cross platform environments and when ever an agent communicates with the master server the factor tool will help the master server to identify the configuration and OS details of the client node

installing the puppet master server   

1. Let's take 2 servers in ubuntu 14.04 as one is puppet master and other is puppet agent as below, set the hostnames and add the respective hostnames in both the servers /etc/hosts file 

root@puppetagent:~# uname -a && hostname

Linux puppetagent.unixchips.com 4.4.0-1066-aws #76-Ubuntu SMP Thu Aug 16 16:21:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

puppetagent.unixchips.com

root@puppetmaster:~# uname -a && hostname

Linux puppetmaster.unixchips.com 4.4.0-1066-aws #76-Ubuntu SMP Thu Aug 16 16:21:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

puppetmaster.unixchips.com

2. Download the puppet labs release in both the nodes 

root@puppetmaster:~# wget https://apt.puppetlabs.com/puppetlabs-release-trusty.deb

--2018-09-17 20:29:29--  https://apt.puppetlabs.com/puppetlabs-release-trusty.deb

Resolving apt.puppetlabs.com (apt.puppetlabs.com)... 52.222.190.242, 52.222.190.106, 52.222.190.188, ...

Connecting to apt.puppetlabs.com (apt.puppetlabs.com)|52.222.190.242|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 16944 (17K) [application/x-debian-package]

Saving to: ‘puppetlabs-release-trusty.deb’

puppetlabs-release-trusty.deb             100%[=====================================================================================>]  16.55K  --.-KB/s    in 0s

2018-09-17 20:29:29 (373 MB/s) - ‘puppetlabs-release-trusty.deb’ saved [16944/16944]

root@puppetmaster:~# sudo dpkg -i puppetlabs-release-trusty.deb

Selecting previously unselected package puppetlabs-release.

(Reading database ... 219442 files and directories currently installed.)

Preparing to unpack puppetlabs-release-trusty.deb ...

Unpacking puppetlabs-release (1.1-1) ...

Setting up puppetlabs-release (1.1-1) ...

3. In the puppet master server run the following 

root@puppetmaster:~# sudo apt-get install puppetmaster-passenger

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following extra packages will be installed:

  apache2 apache2-bin apache2-data augeas-lenses debconf-utils facter hiera

  libapache2-mod-passenger libapr1 libaprutil1 libaprutil1-dbd-sqlite3

  libaprutil1-ldap libaugeas-ruby libaugeas0 libev4 libruby1.9.1 puppet-common

  puppetmaster-common ruby ruby-augeas ruby-json ruby-passenger ruby-rack

  ruby-shadow ruby1.9.1 ssl-cert virt-what

Suggested packages:

  apache2-doc apache2-suexec-pristine apache2-suexec-custom apache2-utils

  augeas-doc augeas-tools ruby-selinux libselinux-ruby1.8 librrd-ruby1.9.1

  librrd-ruby1.8 puppet-el vim-puppet stompserver ruby-stomp libstomp-ruby1.8

..................................................................................

 * Restarting web server apache2                                                                                                                                        AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using puppetmaster.unixchips.com. Set the 'ServerName' directive globally to suppress this message

                                                                                                                                                                 [ OK ]

Processing triggers for libc-bin (2.19-0ubuntu6.14) ...

4. We need to maintain consistent puppet version with in the environment to avoid any issues for the working infrastructure ., for that first we need to stop the apache which is controlling the puppet process 

root@puppetmaster:~# ps -ef |grep -i apache2

root      3724     1  0 21:01 ?        00:00:00 /usr/sbin/apache2 -k start

www-data  3745  3724  0 21:01 ?        00:00:00 /usr/sbin/apache2 -k start

www-data  3746  3724  0 21:01 ?        00:00:00 /usr/sbin/apache2 -k start

root      4047  3936  0 21:40 pts/1    00:00:00 grep --color=auto -i apache2

root@puppetmaster:~# service apache2 stop

 * Stopping web server apache2        

The current running version of the  puppet is 

root@puppetmaster:~# puppet help | tail -n 1

Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations

   (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')

Puppet v3.8.7

Now we need to lock the puppet version as per the current vesrion , add the below lines in /etc/apt/preferences.d/00-puppet.pref

Package: puppet puppet-common puppetmaster-passenger

Pin: version 3.8.7

Pin-Priority: 501

Puppet installation in agent node 

1. First download the package and extract it 

root@puppetagent:~# wget https://apt.puppetlabs.com/puppetlabs-release-trusty.deb

--2018-09-17 21:47:59--  https://apt.puppetlabs.com/puppetlabs-release-trusty.deb

Resolving apt.puppetlabs.com (apt.puppetlabs.com)... 52.222.190.188, 52.222.190.225, 52.222.190.242, ...

Connecting to apt.puppetlabs.com (apt.puppetlabs.com)|52.222.190.188|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 16944 (17K) [application/x-debian-package]

Saving to: ‘puppetlabs-release-trusty.deb’

100%[==============================================================================================================================>] 16,944      --.-K/s   in 0s

2018-09-17 21:47:59 (237 MB/s) - ‘puppetlabs-release-trusty.deb’ saved [16944/16944]

root@puppetagent:~# sudo dpkg -i puppetlabs-release-trusty.deb

Selecting previously unselected package puppetlabs-release.

(Reading database ... 51306 files and directories currently installed.)

Preparing to unpack puppetlabs-release-trusty.deb ...

Unpacking puppetlabs-release (1.1-1) ...

Setting up puppetlabs-release (1.1-1) ...

root@puppetagent:~# sudo apt-get update

Ign http://ap-south-1.ec2.archive.ubuntu.com trusty InRelease

Ign http://apt.puppetlabs.com trusty InRelease

Get:1 http://apt.puppetlabs.com trusty Release.gpg [819 B]

Get:2 http://apt.puppetlabs.com trusty Release [88.9 kB]

Get:3 http://apt.puppetlabs.com trusty/main Sources [49.9 kB]

Get:4 http://apt.puppetlabs.com trusty/dependencies Sources [2,157 B]

Get:5 http://apt.puppetlabs.com trusty/main amd64 Packages [69.0 kB]

Get:6 http://apt.puppetlabs.com trusty/dependencies amd64 Packages [1,180 B]

Get:7 http://ap-south-1.ec2.archive.ubuntu.com trusty-updates InRelease [65.9 kB]

Get:8 http://security.ubuntu.com trusty-security InRelease [65.9 kB]

Hit http://ap-south-1.ec2.archive.ubuntu.com trusty-backports InRelease

Hit http://ap-south-1.ec2.archive.ubuntu.com trusty Release.gpg

Get:9 http://ap-south-1.ec2.archive.ubuntu.com trusty-updates/main Sources [422 kB]

Get:10 http://security.ubuntu.com trusty-security/main Sources [162 kB]

......................................................................................................

Ign http://ap-south-1.ec2.archive.ubuntu.com trusty/universe Translation-en_US

Fetched 12.9 MB in 9s (1,398 kB/s)

Reading package lists... Done

2. Install the puppetagent as below 

root@puppetagent:~# sudo apt-get install puppet

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following extra packages will be installed:

  augeas-lenses debconf-utils facter hiera libaugeas-ruby libaugeas0

  libruby1.9.1 puppet-common ruby ruby-augeas ruby-json ruby-shadow ruby1.9.1

  virt-what

Suggested packages:

  augeas-doc augeas-tools puppet-el vim-puppet ruby-selinux libselinux-ruby1.8

  librrd-ruby1.9.1 librrd-ruby1.8 ri ruby-dev ruby1.9.1-examples ri1.9.1

  graphviz ruby1.9.1-dev ruby-switch

Recommended packages:

  rdoc

..................................................................................................

 * Starting puppet agent

puppet not configured to start, please edit /etc/default/puppet to enable

                                                                                                                                                                 [ OK ]

Processing triggers for libc-bin (2.19-0ubuntu6.14) ...

Processing triggers for ureadahead (0.100.0-16) ...

3. The puppet agent is disabled by default and we need to enable the same by updating the file /etc/default/puppet (by default the below values is "no")

START=yes 

4. Now let's lock the puppet agent to avoid any update 

root@puppetagent:~# puppet help | tail -n 1

Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations

   (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')

Puppet v3.8.7

update the below entries in /etc/apt/preferences.d/00-puppet.pref

Package: puppet puppet-common

Pin: version 3.8.7

Pin-Priority: 501

Puppet master configuration 

1. Login to the puppet master server and edit the /etc/puppet/puppet.conf file by

removing below parameter templatedir=$confdir/templates
in the main session of the master puppet file set the dns_alt_names and certname as below

cert_name = puppet
dns_alt_names = puppet,puppet.unixchips.com

***********************************************************
[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
certname = puppet
dns_alt_names = puppet, puppet.unixchips.com
[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY

*********************************************************************

Make sure we have configured /etc/hosts and /etc/hostname files properly also please use the certname as puppet only as it is pre configured in apache

2. Puppet master itself act as a certificate authority and generates its own certificates which is used to sign certificate requests. Now we have to setup master certificate

remove any existing SSL certificate installed

root@puppetmaster:~# rm -rf /var/lib/puppet/ssl/

Now let's create a new CA certificate using below command

***********************************************************************
root@puppetmaster:~# sudo puppet master --verbose --no-daemonize
Info: Creating a new SSL key for ca
Info: Creating a new SSL certificate request for ca
Info: Certificate Request fingerprint (SHA256): 89:89:5B:92:BC:56:56:14:4A:0C:43:58:14:6C:BC:4F:E4:10:1C:B1:FF:D2:B2:B4:B6:5B:63:4B:5A:89:89:33
Notice: Signed certificate request for ca
Info: Creating a new certificate revocation list
Info: Creating a new SSL key for puppet
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for puppet
Info: Certificate Request fingerprint (SHA256): 3A:08:C2:A3:D1:AC:12:89:16:35:E6:0A:84:76:0C:A2:36:D7:4B:74:7C:5C:D7:3E:77:67:1E:E0:09:92:4A:CC
Notice: puppet has a waiting certificate request
Notice: Signed certificate request for puppet
Notice: Removing file Puppet::SSL::CertificateRequest puppet at '/var/lib/puppet/ssl/ca/requests/puppet.pem'
Notice: Removing file Puppet::SSL::CertificateRequest puppet at '/var/lib/puppet/ssl/certificate_requests/puppet.pem'
Notice: Starting Puppet master version 3.8.7
Error: Could not run: Address already in use - bind(2)

**************************************************************************

we will get an error as "Address already in use" because master puppet runs when apache is running .we can check like below 

root@puppetmaster:~# sudo netstat -ltnp

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      957/sshd

tcp6       0      0 :::22                   :::*                    LISTEN      957/sshd

tcp6       0      0 :::443                  :::*                    LISTEN      1156/apache2

tcp6       0      0 :::8140                 :::*                    LISTEN      1156/apache2

tcp6       0      0 :::80                   :::*                    LISTEN      1156/apache2

Let's stop the apache first

root@puppetmaster:~# sudo service apache2 stop
 * Stopping web server apache2                                                   *
 * The apache2 configtest failed, so we are trying to kill it manually. This is almost certainly suboptimal, so please make sure your system is working as you'd expect now!

Now we can check the details of the certificate which we have created as below 

root@puppetmaster:~# sudo puppet cert list -all

+ "puppet" (SHA256) 4D:3D:B4:D1:B2:C6:FA:BB:3D:EC:42:F7:70:90:C0:FF:02:71:D0:DC:47:37:7D:0C:10:89:A3:ED:18:A2:6D:DF (alt names: "DNS:puppet", "DNS:puppet.unixchips.com")

Now we have to create the main manifest file or site manifest inside the below location 

root@puppetmaster:/# sudo touch /etc/puppet/manifests/site.pp

Start the puppet master as below ( we can ignore the below message )

root@puppetmaster:/var/lib/puppet/ssl/private_keys# sudo service apache2 start

 * Starting web server apache2                                                                                                                                          AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using puppetmaster.unixchips.com. Set the 'ServerName' directive globally to suppress this message

 *

Configure puppet agent 

1. First edit the /etc/puppet/puppet.conf in agent node as below 

[main]

logdir=/var/log/puppet

vardir=/var/lib/puppet

ssldir=/var/lib/puppet/ssl

rundir=/var/run/puppet

factpath=$vardir/lib/facter

#[master]

# These are needed when the puppetmaster is run by passenger

# and can safely be removed if webrick is used.

#ssl_client_header = SSL_CLIENT_S_DN

#ssl_client_verify_header = SSL_CLIENT_VERIFY

[agent]

server = puppetmaster.unixchips.com

server = puppet.unixchips.com

2. Now configure the hostname and /etc/hosts file properly 

root@puppetagent:~# cat /etc/hostname

puppetagent.unixchips.com

root@puppetagent:~# cat /etc/hosts

127.0.0.1 puppetagent puppetagent.unixchips.com

172.31.6.59   puppetmaster puppetmaster.unixchips.com

3. Now start the puppet agent 

root@puppetagent:~# sudo service puppet start

 * Starting puppet agent                                                 [ OK ]

root@puppetagent:~#

4. We need to check the sign request on master by running following command on puppet master 

root@puppetmaster:~# sudo puppet cert list

  "puppetagent.ap-south-1.compute.internal" (SHA256) 54:E9:7C:68:1B:E2:CD:85:52:74:7E:30:FA:AF:14:B6:4A:61:65:43:61:92:FA:15:E8:8C:D0:A7:4B:71:5F:20

root@puppetmaster:~#

We can see cert request is reached from the agent node 

5. We can sign the request using the puppet cert sign command with the hostname of the certificate we want to sign 

root@puppetmaster:~# sudo puppet cert sign puppetagent.ap-south-1.compute.internal

Notice: Signed certificate request for puppetagent.ap-south-1.compute.internal

Notice: Removing file Puppet::SSL::CertificateRequest puppetagent.ap-south-1.compute.internal at '/var/lib/puppet/ssl/ca/requests/puppetagent.ap-south-1.compute.internal.pem'

This inidcats the communication between puppet master and the agent 

Below link will provide some sample manifests which we are using for general purpose

http://unixchips.blogspot.com/2017/08/generally-using-puppet-modules.html

    Thank you for reading 

                                                                                                                                 

  

AWS-cloudwatch configuration

Amazon cloud watch is a robust monitoring mechanism for the entire AWS infrastructure which includes EC2 instances, S3, ELB etc. We can able to track wide verity of metrics like CPU, disk,network traffic, available storage space,memory ( using custom metrics) in cloud watch

AWS also provides access to system and application logs custom alarms that provides real time notification when specific metric appear in logs or when certain events take place .

cloud watch mechanism 

while creating the instances , by default we have below metrics for monitoring in the EC2 instances

1. CPU utilization
2. Disk reads/Disk read operations
3. Disk writes/Disk write operations
4. Network in/Network out
5. Network packets in/Network packets out
6. Status check failed count
7. Instance Status check failed
8.System status check failed
9.CPU credit usage
10. CPU credit balance


So as per that we have only limited resources available in monitoring metrics. We may need to find out the solution to add more metrics on custom level

1. First we need to create the IAM role for cloud watch administration

login to AWS console-services-IAM-roles and create a new role as below

2. click on the EC2 from the services to use this role and click on the permissions we will get the policy details which is ready to attach the role

3. select the policy called "CloudwatchAgentServerPolicy" from the filter policies option

4. Provide the role name as "cloudwatchagentrole" and create the role

5. Now we have to attach this role to an EC2 instance . select the instance which we want to install the agent and go to actions - instance settings-attach/replace IAM  role

6. Login to the instance and download the agent using the below link

https://s3.amazonaws.com/amazoncloudwatch-agent                                                                                        /linux/amd64/latest/AmazonCloudWatchAgent.zip

7. Unzip & Install the agent using the below command

8.Let's configure the metrics which is need to collect and send to the cloud watch , we can configure agent using below command

9. start the agent using below command

10. Now login to the cloudwatch console and click on metrics we can see the CWagent metrics configured

Also if you check the logs from Cloudwatch-Log groups-messages we can see the logs which we have added as a metric in agent from /var/log/messages

Configuring the custom DB metric using boto3


***********************************************************************
from boto.ec2.cloudwatch import connect_to_region

import MySQLdb

import boto.ec2.cloudwatch

db=MySQLdb.connect("dbserver.c5g5m1fcix6x.us-west-2.rds.amazonaws.com","dbadmin","Onm0bile","dbserver")

cursor=db.cursor()

n=cursor.execute("select * from GetActiveSessions")

n1=cursor.fetchone()

s=str(n1)

table=string.maketrans( '', '', )

number=s.translate(table,"(){}<>,L")

numbers=int(number)

reg='ap-northeast-1'

conn_cw=boto.ec2.cloudwatch.connect_to_region(reg,aws_access_key_id=’<access key d>’,aws_secret_access_key=’secret access key>')

conn_cw.put_metric_data(namespace='my_namespace',name='my_metric',value=numbers,dimensions={'InstanceId':'i-a1b2c3d4'})

*****************************************************************************

Custom metric dashboard for DB

Thank you for reading ...