Security Guidelines for Azure Devops

 

Securing your Azure DevOps environment is crucial to ensuring the confidentiality, integrity, and availability of your software development and deployment processes. While I can provide you with a general security checklist, please note that security best practices may change over time, so always refer to the latest documentation and guidelines from Microsoft.

Below are the main points for  securing Azure Devops 

1. Managing users and groups using Role-Based Access Control (RBAC) to define and enforce granular permissions.

Role-Based Access Control (RBAC) is a method of managing user access and permissions
based on their roles within an organization. It helps maintain security by ensuring that users
only have access to the resources and operations relevant to their job responsibilities. In
Azure DevOps, you can use RBAC to assign appropriate permissions to users and groups.

2. Applying the principle of least privilege for granting permissions to minimize potential risks.

The principle of least privilege (PoLP) is a security best practice that involves granting users only the minimum permissions they need to perform their job duties. By applying this principle, you can reduce the risk of unauthorized access, data breaches, and other security incidents.

3. Regularly reviewing user accounts and disabling unnecessary accounts to reduce the attack surface.

Regularly reviewing user accounts and disabling unnecessary accounts is essential to maintain a secure environment in Azure DevOps. By keeping user accounts up to date and removing unused or inactive accounts, you can minimize the risk of unauthorized access and data breaches

4. Implementing strong authentication with Multi-Factor Authentication (MFA) to protect against unauthorized access.

Implementing strong authentication with Multi-Factor Authentication (MFA) is a critical security measure that helps protect your Azure DevOps environment from unauthorized access. MFA requires users to provide at least two forms of verification before granting access, making it much more difficult for attackers to compromise user accounts

5. Integrating centralized identity management using Single Sign-On (SSO) and Azure Active Directory.

Providing centralized identity management using Single Sign-On (SSO) and Azure Active Directory (Azure AD) integration simplifies access control and enhances security in Azure DevOps. SSO allows users to authenticate once and access multiple applications, while Azure AD integration enables centralized management of user accounts and permissions

6. Reducing authentication risks using risk-based policies and Azure AD Identity Protection integration.

Reducing authentication risks with risk-based policies and Azure AD Identity Protectionhelps enhance security in Azure DevOps by detecting and responding to potential threats in real-time. Risk-based policies evaluate user behavior and other factors to identify potential security risks, while Azure AD Identity Protection leverages machine learning algorithms to detect suspicious activities

7. Restricting access with IP-based network security groups and private networks.

Restricting access using IP-based network security groups and private networks helps enhance security in Azure DevOps by limiting access to your resources based on specific IP addresses or address ranges. This approach can help prevent unauthorized access and reduce the attack surface of your environment.

8. Establishing secure communication with on-premises systems using VPN or ExpressRoute.

Establishing secure communication with on-premises systems using VPN or ExpressRoute is essential when you need to integrate Azure DevOps with your existing infrastructure. Both options allow you to create private connections between your on-premises network and Azure, ensuring secure data transfer and reducing exposure to the public internet

9. Protecting and routing network traffic with Azure DDoS Protection and Azure Firewall.

Protecting and routing network traffic with Azure DDoS Protection and Azure Firewall enhances the security of your Azure DevOps environment by safeguarding against Distributed Denial of Service (DDoS) attacks and filtering network traffic based on specific rules.

10. Applying code review processes and utilizing static and dynamic code analysis tools for vulnerability detection.

Applying code review processes to detect security vulnerabilities is essential for ensuring the security and reliability of your Azure DevOps projects. Code reviews help identify potential issues early in the development process, reducing the risk of security breaches and improving overall code quality

11. Establishing secure coding standards and ensuring dependency security.

Using static and dynamic code analysis tools for automatic detection of vulnerabilities is a crucial part of ensuring the security of your Azure DevOps projects. These tools can help identify potential issues early in the development process, reducing the risk of security breaches and improving overall code quality. Select suitable static application security testing (SAST) and dynamic application security testing (DAST) tools based on your organization's requirements, programming languages, and frameworks. Examples of SAST tools include SonarQube, Fortify, and Checkmarx, while examples of DAST tools include OWASP ZAP, Burp Suite, and Arachni.

12. Incorporating security controls and automated tests in Build and Release pipelines.

Adding security controls and automated tests in Build and Release pipelines can help improve the security of your Azure DevOps projects by identifying and addressing vulnerabilities early in the development process. Integrating security checks into your pipelines ensures that security is an integral part of your software development lifecycle.

13. Securing agents with trusted agent pools and implementing Git branch policies and pull request reviews for code security.

Securing agents by using trusted agent pools is essential to ensure the integrity and security of your build and release processes in Azure DevOps. Trusted agent pools help minimize the risk of unauthorized access or tampering with your build and release pipelines

14. Storing credentials, certificates, and access keys securely in Azure Key Vault and configuring access for Azure DevOps pipelines.

Securely storing credentials, certificates, and access keys in Azure Key Vault is crucial for protecting sensitive information and maintaining the security of your Azure DevOps projects. Azure Key Vault helps centralize and manage secrets, making it easier to implement secure access controls and monitor usage

15. Monitoring changes using Azure DevOps audit logs for security, compliance, and operational awareness.

Monitoring changes using Azure DevOps audit logs is essential for maintaining security, compliance, and operational awareness in your DevOps environment. Audit logs provide visibility into activities and changes within your Azure DevOps projects, enabling you to track user behavior, identify potential security issues, and troubleshoot problems

16. Continuously tracking and improving security posture with Azure Policy and Azure Security Center

Continuously tracking and improving your security posture with Azure Policy and Azure Security Center is essential for ensuring the ongoing security and compliance of your Azure DevOps environment. These tools help you define, monitor, and enforce security policies across your Azure resources, providing a comprehensive view of your security posture and facilitating continuous improvement.

17. Conducting internal and external security audits and penetration tests for evaluation and continuous improvement.

Performing internal and external security audits and penetration tests is essential for evaluating the security of your Azure DevOps environment and identifying potential vulnerabilities. Regular audits and tests help you uncover security weaknesses, validate existing security controls, and prioritize remediation efforts

18. Regularly review and update the security configurations of your Azure DevOps services, resources, and tools.

Regularly reviewing and updating the security configurations of your Azure DevOps services, resources, and tools is an essential practice to maintain a secure environment and address evolving threats.

19. Implement secure baselines for your Azure resources and enforce them consistently across your environment.

Implementing secure baselines for your Azure resources and enforcing them consistently across your environment is crucial to maintaining a secure and compliant Azure DevOps setup

20. Use Azure Policy to define and enforce security configurations across your Azure resources.

Using Azure Policy to define and enforce security configurations across your Azure resources is a crucial part of maintaining a secure and compliant environment

21. Continuously monitor configuration changes and assess their impact on your security posture.

Continuously monitoring configuration changes and assessing their impact on your security posture is vital for maintaining a secure environment and addressing potential risks in a timely manner

22. Implement a robust backup and recovery strategy for your critical data, including source code, artifacts, and configuration data.

Implementing a robust backup and recovery strategy for your critical data, including sourcecode, artifacts, and configuration data, is essential for ensuring business continuity andreducing the impact of data loss or corruption. The main points to consider for this are identify critical data, define backup frequency and retention policies, choose appropriate backup methods, use of azure native backup solutions, store backups offsite/multiple locations, encrypt backups etc 

23. Use Azure Backup and Azure Site Recovery to protect your data and applications.

Using Azure Backup and Azure Site Recovery to protect your data and applications is an effective way to ensure business continuity and minimize downtime in the event of data loss or disasters.

24. Regularly test your data recovery processes to ensure they are effective and up to date.

Regularly testing your data recovery processes to ensure they are effective and up to date is crucial for maintaining business continuity and reducing the impact of data loss or corruption.  The main points to consider here is Develop a testing schedule, Test various recovery scenarios, Document test results, Update recovery plans, Train and educate your teams, Review and update testing processes.By regularly testing your data recovery processes, you can ensure they are effective and up to date, helping to maintain business continuity and minimize the impact of data loss or corruption. This proactive approach also supports a culture of continuous improvement and collaboration across teams and helps protect your organization's assets.

25. Establish a disaster recovery plan to minimize downtime and data loss in case of a security breach or system failure.

Establishing a disaster recovery plan is essential to minimize downtime and data loss in case of a security breach or system failure. Main points to be considered here is Identify critical systems and assets, Define recovery objectives, Develop recovery strategies, Document recovery procedures, Test and validate the plan, Train and educate your teams, Review and update the plan. By establishing a disaster recovery plan, you can minimize downtime and data loss in case of a security breach or system failure, helping to maintain business continuity and protect your organization's assets. This proactive approach also supports a culture of continuous improvement and collaboration across teams.

26. Maintain an up-to-date inventory of all Azure DevOps resources, including repositories, pipelines, environments, and tools.

Maintaining an up-to-date inventory of all Azure DevOps resources, including repositories, pipelines, environments, and tools, is crucial for managing and securing your organization's assets effectively. Main points to be considered here is Create a centralized inventory, Include relevant metadata, Implement a tagging strategy, Automate inventory updates,Regularly review and audit your inventory, Integrate with other asset management systems. By maintaining an up-to-date inventory of all Azure DevOps resources, you can better manage and secure your organization's assets, track changes, and enforce access control policies. This proactive approach helps protect your organization's assets and fosters a culture of continuous improvement and collaboration across teams.

27. Use Azure Resource Manager (ARM) templates to manage your Azure resources in a consistent and automated manner.

Using Azure Resource Manager (ARM) templates to manage your Azure resources in a consistent and automated manner is an important best practice for managing infrastructure as code. Main points to be considered here are Standardize resource configurations, Improve collaboration and version control, Automate resource provisioning and updates, Simplify resource management, Validate and test templates, Reuse and share templates. By using Azure Resource Manager (ARM) templates to manage your Azure resources in a consistent and automated manner, you can improve collaboration, simplify resource management, and reduce the potential for human error and inconsistencies. This approach also supports a culture of continuous improvement and collaboration across teams, helping to protect your organization's assets and streamline operations

28. Implement tagging strategies to categorize your Azure resources based on project, team, or other relevant attributes.

Implementing tagging strategies to categorize your Azure resources based on project, team, or other relevant attributes is an essential practice for effective resource management and organization. Main points to be considered here is Define a consistent tagging strategy, Use meaningful and descriptive tags, Enforce tag usage, Monitor and audit tag usage, Update and maintain your tagging strategy, Use tags for cost management and reporting.By implementing tagging strategies to categorize your Azure resources based on project, team, or other relevant attributes, you can improve resource management, organization, and cost allocation. This approach also promotes a culture of collaboration and shared responsibility across teams, helping to protect your organization's assets and streamline operations.

29. Continuously monitor your inventory and resources for any unauthorized changes or access.

Continuously monitoring your inventory and resources for any unauthorized changes or access is crucial for maintaining the security and integrity of your Azure DevOps environment. The main points to be considered here are Use Azure Monitor, Review Azure DevOps audit logs, Implement Azure Security Center, Configure Azure Active Directory (AD) monitoring, Set up intrusion detection and prevention systems, Regularly audit access control and permissions. Use automated tools for monitoring. By continuously monitoring your inventory and resources for unauthorized changes or access, you can proactively detect potential security issues and respond quickly to mitigate risks. This approach helps maintain the security and integrity of your Azure DevOps environment and fosters a culture of shared responsibility and vigilance across your organization.

In conclusion, adopting a comprehensive security approach when using Azure DevOps is crucial for protecting your organization's assets and ensuring the integrity of your development and deployment processes. By following the guidelines outlined above, you can effectively manage access control, authentication, network security, code security, Azure Key Vault usage, and regular auditing to maintain a secure environment.

Puppet Master and agent configuraton

Puppet is a configuration management tool which works on master- agent configuration and communication . It is centralised tool  for configuration and automaton purpose in multi OS environments.

Puppet has two layers: a configuration language to describe how the hosts and services should look, and an abstraction layer that allows the administrator to implement the configuration on a variety of platforms, including Unix, linux, Windows  Administrators can encode the configuration of a service as a policy, which Puppet then monitors and enforces.

Puppet s written in ruby language 

How does the puppet works 

Puppet agent is the daemon which runs in the client servers ( servers where we need the configuration changes) and the main server which is act as a master server will have puppet server installed. Puppet agent will communicate with the puppet server for the configuration changes through the SSL channel and fetch the changes ( default time interval for  the puppet agent for communication with the server for fetching the changes is 30 minutes )

 

As per the above picture there are 3 nodes with puppet agent configured and the puppet master is configured at the centralised server.Now there are multiple steps involved whenever a puppet agent of any node connects to a puppet master server for fetching data. These steps are mentioned below.

Step 1: Whenever a client node connects to the master, the master server analyzes the configuration to be applied to the node, and how to apply that configs on the node.

Step 2:Puppet master server Takes and collects all the resources and configurations to be applied to the node, and compiles it and make it a catalogue. This catalogue is given to the puppet agent of the node.

Step 3: Puppet agent will apply the configuration on the node, according to the catalogue, and then reply back, and submit the report of the configuration applied to the puppet master server.

As i pointed earlier puppet will work on cross platform environments and when ever an agent communicates with the master server the factor tool will help the master server to identify the configuration and OS details of the client node

installing the puppet master server   

1. Let's take 2 servers in ubuntu 14.04 as one is puppet master and other is puppet agent as below, set the hostnames and add the respective hostnames in both the servers /etc/hosts file 

root@puppetagent:~# uname -a && hostname

Linux puppetagent.unixchips.com 4.4.0-1066-aws #76-Ubuntu SMP Thu Aug 16 16:21:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

puppetagent.unixchips.com

root@puppetmaster:~# uname -a && hostname

Linux puppetmaster.unixchips.com 4.4.0-1066-aws #76-Ubuntu SMP Thu Aug 16 16:21:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

puppetmaster.unixchips.com

2. Download the puppet labs release in both the nodes 

root@puppetmaster:~# wget https://apt.puppetlabs.com/puppetlabs-release-trusty.deb

--2018-09-17 20:29:29--  https://apt.puppetlabs.com/puppetlabs-release-trusty.deb

Resolving apt.puppetlabs.com (apt.puppetlabs.com)... 52.222.190.242, 52.222.190.106, 52.222.190.188, ...

Connecting to apt.puppetlabs.com (apt.puppetlabs.com)|52.222.190.242|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 16944 (17K) [application/x-debian-package]

Saving to: ‘puppetlabs-release-trusty.deb’

puppetlabs-release-trusty.deb             100%[=====================================================================================>]  16.55K  --.-KB/s    in 0s

2018-09-17 20:29:29 (373 MB/s) - ‘puppetlabs-release-trusty.deb’ saved [16944/16944]

root@puppetmaster:~# sudo dpkg -i puppetlabs-release-trusty.deb

Selecting previously unselected package puppetlabs-release.

(Reading database ... 219442 files and directories currently installed.)

Preparing to unpack puppetlabs-release-trusty.deb ...

Unpacking puppetlabs-release (1.1-1) ...

Setting up puppetlabs-release (1.1-1) ...

3. In the puppet master server run the following 

root@puppetmaster:~# sudo apt-get install puppetmaster-passenger

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following extra packages will be installed:

  apache2 apache2-bin apache2-data augeas-lenses debconf-utils facter hiera

  libapache2-mod-passenger libapr1 libaprutil1 libaprutil1-dbd-sqlite3

  libaprutil1-ldap libaugeas-ruby libaugeas0 libev4 libruby1.9.1 puppet-common

  puppetmaster-common ruby ruby-augeas ruby-json ruby-passenger ruby-rack

  ruby-shadow ruby1.9.1 ssl-cert virt-what

Suggested packages:

  apache2-doc apache2-suexec-pristine apache2-suexec-custom apache2-utils

  augeas-doc augeas-tools ruby-selinux libselinux-ruby1.8 librrd-ruby1.9.1

  librrd-ruby1.8 puppet-el vim-puppet stompserver ruby-stomp libstomp-ruby1.8

..................................................................................

 * Restarting web server apache2                                                                                                                                        AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using puppetmaster.unixchips.com. Set the 'ServerName' directive globally to suppress this message

                                                                                                                                                                 [ OK ]

Processing triggers for libc-bin (2.19-0ubuntu6.14) ...

4. We need to maintain consistent puppet version with in the environment to avoid any issues for the working infrastructure ., for that first we need to stop the apache which is controlling the puppet process 

root@puppetmaster:~# ps -ef |grep -i apache2

root      3724     1  0 21:01 ?        00:00:00 /usr/sbin/apache2 -k start

www-data  3745  3724  0 21:01 ?        00:00:00 /usr/sbin/apache2 -k start

www-data  3746  3724  0 21:01 ?        00:00:00 /usr/sbin/apache2 -k start

root      4047  3936  0 21:40 pts/1    00:00:00 grep --color=auto -i apache2

root@puppetmaster:~# service apache2 stop

 * Stopping web server apache2        

The current running version of the  puppet is 

root@puppetmaster:~# puppet help | tail -n 1

Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations

   (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')

Puppet v3.8.7

Now we need to lock the puppet version as per the current vesrion , add the below lines in /etc/apt/preferences.d/00-puppet.pref

Package: puppet puppet-common puppetmaster-passenger

Pin: version 3.8.7

Pin-Priority: 501

Puppet installation in agent node 

1. First download the package and extract it 

root@puppetagent:~# wget https://apt.puppetlabs.com/puppetlabs-release-trusty.deb

--2018-09-17 21:47:59--  https://apt.puppetlabs.com/puppetlabs-release-trusty.deb

Resolving apt.puppetlabs.com (apt.puppetlabs.com)... 52.222.190.188, 52.222.190.225, 52.222.190.242, ...

Connecting to apt.puppetlabs.com (apt.puppetlabs.com)|52.222.190.188|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 16944 (17K) [application/x-debian-package]

Saving to: ‘puppetlabs-release-trusty.deb’

100%[==============================================================================================================================>] 16,944      --.-K/s   in 0s

2018-09-17 21:47:59 (237 MB/s) - ‘puppetlabs-release-trusty.deb’ saved [16944/16944]

root@puppetagent:~# sudo dpkg -i puppetlabs-release-trusty.deb

Selecting previously unselected package puppetlabs-release.

(Reading database ... 51306 files and directories currently installed.)

Preparing to unpack puppetlabs-release-trusty.deb ...

Unpacking puppetlabs-release (1.1-1) ...

Setting up puppetlabs-release (1.1-1) ...

root@puppetagent:~# sudo apt-get update

Ign http://ap-south-1.ec2.archive.ubuntu.com trusty InRelease

Ign http://apt.puppetlabs.com trusty InRelease

Get:1 http://apt.puppetlabs.com trusty Release.gpg [819 B]

Get:2 http://apt.puppetlabs.com trusty Release [88.9 kB]

Get:3 http://apt.puppetlabs.com trusty/main Sources [49.9 kB]

Get:4 http://apt.puppetlabs.com trusty/dependencies Sources [2,157 B]

Get:5 http://apt.puppetlabs.com trusty/main amd64 Packages [69.0 kB]

Get:6 http://apt.puppetlabs.com trusty/dependencies amd64 Packages [1,180 B]

Get:7 http://ap-south-1.ec2.archive.ubuntu.com trusty-updates InRelease [65.9 kB]

Get:8 http://security.ubuntu.com trusty-security InRelease [65.9 kB]

Hit http://ap-south-1.ec2.archive.ubuntu.com trusty-backports InRelease

Hit http://ap-south-1.ec2.archive.ubuntu.com trusty Release.gpg

Get:9 http://ap-south-1.ec2.archive.ubuntu.com trusty-updates/main Sources [422 kB]

Get:10 http://security.ubuntu.com trusty-security/main Sources [162 kB]

......................................................................................................

Ign http://ap-south-1.ec2.archive.ubuntu.com trusty/universe Translation-en_US

Fetched 12.9 MB in 9s (1,398 kB/s)

Reading package lists... Done

2. Install the puppetagent as below 

root@puppetagent:~# sudo apt-get install puppet

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following extra packages will be installed:

  augeas-lenses debconf-utils facter hiera libaugeas-ruby libaugeas0

  libruby1.9.1 puppet-common ruby ruby-augeas ruby-json ruby-shadow ruby1.9.1

  virt-what

Suggested packages:

  augeas-doc augeas-tools puppet-el vim-puppet ruby-selinux libselinux-ruby1.8

  librrd-ruby1.9.1 librrd-ruby1.8 ri ruby-dev ruby1.9.1-examples ri1.9.1

  graphviz ruby1.9.1-dev ruby-switch

Recommended packages:

  rdoc

..................................................................................................

 * Starting puppet agent

puppet not configured to start, please edit /etc/default/puppet to enable

                                                                                                                                                                 [ OK ]

Processing triggers for libc-bin (2.19-0ubuntu6.14) ...

Processing triggers for ureadahead (0.100.0-16) ...

3. The puppet agent is disabled by default and we need to enable the same by updating the file /etc/default/puppet (by default the below values is "no")

START=yes 

4. Now let's lock the puppet agent to avoid any update 

root@puppetagent:~# puppet help | tail -n 1

Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations

   (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')

Puppet v3.8.7

update the below entries in /etc/apt/preferences.d/00-puppet.pref

Package: puppet puppet-common

Pin: version 3.8.7

Pin-Priority: 501

Puppet master configuration 

1. Login to the puppet master server and edit the /etc/puppet/puppet.conf file by

removing below parameter templatedir=$confdir/templates
in the main session of the master puppet file set the dns_alt_names and certname as below

cert_name = puppet
dns_alt_names = puppet,puppet.unixchips.com

***********************************************************
[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
certname = puppet
dns_alt_names = puppet, puppet.unixchips.com
[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY

*********************************************************************

Make sure we have configured /etc/hosts and /etc/hostname files properly also please use the certname as puppet only as it is pre configured in apache

2. Puppet master itself act as a certificate authority and generates its own certificates which is used to sign certificate requests. Now we have to setup master certificate

remove any existing SSL certificate installed

root@puppetmaster:~# rm -rf /var/lib/puppet/ssl/

Now let's create a new CA certificate using below command

***********************************************************************
root@puppetmaster:~# sudo puppet master --verbose --no-daemonize
Info: Creating a new SSL key for ca
Info: Creating a new SSL certificate request for ca
Info: Certificate Request fingerprint (SHA256): 89:89:5B:92:BC:56:56:14:4A:0C:43:58:14:6C:BC:4F:E4:10:1C:B1:FF:D2:B2:B4:B6:5B:63:4B:5A:89:89:33
Notice: Signed certificate request for ca
Info: Creating a new certificate revocation list
Info: Creating a new SSL key for puppet
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for puppet
Info: Certificate Request fingerprint (SHA256): 3A:08:C2:A3:D1:AC:12:89:16:35:E6:0A:84:76:0C:A2:36:D7:4B:74:7C:5C:D7:3E:77:67:1E:E0:09:92:4A:CC
Notice: puppet has a waiting certificate request
Notice: Signed certificate request for puppet
Notice: Removing file Puppet::SSL::CertificateRequest puppet at '/var/lib/puppet/ssl/ca/requests/puppet.pem'
Notice: Removing file Puppet::SSL::CertificateRequest puppet at '/var/lib/puppet/ssl/certificate_requests/puppet.pem'
Notice: Starting Puppet master version 3.8.7
Error: Could not run: Address already in use - bind(2)

**************************************************************************

we will get an error as "Address already in use" because master puppet runs when apache is running .we can check like below 

root@puppetmaster:~# sudo netstat -ltnp

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name

tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      957/sshd

tcp6       0      0 :::22                   :::*                    LISTEN      957/sshd

tcp6       0      0 :::443                  :::*                    LISTEN      1156/apache2

tcp6       0      0 :::8140                 :::*                    LISTEN      1156/apache2

tcp6       0      0 :::80                   :::*                    LISTEN      1156/apache2

Let's stop the apache first

root@puppetmaster:~# sudo service apache2 stop
 * Stopping web server apache2                                                   *
 * The apache2 configtest failed, so we are trying to kill it manually. This is almost certainly suboptimal, so please make sure your system is working as you'd expect now!

Now we can check the details of the certificate which we have created as below 

root@puppetmaster:~# sudo puppet cert list -all

+ "puppet" (SHA256) 4D:3D:B4:D1:B2:C6:FA:BB:3D:EC:42:F7:70:90:C0:FF:02:71:D0:DC:47:37:7D:0C:10:89:A3:ED:18:A2:6D:DF (alt names: "DNS:puppet", "DNS:puppet.unixchips.com")

Now we have to create the main manifest file or site manifest inside the below location 

root@puppetmaster:/# sudo touch /etc/puppet/manifests/site.pp

Start the puppet master as below ( we can ignore the below message )

root@puppetmaster:/var/lib/puppet/ssl/private_keys# sudo service apache2 start

 * Starting web server apache2                                                                                                                                          AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using puppetmaster.unixchips.com. Set the 'ServerName' directive globally to suppress this message

 *

Configure puppet agent 

1. First edit the /etc/puppet/puppet.conf in agent node as below 

[main]

logdir=/var/log/puppet

vardir=/var/lib/puppet

ssldir=/var/lib/puppet/ssl

rundir=/var/run/puppet

factpath=$vardir/lib/facter

#[master]

# These are needed when the puppetmaster is run by passenger

# and can safely be removed if webrick is used.

#ssl_client_header = SSL_CLIENT_S_DN

#ssl_client_verify_header = SSL_CLIENT_VERIFY

[agent]

server = puppetmaster.unixchips.com

server = puppet.unixchips.com

2. Now configure the hostname and /etc/hosts file properly 

root@puppetagent:~# cat /etc/hostname

puppetagent.unixchips.com

root@puppetagent:~# cat /etc/hosts

127.0.0.1 puppetagent puppetagent.unixchips.com

172.31.6.59   puppetmaster puppetmaster.unixchips.com

3. Now start the puppet agent 

root@puppetagent:~# sudo service puppet start

 * Starting puppet agent                                                 [ OK ]

root@puppetagent:~#

4. We need to check the sign request on master by running following command on puppet master 

root@puppetmaster:~# sudo puppet cert list

  "puppetagent.ap-south-1.compute.internal" (SHA256) 54:E9:7C:68:1B:E2:CD:85:52:74:7E:30:FA:AF:14:B6:4A:61:65:43:61:92:FA:15:E8:8C:D0:A7:4B:71:5F:20

root@puppetmaster:~#

We can see cert request is reached from the agent node 

5. We can sign the request using the puppet cert sign command with the hostname of the certificate we want to sign 

root@puppetmaster:~# sudo puppet cert sign puppetagent.ap-south-1.compute.internal

Notice: Signed certificate request for puppetagent.ap-south-1.compute.internal

Notice: Removing file Puppet::SSL::CertificateRequest puppetagent.ap-south-1.compute.internal at '/var/lib/puppet/ssl/ca/requests/puppetagent.ap-south-1.compute.internal.pem'

This inidcats the communication between puppet master and the agent 

Below link will provide some sample manifests which we are using for general purpose

http://unixchips.blogspot.com/2017/08/generally-using-puppet-modules.html

    Thank you for reading 

                                                                                                                                 

  

Python in AWS management using boto3

Python is a powerfull scripting language which is using for many automation purpose in system management . In this blog i am introducing a python module called boto (boto3) for AWS management purpose like creating instances, S3 bucket's , transferring files to S3 bucket etc..

 Boto is the Amazon Web Services (AWS) SDK for Python, which allows Python developers to write software that makes use of Amazon services like S3 and EC2. Boto provides an easy to use, object-oriented API as well as low-level direct service access.


1. Installing the boto3 in your local system

First install the python-pip which is used to install boto3

*************************************************
unixchips@unixchips:~$ sudo apt install python-pip
[sudo] password for unixchips:
Reading package lists... Done
Building dependency tree     
Reading state information... Done
The following packages were automatically installed and are no longer required:
  linux-headers-4.10.0-28 linux-headers-4.10.0-28-generic
  linux-headers-4.13.0-32 linux-headers-4.13.0-32-generic
  linux-headers-4.13.0-36 linux-headers-4.13.0-36-generic
  linux-headers-4.13.0-37 linux-headers-4.13.0-37-generic
  linux-headers-4.13.0-38 linux-headers-4.13.0-38-generic
  linux-headers-4.13.0-39 linux-headers-4.13.0-39-generic
  linux-headers-4.13.0-41 linux-headers-4.13.0-41-generic
  linux-headers-4.13.0-43 linux-headers-4.13.0-43-generic
  linux-headers-4.15.0-29 linux-headers-4.15.0-29-generic
  linux-headers-4.15.0-30 linux-headers-4.15.0-30-generic
  linux-image-4.10.0-28-generic linux-image-4.13.0-32-generic
  linux-image-4.13.0-36-generic linux-image-4.13.0-37-generic
  linux-image-4.13.0-38-generic linux-image-4.13.0-39-generic
  linux-image-4.13.0-41-generic linux-image-4.13.0-43-generic
  linux-image-4.15.0-24-generic linux-image-4.15.0-29-generic
  linux-image-4.15.0-30-generic linux-image-extra-4.10.0-28-generic
  linux-image-extra-4.13.0-32-generic linux-image-extra-4.13.0-36-generic
  linux-image-extra-4.13.0-37-generic linux-image-extra-4.13.0-38-generic
  linux-image-extra-4.13.0-39-generic linux-image-extra-4.13.0-41-generic
  linux-image-extra-4.13.0-43-generic linux-image-extra-4.13.0-45-generic
  linux-modules-4.15.0-24-generic linux-modules-4.15.0-29-generic
  linux-modules-4.15.0-30-generic
...........................................................

Now we have to install awscli ( command line tool for aws management) and boto3 using below command unixchips@unixchips:~$ pip install awscli boto3 -U --ignore-installed six
Collecting awscli
  Downloading https://files.pythonhosted.org/packages/e6/8e/129966e5ae7e14a301fe58e81b7ce6dd762745518b6e3f987fb1d1df55a1/awscli-1.16.1-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 654kB/s
Collecting boto3
  Downloading https://files.pythonhosted.org/packages/d6/bc/5393e095b03e871055efae27bac7d314c7f62ab05fea098441052c2afdbb/boto3-1.8.1-py2.py3-none-any.whl (128kB)
    100% |████████████████████████████████| 133kB 1.8MB/s
Collecting six
  Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting docutils>=0.10 (from awscli)
  Downloading https://files.pythonhosted.org/packages/50/09/c53398e0005b11f7ffb27b7aa720c617aba53be4fb4f4f3f06b9b5c60f28/docutils-0.14-py2-none-any.whl (543kB)
    100% |████████████████████████████████| 552kB 1.2MB/s
Collecting PyYAML<=3.13,>=3.10 (from awscli)
  Downloading https://files.pythonhosted.org/packages/9e/a3/1d13970c3f36777c583f136c136f804d70f500168edc1edea6daa7200769/PyYAML-3.13.tar.gz (270kB)
    100% |████████████████████████████████| 276kB 1.9MB/s
Collecting s3transfer<0.2.0,>=0.1.12 (from awscli)
  Downloading https://files.pythonhosted.org/packages/d7/14/2a0004d487464d120c9fb85313a75cd3d71a7506955be458eebfe19a6b1d/s3transfer-0.1.13-py2.py3-none-any.whl (59kB)
    100% |████████████████████████████████| 61kB 4.0MB/s
Collecting rsa<=3.5.0,>=3.1.2 (from awscli)
  Downloading https://files.pythonhosted.org/packages/e1/ae/baedc9cb175552e95f3395c43055a6a5e125ae4d48a1d7a924baca83e92e/rsa-3.4.2-py2.py3-none-any.whl (46kB)
    100% |████████████████████████████████| 51kB 3.4MB/s
Collecting colorama<=0.3.9,>=0.2.5 (from awscli)


*************************************************************

2. Next step is to create a user with programmatic access in AWS . This user should give full permission with respect to EC2, S3, RDS level . Go to services-IAM-users and click add user option

3. Now copy the access-key and secret-access-key of the newly created user from IAM-user-security credentials and paste it in ~.aws/credentials

unixchips@unixchips:~/.aws$ cat credentials
[default]
aws_access_key_id = ************
aws_secret_access_key = **************************

(set the permission for the credential file so that other's doesn't have access on it)

Script to find EC2 status 

For sample testing i have created an EC2 instance and forcefully shutdown that, we will see how can we get the instance status using boto script

sample Script 
************************************************
unixchips@unixchips:~$ cat list_instances.py
#!/usr/bin/env python
import boto3
import os
import argparse
import sys
session = boto3.Session(profile_name='default')
default_ec2_client = session.client('ec2')


ec2 = boto3.resource('ec2')
for instance in ec2.instances.all():
    print instance.id, instance.state

*************************************************

output:

unixchips@unixchips:~$ ./list_instances.py
i-0fb8b62e337e1ab3f {u'Code': 80, u'Name': 'stopped'}

( here the AWS credentials are saved in side the file in the tag of default)

Create an instance using boto3

For creating the instance we need to give the AMI details which is using . So we have to pass the AMI details in the script 

AMI id will be get as below from AWS console 

sample script

***************************************************

unixchips@unixchips:~$ cat create_instances.py 

#!/usr/bin/env python

import boto3

import os

import argparse

import sys

session = boto3.Session(profile_name='default')

default_ec2_client = session.client('ec2')

ec2 = boto3.resource('ec2')

instance = ec2.create_instances(

    ImageId='ami-6cd6f714',

    MinCount=1,

    MaxCount=1,

    InstanceType='t2.micro')

print instance[0].id

**************************************************

Output:

unixchips@unixchips:~$ ./create_instances.py 

i-039933493d9fbfa61

Also if we check the console we can see that one instance is created automatically 

Script to terminate an instance 

Now let us terminate an instance using the script 

Sample script

******************************************************

unixchips@unixchips:~$ cat term_instance.py 

#!/usr/bin/env python

import boto3

import os

import argparse

import sys

session = boto3.Session(profile_name='default')

default_ec2_client = session.client('ec2')

ec2 = boto3.resource('ec2')

for instance_id in sys.argv[1:]:

    instance = ec2.Instance(instance_id)

    response = instance.terminate()

    print response

*******************************************************

In this case we need to pass the instance id as argument which we will get from list_instance.py script 

output:

unixchips@unixchips:~$ ./list_instances.py 

i-039933493d9fbfa61 {u'Code': 16, u'Name': 'running'}

i-0fb8b62e337e1ab3f {u'Code': 80, u'Name': 'stopped'}

unixchips@unixchips:~$ ./term_instance.py i-039933493d9fbfa61

{u'TerminatingInstances': [{u'InstanceId': 'i-039933493d9fbfa61', u'CurrentState': {u'Code': 32, u'Name': 'shutting-down'}, u'PreviousState': {u'Code': 16, u'Name': 'running'}}], 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': '5fc8cc94-bc21-42a2-a673-82afaede516a', 'HTTPHeaders': {'transfer-encoding': 'chunked', 'content-type': 'text/xml;charset=UTF-8', 'vary': 'Accept-Encoding', 'date': 'Tue, 28 Aug 2018 20:20:34 GMT', 'server': 'AmazonEC2'}}}

Script to create the bucket

Now let's create a bucket using the boto script 

Sample script

****************************************************

unixchips@unixchips:~$ cat create_s3.py
#!/usr/bin/env python

import boto3

import os

import argparse

import sys

session = boto3.Session(profile_name='default')

default_s3_client = session.client('s3')
s3 = boto3.resource('s3')
for bucket_name in sys.argv[1:]:
    try:
        response = s3.create_bucket(Bucket=bucket_name, CreateBucketConfiguration={
    'LocationConstraint': 'us-west-2'})
        print response
    except Exception as error:
        print error

*************************************************************

output:

unixchips@unixchips:~$ ./create_s3.py pythonbucket1-$(date +%F-%s)

s3.Bucket(name='pythonbucket1-2018-08-29-1535490688')

From the console we can see that pythonbucket1 is created 

Script to list all buckets and its contents 

Sample script 

********************************************************

#!/usr/bin/env python

import boto3

import os

import argparse

import sys

session = boto3.Session(profile_name='default')

default_s3_client = session.client('s3')

s3 = boto3.resource('s3')

for bucket in s3.buckets.all():

    print bucket.name

    print "---"

    for item in bucket.objects.all():

        print "\t%s" % item.key

***************************************************

Output:

unixchips@unixchips:~$ ./list_s3.py 

cf-templates-1heczgkj9avl7-ap-south-1

---

20181655Fc-autoscalv1.1gywvpp3fiw

20181657ke-designer/template1sgfsgkbkga8

2018165IdC-autoscalv1.2

20181661uh-ec2cloudformation.txt

201816657J-ec2cloudformation.txt

2018166Fiz-ec2cloudformation.txt

2018166UrA-ec2cloudformation

2018166V5J-ec2cloudformation

2018166VaD-ec2cloudformation.txt

2018166jfs-ec2cloudformation.txt

2018166kWD-ec2cloudformation.txt

2018166vPH-ec2cloudformation.txt

pythonbucket1-2018-08-29-1535490688

---

unixchips1

---

cloudformer.template

test1.txt

test3.txt

test4.txt

unixchips2

---

test2.txt

test3.txt

test4.txt

*****************************************************

Script to put a file to S3 bucket 

sample script 

************************************************************

#!/usr/bin/env python

import boto3

import os

import argparse

import sys

session = boto3.Session(profile_name='default')

default_s3_client = session.client('s3')

s3 = boto3.resource('s3')

bucket_name = sys.argv[1]

object_name = sys.argv[2]

try:

    response = s3.Object(bucket_name, object_name).put(Body=open(object_name, 'rb'))

    print response

except Exception as error:

    print error

***********************************************

now let's create some text files 

unixchips@unixchips:~$ touch file{1,2,3,4}.txt

-rw-rw-r-- 1 unixchips unixchips     0 Aug 29 12:30 file4.txt

-rw-rw-r-- 1 unixchips unixchips     0 Aug 29 12:30 file3.txt

-rw-rw-r-- 1 unixchips unixchips     0 Aug 29 12:30 file2.txt

-rw-rw-r-- 1 unixchips unixchips     0 Aug 29 12:30 file1.txt

output:

(we need to pass S3 bucket name and file details as parameters)

unixchips@unixchips:~$ ./filecopy_s3.py pythonbucket1-2018-08-29-1535490688 file1.txt 

{u'ETag': '"d41d8cd98f00b204e9800998ecf8427e"', 'ResponseMetadata': {'HTTPStatusCode': 200, 'RetryAttempts': 0, 'HostId': 'H2f4CkfA/FOEhgrdsZtlKx62qtBZ5GpSBDkFnPmLiDxFKd/OWjsNhiu8jB8WuL/dRlwQUDvyZtI=', 'RequestId': '0495618196C7D050', 'HTTPHeaders': {'content-length': '0', 'x-amz-id-2': 'H2f4CkfA/FOEhgrdsZtlKx62qtBZ5GpSBDkFnPmLiDxFKd/OWjsNhiu8jB8WuL/dRlwQUDvyZtI=', 'server': 'AmazonS3', 'x-amz-request-id': '0495618196C7D050', 'etag': '"d41d8cd98f00b204e9800998ecf8427e"', 'date': 'Wed, 29 Aug 2018 07:01:06 GMT'}}}

Delete bucket contents 

Sample script 

*******************************************************

#!/usr/bin/env python

import boto3

import os

import argparse

import sys

session = boto3.Session(profile_name='default')

default_s3_client = session.client('s3')

s3 = boto3.resource('s3')

for bucket_name in sys.argv[1:]:

    bucket = s3.Bucket(bucket_name)

    for key in bucket.objects.all():

        try:

            response = key.delete()

            print response

        except Exception as error:

            print error

******************************************************

output:-

unixchips@unixchips:~$ ./delete_content_s3.py pythonbucket1-2018-08-29-1535490688

(we need to pass bucket name along with the script)

{'ResponseMetadata': {'HTTPStatusCode': 204, 'RetryAttempts': 0, 'HostId': '1oJ8Bs08KlxrEjbqEXVjgy7/U5EvySvxXf3wooMIZmOKGA9XzWc20K128fwrpsBkrB/qwIlsWvs=', 'RequestId': '2E0B2A932539FDE4', 'HTTPHeaders': {'x-amz-id-2': '1oJ8Bs08KlxrEjbqEXVjgy7/U5EvySvxXf3wooMIZmOKGA9XzWc20K128fwrpsBkrB/qwIlsWvs=', 'date': 'Wed, 29 Aug 2018 07:22:09 GMT', 'x-amz-request-id': '2E0B2A932539FDE4', 'server': 'AmazonS3'}}}

{'ResponseMetadata': {'HTTPStatusCode': 204, 'RetryAttempts': 0, 'HostId': '9Oc5R8xzAmrCJtd+EBeEQLosTRsJOH9LnEmZW5PzK3cDAkn3BCU1byvVmivpC3ssqS2Car/z/QM=', 'RequestId': '6B30F1635C22C516', 'HTTPHeaders': {'x-amz-id-2': '9Oc5R8xzAmrCJtd+EBeEQLosTRsJOH9LnEmZW5PzK3cDAkn3BCU1byvVmivpC3ssqS2Car/z/QM=', 'date': 'Wed, 29 Aug 2018 07:22:09 GMT', 'x-amz-request-id': '6B30F1635C22C516', 'server': 'AmazonS3'}}}

{'ResponseMetadata': {'HTTPStatusCode': 204, 'RetryAttempts': 0, 'HostId': 'D4nIam0anNHmLOPG5BBCQUFNCawIM5gttQALtP4IX5iLY0gPl78jy/zo+UcJ6ahOTTKLAJ3he1o=', 'RequestId': 'C97C79B4F0DCAB34', 'HTTPHeaders': {'x-amz-id-2': 'D4nIam0anNHmLOPG5BBCQUFNCawIM5gttQALtP4IX5iLY0gPl78jy/zo+UcJ6ahOTTKLAJ3he1o=', 'date': 'Wed, 29 Aug 2018 07:22:10 GMT', 'x-amz-request-id': 'C97C79B4F0DCAB34', 'server': 'AmazonS3'}}}

{'ResponseMetadata': {'HTTPStatusCode': 204, 'RetryAttempts': 0, 'HostId': 'WIFcb/N7KWR/rJUNrLnn4b3L51At9yge2WQ2XWkw9m98HI54UolBxz3rGme+uoal2mWxXd3Exio=', 'RequestId': 'BCB23562E691A69F', 'HTTPHeaders': {'x-amz-id-2': 'WIFcb/N7KWR/rJUNrLnn4b3L51At9yge2WQ2XWkw9m98HI54UolBxz3rGme+uoal2mWxXd3Exio=', 'date': 'Wed, 29 Aug 2018 07:22:10 GMT', 'x-amz-request-id': 'BCB23562E691A69F', 'server': 'AmazonS3'}}

 

Delete bucket...

Sample script

#!/usr/bin/env python

import boto3

import os

import argparse

import sys

session = boto3.Session(profile_name='default')

default_s3_client = session.client('s3')

s3 = boto3.resource('s3')

for bucket_name in sys.argv[1:]:

    bucket = s3.Bucket(bucket_name)

try:

    response = bucket.delete()

    print response

except Exception as error:

    print error

**********************************************************

output:-

(we need to pass bucket name as the parameter) 

unixchips@unixchips:~$ ./delete_s3.py pythonbucket1-2018-08-29-1535490688

{'ResponseMetadata': {'HTTPStatusCode': 204, 'RetryAttempts': 0, 'HostId': 'oDeaICQ/pP4tNk4Z4SW4LrObKYZIYnQUyPX3UWcU936qskOvPLO1SLesC43orgMZijwjQakpKuY=', 'RequestId': '0ECE4D2E59AE3C43', 'HTTPHeaders': {'x-amz-id-2': 'oDeaICQ/pP4tNk4Z4SW4LrObKYZIYnQUyPX3UWcU936qskOvPLO1SLesC43orgMZijwjQakpKuY=', 'date': 'Wed, 29 Aug 2018 09:28:33 GMT', 'x-amz-request-id': '0ECE4D2E59AE3C43', 'server': 'AmazonS3'}}}

Database creation script 

Sample script

*****************************************************************

#!/usr/bin/env python

import boto3

import os

import argparse

import sys

session = boto3.Session(profile_name='default')

default_s3_client = session.client('rds')

rds = boto3.client('rds')

try:

    response = rds.create_db_instance(

        DBInstanceIdentifier='dbserver',

        MasterUsername='dbadmin',

        MasterUserPassword='******',

        DBInstanceClass='db.t2.micro',

        Engine='mariadb',

        AllocatedStorage=5)

    print response

except Exception as error:

    print error

******************************************************************

output:-

unixchips@unixchips:~$ ./create_db.py 

{u'DBInstance': {u'PubliclyAccessible': True, u'MasterUsername': 'dbadmin', u'MonitoringInterval': 0, u'LicenseModel': 'general-public-license', u'VpcSecurityGroups': [{u'Status': 'active', u'VpcSecurityGroupId': 'sg-3fb4834e'}], u'CopyTagsToSnapshot': False, u'OptionGroupMemberships': [{u'Status': 'in-sync', u'OptionGroupName': 'default:mariadb-10-1'}], u'PendingModifiedValues': {u'MasterUserPassword': '****'}, u'Engine': 'mariadb', u'MultiAZ': False, u'DBSecurityGroups': [], u'DBParameterGroups': [{u'DBParameterGroupName': 'default.mariadb10.1', u'ParameterApplyStatus': 'in-sync'}], u'PerformanceInsightsEnabled': False, u'AutoMinorVersionUpgrade': True, u'PreferredBackupWindow': '13:23-13:53', u'DBSubnetGroup': {u'Subnets': [{u'SubnetStatus': 'Active', u'SubnetIdentifier': 'subnet-8fb4f5d5', u'SubnetAvailabilityZone': {u'Name': 'us-west-2c'}}, {u'SubnetStatus': 'Active', u'SubnetIdentifier': 'subnet-d9711ba0', u'SubnetAvailabilityZone': {u'Name': 'us-west-2b'}}, {u'SubnetStatus': 'Active', u'SubnetIdentifier': 'subnet-69188c22', u'SubnetAvailabilityZone': {u'Name': 'us-west-2a'}}], u'DBSubnetGroupName': 'default', u'VpcId': 'vpc-429b8b3b', u'DBSubnetGroupDescription': 'default', u'SubnetGroupStatus': 'Complete'}, u'ReadReplicaDBInstanceIdentifiers': [], u'AllocatedStorage': 5, u'DBInstanceArn': 'arn:aws:rds:us-west-2:891070219291:db:dbserver', u'BackupRetentionPeriod': 1, u'PreferredMaintenanceWindow': 'sat:07:02-sat:07:32', u'DBInstanceStatus': 'creating', u'IAMDatabaseAuthenticationEnabled': False, u'EngineVersion': '10.1.34', u'DomainMemberships': [], u'StorageType': 'standard', u'DbiResourceId': 'db-SF625J6QYBSD6HHUXYUET65YEE', u'CACertificateIdentifier': 'rds-ca-2015', u'StorageEncrypted': False, u'DBInstanceClass': 'db.t2.micro', u'DbInstancePort': 0, u'DBInstanceIdentifier': 'dbserver'}, 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': '949eccc0-d3cb-4f62-a0b9-2b465b9c1055', 'HTTPHeaders': {'x-amzn-requestid': '949eccc0-d3cb-4f62-a0b9-2b465b9c1055', 'content-type': 'text/xml', 'content-length': '3647', 'vary': 'Accept-Encoding', 'date': 'Wed, 29 Aug 2018 21:08:10 GMT'}}}

Delete DB instance 

Sample script

*****************************************************************
unixchips@unixchips:~$ cat delete_db.py
#!/usr/bin/env python

import boto3

import os

import argparse

import sys

session = boto3.Session(profile_name='default')

default_s3_client = session.client('rds')
rds = boto3.client('rds')
try:
    response = rds.delete_db_instance(
        DBInstanceIdentifier='dbserver',
        SkipFinalSnapshot=True)
    print response
except Exception as error:
    print error

*********************************************************

output:-

unixchips@unixchips:~$ ./delete_db.py 

{u'DBInstance': {u'PubliclyAccessible': True, u'MasterUsername': 'dbadmin', u'MonitoringInterval': 0, u'LicenseModel': 'general-public-license', u'VpcSecurityGroups': [{u'Status': 'active', u'VpcSecurityGroupId': 'sg-3fb4834e'}], u'InstanceCreateTime': datetime.datetime(2018, 8, 29, 21, 8, 10, 389000, tzinfo=tzutc()), u'CopyTagsToSnapshot': False, u'OptionGroupMemberships': [{u'Status': 'in-sync', u'OptionGroupName': 'default:mariadb-10-1'}], u'PendingModifiedValues': {u'MasterUserPassword': '****'}, u'Engine': 'mariadb', u'MultiAZ': False, u'DBSecurityGroups': [], u'DBParameterGroups': [{u'DBParameterGroupName': 'default.mariadb10.1', u'ParameterApplyStatus': 'in-sync'}], u'PerformanceInsightsEnabled': False, u'AutoMinorVersionUpgrade': True, u'PreferredBackupWindow': '13:23-13:53', u'DBSubnetGroup': {u'Subnets': [{u'SubnetStatus': 'Active', u'SubnetIdentifier': 'subnet-8fb4f5d5', u'SubnetAvailabilityZone': {u'Name': 'us-west-2c'}}, {u'SubnetStatus': 'Active', u'SubnetIdentifier': 'subnet-d9711ba0', u'SubnetAvailabilityZone': {u'Name': 'us-west-2b'}}, {u'SubnetStatus': 'Active', u'SubnetIdentifier': 'subnet-69188c22', u'SubnetAvailabilityZone': {u'Name': 'us-west-2a'}}], u'DBSubnetGroupName': 'default', u'VpcId': 'vpc-429b8b3b', u'DBSubnetGroupDescription': 'default', u'SubnetGroupStatus': 'Complete'}, u'ReadReplicaDBInstanceIdentifiers': [], u'AllocatedStorage': 5, u'DBInstanceArn': 'arn:aws:rds:us-west-2:891070219291:db:dbserver', u'BackupRetentionPeriod': 1, u'PreferredMaintenanceWindow': 'sat:07:02-sat:07:32', u'Endpoint': {u'HostedZoneId': 'Z1PVIF0B656C1W', u'Port': 3306, u'Address': 'dbserver.c5g5m1fcix6x.us-west-2.rds.amazonaws.com'}, u'DBInstanceStatus': 'deleting', u'IAMDatabaseAuthenticationEnabled': False, u'EngineVersion': '10.1.34', u'AvailabilityZone': 'us-west-2b', u'DomainMemberships': [], u'StorageType': 'standard', u'DbiResourceId': 'db-SF625J6QYBSD6HHUXYUET65YEE', u'CACertificateIdentifier': 'rds-ca-2010', u'StorageEncrypted': False, u'DBInstanceClass': 'db.t2.micro', u'DbInstancePort': 0, u'DBInstanceIdentifier': 'dbserver'}, 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': '3dfd0ff6-5f1b-4b79-aea7-4eb868a5bbe1', 'HTTPHeaders': {'x-amzn-requestid': '3dfd0ff6-5f1b-4b79-aea7-4eb868a5bbe1', 'content-type': 'text/xml', 'content-length': '3963', 'vary': 'Accept-Encoding', 'date': 'Wed, 29 Aug 2018 21:12:35 GMT'}}}

We have many more options to automate the AWS management using boto and i will explain in detail in another post.

Thank you for reading ....

Python OS functions explained

The OS module in python is very useful for testing the functionality of underlying operating system like linux,Mac,windows etc

before applying these modules we need to import the os module using the command  import os

The main os functions are given below

os.system ()- this is for executing shell command 

ex: os.system ("rpm --ivh aide*")  - this will install the aide rpm in the system 

os.stat () - This command will give the status of a file 

ex: print "getting the status of: ", os.stat('/usr/bin/python')

getting the status of:  posix.stat_result(st_mode=33261, st_ino=1051053, st_dev=2054, st_nlink=1, st_uid=0, st_gid=0, st_size=3542008, st_atime=1532933669, st_mtime=1511456897, st_ctime=1519176316)

os.environ() - Get the users environment

ex: 

import os

ux = os.environ['HOME']
print ux

out:/home/unixchips

os.chdir() # Move focus to a different directory

ex:

import os

print os.chdir('/home/unixchips/Desktop')

print (os.getcwd())

output: /home/unixchips/Desktop

os.getgid() # Return the real group id of the current process

ex:

import os

print os.getgid()

output:1000

os.getuid() # Return the current process’s user id

ex:

import os

print os.getuid()

output: 1000

os.getpid() # Returns the real process ID of the current process

ex:

import os

print os.getpid()

output:18616

getpass.getuser()# Return the name of the user logged

ex:

import getpass

print (getpass.getuser())

output:unixchips

os.access() # Check file status 

ex:

import os

path = '/tmp/test.txt'

print os.access(path, os.R_OK)

output:True

os.chmod() # Change the mode of path to the numeric mode

in this case you need to import a module called stat along with OS which will act a parameter , below are the parameter's used to change the ownership of a file 

stat.S_ISUID − Set user ID on execution.


stat.S_ISGID − Set group ID on execution.


stat.S_ENFMT − Record locking enforced.


stat.S_ISVTX − Save text image after execution.


stat.S_IREAD − Read by owner.


stat.S_IWRITE − Write by owner.


stat.S_IEXEC − Execute by owner.


stat.S_IRWXU − Read, write, and execute by owner.


stat.S_IRUSR − Read by owner.


stat.S_IWUSR − Write by owner.


stat.S_IXUSR − Execute by owner.


stat.S_IRWXG − Read, write, and execute by group.


stat.S_IRGRP − Read by group.


stat.S_IWGRP − Write by group.


stat.S_IXGRP − Execute by group.


stat.S_IRWXO − Read, write, and execute by others.


stat.S_IROTH − Read by others.


stat.S_IWOTH − Write by others.


stat.S_IXOTH − Execute by others.

ex:

import os, stat

os.chmod('/tmp/test.txt',stat.S_IRWXU)

(this will provide executable permission to the /tmp/test.txt by the owner )

output: -rwx------ 1 unixchips unixchips    0 Jul 31 23:39 test.txt

os.chown() # Change the owner and group id

(you need to perform this as a root user only)

ex:

import os, sys

os.chown("/tmp/test.txt", 1001, 1001)

output: -rwx------ 1 ratheesh  ratheesh     0 Jul 31 23:39 test.txt

os.umask(mask) # Set the current numeric umask

import os











os.umask(0777)



open("/tmp/sample.txt", "w").close()

output:                ---------- 1 root      root         0 Aug  1 21:04 sample.txt

os.getsize() # Get the size of a file

import os


print os.path.getsize("/tmp/test.txt")

output: 20 (as it is a 20B file)

os.environ() # Get the users environment

suppose if we want to get the home directory of current user root below program can be used 

ex:

import os

print os.environ.get('HOME')

output:root

to get all the environment variables of a user below program 

import os

for param in os.environ.keys():

    print "%20s %s" % (param,os.environ[param])

output
*******************************************************************************

/root

root@unixchips:~# nano env.py

root@unixchips:~# nano env1.py

root@unixchips:~# python env1.py 

                LANG en_IN

                TERM xterm-256color

               SHELL /bin/bash

           LESSCLOSE /usr/bin/lesspipe %s %s

          XAUTHORITY /home/unixchips/.Xauthority

            LANGUAGE en_IN:en

               SHLVL 1

QT_QPA_PLATFORMTHEME appmenu-qt5

            LESSOPEN | /usr/bin/lesspipe %s

                 PWD /root

             LOGNAME root

                USER root

                PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin

                MAIL /var/mail/root

           LS_COLORS rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:

                HOME /root

             DISPLAY :0

                   _ /usr/bin/python

********************************************************************************

os.uname() # Return information about the current operating system

ex:

import os

print os.uname()

output:   ('Linux', 'unixchips', '4.15.0-24-generic', '#26~16.04.1-Ubuntu SMP Fri Jun 15 14:35:08 UTC 2018', 'x86_64')

os.chroot(path) # Change the root directory of the current process to path & os.getcwd

ex:

import os, sys

os.chdir("/tmp")

print os.getcwd()

output: /tmp

os.chroot(path) # Change the root directory of the current process to path

ex:

import os

print os.listdir('/tmp')

output: (will display the contents of the /tmp directory)

['.XIM-unix', 'systemd-private-5ccc0291bc60473d8d8fc248cff92fd3-systemd-timesyncd.service-OnYY3G', 'gnome-software-D9QEMZ', 'gnome-software-E1KUMZ', 'Temp-8984a84e-9508-434e-8180-a9cfbaa969c8', '.font-unix', 'gnome-software-VTVXMZ', 'gnome-software-N6ZDMZ', 'sample.txt', 'test.txt', '.wine-1000', '.Test-unix', 'gnome-software-3RD9MZ', '.org.chromium.Chromium.KfRkcm', 'systemd-private-5ccc0291bc60473d8d8fc248cff92fd3-rtkit-daemon.service-wVPLTC', 'unity_support_test.0', '.X0-lock', 'gnome-software-UMBENZ', 'gnome-software-IY0GNZ', 'gnome-software-4GZUMZ', '.ICE-unix', 'systemd-private-5ccc0291bc60473d8d8fc248cff92fd3-colord.service-BSHI7z', '.X11-unix', 'config-err-SiAh3i', 'gnome-software-3657MZ', '.org.chromium.Chromium.dI8BqE', 'gnome-software-SSMSMZ', 'gnome-software-XX9GMZ']

os.getloadavg() # Show queue averaged over the last 1, 5, and 15 minutes

ex:

import os

print os.getloadavg()

output: (which will display load average of  1,5,15 minutes)

(0.75, 0.69, 0.59) 

os.path.exists()# Check if a path exists

ex:

import os

print os.path.exists('/tmp/test.txt')

output: true

os.walk() # Print out all directories, sub-directories and files

The syntax of the os.walk will be 

os.walk(top[, topdown=True[, onerror=None[, followlinks=False]]])

• top − Each directory rooted at directory, yields 3-tuples, i.e., (dirpath, dirnames, filenames)
• topdown − If optional argument topdown is True or not specified, directories are scanned from top-down. If topdown is set to False, directories are scanned from bottom-up.
• onerror − This can show error to continue with the walk, or raise the exception to abort the walk.
• followlinks − This visits directories pointed to by symlinks, if set to true.

ex:

import os

for root, dirs, files in os.walk("/tmp", topdown=False):

   for name in files:

      print(os.path.join(root, name))

   for name in dirs:

      print(os.path.join(root, name))

output: ( List all directories /subdrectories  and files 

/tmp/systemd-private-5ccc0291bc60473d8d8fc248cff92fd3-systemd-timesyncd.service-OnYY3G/tmp

/tmp/gnome-software-D9QEMZ/debconf.socket

/tmp/gnome-software-E1KUMZ/debconf.socket

/tmp/gnome-software-VTVXMZ/debconf.socket

/tmp/gnome-software-N6ZDMZ/debconf.socket

/tmp/.wine-1000/server-806-3e0902/lock

/tmp/.wine-1000/server-806-3e0902

/tmp/gnome-software-3RD9MZ/debconf.socket

/tmp/.org.chromium.Chromium.KfRkcm/SingletonCookie

/tmp/.org.chromium.Chromium.KfRkcm/SingletonSocket

/tmp/systemd-private-5ccc0291bc60473d8d8fc248cff92fd3-rtkit-daemon.service-wVPLTC/tmp

/tmp/gnome-software-UMBENZ/debconf.socket

/tmp/gnome-software-IY0GNZ/debconf.socket

/tmp/gnome-software-4GZUMZ/debconf.socket

/tmp/.ICE-unix/2087

/tmp/systemd-private-5ccc0291bc60473d8d8fc248cff92fd3-colord.service-BSHI7z/tmp

/tmp/.X11-unix/X0

/tmp/gnome-software-3657MZ/debconf.socket

................................................

os.mkdir(path) # Create a directory named path with numeric mode mode

ex: 

import os 

os.mkdir('/tmp/ratheesh')

output: 

drwxr-xr-x 2 root root 4096 Aug  3 14:09 /tmp/ratheesh/

os.rmdir(path) # Remove (delete) the directory path

ex:

import os

os.rmdir('/tmp/ratheesh')

output:

root@unixchips:~# ls ld /tmp/ratheesh

ls: cannot access 'ld': No such file or directory

ls: cannot access '/tmp/ratheesh': No such file or directory

os.rename(src, dst) # Rename the file or directory src to dst

ex:

import os

os.rename('/tmp/test.txt', '/tmp/test1.txt')

output:

-rwx------ 1 ratheesh ratheesh 20 Aug  1 21:11 /tmp/test1.txt