Unixchips

Thursday, October 11, 2018

openstack command line administration PART-2 (glance/cinder/manila)

Manage Images (Glance)

cloud operator assigns the roles to users and determines who can upload and manage the images .Only cloud operators or administrators can upload and manage images.
We can upload the images through glance client or image service API

To list the available images

$ glance image-list
************************************
+--------------------------------------+---------------------------------
+-------------+------------------+----------+--------+
| ID
| Name
|
Disk Format | Container Format | Size Status |
+--------------------------------------+---------------------------------
+-------------+------------------+----------+--------+
| 397e713c-b95b-4186-ad46-6126863ea0a9 | cirros-0.3.1-x86_64-uecami
| ami 25165824 | active |
| df430cc2-3406-4061-b635-a51c16e488ac | cirros-0.3.1-x86_64-uec-kernel |
aki
| aki
| 4955792 | active |
| 3cf852bd-2332-48f4-9ae4-7d926d50945e | cirros-0.3.1-x86_64-uec-ramdisk |
ari
| ari
| 3714968 | active |
| 7e5142af-1253-4634-bcc6-89482c5f2e8a | myCirrosImage
|
ami
| ami
| 14221312 | active |
+--------------------------------------+---------------------------------

To filter only the cirrors image

$ glance image-list | grep 'cirros'
*********************************
| 397e713c-b95b-4186-ad46-6126863ea0a9 | cirros-0.3.1-x86_64-uec
|
ami
| ami
| 25165824 | active |
| df430cc2-3406-4061-b635-a51c16e488ac | cirros-0.3.1-x86_64-uec-kernel |
aki
| aki
| 4955792 | active |
| 3cf852bd-2332-48f4-9ae4-7d926d50945e | cirros-0.3.1-x86_64-uec-ramdisk |
ari
| ari
| 3714968 | active |

To get the image details by name or id

$ glance image-show myCirrosImage

+---------------------------------------+---------------------------------

-----+

| Property

| Value

+---------------------------------------+---------------------------------

-----+

| Property 'base_image_ref'

| 397e713c-b95b-4186-ad46-

6126863ea0a9 |

| Property 'image_location'

| snapshot

| Property 'image_state'

| available

| Property 'image_type'

| snapshot

| Property 'instance_type_ephemeral_gb' | 0

| Property 'instance_type_flavorid'

| 2

| Property 'instance_type_id'

| 5

| Property 'instance_type_memory_mb'

| 2048

| Property 'instance_type_name'

| m1.small

| Property 'instance_type_root_gb'

| 20

| Property 'instance_type_rxtx_factor' | 1

| Property 'instance_type_swap'

| 0

| Property 'instance_type_vcpu_weight' | None

| Property 'instance_type_vcpus'

| 1

| Property 'instance_uuid'

| 84c6e57d-a6b1-44b6-81eb-

fcb36afd31b5 |

| Property 'kernel_id'

| df430cc2-3406-4061-b635-

a51c16e488ac |

| Property 'owner_id'

| 66265572db174a7aa66eba661f58eb9e

| Property 'ramdisk_id'

| 3cf852bd-2332-48f4-9ae4-

7d926d50945e |

| Property 'user_id'

| 376744b5910b4b4da7d8e6cb483b06a8

| checksum

| 8e4838effa1969ad591655d6485c7ba8

| container_format

| ami

| created_at

| 2013-07-22T19:45:58

| deleted

| False

| disk_format

| ami

| id

| 7e5142af-1253-4634-bcc6-

89482c5f2e8a |

| is_public

| False

min_ram

| 0

| name

| myCirrosImage

| owner

| 66265572db174a7aa66eba661f58eb9e

| protected

| False

| size

| 14221312

| status

| active

| updated_at

| 2013-07-22T19:46:42

+---------------------------------------+---------------------------------

The location metadata of the images can be defined in following method

edit the /etc/glance/glance.conf as below

show_multiple_locations = True

filesystem_store_metadata_file = filePath, where filePath points to a JSON

file that defines the mount point for OpenStack images on your system and a unique ID.

For example:

[{

"id": "b9d69795-5951-4cb0-bb5c-29491e1e2daf",

"mountpoint": "/var/lib/glance/images/"

}]

After you restart the Image Service, you can use the following syntax to view the image's

location information:

$ glance --os-image-api-version=2 image-show 2d9bb53f-70ea-4066-

a68b-67960eaae673

To upload a cent Os 6.3 image in qcow format and configure the public access

$ glance image-create --name centos63-image --disk-format=qcow2 \ --

container-format=bare --is-public=True --file=./centos63.qcow2

To update the image using name or ID

Below are the parameters which we can update on images

--name NAME The name of the image.

--disk-format DISK_FORMAT The disk format of the image. Acceptable formats are

ami, ari, aki, vhd, vmdk, raw, qcow2, vdi, and iso.

--container-format

CONTAINER_FORMAT The container format of the image. Acceptable formats

are ami, ari, aki, bare, and ovf.

--owner TENANT_ID The tenant who should own the image.

--size SIZE The size of image data, in bytes.

--min-disk DISK_GB The minimum size of disk needed to boot image, in

gigabytes.

--min-ram DISK_RAM The minimum amount of ram needed to boot image, in

megabytes.

--location IMAGE_URL The URL where the data for this image resides. For

example, if the image data is stored in swift, you could

specify

swift://account:key@example.com/container/ob

--file FILE Local file that contains disk image to be uploaded during

update. Alternatively, you can pass images to the client

through stdin.

--checksum CHECKSUM Hash of image data to use for verification.

--copy-from IMAGE_URL Similar to --location in usage, but indicates that the

Glance server should immediately copy the data and

store it in its configured image store.

--is-public [True|False] Makes an image accessible to the public.

--is-protected [True|False] Prevents an image from being deleted.

--property KEY=VALUE Arbitrary property to associate with image. Can be used

multiple times.

--purge-props Deletes all image properties that are not explicitly set in

the update request. Otherwise, those properties not

referenced are preserved.

--human-readable Prints image size in a human-friendly format.

$ glance image-update \ --property hw_vif_model=e1000 f16-x86_64-

openstack-sda

Below are the valid vif models

libvirt_type setting | Supported model values

qemu or kvm virtio , ne2k_pci, pcnet, rtl8139, e1000

Create images using nova

To create an image first we need to list out the instances

$ nova list

+--------------------------------------+----------------------+--------+--

----------+-------------+------------------+

| ID

| Name

| Status |

Task State | Power State | Networks

+--------------------------------------+----------------------+--------+--

----------+-------------+------------------+

| 84c6e57d-a6b1-44b6-81eb-fcb36afd31b5 | myCirrosServer

| ACTIVE |

None

| Running

| private=10.0.0.3 |

+--------------------------------------+----------------------+--------+--

To create a snapshot of the instance

$ nova image-create myCirrosServer myCirrosImage

We can check the details of the image created using below command

$ nova image-show myCirrosImage

+-------------------------------------+-----------------------------------

---+

| Property

| Value

+-------------------------------------+-----------------------------------

---+

| metadata owner_id

| 66265572db174a7aa66eba661f58eb9e

| minDisk

| 0

| metadata instance_type_name

| m1.small

| metadata instance_type_id

| 5

| metadata instance_type_memory_mb

| 2048

| id

| 7e5142af-1253-4634-bcc6-

89482c5f2e8a |

| metadata instance_type_root_gb

| 20

| metadata instance_type_rxtx_factor | 1

| metadata ramdisk_id

| 3cf852bd-2332-48f4-9ae4-

7d926d50945e |

| metadata image_state

| available

| metadata image_location

| snapshot

| minRam

| 0

| metadata instance_type_vcpus

| 1

| status

| ACTIVE

| updated

| 2013-07-22T19:46:42Z

| metadata instance_type_swap

| 0

| metadata instance_type_vcpu_weight | None

| metadata base_image_ref

| 397e713c-b95b-4186-ad46-

6126863ea0a9 |

| progress

| 100

| metadata instance_type_flavorid

| 2

| OS-EXT-IMG-SIZE:size

| 14221312

| metadata image_type

| snapshot

| metadata user_id

| 376744b5910b4b4da7d8e6cb483b06a8

| name

| myCirrosImage

| created

| 2013-07-22T19:45:58Z

| metadata instance_uuid

| 84c6e57d-a6b1-44b6-81eb-

fcb36afd31b5 |

| server

| 84c6e57d-a6b1-44b6-81eb-

To launch an instance from the image

$ nova boot newServer --image 7e5142af-1253-4634-bcc6-89482c5f2e8a \ --

flavor 3

+-------------------------------------+-----------------------------------

---+

| Property

| Value

+-------------------------------------+-----------------------------------

---+

| OS-EXT-STS:task_state

| scheduling

| image

| myCirrosImage

| OS-EXT-STS:vm_state

| building

| OS-EXT-SRV-ATTR:instance_name

| instance-00000007

| flavor

| m1.medium

| id

| d7efd3e4-d375-46d1-9d57-

372b6e4bdb7f |

| security_groups

| [{u'name': u'default'}]

| user_id

| 376744b5910b4b4da7d8e6cb483b06a8

| OS-DCF:diskConfig

| MANUAL

| accessIPv4

| accessIPv6

| progress

| 0

| OS-EXT-STS:power_state

| 0

| OS-EXT-AZ:availability_zone

| nova

| config_drive

| status

| BUILD

| updated

| 2013-07-22T19:58:33Z

| hostId

| OS-EXT-SRV-ATTR:host

| None

| key_name

| None

| OS-EXT-SRV-ATTR:hypervisor_hostname | None

| name

| newServer

| adminPass

| jis88nN46RGP

| tenant_id

| 66265572db174a7aa66eba661f58eb9e

| created

| 2013-07-22T19:58:33Z

| metadata

| {}

(important we cannot create a snapshot form an instance which has an attached volume. We need to detach the volume first and create the snapshot)

Below is the procedure for snapshot creation of an instance

Shut down the source VM before you take the snapshot to ensure that all data is flushed to disk. If necessary, list the instances to view the instance name:

$ nova list

+--------------------------------------+------------+--------+------------------------------+

+--------------------------------------+------------+--------+------------------------------+

+--------------------------------------+------------+--------+------------------------------+

Use the nova stop command to shut down the instance:

$ nova stop myInstance

Use the nova list command to confirm that the instance shows a SHUTOFF status:

$ nova list

+--------------------------------------+------------+---------+------------------+

+--------------------------------------+------------+---------+------------------+

+--------------------------------------+------------+---------+------------------+

Use the nova image-create command to take a snapshot:

$ nova image-create --poll myInstance myInstanceSnapshot
Instance snapshotting... 50% complete

Use the nova image-list command to check the status until the status is ACTIVE:

$ nova image-list

+--------------------------------------+---------------------------------+--------+--------+

+--------------------------------------+---------------------------------+--------+--------+

+--------------------------------------+---------------------------------+--------+--------+

Download the snapshot as an image

$ glance image-download --file snapshot.raw f30b204e-1ce6-40e7-b8d9-b353d4d84e7d

Copy the image to the new machine/environment using scp or http method and import the snapshot

$ glance --os-image-api-version 1 image-create \

--container-format bare --disk-format qcow2 --copy-from IMAGE_URL

Boot a new instance from the snapshot

$nova boot --flavor m1.tiny --image myInstanceSnapshot myNewInstance

Manage volumes (cinder)

Migrate a volume

According to the below cases we can migrate a volume

Bring down a physical storage device for maintenance without disrupting workloads.
Modify the properties of a volume.
Free up space in a thinly-provisioned back end.

$ cinder migrate volumeID destinationHost --force-host-copy True|False

To create a volume

List the images and note the id of the imagesfrom which we are creating the volume

$nova image-list

+-----------------------+---------------------------------+--------+--------------------------+
| ID | Name | Status | Server |
+-----------------------+---------------------------------+--------+--------------------------+
| 397e713c-b95b-4186... | cirros-0.3.2-x86_64-uec | ACTIVE | |
| df430cc2-3406-4061... | cirros-0.3.2-x86_64-uec-kernel | ACTIVE | |
| 3cf852bd-2332-48f4... | cirros-0.3.2-x86_64-uec-ramdisk | ACTIVE | |
| 7e5142af-1253-4634... | myCirrosImage | ACTIVE | 84c6e57d-a6b1-44b6-81... |
| 89bcd424-9d15-4723... | mysnapshot | ACTIVE | f51ebd07-c33d-4951-87... |
+-----------------------+---------------------------------+--------+--------------------------+

List the availability zones and note the id of the AZ

$ cinder availability-zone-list

+------+-----------+
| Name | Status |
+------+-----------+
| nova | available |
+------+-----------+

Now create a volume with 8GB size

cinder create 8 --display-name my-new-volume --image-id 397e713c-b95b-4186-ad46-6126863ea0a9 --availability-zone nova

+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2013-07-25T17:02:12.472269 |
| display_description | None |
| display_name | my-new-volume |
| id | 573e024d-5235-49ce-8332-be1576d323f8 |
| image_id | 397e713c-b95b-4186-ad46-6126863ea0a9 |
| metadata | {} |
| size | 8 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| volume_type | None |
+---------------------+--------------------------------------+

Verify that cinder volume is created successfully

(if the status of the volume is error, we need to check the quota details we may exceeded the quota)

To attach a volume to an instance

$ nova volume-attach 84c6e57d-a6b1-44b6-81eb-fcb36afd31b5 573e024d-5235-49ce-8332-be1576d323f8 /dev/vdb

+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdb |
| serverId | 84c6e57d-a6b1-44b6-81eb-fcb36afd31b5 |
| id | 573e024d-5235-49ce-8332-be1576d323f8 |
| volumeId | 573e024d-5235-49ce-8332-be1576d323f8 |
+----------+--------------------------------------+

To get the information about the attached volume

cinder show 573e024d-5235-49ce-8332-be1576d323f8
+------------------------------+------------------------------------------+
| Property | Value |
+------------------------------+------------------------------------------+
| attachments | [{u'device': u'/dev/vdb', |
| | u'server_id': u'84c6e57d-a |
| | u'id': u'573e024d-... |
| | u'volume_id': u'573e024d... |
| availability_zone | nova |
| bootable | true |
| created_at | 2013-07-25T17:02:12.000000 |
| display_description | None |
| display_name | my-new-volume |
| id | 573e024d-5235-49ce-8332-be1576d323f8 |
| metadata | {} |
| os-vol-host-attr:host | devstack |
| os-vol-tenant-attr:tenant_id | 66265572db174a7aa66eba661f58eb9e |
| size | 8 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| volume_image_metadata | {u'kernel_id': u'df430cc2..., |
| | u'image_id': u'397e713c..., |
| | u'ramdisk_id': u'3cf852bd..., |
| |u'image_name': u'cirros-0.3.2-x86_64-uec'}|
| volume_type | None |
+------------------------------+------------------------------------------+

To resize the volume we need to follow the below steps

First detach the volume using below command

To delete a volume

To delete a volume we need to detach the volume from the instance first as explained above then delete the volume using below command

To transfer a volume

Accept a volume transfer

Create and manage file shares using manila client

To create a share network (We need to pass nutron id and subnet id along with this to configure the network properties of the share)

3. To create a share

$ manila create NFS 1 \

--name myshare \

--description "My Manila share" \

--share-network mysharenetwork \

--share-type default

+-----------------------------+--------------------------------------+

| Property | Value |

+-----------------------------+--------------------------------------+

| status | creating |

| share_type_name | default |

| description | My Manila share |

| availability_zone | None |

| share_network_id | c895fe26-92be-4152-9e6c-f2ad230efb13 |

| share_server_id | None |

| host | |

| access_rules_status | active |

| snapshot_id | None |

| is_public | False |

| task_state | None |

| snapshot_support | True |

| id | 8d8b854b-ec32-43f1-acc0-1b2efa7c3400 |

| size | 1 |

| name | myshare |

| share_type | bf6ada49-990a-47c3-88bc-c0cb31d5c9bf |

| has_replicas | False |

| replication_type | None |

| created_at | 2016-03-24T14:15:34.000000 |

| share_proto | NFS |

| consistency_group_id | None |

| source_cgsnapshot_member_id | None |

source_cgsnapshot_member_id | None |
project_id | 907004508ef4447397ce6741a8f037c1 |
| metadata | {} |
+-----------------------------+--------------------------------------+

4. To show the share

$ manila show myshare
+-----------------------------+---------------------------------------------------------------+
| Property | Value |
+-----------------------------+---------------------------------------------------------------+
| status | available |
| share_type_name | default |
| description | My Manila share |
| availability_zone | nova |
| share_network_id | c895fe26-92be-4152-9e6c-f2ad230efb13 |
| export_locations | |
| | path = 10.254.0.3:/share-e1c2d35e-fe67-4028-ad7a-45f668732b1d |
| | preferred = False |
| | is_admin_only = False |
| | id = b6bd76ce-12a2-42a9-a30a-8a43b503867d |
| | share_instance_id = e1c2d35e-fe67-4028-ad7a-45f668732b1d |
| | path = 10.0.0.3:/share-e1c2d35e-fe67-4028-ad7a-45f668732b1d |
| | preferred = False |
| | is_admin_only = True |
| | id = 6921e862-88bc-49a5-a2df-efeed9acd583 |
| | share_instance_id = e1c2d35e-fe67-4028-ad7a-45f668732b1d |
| share_server_id | 2e9d2d02-883f-47b5-bb98-e053b8d1e683 |
| host | nosb-devstack@london#LONDON |
| access_rules_status | active |
| snapshot_id | None |
| is_public | False |
| task_state | None |
| snapshot_support | True |
| id | 8d8b854b-ec32-43f1-acc0-1b2efa7c3400 |
| size | 1 |
| name | myshare |
| share_type | bf6ada49-990a-47c3-88bc-c0cb31d5c9bf |
| has_replicas | False |
| replication_type | None |
| created_at | 2016-03-24T14:15:34.000000 |
| share_proto | NFS |
| consistency_group_id | None |
| source_cgsnapshot_member_id | None |
| project_id | 907004508ef4447397ce6741a8f037c1 |
| metadata | {}

5. To list the share

$ manila list
+--------------------------------------+---------+------+-------------+-----------+-----------+-----------------+-----------------------------+-------------------+
| ID | Name | Size | Share Proto | Status | Is Public | Share Type Name | Host | Availability Zone |
+--------------------------------------+---------+------+-------------+-----------+-----------+-----------------+-----------------------------+-------------------+
| 8d8b854b-ec32-43f1-acc0-1b2efa7c3400 | myshare | 1 | NFS | available | False | default | nosb-devstack@london#LONDON | nova |
+--------------------------------------+---------+------+-------------+-----------+-----------+-----------------+-----------------------------+-------

6. To list the export locations

$ manila share-export-location-list myshare
+--------------------------------------+--------------------------------------------------------+-----------+
| ID | Path | Preferred |
+--------------------------------------+--------------------------------------------------------+-----------+
| 6921e862-88bc-49a5-a2df-efeed9acd583 | 10.0.0.3:/share-e1c2d35e-fe67-4028-ad7a-45f668732b1d | False |
| b6bd76ce-12a2-42a9-a30a-8a43b503867d | 10.254.0.3:/share-e1c2d35e-fe67-4028-ad7a-45f668732b1d | False |
+--------------------------------------+--------------------------------------------------------+-----------+

7. Allow read write access to the share

$ manila access-allow myshare ip 10.0.0.0/24
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| share_id | 8d8b854b-ec32-43f1-acc0-1b2efa7c3400 |
| access_type | ip |
| access_to | 10.0.0.0/24 |
| access_level | rw |
| state | new |
| id | 0c8470ca-0d77-490c-9e71-29e1f453bf97 |
+--------------+--------------------------------------+

We can check the access details by using below command

$ manila access-list myshare
+--------------------------------------+-------------+-------------+--------------+--------+
| id | access_type | access_to | access_level | state |
+--------------------------------------+-------------+-------------+--------------+--------+
| 0c8470ca-0d77-490c-9e71-29e1f453bf97 | ip | 10.0.0.0/24 | rw | active |
+--------------------------------------+-------------+-------------+--------------+--------+

8. To give read only access to the share

$ manila access-allow myshare ip 20.0.0.0/24 --access-level ro
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| share_id | 8d8b854b-ec32-43f1-acc0-1b2efa7c3400 |
| access_type | ip |
| access_to | 20.0.0.0/24 |
| access_level | ro |
| state | new |
| id | f151ad17-654d-40ce-ba5d-98a5df67aadc |
+--------------+--------------------------------------+

We can check the access list using below command

$ manila access-list myshare
+--------------------------------------+-------------+-------------+--------------+--------+
| id | access_type | access_to | access_level | state |
+--------------------------------------+-------------+-------------+--------------+--------+
| 0c8470ca-0d77-490c-9e71-29e1f453bf97 | ip | 10.0.0.0/24 | rw | active |
| f151ad17-654d-40ce-ba5d-98a5df67aadc | ip | 20.0.0.0/24 | ro | active |
+--------------------------------------+-------------+-------------+--------------+--------+

9. To create a snapshot of a share

$ manila snapshot-create --name mysnapshot --description "My Manila snapshot" myshare
+-------------------+--------------------------------------+
| Property | Value |
+-------------------+--------------------------------------+
| status | creating |
| share_id | 8d8b854b-ec32-43f1-acc0-1b2efa7c3400 |
| description | My Manila snapshot |
| created_at | 2016-03-24T14:39:58.232844 |
| share_proto | NFS |
| provider_location | None |
| id | e744ca47-0931-4e81-9d9f-2ead7d7c1640 |
| size | 1 |
| share_size | 1 |
| name | mysnapshot |
+-------------------+--------------------------------------+

We can check the snapshot details from list snapshot command

$ manila snapshot-list
+--------------------------------------+--------------------------------------+-----------+------------+------------+
| ID | Share ID | Status | Name | Share Size |
+--------------------------------------+--------------------------------------+-----------+------------+------------+
| e744ca47-0931-4e81-9d9f-2ead7d7c1640 | 8d8b854b-ec32-43f1-acc0-1b2efa7c3400 | available | mysnapshot | 1 |
+--------------------------------------+--------------------------------------+-----------+------------+------------+

10. To create a share from the snapshot

$ manila create NFS 1 \
--snapshot-id e744ca47-0931-4e81-9d9f-2ead7d7c1640 \
--share-network mysharenetwork \
--name mysharefromsnap
+-----------------------------+--------------------------------------+
| Property | Value |
+-----------------------------+--------------------------------------+
| status | creating |
| share_type_name | default |
| description | None |
| availability_zone | nova |
| share_network_id | c895fe26-92be-4152-9e6c-f2ad230efb13 |
| share_server_id | None |
| host | nosb-devstack@london#LONDON |
| access_rules_status | active |
| snapshot_id | e744ca47-0931-4e81-9d9f-2ead7d7c1640 |
| is_public | False |
| task_state | None |
| snapshot_support | True |
| id | e73ebcd3-4764-44f0-9b42-fab5cf34a58b |
| size | 1 |
| name | mysharefromsnap |
| share_type | bf6ada49-990a-47c3-88bc-c0cb31d5c9bf |
| has_replicas | False |
| replication_type | None |
| created_at | 2016-03-24T14:41:36.000000 |
| share_proto | NFS |
| consistency_group_id | None |
| source_cgsnapshot_member_id | None |
| project_id | 907004508ef4447397ce6741a8f037c1 |
| metadata | {} |
+-----------------------------+--------------------------------------+

List shares

$ manila list
+--------------------------------------+-----------------+------+-------------+-----------+-----------+-----------------+-----------------------------+-------------------+
| ID | Name | Size | Share Proto | Status | Is Public | Share Type Name | Host | Availability Zone |
+--------------------------------------+-----------------+------+-------------+-----------+-----------+-----------------+-----------------------------+-------------------+
| 8d8b854b-ec32-43f1-acc0-1b2efa7c3400 | myshare | 1 | NFS | available | False | default | nosb-devstack@london#LONDON | nova |
| e73ebcd3-4764-44f0-9b42-fab5cf34a58b | mysharefromsnap | 1 | NFS | available | False | default | nosb-devstack@london#LONDON | nova |
+--------------------------------------+-----------------+------+-------------+-----------+-----------+-----------------+-----------------------------

More commands will be provided on next session

Thank you for reading

Tuesday, October 2, 2018

openstack command line administration - PART1

Here we are going through some openstack administration commands which is used in day to day operations , even though we have horizon graphical user interface many of us prefer to use commands for a better visibility .. let's start

Each openstack services has its own command line client in some commands we can put debug parameter to identify the API requests for the command

Following are the command line clients are available for the respective API's

ceilometer (python-ceilometerclient). Client for the Telemetry API that lets you create and collect measurements across OpenStack.

cinder (python-cinderclient). Client for the Block Storage Service API that lets you create and manage volumes.

glance (python-glanceclient). Client for the Image Service API that lets you create and manage images.

heat (python-heatclient). Client for the Orchestration API that lets you launch stacks from templates, view details of running stacks including events and resources, and update and delete stacks.

keystone (python-keystoneclient). Client for the Identity Service API that lets you create and manage users, tenants, roles, endpoints, and credentials.

neutron (python-neutronclient). Client for the Networking API that lets you configure networks for guest servers. This client was previously known as quantum.

nova (python-novaclient). Client for the Compute API and its extensions. Use to create and manage images, instances, and flavors.

swift (python-swiftclient). Client for the Object Storage API that lets you gather statistics, list items,update metadata, upload, download and delete files stored by the Object Storage service. Provides access to a swift installation for ad hoc processing.

To get the version of the above mentioned clients

$ nova --version

2.14.1.17

$ keystone --version

0.3.1.73

To get the help for client commands

$ swift help

Usage: swift [--version] [--help] [--snet] [--verbose]

[--debug] [--quiet] [--auth <auth_url>]

[--auth-version <auth_version>] [--user <username>]

[--key <api_key>] [--retries <num_retries>]

[--os-username <auth-user-name>] [--os-password <auth-password>]

[--os-tenant-id <auth-tenant-id>]

[--os-tenant-name <auth-tenant-name>]

[--os-auth-url <auth-url>] [--os-auth-token <auth-token>]

[--os-storage-url <storage-url>] [--os-region-name <region-name>]

[--os-service-type <service-type>]

[--os-endpoint-type <endpoint-type>]

[--os-cacert <ca-certificate>] [--insecure]

[--no-ssl-compression]

<subcommand> ...

Command-line interface to the OpenStack Swift API.

Positional arguments:

delete Delete a container or objects within a container

download Download objects from containers

list Lists the containers for the account or the objects

for a container

post Updates meta information for the account, container,

or object

stat Displays information for the account, container,

or object

upload Uploads files or directories to the given container

Examples:

swift -A https://auth.api.rackspacecloud.com/v1.0 -U user -K api_key stat -v

swift --os-auth-url https://api.example.com/v2.0 --os-tenant-name tenant \

--os-username user --os-password password list

swift --os-auth-token 6ee5eb33efad4e45ab46806eac010566 \

--os-storage-url https://10.1.5.2:8080/v1/AUTH_ced809b6a4baea7aeab61a \

list

swift list --lh

Users/Tenants/Roles - keystone commands

To get the service details

$keystone service-list

To get the details of a service

$ keystone service-get 08741d8ed88242ca88d1f61484a0fe3b

To delete a service

$ keystone service-delete 08741d8ed88242ca88d1f61484a0fe3b

To list the tanents ( projects) with their id,name and status

$ keystone tenant-list

+----------------------------------+--------------------+---------+

| id | name | enabled |

+----------------------------------+--------------------+---------+

| f7ac731cc11f40efbc03a9f9e1d1d21f | admin | True |

| c150ab41f0d9443f8874e32e725a4cc8 | alt_demo | True |

| a9debfe41a6d4d09a677da737b907d5e | demo | True |

| 9208739195a34c628c58c95d157917d7 | invisible_to_admin | True |

| 3943a53dc92a49b2827fae94363851e1 | service | True |

| 80cab5e1f02045abad92a2864cfd76cb | test_project | True |

+----------------------------------+--------------------+---------+

To create a new project

$ keystone tenant-create --name new-project --description 'my new project'

+-------------+----------------------------------+

| Property | Value |

+-------------+----------------------------------+

| description | my new project |

| enabled | True |

| id | 1a4a0618b306462c9830f876b0bd6af2 |

| name | new-project |

+-------------+----------------------------------+

To temporarily disable a project

$ keystone tenant-update PROJECT_ID --enabled false

To enable a disabled project

$ keystone tenant-update PROJECT_ID --enabled true

To update the name of the project

$ keystone tenant-update PROJECT_ID --name project-new

To delete a project

$ keystone tenant-delete PROJECT_ID

To verify the changes above

$ keystone tenant-get 1a4a0618b306462c9830f876b0bd6af2

+-------------+----------------------------------+

| Property | Value |

+-------------+----------------------------------+

| description | my new project |

| enabled | True |

| id | 1a4a0618b306462c9830f876b0bd6af2 |

| name | project-new |

+-------------+----------------------------------+

To list all users

$keystone user-list

+----------------------------------+----------+---------+-----------------

-----+

| id | name | enabled | email

+----------------------------------+----------+---------+-----------------

-----+

| 352b37f5c89144d4ad0534139266d51f | admin | True |

admin@example.com |

| 86c0de739bcb4802b8dc786921355813 | demo | True |

demo@example.com |

| 32ec34aae8ea432e8af560a1cec0e881 | glance | True |

glance@example.com |

| 7047fcb7908e420cb36e13bbd72c972c | nova | True |

nova@example.com |

+----------------------------------+----------+---------+-----------------

-----+

To create a user

$ keystone user-create --name USER_NAME --tenant_id TENANT_ID --pass

PASSWORD

$ keystone user-create --name demo --tenant_id

1a4a0618b306462c9830f876b0bd6af2 --pass myPASS

+----------+----------------------------------+

| Property | Value |

+----------+----------------------------------+

| email | |

| enabled | True |

| id | 6e5140962b424cb9814fb172889d3be2 |

| name | new-user |

| tenantId | 1a4a0618b306462c9830f876b0bd6af2 |

+----------+----------------------------------+

To temporarily disable a user ID

$ keystone user-update USER_ID --enabled false

To enable a disabled user account

$ keystone user-update USER_ID --enabled true

Users can be part of multiple projects with the help of assigned roles.

To list the available roles

$ keystone role-list

To create a new role

$ keystone role-create --name new-role

+----------+----------------------------------+
| Property |Value
|
+----------+----------------------------------+
|
id bef1f95537914b1295da6aa038ef4de6 |
|
name new-role
|
+----------+----------------------------------+

To assign a a role to a user tenant pair

$ keystone user-role-add --user demo --role new-role --tenant test-
project
(here we will add role new-role to demo user and tenant pair)

$ keystone user-role-list --user demo --tenant test-project

+----------------------------------+----------+---------------------------
-------+----------------------------------+
|
id name user_id tenant_id
|
+----------------------------------+----------+---------------------------
-------+----------------------------------+
| bef1f95537914b1295da6aa038ef4de6 | new-role |86c0de739bcb4802b8dc786921355813 | 80cab5e1f02045abad92a2864cfd76cb |
+----------------------------------+----------+---------------------------
-------+----------------------------------+

Manage Project Security&Networking ( Nutron commands)

Security groups are the IP rules which will control the in and out traffic to projects. Whether the security rules needs to be applied to all projects which will share the network or to individual projects depends up on the allow_same_net_traffic option in the /etc/nova/nova.conf.

True (default), hosts on the same subnet are not filtered and are allowed to pass all types of trafficbetween them. On a flat network, this allows all instances from all projects unfiltered communication.With VLAN networking, this allows access between instances within the same project. You can alsosimulate this setting by configuring the default security group to allow all traffic from the subnet.
False, security groups are enforced for all connections

Before proceeding the security group commands please make sure system variables are set for the user and the tenant which we will check the security rules

export OS_USERNAME=demo00
export OS_TENANT_NAME=tenant01

To list security groups

To view the details of a security group

To create a security group

neutron security-group-create [securitygroupname]

$ neutron secgroup-create global_http "Allows Web traffic anywhere on the
Internet."
+--------------------------------------+-------------+--------------------
--------------------------+
| Id Name Description
|
+--------------------------------------+-------------+--------------------
--------------------------+
| 1578a08c-5139-4f3e-9012-86bd9dd9f23b | global_http | Allows Web traffic
anywhere on the Internet. |
+--------------------------------------+-------------+--------------------

To create a security group rule

neutron security-group-rule-create [Security-groupID]

$ nutron security-group-rule-create global_http tcp 80 80 0.0.0.0/0
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range | Source Group |
+-------------+-----------+---------+-----------+--------------+
| tcp 80 80 0.0.0.0/0 |
|
+-------------+-----------+---------+-----------+--------------+

$ nutron security-group-add-rule global_http tcp 443 443 0.0.0.0/0

$ nutron security-group-list-rules global_http
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range | Source Group |
+-------------+-----------+---------+-----------+--------------+
| tcp 80 80 0.0.0.0/0 |
|
| tcp 443 443 0.0.0.0/0 |
|
+-------------+-----------+---------+-----------+--------------+

To delete a security group

neutron security-group-delete [securitygroupID/name]

neutron secrity-group-delete global_http

To delete a security group rule

neutron security-group-rule-delete [secrity group name-rule name]

$neutron security-group-rule-delete global_http tcp 443 443 0.0.0.0/0

----------------------------------------------------------------------

To update a security group

neutron security-group-update [securitygroupname]

$neutron security-group-update global_http tcp 443 80 0.0.0.0/0
$ nutron security-group-list-rules global_http

----------------------------------------------------------------------

To list the routers

$neutron router-list

+----------------------- +--------------------------------------+

| Id | name |

+-----------------------+--------------------------------------+

1e4g48d3-a9d0-3567-3f1c-29cd8b83345d

tuxfixer-router

---------------------------------------------------------------------

To create a router

$neutron router-create tuxfixer-router

Created a new router:

+-----------------------+--------------------------------------+

| Field | Value |

+-----------------------+--------------------------------------+

| admin_state_up | True |

| external_gateway_info | |

| id | 1e4g48d3-a9d0-3567-3f1c-29cd8b83345d |

| name | tuxfixer-router |

| routes | |

| status | ACTIVE |

| tenant_id | dbe3cf30f46b446fcfe84b205459780d |

To add an interface to a router

$neutron router-interface-add tuxfixer-router tuxfixer-subnet

Added interface 445d79cb-3dcf-5f88-963c-aa054f7ce758 to router tuxfixer-router

To set the gateway for the router

$neutron router-gateway-set tuxfixer-router ext-net

Set gateway for router tuxfixer-router

To view the details of newly created router

$neutron router-port-list tuxfixer-router

+--------------------------------------+------+-------------------+------------------------------------------------------------------------------------+

+--------------------------------------+------+-------------------+------------------------------------------------------------------------------------+

| |1e4g48d3-a9d0-3567-3f1c-29cd8b83345d | fc:16:4d:13:32:21 | {"subnet_id": "f6523637-7162-449d-b12c-e1f0eda6196d", "ip_address": "192.168.5.1"} |

To create the network

$ neutron net-create tuxfixer-net

Created a new network:

+-----------------+--------------------------------------+

| Field | Value |

+-----------------+--------------------------------------+

| admin_state_up | True |

| id | 2c0dh763-3fd4-2f8c-743f-7h0j35cv6cde |

| name | tuxfixer-net |

| router:external | False |

| shared | False |

| status | ACTIVE |

| subnets | |

| tenant_id | dbe3cf30f46b446fcfe84b205459780d |

+-----------------+--------------------------------------

To create a subnet for the network

neutron subnet-create --name <SUBNET-NAME> <NET-NAME> <GATEWAY IP><SUBNET-CIDR>

$neutron subnet-create tuxfixer-net --name tuxfixer-subnet --gateway 192.168.5.1 192.168.5.0/24

Created a new subnet:

+-------------------+--------------------------------------------------+

| Field | Value |

+-------------------+--------------------------------------------------+

| allocation_pools | {"start": "192.168.5.2", "end": "192.168.5.254"} |

| cidr | 192.168.5.0/24 |

| dns_nameservers | |

| enable_dhcp | True |

| gateway_ip | 192.168.5.1 |

| host_routes | |

| id | ac05bc74-eade-4811-8e7b-8de021abe0c1 |

| ip_version | 4 |

| ipv6_address_mode | |

| ipv6_ra_mode | |

| name | tuxfixer-subnet |

| network_id | 2c0dh763-3fd4-2f8c-743f-7h0j35cv6cde |

| tenant_id | dbe3cf30f46b446fcfe84b205459780d |

Network quotas

Quota related details like network,subnet,port will be mentioned in openstack network configuration file nutron.conf. If we need to remove any particular item from the quota we have to remove from quota_items in nutron.conf

**********************************************************
[quotas]
# resource name(s) that are supported in quota features
quota_items = network,subnet,port
# number of networks allowed per tenant, and minus means unlimited
quota_network = 10
# number of subnets allowed per tenant, and minus means unlimited
quota_subnet = 10
# number of ports allowed per tenant, and minus means unlimited
quota_port = 50
# default driver to use for quota checks
quota_driver = neutron.quota.ConfDriver

***********************************************************
For L3 quotas like router quotas we need to define as below in nutron.conf

************************************************************
[quotas]
# number of routers allowed per tenant, and minus means unlimited
quota_router = 10
# number of floating IPs allowed per tenant, and minus means unlimited
quota_floatingip = 50

************************************************************

For security group quotas
*************************************************
[quotas]
# number of security groups per tenant, and minus means unlimited
quota_security_group = 10
# number of security rules allowed per tenant, and minus means unlimited
quota_security_group_rule = 100
***********************************************

To show the quota details per tenant

To get the quota detail of a particular tenant id

To update the quota of a particular tenant

update the multiple values of the quota

$ neutron quota-update --tenant_id 6f88036c45344d9999a1f971e4882723 --network 3

--subnet 3 --port 3 -- --floatingip 3 --router 3

+------------+-------+

| Field Value |

+------------+-------+

| floatingip | 3

| network 3

| port 3

| router 3

| subnet 3

+------------+-------+

Floating IP details

To list floatingIP for this tenant

To list all floatingIP pools

$ nova floating-ip-pool-list

+--------+

| name |

+--------+

| public |

| test |

+--------+

To allocating a new floatingIP

$ nova floating-ip-create pblic

+--------------+-------------+----------+--------+

+--------------+-------------+----------+--------+

+--------------+-------------+----------+--------+

To disallocate a floating ip

$nova floating-ip-delete 172.24.4.255

To attach a floating ip to an instance

$nova floating-ip-associate VM1 172.24.4.225

$nova list

+------------------+------+--------+------------+-------------+-------------------------------+

+------------------+------+--------+------------+-------------+-------------------------------+

| 42290b01-0968... | VM2 | SHUTOFF| - | Shutdown | private=10.0.0.4 |

+------------------+------+--------+------------+-------------+-------------------------------+

To detach a floating ip from an instance

$ nova floating-ip-delete FLOATING_IP_ADDRESS

$nova floating-ip-delete 172.24.4.255

$nova list

+------------------+------+--------+------------+-------------+-------------------------------+

+------------------+------+--------+------------+-------------+-------------------------------+

| 42290b01-0968... | VM2 | SHUTOFF| - | Shutdown | private=10.0.0.4 |

+------------------+------+--------+------------+-------------+-------------------------------+

More commands will be explained in another session

Thank you for reading

Wednesday, September 19, 2018

Puppet Master and agent configuraton

Puppet is a configuration management tool which works on master- agent configuration and communication . It is centralised tool for configuration and automaton purpose in multi OS environments.

Puppet has two layers: a configuration language to describe how the hosts and services should look, and an abstraction layer that allows the administrator to implement the configuration on a variety of platforms, including Unix, linux, Windows Administrators can encode the configuration of a service as a policy, which Puppet then monitors and enforces.

Puppet s written in ruby language

How does the puppet works

Puppet agent is the daemon which runs in the client servers ( servers where we need the configuration changes) and the main server which is act as a master server will have puppet server installed. Puppet agent will communicate with the puppet server for the configuration changes through the SSL channel and fetch the changes ( default time interval for the puppet agent for communication with the server for fetching the changes is 30 minutes )

As per the above picture there are 3 nodes with puppet agent configured and the puppet master is configured at the centralised server.Now there are multiple steps involved whenever a puppet agent of any node connects to a puppet master server for fetching data. These steps are mentioned below.

Step 1: Whenever a client node connects to the master, the master server analyzes the configuration to be applied to the node, and how to apply that configs on the node.

Step 2:Puppet master server Takes and collects all the resources and configurations to be applied to the node, and compiles it and make it a catalogue. This catalogue is given to the puppet agent of the node.

Step 3: Puppet agent will apply the configuration on the node, according to the catalogue, and then reply back, and submit the report of the configuration applied to the puppet master server.

As i pointed earlier puppet will work on cross platform environments and when ever an agent communicates with the master server the factor tool will help the master server to identify the configuration and OS details of the client node

installing the puppet master server

1. Let's take 2 servers in ubuntu 14.04 as one is puppet master and other is puppet agent as below, set the hostnames and add the respective hostnames in both the servers /etc/hosts file

root@puppetagent:~# uname -a && hostname

Linux puppetagent.unixchips.com 4.4.0-1066-aws #76-Ubuntu SMP Thu Aug 16 16:21:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

puppetagent.unixchips.com

root@puppetmaster:~# uname -a && hostname

Linux puppetmaster.unixchips.com 4.4.0-1066-aws #76-Ubuntu SMP Thu Aug 16 16:21:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

puppetmaster.unixchips.com

2. Download the puppet labs release in both the nodes

root@puppetmaster:~# wget https://apt.puppetlabs.com/puppetlabs-release-trusty.deb

--2018-09-17 20:29:29-- https://apt.puppetlabs.com/puppetlabs-release-trusty.deb

Resolving apt.puppetlabs.com (apt.puppetlabs.com)... 52.222.190.242, 52.222.190.106, 52.222.190.188, ...

Connecting to apt.puppetlabs.com (apt.puppetlabs.com)|52.222.190.242|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 16944 (17K) [application/x-debian-package]

Saving to: ‘puppetlabs-release-trusty.deb’

puppetlabs-release-trusty.deb 100%[=====================================================================================>] 16.55K --.-KB/s in 0s

2018-09-17 20:29:29 (373 MB/s) - ‘puppetlabs-release-trusty.deb’ saved [16944/16944]

root@puppetmaster:~# sudo dpkg -i puppetlabs-release-trusty.deb

Selecting previously unselected package puppetlabs-release.

(Reading database ... 219442 files and directories currently installed.)

Preparing to unpack puppetlabs-release-trusty.deb ...

Unpacking puppetlabs-release (1.1-1) ...

Setting up puppetlabs-release (1.1-1) ...

3. In the puppet master server run the following

root@puppetmaster:~# sudo apt-get install puppetmaster-passenger

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following extra packages will be installed:

apache2 apache2-bin apache2-data augeas-lenses debconf-utils facter hiera

libapache2-mod-passenger libapr1 libaprutil1 libaprutil1-dbd-sqlite3

libaprutil1-ldap libaugeas-ruby libaugeas0 libev4 libruby1.9.1 puppet-common

puppetmaster-common ruby ruby-augeas ruby-json ruby-passenger ruby-rack

ruby-shadow ruby1.9.1 ssl-cert virt-what

Suggested packages:

apache2-doc apache2-suexec-pristine apache2-suexec-custom apache2-utils

augeas-doc augeas-tools ruby-selinux libselinux-ruby1.8 librrd-ruby1.9.1

librrd-ruby1.8 puppet-el vim-puppet stompserver ruby-stomp libstomp-ruby1.8

..................................................................................

* Restarting web server apache2 AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using puppetmaster.unixchips.com. Set the 'ServerName' directive globally to suppress this message

[ OK ]

Processing triggers for libc-bin (2.19-0ubuntu6.14) ...

4. We need to maintain consistent puppet version with in the environment to avoid any issues for the working infrastructure ., for that first we need to stop the apache which is controlling the puppet process

root@puppetmaster:~# ps -ef |grep -i apache2

root 3724 1 0 21:01 ? 00:00:00 /usr/sbin/apache2 -k start

www-data 3745 3724 0 21:01 ? 00:00:00 /usr/sbin/apache2 -k start

www-data 3746 3724 0 21:01 ? 00:00:00 /usr/sbin/apache2 -k start

root 4047 3936 0 21:40 pts/1 00:00:00 grep --color=auto -i apache2

root@puppetmaster:~# service apache2 stop

* Stopping web server apache2

The current running version of the puppet is

root@puppetmaster:~# puppet help | tail -n 1

Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations

(at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')

Puppet v3.8.7

Now we need to lock the puppet version as per the current vesrion , add the below lines in /etc/apt/preferences.d/00-puppet.pref

Package: puppet puppet-common puppetmaster-passenger

Pin: version 3.8.7

Pin-Priority: 501

Puppet installation in agent node

1. First download the package and extract it

root@puppetagent:~# wget https://apt.puppetlabs.com/puppetlabs-release-trusty.deb

--2018-09-17 21:47:59-- https://apt.puppetlabs.com/puppetlabs-release-trusty.deb

Resolving apt.puppetlabs.com (apt.puppetlabs.com)... 52.222.190.188, 52.222.190.225, 52.222.190.242, ...

Connecting to apt.puppetlabs.com (apt.puppetlabs.com)|52.222.190.188|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 16944 (17K) [application/x-debian-package]

Saving to: ‘puppetlabs-release-trusty.deb’

100%[==============================================================================================================================>] 16,944 --.-K/s in 0s

2018-09-17 21:47:59 (237 MB/s) - ‘puppetlabs-release-trusty.deb’ saved [16944/16944]

root@puppetagent:~# sudo dpkg -i puppetlabs-release-trusty.deb

Selecting previously unselected package puppetlabs-release.

(Reading database ... 51306 files and directories currently installed.)

Preparing to unpack puppetlabs-release-trusty.deb ...

Unpacking puppetlabs-release (1.1-1) ...

Setting up puppetlabs-release (1.1-1) ...

root@puppetagent:~# sudo apt-get update

Ign http://ap-south-1.ec2.archive.ubuntu.com trusty InRelease

Ign http://apt.puppetlabs.com trusty InRelease

Get:1 http://apt.puppetlabs.com trusty Release.gpg [819 B]

Get:2 http://apt.puppetlabs.com trusty Release [88.9 kB]

Get:3 http://apt.puppetlabs.com trusty/main Sources [49.9 kB]

Get:4 http://apt.puppetlabs.com trusty/dependencies Sources [2,157 B]

Get:5 http://apt.puppetlabs.com trusty/main amd64 Packages [69.0 kB]

Get:6 http://apt.puppetlabs.com trusty/dependencies amd64 Packages [1,180 B]

Get:7 http://ap-south-1.ec2.archive.ubuntu.com trusty-updates InRelease [65.9 kB]

Get:8 http://security.ubuntu.com trusty-security InRelease [65.9 kB]

Hit http://ap-south-1.ec2.archive.ubuntu.com trusty-backports InRelease

Hit http://ap-south-1.ec2.archive.ubuntu.com trusty Release.gpg

Get:9 http://ap-south-1.ec2.archive.ubuntu.com trusty-updates/main Sources [422 kB]

Get:10 http://security.ubuntu.com trusty-security/main Sources [162 kB]

......................................................................................................

Ign http://ap-south-1.ec2.archive.ubuntu.com trusty/universe Translation-en_US

Fetched 12.9 MB in 9s (1,398 kB/s)

Reading package lists... Done

2. Install the puppetagent as below

root@puppetagent:~# sudo apt-get install puppet

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following extra packages will be installed:

augeas-lenses debconf-utils facter hiera libaugeas-ruby libaugeas0

libruby1.9.1 puppet-common ruby ruby-augeas ruby-json ruby-shadow ruby1.9.1

virt-what

Suggested packages:

augeas-doc augeas-tools puppet-el vim-puppet ruby-selinux libselinux-ruby1.8

librrd-ruby1.9.1 librrd-ruby1.8 ri ruby-dev ruby1.9.1-examples ri1.9.1

graphviz ruby1.9.1-dev ruby-switch

Recommended packages:

rdoc

..................................................................................................

* Starting puppet agent

puppet not configured to start, please edit /etc/default/puppet to enable

[ OK ]

Processing triggers for libc-bin (2.19-0ubuntu6.14) ...

Processing triggers for ureadahead (0.100.0-16) ...

3. The puppet agent is disabled by default and we need to enable the same by updating the file /etc/default/puppet (by default the below values is "no")

START=yes

4. Now let's lock the puppet agent to avoid any update

root@puppetagent:~# puppet help | tail -n 1

Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations

(at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')

Puppet v3.8.7

update the below entries in /etc/apt/preferences.d/00-puppet.pref

Package: puppet puppet-common

Pin: version 3.8.7

Pin-Priority: 501

Puppet master configuration

1. Login to the puppet master server and edit the /etc/puppet/puppet.conf file by

removing below parameter templatedir=$confdir/templates
in the main session of the master puppet file set the dns_alt_names and certname as below

cert_name = puppet
dns_alt_names = puppet,puppet.unixchips.com

***********************************************************
[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
certname = puppet
dns_alt_names = puppet, puppet.unixchips.com
[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY

*********************************************************************

Make sure we have configured /etc/hosts and /etc/hostname files properly also please use the certname as puppet only as it is pre configured in apache

2. Puppet master itself act as a certificate authority and generates its own certificates which is used to sign certificate requests. Now we have to setup master certificate

remove any existing SSL certificate installed

root@puppetmaster:~# rm -rf /var/lib/puppet/ssl/

Now let's create a new CA certificate using below command

***********************************************************************
root@puppetmaster:~# sudo puppet master --verbose --no-daemonize
Info: Creating a new SSL key for ca
Info: Creating a new SSL certificate request for ca
Info: Certificate Request fingerprint (SHA256): 89:89:5B:92:BC:56:56:14:4A:0C:43:58:14:6C:BC:4F:E4:10:1C:B1:FF:D2:B2:B4:B6:5B:63:4B:5A:89:89:33
Notice: Signed certificate request for ca
Info: Creating a new certificate revocation list
Info: Creating a new SSL key for puppet
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for puppet
Info: Certificate Request fingerprint (SHA256): 3A:08:C2:A3:D1:AC:12:89:16:35:E6:0A:84:76:0C:A2:36:D7:4B:74:7C:5C:D7:3E:77:67:1E:E0:09:92:4A:CC
Notice: puppet has a waiting certificate request
Notice: Signed certificate request for puppet
Notice: Removing file Puppet::SSL::CertificateRequest puppet at '/var/lib/puppet/ssl/ca/requests/puppet.pem'
Notice: Removing file Puppet::SSL::CertificateRequest puppet at '/var/lib/puppet/ssl/certificate_requests/puppet.pem'
Notice: Starting Puppet master version 3.8.7
Error: Could not run: Address already in use - bind(2)

**************************************************************************

we will get an error as "Address already in use" because master puppet runs when apache is running .we can check like below

root@puppetmaster:~# sudo netstat -ltnp

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 957/sshd

tcp6 0 0 :::22 :::* LISTEN 957/sshd

tcp6 0 0 :::443 :::* LISTEN 1156/apache2

tcp6 0 0 :::8140 :::* LISTEN 1156/apache2

tcp6 0 0 :::80 :::* LISTEN 1156/apache2

Let's stop the apache first

root@puppetmaster:~# sudo service apache2 stop
* Stopping web server apache2 *
* The apache2 configtest failed, so we are trying to kill it manually. This is almost certainly suboptimal, so please make sure your system is working as you'd expect now!

Now we can check the details of the certificate which we have created as below

root@puppetmaster:~# sudo puppet cert list -all

+ "puppet" (SHA256) 4D:3D:B4:D1:B2:C6:FA:BB:3D:EC:42:F7:70:90:C0:FF:02:71:D0:DC:47:37:7D:0C:10:89:A3:ED:18:A2:6D:DF (alt names: "DNS:puppet", "DNS:puppet.unixchips.com")

Now we have to create the main manifest file or site manifest inside the below location

root@puppetmaster:/# sudo touch /etc/puppet/manifests/site.pp

Start the puppet master as below ( we can ignore the below message )

root@puppetmaster:/var/lib/puppet/ssl/private_keys# sudo service apache2 start

* Starting web server apache2 AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using puppetmaster.unixchips.com. Set the 'ServerName' directive globally to suppress this message

Configure puppet agent

1. First edit the /etc/puppet/puppet.conf in agent node as below

[main]

logdir=/var/log/puppet

vardir=/var/lib/puppet

ssldir=/var/lib/puppet/ssl

rundir=/var/run/puppet

factpath=$vardir/lib/facter

#[master]

# These are needed when the puppetmaster is run by passenger

# and can safely be removed if webrick is used.

#ssl_client_header = SSL_CLIENT_S_DN

#ssl_client_verify_header = SSL_CLIENT_VERIFY

[agent]

server = puppetmaster.unixchips.com

server = puppet.unixchips.com

2. Now configure the hostname and /etc/hosts file properly

root@puppetagent:~# cat /etc/hostname

puppetagent.unixchips.com

root@puppetagent:~# cat /etc/hosts

127.0.0.1 puppetagent puppetagent.unixchips.com

172.31.6.59 puppetmaster puppetmaster.unixchips.com

3. Now start the puppet agent

root@puppetagent:~# sudo service puppet start

* Starting puppet agent [ OK ]

root@puppetagent:~#

4. We need to check the sign request on master by running following command on puppet master

root@puppetmaster:~# sudo puppet cert list

"puppetagent.ap-south-1.compute.internal" (SHA256) 54:E9:7C:68:1B:E2:CD:85:52:74:7E:30:FA:AF:14:B6:4A:61:65:43:61:92:FA:15:E8:8C:D0:A7:4B:71:5F:20

root@puppetmaster:~#

We can see cert request is reached from the agent node

5. We can sign the request using the puppet cert sign command with the hostname of the certificate we want to sign

root@puppetmaster:~# sudo puppet cert sign puppetagent.ap-south-1.compute.internal

Notice: Signed certificate request for puppetagent.ap-south-1.compute.internal

Notice: Removing file Puppet::SSL::CertificateRequest puppetagent.ap-south-1.compute.internal at '/var/lib/puppet/ssl/ca/requests/puppetagent.ap-south-1.compute.internal.pem'

This inidcats the communication between puppet master and the agent

Below link will provide some sample manifests which we are using for general purpose

http://unixchips.blogspot.com/2017/08/generally-using-puppet-modules.html

Thank you for reading

Thursday, September 13, 2018

AWS-cloudwatch configuration

Amazon cloud watch is a robust monitoring mechanism for the entire AWS infrastructure which includes EC2 instances, S3, ELB etc. We can able to track wide verity of metrics like CPU, disk,network traffic, available storage space,memory ( using custom metrics) in cloud watch

AWS also provides access to system and application logs custom alarms that provides real time notification when specific metric appear in logs or when certain events take place .

cloud watch mechanism

while creating the instances , by default we have below metrics for monitoring in the EC2 instances

1. CPU utilization
2. Disk reads/Disk read operations
3. Disk writes/Disk write operations
4. Network in/Network out
5. Network packets in/Network packets out
6. Status check failed count
7. Instance Status check failed
8.System status check failed
9.CPU credit usage
10. CPU credit balance

So as per that we have only limited resources available in monitoring metrics. We may need to find out the solution to add more metrics on custom level

1. First we need to create the IAM role for cloud watch administration

login to AWS console-services-IAM-roles and create a new role as below

2. click on the EC2 from the services to use this role and click on the permissions we will get the policy details which is ready to attach the role

3. select the policy called "CloudwatchAgentServerPolicy" from the filter policies option

4. Provide the role name as "cloudwatchagentrole" and create the role

5. Now we have to attach this role to an EC2 instance . select the instance which we want to install the agent and go to actions - instance settings-attach/replace IAM role

6. Login to the instance and download the agent using the below link

https://s3.amazonaws.com/amazoncloudwatch-agent /linux/amd64/latest/AmazonCloudWatchAgent.zip

7. Unzip & Install the agent using the below command

8.Let's configure the metrics which is need to collect and send to the cloud watch , we can configure agent using below command

9. start the agent using below command

10. Now login to the cloudwatch console and click on metrics we can see the CWagent metrics configured

Also if you check the logs from Cloudwatch-Log groups-messages we can see the logs which we have added as a metric in agent from /var/log/messages

Configuring the custom DB metric using boto3

***********************************************************************
from boto.ec2.cloudwatch import connect_to_region

import MySQLdb

import boto.ec2.cloudwatch

db=MySQLdb.connect("dbserver.c5g5m1fcix6x.us-west-2.rds.amazonaws.com","dbadmin","Onm0bile","dbserver")

cursor=db.cursor()

n=cursor.execute("select * from GetActiveSessions")

n1=cursor.fetchone()

s=str(n1)

table=string.maketrans( '', '', )

number=s.translate(table,"(){}<>,L")

numbers=int(number)

reg='ap-northeast-1'

conn_cw=boto.ec2.cloudwatch.connect_to_region(reg,aws_access_key_id=’<access key d>’,aws_secret_access_key=’secret access key>')

conn_cw.put_metric_data(namespace='my_namespace',name='my_metric',value=numbers,dimensions={'InstanceId':'i-a1b2c3d4'})

*****************************************************************************

Custom metric dashboard for DB

Thank you for reading ...