We have created a Python Automation Menu for Automating multiple technologies such as Hadoop, Docker, Amazon Web Server, Apache Webserver and Logical Volume Management

Brief Details about the technologies used in the project

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.

Docker is an open platform for developing, shipping, and running applications. Docker enables you to separate your applications from your infrastructure so you can deliver software quickly. With Docker, you can manage your infrastructure in the same ways you manage your applications.

Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform, offering over 175 fully-featured services from data centres globally. Millions of customers — including the fastest-growing startups, largest enterprises, and leading government agencies — are using AWS to lower costs, become more agile, and innovate faster.

Apache is an open-source and free web server software that powers around 40% of websites around the world. The official name is Apache HTTP Server, and it’s maintained and developed by the Apache Software Foundation.

In Linux, Logical Volume Manager (LVM) is a device mapper framework that provides logical volume management for the Linux kernel.

Services provided for different technologies are

  • NameNode Configure
  • DataNode Configure
  • Check Report
  • Launch Container
  • Stop Running Container
  • Pull Docker Images
  • Check Running Container
  • Check All Container
  • Check Available Images
  • Remove Image
  • Remove all Images
  • Remove a Container
  • Remove all Container
  • Launch Instance
  • List the Instance
  • Stop an Instance
  • Start an Instance
  • Terminate an Instance
  • Create a Volume
  • Attach the Volume
  • Detach the Volume
  • Delete a Volume
  • Create an S3 bucket
  • List the S3 bucket
  • Upload in the S3 bucket
  • Delete S3 bucket
  • Create CloudFront Distribution
  • Delete CloudFront Distribution
  • List all Attached Disk
  • List all Physical Volume
  • List all Volume Group
  • List all Logical Volume
  • Create Physical Volume
  • Create Volume Group
  • Create Logical Volume
  • Extend Volume
  • Install HTTPD
  • Configure Webserver
  • Stop WebServer

Let’s Checkout detail working of this python program technology-wise used in this Project

Hadoop

Here is the Main Menu for Hadoop

Let’s Explore it options

NameNode Configure

This option is used to completely configure Namenode and start the services. It only asks for the directory in which want to use for NameNode.

DataNode Configure

Check Report

This gives the report about the number of data node connected, capacity and other information about the cluster

Docker

Here is the main menu for the docker program

Let’s Explore it options

Launch Container

This option is used to launch the container. For this, we have only had to provide it with the container name of our choice and we have to give the name of the image of which we want to launch the container.

Stop Running Container

This option is for stopping the running container after its use. Here we have to give the name of the container we want to stop

Pull Docker Image

Check Running Container

Check All Container

In this option, we can check details of all container (running and stopped both)

Check Available Image

Remove Container

Remove Image

LVM Partition

The main menu for the LVM Partition is

List all Attached Disk

List all Physical Volume

List all Volume Group

List all Logical Volume

Create Physical Volume

Create Volume Group

Create Logical Volume

AWS

We are exploring the AWS functionality in more the detail due to it is the best part of this automation program

In order to Integrate the AWS command-line interface, we have to create a CLI user in IAM. After creating user we get AWS Access Key ID and AWS Secret Access Key which is important for interacting command line and integrating with python.

Here is the Menu for AWS

Let’s get into it one-by-one, Firstly we’re gonna go through EC2 also known as Compute as a service.

To run different services of aws in command line we have written lengthy commands like

aws ec2 run-instances image-id ami-052c08d78def0ac62 instance-type t2.micro count 1 key-name keyname

same we have to do in rest of menu, So in order save time we have created python to run these commands in the background. Now we have to choose from what we to do.

Now, to launch an Instance we have to just press “1”, and choose the instance of my choice and the instance is created.

List of Running Instance,

To Change the name of Instance we have to provide with Instance-name

and the Instance name gets changed

To stop an Instance press 4 and provide Instance-id

and Instance gets stopped.

To start an Instance press 5 and provide Instance-id

now after work done let’s Terminate Instance by providing Instance-id

Elastic Block Storage or EBS is Storage as a Service to Extra hard-drive of Operating-System. To create a volume Press 7 in the python program

And hard-disk of 5GB gets created

Now let’s Attach the volume to the instance

we can that Instance attached or not by the command fdisk -l

Detach an Instance Press 9 we have to provide Volume-id and Instance-id.

Finally, After completing the task Deleting the Volume, provide Volume-id

S3 stands for Simple Storage Service, S3 is another Storage as a service, it basically provides object storage. To Create S3 bucket press 11 in given python program.

and bucket gets created

press 12 to list the buckets

Now, we have to place Object inside the bucket, press 13 and provide a file with the file path and bucket name incase of more than one bucket

File get placed in the bucket

After the Work is done, We can Delete files from the bucket Press 14 and give the bucket name and you file gets deleted.

After Deleting the files delete the bucket

To create Cloudfront distribution Press 16 in the given Program

CloudFront gets created with filename provided as mention in the above image

To describe CloudFront Distribution press 17

To Delete CloudFront Distribution we have to disable it manually and provide with the Distribution-id and Etag as visible in the above image

Webserver

The main menu for Apache Web Server

In order to configure webserver Press 1 and behind the scene it run background command as required

After webserver gets configured, now is the time to check the configuration of the webserver, but before configuration let’s make a normal webpage in CLI

This normal webpage, now Press 2 in a given python program and provide it with a file with its path

Yuppiieee !! webserver configured, just paste your IP address in web-browser

After Your work is done now when you want to destroy your web server just Press 3 in the python program

and webserver gets destroyed

☺ Thanks For Reading ☺