We have created a Python Automation Menu for Automating multiple technologies such as Hadoop, Docker, Amazon Web Server, Apache Webserver and Logical Volume Management
Brief Details about the technologies used in the project
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.
Docker is an open platform for developing, shipping, and running applications. Docker enables you to separate your applications from your infrastructure so you can deliver software quickly. With Docker, you can manage your infrastructure in the same ways you manage your applications.
Amazon Web Server:
Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform, offering over 175 fully-featured services from data centres globally. Millions of customers — including the fastest-growing startups, largest enterprises, and leading government agencies — are using AWS to lower costs, become more agile, and innovate faster.
Apache is an open-source and free web server software that powers around 40% of websites around the world. The official name is Apache HTTP Server, and it’s maintained and developed by the Apache Software Foundation.
Logical Volume Manager:
In Linux, Logical Volume Manager (LVM) is a device mapper framework that provides logical volume management for the Linux kernel.
Services provided for different technologies are
- NameNode Configure
- DataNode Configure
- Check Report
- Launch Container
- Stop Running Container
- Pull Docker Images
- Check Running Container
- Check All Container
- Check Available Images
- Remove Image
- Remove all Images
- Remove a Container
- Remove all Container
Amazon Web Server
- Launch Instance
- List the Instance
- Stop an Instance
- Start an Instance
- Terminate an Instance
- Create a Volume
- Attach the Volume
- Detach the Volume
- Delete a Volume
- Create an S3 bucket
- List the S3 bucket
- Upload in the S3 bucket
- Delete S3 bucket
- Create CloudFront Distribution
- Delete CloudFront Distribution
Logical Volume Management
- List all Attached Disk
- List all Physical Volume
- List all Volume Group
- List all Logical Volume
- Create Physical Volume
- Create Volume Group
- Create Logical Volume
- Extend Volume
- Install HTTPD
- Configure Webserver
- Stop WebServer
Let’s Checkout detail working of this python program technology-wise used in this Project
Here is the Main Menu for Hadoop
Let’s Explore it options
This option is used to completely configure Namenode and start the services. It only asks for the directory in which want to use for NameNode.
This gives the report about the number of data node connected, capacity and other information about the cluster
Here is the main menu for the docker program
Let’s Explore it options
This option is used to launch the container. For this, we have only had to provide it with the container name of our choice and we have to give the name of the image of which we want to launch the container.
Stop Running Container
This option is for stopping the running container after its use. Here we have to give the name of the container we want to stop
Pull Docker Image
Check Running Container
Check All Container
In this option, we can check details of all container (running and stopped both)
Check Available Image
The main menu for the LVM Partition is
List all Attached Disk
List all Physical Volume
List all Volume Group
List all Logical Volume
Create Physical Volume
Create Volume Group
Create Logical Volume
We are exploring the AWS functionality in more the detail due to it is the best part of this automation program
In order to Integrate the AWS command-line interface, we have to create a CLI user in IAM. After creating user we get AWS Access Key ID and AWS Secret Access Key which is important for interacting command line and integrating with python.
Here is the Menu for AWS
Let’s get into it one-by-one, Firstly we’re gonna go through EC2 also known as Compute as a service.
To run different services of aws in command line we have written lengthy commands like
aws ec2 run-instances image-id ami-052c08d78def0ac62 instance-type t2.micro count 1 key-name keyname
same we have to do in rest of menu, So in order save time we have created python to run these commands in the background. Now we have to choose from what we to do.
Now, to launch an Instance we have to just press “1”, and choose the instance of my choice and the instance is created.
List of Running Instance,
To Change the name of Instance we have to provide with Instance-name
and the Instance name gets changed
To stop an Instance press 4 and provide Instance-id
and Instance gets stopped.
To start an Instance press 5 and provide Instance-id
now after work done let’s Terminate Instance by providing Instance-id
Elastic Block Storage or EBS is Storage as a Service to Extra hard-drive of Operating-System. To create a volume Press 7 in the python program
And hard-disk of 5GB gets created
Now let’s Attach the volume to the instance
we can that Instance attached or not by the command fdisk -l
Detach an Instance Press 9 we have to provide Volume-id and Instance-id.
Finally, After completing the task Deleting the Volume, provide Volume-id
S3 stands for Simple Storage Service, S3 is another Storage as a service, it basically provides object storage. To Create S3 bucket press 11 in given python program.
and bucket gets created
press 12 to list the buckets
Now, we have to place Object inside the bucket, press 13 and provide a file with the file path and bucket name incase of more than one bucket
File get placed in the bucket
After the Work is done, We can Delete files from the bucket Press 14 and give the bucket name and you file gets deleted.
After Deleting the files delete the bucket
To create Cloudfront distribution Press 16 in the given Program
CloudFront gets created with filename provided as mention in the above image
To describe CloudFront Distribution press 17
To Delete CloudFront Distribution we have to disable it manually and provide with the Distribution-id and Etag as visible in the above image
The main menu for Apache Web Server
In order to configure webserver Press 1 and behind the scene it run background command as required
After webserver gets configured, now is the time to check the configuration of the webserver, but before configuration let’s make a normal webpage in CLI
This normal webpage, now Press 2 in a given python program and provide it with a file with its path
Yuppiieee !! webserver configured, just paste your IP address in web-browser
After Your work is done now when you want to destroy your web server just Press 3 in the python program
and webserver gets destroyed