If we stood back and took a look at the way the biggest company proceed in the DevOps world, we will confirm that the automation is a crucial deal.
When we say DevOps, we say continuous development, continuous integration, continuous testing and continuous deployment.
Certainly, DevOps is not just automation but automation will ensure the “continuous” job.
Why do we automate ?
The cloud puts the limitations of hardware aside, there are a lot more moving parts that we have to consider. There are no “completion dates” on a lot of cloud projects. Automation allows us to efficiently manage the fact that infrastructure changes all the time.
These infrastructure changes were managed by throwing people and money at problems.
You needed more hardware, or a bigger staff, and that meant more complex infrastructure buildout and development processes.
Nowadays, we develop sharper and smarter software to control complex systems and deployments. In other words, we automate.
Security and Automation ?
When talking about automation, we assume it automatically increases the security risk.
But in fact, we find that automation reduces the risk of potential security flaws since we have a controlled build and update process.
If we need to push out a simple patch to our environment, our Puppet or Ansible architecture, for example, will allow us to do this in a very standardized way. We can update a central script and make sure this is pushed out to all environments, and if for some reason an environment gets launched with an old configuration, it is automatically changed when it checks into Puppet.
We can build and update a system with a small and even no direct intervention at all in an environment, which of course reduces the chance of human errors.
Automation or Orchestration ?
We have to differentiate and recognize the slight line that separate automation from orchestration.
A simple automation cannot achieve the optimization process since it is related to a single task such as launching a web server, configuring a web server, stopping/starting a service.
Orchestration, however, is concerned with automating the execution of a workflow, a process or we might call it a succession of jobs. A provisioning or even updating a database process might be a bunch of tasks that should be executed in a significant order and that might involve multiple systems.
The goal of orchestration does not consist in automatically execute a service, which brings speed to the deployment process and gets applications into production faster. It also affords an opportunity to pipeline and optimize the processes to run for a greater gains in the daily job.
Puppet, Ansible, Jenkins and many others tools could be used in the automation process.
Ansible, C2O, Orc and others could be used for the orchestration process.
I am just mentioning some samples of the automation/orchestration tools and I will let you dive in further for details !!!
Why did I choose to talk about Automation?
I have been working with a “Big data” team within a company that handles a really large amount of data related to the “automobile” field.
On one hand, the fact of handling an extremely increasing amount of data pushed this company to migrate a part of their information system from the on-premise datacenter to the cloud in order to have no limitation in terms of memory and data storage.
The cloud Azure was the choice.
On the other hand, minimize the human intervention on the production environment was the follow up. As a result , we involved automation and we used it for many purposes such as :
-Migrating the data to the cloud;
-Running the needed jobs on the cloud;
-Handling the cloud infrastructure to save money by controlling the start/stop of virtual machines, hdinsight clusters, …
-Monitoring the cloud infrastructure.
So as you see here, the automation was used for different aims using different tools.
How did we do it in a customer environment ?
Involved in his architecture process, we have chosen to start creating a gateway server on which we installed an Azure CLI. This will ensure the communication between the cloud and the on-premise Datalake. Then, we’ve covered both DevOps topics:
-The platform and its packages were automated via Puppet: Users access and rights, NFS shared folders, java version, apache tomcat, …
The same tool was used for the Azure virtual machines as well.
-To automate the start/stop process of the cloud virtual machines on a definite schedule, we have proceeded with the Start/Stop VMs during off-hours based on solution provided by Azure Microsoft
This solution saves money by making sure VMs are off when not being used. (note: more than 50% of the time is off-time for more than 80% companies in France).
C2O was our orchestrator.
We have prepared a bunch of C2O graphs, each one executing a succession of steps Example of a C2O graph:
1-Extract database tables
2-Start a hdinsight cluster
3-Upload the files into the cloud
4-Perform CRUD operations and/or spark jobs on the cloud
5-Terminate the hdinsight Cluster
The orchestration was the perfect solution to start a whole workflow without any human intervention …
As far as I learned … Whether we automate or orchestrate, we are definitely facilitating and protecting the DevOps World.