Cloud automation refers to the practice of leveraging automation technology to automate cloud operations. Cloud automation should be more aptly named automation of the cloud because it is an evolutionary approach to managing cloud operations rather than just the technology that enables it. By automating their cloud operations, organizations can quickly complete previously manually tasks with significantly reduced errors, such as deploying code, backing up data stores, and rolling out configuration changes to environments.
These tasks done infrequently may be manageable, however, automation shows its advantage by enabling IT teams to architect solutions that leverage the speed and accuracy possible only with machines. For example, major cloud service providers, Google, AWS, Azure, IBM, all deploy their versions of cloud provisioning and orchestration software which makes modern cloud computing, with its rapid flexibility and scalability, possible. While cloud computing was initially manually managed—administrators themselves would need to create and manage virtual machines, make back-ups, etc.—the number of users and companies rooting their IT in the public cloud created the demand and necessity for more efficient workflows. CSPs turned to automation, which helped them power the cloud orchestration and provisioning software that is responsible for coordinating and managing cloud resources with cloud environments that serve a multitude of users.
In DevOps, automation plays a key component in their continuous integration/continuous deployment (CI/CD) workflow. By automating aspects of the building, testing, and deployment activities, the gap between development and operations is bridged, and teams can rapidly respond to user demands with releases and updates to address issues, a process that can handle deployments multiple times a day. In combination with other automations, like orchestration software, entire data centers can be rapidly deployed and updated in the fraction of the time that it could be performed manually, and with reduced chance of errors.
Modern cloud computing is made possible through the abstraction advantages of virtualization. But, cloud virtualization entails a complex set of management practices bottlenecked by manual workflows. The logical solution is cloud automation, which programmatically enables rapid scaling and provisioning, system wide configurations, creation and destruction of VMs, and performance monitoring.
Cloud automation serves business goals as well. The costs of operating in the cloud on third-party platforms has become competitive, and for cloud consumers, it has made migrating to the cloud an effective cost strategy. Many of the top tier providers offer cloud automation that increases productivity, promotes cost savings, and reduces risk of errors. These solutions also address managing technology stacks, which grow taller and more complex with every newly integrated system. Often these integrations are challenged by real-time management concerns, a factor that with manual active management is risky, but a problem that automation is ideally suited to solve.
Automation tools streamline cloud computing workflows, relieving burdens, risks, and costs from IT staff. A successful cloud automation strategy will reduce or eliminate the need for many repetitive tasks; it will push redundant decision making to the system which can make and execute decisions faster than relaying them for a human troubleshooting response; it will allow IT staff to focus on innovations and higher level decision making, and automations can extend other domains in the cloud such as performance monitoring and diagnostics, and security.
Cloud automation is ideal for automating routine operations and sets of workflows, including infrastructure provisioning, identify provisioning and management, application deployment, monitoring and remediation, managing hybrid cloud and multi-cloud configurations, and discovering data and classifying it. Among these workflow sets, automation can unburden manual workflows from routine activities such as:
- Cloning — Automation can easily clone VMs. Further, it can be used in tandem with a template creation activity, archival activity, or remote replication activity.
- Create Internal APIs — Internal APIs can be made using automation and self-service operations.
- Permission Settings — Automations are ideal for setting permission across a multitude of files, folders, subfolders, users, and systems.
- Resource Access — Defining access to resources can be automated and integrated with other automation sets, such as automation runbooks for provisioning.
- Restarts — Like shutdowns, restarts can also be automated.
- Scaling — Scaling is enabled using automations that respond to threshold demands, automatically provisioning new resources where they are needed.
- Shutdowns — Shutting systems down in bulk, such as executing containment actions to protect systems, can safely be performed in a controlled fashion using automated scripts.
- Snapshots — Automation can be delegated to take snapshots of VMs. In like fashion, VMs can be removed when not needed. A snapshot can be taken, the VM can be reconfigured or set up, and if it runs smoothly, automation can remove the snapshot.
- User Account Creation — Using given information, possible from a web form, automation can automatically create a user account. Other integrations can be added for verification, or further security set up depending on the application.
- User Account Management — After user account creation, additional automations can reset, disable, unlock, and activate accounts.
Cloud automation is a general class of IT practices and technology leveraging automated workflows to complete administrative tasks that maintain cloud operations more efficiently than manually workflows. Cloud automation is used in both cloud provisioning and cloud orchestration. Automation uses scripts, schedules, and automation tools to programmatically perform workflows. Whereas orchestration manages the automated workflows, which in orchestrations are called runbooks composed of steps called activities.
Cloud orchestration integrates several systems, e.g. ticketing systems, or container orchestration systems. In the case of a ticketing system, IT staff can respond to a trouble ticket by sending the appropriate information that solves the problem. The automation then refers to a runbook, a predefined set of automated workflows, and supplies any necessary details included in the ticket, and logs its changes in an integrated change management system. In this scenario, the orchestration is mapped to various runbook workflows, which in turn take variables, and follow dependencies, to automatically execute steps.
Automation of the cloud benefits teams by combining the advantages of the cloud and the advantages of automation.
- Reduced Errors — By removing manual processes and replacing them with automation, there are fewer points of error, especially when testing is built into automation.
- Lower Infrastructure Costs — Humans are expensive solutions for many minor administrative tasks. Automation lowers costs by reducing errors and activities that reclaim valuable and expensive IT staff time.
- Enhanced Security — Automations can help secure sensitive workflows as well as enable real-time surveillance.
- Helps Streamline DevOps Workflows — DevOps has proven to be the new standard in development, and automation that streamlines the deployment process enables modern DevOps practices.
- Cloud Scalability — Simply, without automation, scaling cloud resources is not practical. Automation handles the complexities of provisioning and orchestrating environments for multiple users.
- Intelligent Backups — Backups are paramount in cases of server failures or cyber-attacks. Automation capabilities enable intelligent backups of storage, syncing, and disaster recovery.
- Improved Governance — Manually managing virtual systems, while possible, can become frustratingly difficult when environments multiply. For example, zombie cloud resources are those cloud services spun up but forgotten about only to be left to consume valuable resources. Automation used in visibility and monitoring software can detect and close these zombies.
Cloud automation comes in many varieties. Some general automation tools include:
- Application Release Orchestration (ARO) Tools — Another tool highly associated with DevOps, ARO tools comprehensively layout an application's release pipeline so teams can easily manage the project progress, scheduling and assigning tasks, through visualizations. These tools also integrate or natively manage a CI/CD framework that allows teams to deliver continuously.
- Cloud Infrastructure Automation Software — Referred to as "Infrastructure as Code", organizations install cloud infrastructure automation software to provision servers and data centers using metadata files rather than to physically configure each server. These tools are strongly associated with DevOps practices, as they allow developers to create templates for the reuse of preconfigured infrastructure.
- Container Orchestration Software — Container orchestration software enables the automated deployment of containers that can be used in microservices approaches. Container orchestration integrated with hypervisors and provisioning automation allow the scaling capabilities of cloud computing.
- Data Warehouse Automation Software — Data warehouse automation software automates and streamlines the entire data warehouse. This includes ETL functionality, scaling, analysis and insights, and preparation for external use.
- IT Resilience Orchestration Automation (ITRO) Software — Failovers are a problem that should be expected and can be planned. ITRO software orchestrates service failovers through automated infrastructure operations to ensure that when systems failover that services remain available.
Automation platforms perform automated functionality as well, and will fall under one or more of the above categories. Platforms may be more encapsulated, for example, while developers may write simple scripts to perform simple tasks, platforms aim to bundle essential tools in one package. Each major cloud provider, Google, Microsoft, AWS, and IBM all offer automation services, but below are other popular platforms integrated into many departmental toolboxes.
- Kubernetes — Kubernetes is a popular open-source system for automating the management of containers. Kubernetes builds upon the extensive experience of Google, which has developed its own container-management systems, incorporating the best-of-breed practices.
- Puppet Enterprise — Puppet pioneered Infrastructure as Code (IaC), and is one of the most used configuration management tools. With a GUI interface, Puppet Enterprise can automate tasks for massive workloads, up to 20k nodes.
- Ansible Tower — Ansible, developed by RedHat, and Ansible Tower, the enterprise version, are both open-source configuration management software. Ansible tower upgrades the power in Ansible and provides a web interface for easy management of cloud automations.
- VMware vRealize — vRealize automates infrastructure, workflow, and recovery management, for single or multi-clouds.
- CFEngine — CFEngine is a highly compatible tool that approaches management using autonomous agents that actively monitor up to 5,000 hosts rather than use a central management server.
Organizations use cloud automation for three general uses: managing their infrastructure with DevOps; scaling their production stacks; and sharing and enforcing best practices. To these ends, several leading brands have implemented automated cloud systems to achieve exceptional levels of scale, all while making it look easily doable.
Netflix, thanks to its cloud scaling innovations, many of which have become de facto best practices, purports to push deployments hundreds of times a day. They do this using an open-source automation software, Spinnaker, to push deployments in a multi-cloud environment. It does such a good job that Google, Microsoft, and even Target are using it.
Like Netflix, Amazon also develops many of their automation innovations in-house. Amazon's IT automation confidence rose even further after they moved their operations to the AWS cloud in 2010, and leveraged cloud advantages into expanding operations to tens of thousands of servers, serving millions of users.
Belying the giant's size, Facebook also utilizes open-source code to automatically deploy and scale its servers; the tool they use is Opscode. As well, the configurations of thousands of servers that serve billions of Facebook users are automated by Chef configuration management software, which is used by Google, CapitalOne, IBM, HP, and many more.
A defining characteristic of DataOps is the numerous roles that interact and contribute to the accumulation, processing, and use of a company's data assets. Towards the extreme, companies whose data assets are their main value proposition have the most immediate need to understand the people engaging with proprietary information. These DataOps roles can be generally classified as data consumers, data preparers, and data suppliers.
Data Suppliers — Data suppliers are the end data owners, like database administrators, responsible for data management, processing, and user access control of a company’s DataOps.
Data Preparers — Due to the ever-complicating nature of DataOps, a middle ground of roles is developing between data engineers, data suppliers, and data consumers. Data engineers build the pipelines that refine raw data into new usable, valuable, and monetizable data. Data curators is a developing role that begins with the needs of consumers to optimize accordingly DataOps content to businesses the needed context for enhancing final assets. Another developing role due to heightened requirements around data governance is the data steward which is responsible for developing company data governance policies and ensuring compliance.
Data Consumers — Data Consumers receive the final data output and the largest group that interacts with DataOps assets. Many roles have emerged: data scientists apply data to solve business problems, data citizens are frontline workers in need of real-time information, and data developers need accurate DataOps as they build business applications that use those pipelines.
