Blog Posts BPMN Service Oriented Architecture (SOA)

Future of Automation in DevOps

Blog: Good eLearning

Ah, DevOps. With such a simple surface-level concept, the combination of Development and Operations work, it can be surprising to learn that actually implementing DevOps can be quite a complex affair. The skills that make up cross-functional DevOps teams can vary significantly between different businesses and departments, as can the organizational elements that influence exactly what kind of shape a DevOps culture ultimately takes. The constant evolution of the Digital Age has also consistently created opportunities to expand the scope of DevOps, to the extent that the methodology can now have far more of an impact on the wider business.

While the potential of DevOps has become more complex over time, its most central elements have remained largely consistent. One of the most essential of these is ‘automation’, the act of replacing inefficient and drawn-out manual tasks with automated processes. This can save a great deal of time for specialists to apply elsewhere and also increases the reliability of tasks and services in a way that guarantees a higher level of quality for end-users.

Still, while the concept of automation may seem straightforward, especially for folks who know their industrial history, applying it on the same scale most DevOps cultures do is not as easy as it sounds. There is always a human element, in the form of a DevOps engineer, deciding what to optimize, to what extent, and for what purpose. The engineer must also have a strong contextual awareness of the pipeline in question, as well as the capabilities offered by DevOps tools.

The focus of automation in DevOps has also evolved significantly over time. As new legislation emerges, organizations can find themselves obliged to invest heavily in areas of a pipeline like security, which may not produce as much value in a traditional sense. Other developments, such as the release of new DevOps tools, can also make it feasible to automate pipeline elements that may previously have been left alone.

In short, automation in DevOps can be a lot more complicated than it sounds – though that shouldn’t put you off! Businesses have a great deal to gain by adopting DevOps, providing they know what they’re in for before getting started.

Good e-Learning recently reached out to the DevOps Institute and its subject matter experts regarding automation in DevOps. What could they tell us about their own experiences? What do many people get wrong about automation in DevOps? Most importantly, how has the concept been evolving in recent years?

Let’s take a look at what DevOps engineers have to say about automation!

What can automation be used for in a typical organization?

Tracy Ragan: “Collaboration, streamlining, and the need for metrics generally start the discussion around ‘automation.’ In reality, most developers have ‘automated’ many of their tasks, but that ‘automation’ is known only to them. Scripting is the basis for this, such as build and deployment scripts. When we think about DevOps automation, we are maturing the siloed scripted processes into something everyone can use”

Mark Levy: “One of the main principles of Continuous Delivery is “Automate almost everything”. Most of the current focus has been on automating repeatable manual tasks that are process or event-driven. Pretty much every build and deployment process can be automated with enough ingenuity. Some things are harder or in the past were impossible to automate, but with the convergence of AI/ML and automation, the areas of automation are dramatically expanding as new and adaptive automation systems are able to embed real-time intelligence to look at trends, analyze and correlate data and events across entire value streams to predict and prevent issues in real-time. This convergence of ML, AI, and automation will present a significant advantage for companies utilizing DevOps practices.”

Mark Peters: “Automation can be used for any process which adds value. The short answer is, if you do it twice, you should automate. However, you can only automate those functions that add value to the overall business process. Easy candidates for automation are pipeline execution with associated tests, scans, static, and dynamic analysis for security, run time tests, merging successful code, and deployment. One area normally left behind is the production of useful metrics from processes that should be automated, beginning with lead time to change, deployment frequency, mean time to restore, and change failure rate.”

What are the benefits of automation in a DevOps pipeline?

Tracy Ragan: “DevOps at scale. Breaking down the silos, centralizing tasks, and clarifying what is needed to move a software update from development through production release is the core of what automation does. What it delivers is speed with guardrails. Agile practices and a cloud-native microservices architecture necessitate amping up the speed of a change through the lifecycle pipeline. We can’t rely on humans and brittle scripts to support this level of change. Automation is a must.”

Mark Levy: “Automation increases IT’s responsiveness to the business as well as improves operational efficiency. Computers move bits better than humans. The objective of continuous delivery is to create a reliable, repeatable, secure way to deliver software. Automation fits the bill by reducing lead times and performing routine, codifiable tasks. Automation also enables lower-skilled IT workers to perform complex tasks. Software delivery was once the domain of highly skilled experts. However, with automation, those experts can be redeployed to create actual customer value.”

How do you decide whether a process should be automated? What makes a process unsuitable for automation?

Tracy Ragan: “Any task that a developer has decided to capture in a script, and execute the script versus going through manual steps, should be automated. Any task that creates a bottleneck in your software delivery process should be automated. Is there anything that should not be automated in this realm? I doubt it.”

Mark Levy: “Some tasks can’t be automated, but with AI/ML, the list is getting smaller and smaller. When problem-solving skills, adaptability, and creativity are required, people still have an advantage, but that is changing. Fully automated deployment pipelines are getting pretty common.”

What are the most valuable/ widely used tools for automation?

Tracy Ragan: “This question depends on what you are automating. For orchestration of your pipeline, look for continuous delivery tooling (Jenkins, CircleCI, OpsEra, Keptn). These tools orchestrate everything else in the pipeline. Build Automation tools improve the compile/link step, the first step in a monolithic pipeline (Maven, Ant, and OpenMake Meister.) Automated configuration management is needed in microservice implementations for tracking relationships before a deployment or test(Ortelius and DeployHub). Continuous deployment tools move changes out to endpoints (ArgoCD, Helm, Xebia). Once deployed, you are ready to automate testing (Tricentis, Eggplant, SauceLabs). And never forget security scanning, library management, and infrastructure automation. All of this can be automated with open-source or commercial solutions.”

Mark Levy: “The most common automation tools center around the Continuous Delivery process, which focuses on automating the build, test, deploy, and release processes. If you view DevOps as something broader (which I do), there are many other areas that extend DevOps toolchain automation.”

What are the most common reasons for automation going wrong?

Tracy Ragan: “If you are going to automate, go all the way. Think about reducing your reliance on executing one-off scripts from your automation orchestration engine. Adding value streams to the process will give you feedback needed to continually improve. One-off scripts can impede your progress and hide your value.”

Mark Peters: “Automating without control factors. Automating and not knowing why. Following automation that worked with a different value proposition.”

Mark Levy: “There are many reasons for automation problems. First off, automating existing poorly developed processes is very common. Sometimes automating a process only moves the constraint someplace else or makes lead times actually grow.”

How does automation help drive digital transformation?

Tracy Ragan: “Digital Transformation generally refers to the new Cloud Native architecture with decomposed applications running microservices. In this architecture, moving changes to production occurs all day long, not once a day or once a week. If you don’t automate, you will not achieve the business agility that digital transformation promises.”

Mark Levy: “With digital transformation, IT has become the deployment pipeline that delivers digital assets to the customer. The faster IT can deliver these assets, the bigger the competitive edge the business has over rivals. With automation, business decisions and feedback can be implemented faster, and new revenue streams can come online faster.”

What oversight is required for successful automation?

Tracy Ragan: “Metric, metrics, and did I mention metrics? If you can measure it, there is no way of telling if your automation is working. Know what you want to achieve and measure it.”

Mark Peters: “I’d expand to measurements as the three areas seem to be logs, traces, and metrics. Automation can help with all of those.”

Mark Levy: “A CIO of a large healthcare company once said, “Nothing is less productive, and more demotivating, than to make something more efficient and effective that should not be done at all”. Successful automation not only has to be measured in terms of outputs (lead times, MTTR, Change fail percentage) but also outcomes. Outputs are great for measuring capabilities, but outcomes show you how you impact the business.”

What quick wins can automation achieve?

Tracy Ragan: “Clarity – knowing what is coming down the pipeline, knowing what you just released, and knowing the impact of a change. These insights are easy to achieve once you begin automating.

“Automation can help you build momentum if you are trying to internally build a business case for continuous delivery/ DevOps. Automated deployments can happen quickly, and the success can give you a tailwind for your DevOps journey.”

Are there human elements that automation cannot replace?

Tracy Ragan: “Automation cannot build itself. Humans must be driving the requirements and tweaking the process to make sure it is achieving the goals.”

Mark Peters: “Yes, currently, automation is poor at solving abstract problems or integrating new solutions into practices. Unsupervised learning allows ML models to identify and work through challenging issues, but humans are still required.”

How is AI/ machine learning influencing DevOps automation?

Tracy Ragan: “In the future of Cloud Native and microservices, machine learning around the core metadata about microservice usage and value will be key. This can lead to understanding risks associated with highly reused services and the cost associated with microservice sprawl.”

Mark Peters: “Keep returning to the fact AI/ML learning is a wide facet of most functions, no different in DevOps. A previous AI/ML article discussed how, when robots were first used in factories, they were kept in worker-safe zones. Now, workers and robots interact, letting robots do robot things and humans do human things. The same will hold true in DevOps. Automation will take over, and humans will inspect or tweak those items where automation does not apply.”

How has automation been affected by lockdown and distance working?

Tracy Ragan: “Automation has proven to be critical in a COVID world. In fact, it has driven companies to consider what more can be done through automation.”

As digital transformation continues, will automation and DevOps have more of a part to play across enterprises? Should they?

Tracy Ragan: “As we speed up releases using independently deployed microservices, automation will be 100% required. This new architecture can’t be done manually. This is becoming evident as companies add more Site Reliability Engineers to solve the problem, and they are all reporting high levels of stress.”

Mark Peters: “If you want speed, you need automation. If you want deployments, you need automation. If you want basic security, you need automation. If you are building a nuclear reactor, and need surety, you still need automation. Automation accelerates multiple practices. Simply management theory demands if you can automate and remove lower-level costs to add higher-level solutions, you do so.”

How important has automation become for remaining competitive?

Tracy Ragan: “One of the goals of automation is business agility. The ability to get new features to end-users is not just ‘nice to have’, it is essential. Consumers do not understand why a particular service portal is not working – and they don’t care. Consumers are driving 100% fault tolerance and auto-scaling. As consumers, we will move to another online service if our current one is constantly unavailable. Just look at the success of Ally bank compared to its competitors, who were slower to move to a 100% online service.”

Mark Peters: “Winning competitions means creating more value than a competitor. If your process creates more value for the customer, you win. Automation is one of the behind the scene factors.”

Why is automation important for security?

Tracy Ragan: “Security is detailed. Asking a human to scan containers and clusters is impossible. This must be done by a computer.”

Mark Peters: “Automation has the same impacts on security as in every other area. It frees humans to think about complicated problems while providing data about the simpler areas. Using automation enables the security expert to focus on areas not currently fixed, excepted by policy, or created by human behaviors.”

What impact will AIOps have on automation?

Tracy Ragan: “AIOps will need its own type of automation and workflows that are different from DevOps. Scheduling, tracking, and metrics are the same, but the tasks are different.”

Mark Peters: “Technically, AI processes are automation. Employing an AI constitutes automating a process across DevOps.”

Tracy Ragan

Tracy is a specialist with over 25 years of experience working in DevOps cultures. She is currently the CEO of DeplopyHub, as well as a Board Member for the Continuous Delivery Foundation.

“As we move into managing microservices in high-performance computing, we must fully embrace automation and start using technology to replace the manual steps that worked in monolithic practices. DevOps at scale will be essential to meet the demand.”

Mark Peters

Mark is the Lead Information Assurance/ Security Engineer for the Technica Corporation, as well as a Global Ambassador for the DevOps Institute. He has over 25 years of experience as a Cybersecurity specialist, as well as more than three years working with DevOps.

“DevOps is the culture, the way to think about the processes and tools one employs in creating value. If you think merely about the tools, you aren’t thinking deeply enough to solve the problem, in the “every nail needs a hammer” sense.”

The DevOps Institute

The DevOps Institute is a professional membership association whose mission is to advance the human elements of DevOps by creating a safe and interactive environment where our members can network, gain knowledge, grow their careers, support enterprise transformation, and celebrate professional achievements. We connect and enable the global DevOps community to drive change in the digital age. DevOps Institute ambassadors are pioneers in DevOps and some of the world’s most foremost thinkers and advanced practitioners of DevOps ways of working who volunteer to share their wisdom and expertise with the humans of DevOps, globally.