Automation and Orchestration tools have helped IT focus more on creating value to customers and users, and less on keeping the lights on. These automation and orchestration tools, combined with the cloud-based infrastructure enables streamlined workflows, scalability, and performance we come to expect, but they also create new concentrations of risk in our infrastructures.
IT has long operated in a mode of prioritizing security activities: generally “production” systems are prioritized over “lab” systems, and for good reason. Lab systems were generally not mission critical to an organization and were typically squirreled away in the bowels of an organization’s network.
I often see organizations employing devops continuing to focus on protecting the “production” environment: the business applications customer-facing web applications, and so on. In fact, automation, orchestration, and cloud infrastructure creates new and innovative security capabilities, such as spinning up new servers that are patched, tested, and then rotated in to replace the existing, unpatched servers, which are summarily destroyed upon being replaced. Unfortunately, I also see these orchestration and automation tools treated like legacy “lab” systems.
Quite the contrary, these automation and orchestration systems should be treated with at least the same level of diligence as the platforms they manage, hopefully for obvious reasons.
A public example of what can happen is the recent breach of Matrix.org’s Jenkins server. Note: the Matrix.org team should be commended for their transparency and speed in responding to this incident. I do not claim to know the reasons that their Jenkins server had unpatched vulnerabilities or the details of how those vulnerabilities were exploited, only that this situation aligns with my observations from other organizations.
Orchestration and automation tools are an very attractive target for our adversaries since they are a) generally not well protected or monitored, and b) enable rapid attacks on an organization’s most important and sensitive systems. I implore you to work with your respective IT teams to ensure that these tools are managed and protected appropriately. And yes, Active Directory is one of these tools.