Jumbo Shrimp. Deafening Silence. Seriously Funny. All great examples of oxymorons.
If some in the DevOps community were to be believed, we should add Enterprise DevOps to that list as well. Yet over the last couple of years, we’ve been successfully transforming Ipreo into a DevOps friendly enterprise software company. The results have been amazing.
We now have application teams doing zero downtime deployments. We do over 25,000 TeamCity builds and hundreds of deploys per day across all our products and environments. Bugs are down and engineering morale is up as they are learning modern tools and are rapidly iterating in ways that were impossible before. We’re even leveraging public cloud infrastructure.
What’s an Enterprise Software Company?
There’s no one definition for this phrase and no one attribute that pushes you into enterprise territory. Here are a few key dimensions to consider:
- Number and/or size of customers: Enterprises have large numbers of customers, or their customers are governments or enterprises themselves.
- Number of products and/or lines of business: You probably don’t just sell one product if you’re an enterprise company. Those products likely have integrations together as well.
- How critical is the service you provide? An enterprise usually provides a service that is more than just a value-add or efficiency gain for your customers. Customers rely on you to run their business and they couldn’t operate without you.
- An enterprise software company usually has to deal with complex legal and regulatory landscapes, thanks to the countries in which they do business.
- Complex technology landscape. Enterprises frequently grow through mergers and acquisitions (M&A), often resulting in mixed platforms, tools, and application stacks.
- Security concerns come first. Solutions are built with security at the forefront. Networks, access, and permissions are locked down with only the bare minimum opened up. Silo’d teams are usually the easiest short-term way to satisfy concerns, and efficiency is often sacrificed for this “easy path.”
What is DevOps?
Everyone has their own definition of DevOps. At Ipreo, DevOps is a culture of continuous improvement applied to the code, build, and deploy process. It is a constant de-risking of how we deploy and monitor our applications. Many DevOps definitions focus on the tools. We have our fair share, but tools are an enablement of our culture, and culture comes first.
The great news about these definitions is that there’s actually nothing here that makes these terms contradictory. Enterprise DevOps is not an oxymoron. The problem is that many enterprise software companies set out to “adopt DevOps” and go about it all wrong. They build a “DevOps team,” and they never address the underlying culture and all the friction that comes along with it.
In my next few posts I’ll explain a few of the things that have been key to making DevOps work for us. First up: teaching people to fish…
Foundation of the culture
If you want to feed a man for a day, give him a fish. If you want to feed him for the rest of his life, teach him how to fish. It’s an old adage, but it points to a key problem in the traditional enterprise.
Traditionally, a release management team would facilitate the build and deploy process for an application in an enterprise. If you have one application, that’s workable, but an enterprise will inevitably have lots of diverse applications to support. Microservices refactoring is a common enterprise project that will create a huge load on a central team. You simply cannot scale a central team to meet the demand in a cost-effective way.
If you’ve read The Phoenix Project by Gene Kim then you’re familiar with Little’s Law. The principle is that the more utilized a work center is, the greater the delay for work flowing through it. It’s what causes a 20 minute task to take two weeks to complete.
Rather than trying to squeeze throughput from a broken model, you have to change the model.
Internal Tools paves the road ahead
You need an internal tools team that will own and control a set of tools that your development teams will use on their DevOps journey. That set of tools will continually evolve as technology and needs change.
Rather than try to have a central team create and maintain the build process for dozens of applications maintained by several dozen teams, you have a team provide the build tool that those teams can safely consume. Rather than rely on a central team to try to learn how to deploy all those applications, keep up with the changes to that process over time as the applications evolve, and meet the scheduling demands of all those teams, you need a tool those teams can use to do their own deployments.
Here’s what our toolbox looks like at Ipreo:
What qualifies as DevOps work? Improving the build process, de-risking the deploy process through automation, implementing automatic rollbacks, introspection of open source dependencies, tweaking the branching strategy to make testing or deploying easier, adding feature flagging, implementing better monitoring… the list goes on and on. There’s no better group of people to do that than the people building the applications. No central team will ever be able to keep up with the rate of change across dozens of teams like you’ll find in an enterprise.
But won’t developers make mistakes? What if they use these tools wrong?
Don’t worry, they will. So what? They’ll figure it out. They’ll learn. The application teams get together every two weeks to retrospect and decide what to improve on next. The trade-off is a no-brainer. Ipreo has looked at this from both sides for years now. The reality is that there are two kinds of risk: inherent and residual. You can only squash one of them.
For example, an inherent risk in giving teams their own deploy automation is that they might push out a change that the product folks didn’t know about or approve of. You can easily mitigate that by having the tools team ensure the pipelines created include a manual approval step from a non-developer, and that the build/deploy process has great traceability. You also make sure that the deploy pipeline is just promoting the same artifact that has been tested, so it’s not possible to deploy a change that hasn’t already been reviewed. Sprinkle in static code analysis and you’ve reduced the residual risk to business and market accepted levels.
There may still be residual risk. What if the deploy process fails for some reason? That danger exists with or without the automation, and exists regardless of who clicks the button to do the deployment.
As long as you have some path for the application team to escalate and get help in the case of a failed deploy then you’ve reduced that risk as much as you feasibly can. You cannot eliminate all risk, and that should never be your goal. You want to get rid of as much inherent risk as possible, then recognize the residual risk, plan around it as best you can, and accept it.
Help your teams learn how to think about the changes they are making in a way that works for your business, give them the tools they need to be successful, and then turn them loose. Incentivize and enable continuous improvement in your code, build, and deploy process, and you’ll be on your way to a DevOps culture in no time.