DevOps in the enterprise (continued)

I have already posed that one of the biggest challenges for the enterprise is sheer scale. Scale in terms of tin, services, process and people and this has a serious impact on transformation.

With hundreds or thousands of servers I can assure you that no-one has a complete map of the assets let alone the configuration of a global infrastructure. Sure, configuration management databases (CMDB) exist but they are no solution.
Centralised IT teams distributed according to an ITIL model have been bolstered by shadow IT which operates out of business units diluting knowledge and completeness.

I know that many industries are forced to comply with regulation . Many have defined standards to try and lower cost and achieve ‘compliance’. But I fear that many realise that their previous efforts have only gone so far. I have long held the opinion that if a privileged user is able to access a system it can no longer be guaranteed to comply to standards; this, in and of itself, is one of the reasons why Infrastructure as Code and immutable systems are so compelling.

And with this in mind the starting point for introducing DevOps to a legacy estate has to be a Discovery process.

Discovery is the first phase or in more traditional IT automation circles using an approach which I first used in 2005, a passive deployment.

HP, IBM, CA etc. will sell you Discovery tools but can you also use the emerging DevOps options?

Tools that are based on Promise Theory do not do passive although this may change and CFEngine comes close already, their goal is to implement a representation of its’ desired state from a policy. You guessed it Chef and Puppet are really based on this principle so you have a challenge.

Discovery using traditional tools uses approaches that are not popular in the enterprise; nmap and fingerprinting etc. with agents so they may work but you have no certainty.

You tackle the problem but breaking down the scale. Selecting a small footprint of Unix|Windows boxes, a specific application or some other logical divide.

You must first look for, and find, patterns and analyse these so you can find the most suitable target and start from there.
The second phase is to identify relationships which link patterns, in a computing world this becomes relationships between load balancers or connecting a web server to its’ application server.

Visual / mapping tools are a great way to start this discovery if it is available but they need to be able to exploit the discovery techniques described above. A good engineer can assimilate this information using scripts, spreadsheets and the like.

Note that Discovery takes time and will delay the implementation and adoption of your DevOps tools. This time will however be a very good investment!!!

If you have no tested content ready to deploy immediately then leave deploying the agent for later unless you can benefit by low-level infrastructure data like CPU count, RAM etc. Because if you have an agent running and someone inadvertently attaches a policy then you will have issues.

Testing your automation

I was prompted to write this after enquiring why the team was applauding in the background while I was on a conference call the other day and I recognised that a deploy is still fraught and laden with unnecessary angst.

It’s time for an admission!

A long time ago I was part of a team that spent six months automating the delivery of a major platform for a client.

We automated everything, tested it and waited to deployment day.

So far so good…..

We tested every unit, tested everything in isolation, then integration tested in a range of pre-production environments.

But it went horribly wrong on release day!


Storage, security, authentication, routing, name services, timing and perhaps just poor planning or bad luck.

Actually it was a combination of these things that I should have foreseen.

To name just a few root causes.
1) firewalls
2) routing and name service deltas
3) latency
4) user permissions.

All sounds simple and straightforward but all caused problems plus perhaps 20 other things. Our unit and integration tests did not protect us.

The solution; we developed a framework called probes. Non-destructive tests, some as simple as a traceroute or ping to be run repeatedly in the weeks, days, hours and immediately before and after deployment to help determine readiness and identify failure. We choreographed the probes using iConclude, now HP Operations Orchestrator, in a matter of days and it was one of the most pleasing aspects of our project.

Anytime we needed to check status run the probe suite and validate our tests. Prior to the deploy run the probe suite and look for red alarms. No alarms then all pre-conditions were met. Run the probe at deployment time and it also raised and closed tickets in the release system and updated the monitoring tools. Deployment down from 48 hours of overtime to minutes on a normal day.

Sounds simple. It was simple!

As I look at the current state of DevOps I cannot see anything as elegant as this being developed.

I think there is huge value in this approach and perhaps people are doing it so I will continue to look and hope to see it soon.

Oh, and by the way, probes were our friend in finding problems with the solution once it had gone live.

DevOps and what you wont hear

DevOps is about Automation, the ‘A’ in the CAMS acronym so you really need to understand the following to make a success of it.

Data, Data, Data – metrics and instrumentation are key to planning, execution and understanding progress

Necessity is the mother of invention; if it ain’t broke don’t fix it <Powershell or bash might be the right answer>

Legacy businesses may be able to use existing automation tools as effectively as emerging tools for automation

Choreographing a service is more important than automating a server

Redressing the failures of historical behaviour is hard nee impossible; tread carefully with legacy

Open source changes rapidly; keeping current needs a process

Workflow ideas are emerging but confused in the Configuration Management community

You write positive actions when building something. Removing something needs a positive action too. Is an immutable infrastructure practical for you?

Physical infrastructure is harder than virtualisation which is harder than cloud

Community doesn’t replace your own expertise

It is all about infrastructure as code – consider your need for a UI and how your user experience will benefit | suffer

You can test from the inside out or the outside in; which is better is yet to be decided

It Started Off As A Post On DevOps In The Enterprise

As the title says; I started writing this post to explore the suitability of DevOps practices and the Enterprise but then realised that the question that needed addressing first was a little different.

What should the Enterprise expect to gain from adopting DevOps and how should they go about it?

IT departments in the Enterprise are increasingly adopting tooling that is associated with DevOps but are they adopting the practices that will yield future benefits?

Let’s not forget that DevOps is an idea that extends on the themes of source management, automation and collaboration across a number of teams involved in architecture, development and operations.

So DevOps is quite complicated and like ITIL, 6Sigma etc. DevOps is not immediate. You do not become an organisation capable of functioning in a DevOps manner overnight, it is something that takes work to adapt to and adopt. While adopting the practices of these processes will improve future performance when done properly it is about future performance. The controls, tools, policies and learnings do not affect what has gone before and may have been considered ‘finished’.

For future services the benefits of a DevOps approach should be consistent with the early proponents and therefore easy to categorise and quantify. But in the enterprise how much of the budget, effort and intellect goes into maintaining the status quo and how much into future?

If 80% of the IT budget goes into maintaining services then the benefits will not be seen until technology refresh occurs, this does not necessarily mean replacing the service or even the hardware but it certainly does mean change.

You have the (technical) debt from previous activities to deal with and technical debt in the enterprise is significant by any measure!

So in the enterprise I summarise that the benefits of a DevOps approach are there in the long term and the key to gaining early value is closely linked to your approach to maintenance and how well you have delivered your platforms historically. I will explore this in future posts.

In the short-term the costs will exceed the benefits. Clearly you may be investing in tools and training and if you are wise you will also perform some [process] integration / automation on top of the relatively simple concepts of configuration management.

 DevOps Value Map

Once you have a ‘top team’ of proficient specialists you should look to prove your process and capabilities by taking on a project, product or deliver a component using a DevOps approach. Of course once you have one project in production which has been delivered using a DevOps approach you need to ensure that you have the organisational construct and management vision to support the service ; building this competency takes time and great leadership and understanding and we will visit this in another post.

As your competency grows look at the standard MTTF, MTBF, MTTR metrics and determine where to attack the technical debt and be prepared for a journey.

The series continues.