In this 7-part series we’ll look at some common misconceptions about the process of deploying software in today’s unforgiving world. Over the next few posts we tackle these myths head on and show how there is a better way.
Releasing software into the wild is exciting and terrifying. When it goes well: we party. When it doesn’t: we spend the weekend without sleep, showers, food or sleep. Wait! Did I mention no sleep already?
Too often the reason our deployments fail is because we fall into the same traps over and over again. We never have time to step back and do it right so we keep on doing it the best we can and that is where the errors creep in.
Here are five common traps we fall into that are easy to avoid and inexpensive to solve.
There’s your whole problem. It is true that what is being deployed is (or at least should be) different each time you deploy but how it is deployed needs to be standardized and familiar so it becomes repeatable and predictable. Each time you update an application it likely that it has the same topology, the same dependencies, the same footprint and the same risks.
This is a common problem too. In the Internet of Things every device on every hip, in every pocket and buried on our phone is a unique configuration of versions and patches and operating systems. Every server has custom settings and distinctive considerations that need to be accommodated. It is beyond any human’s ability to track and manage the discrepancies amongst so many deployment targets. But yet we try with our spreadsheets and notebooks.
On a Sunday morning, at 3:00 am, no one wants to modify the 27 deployment scripts and the 81 server config files or follow the defined procedures to stop the 14 databases and quiesce the 8 transaction queues just to make a simple change. A skilled developer can do what’s need by writing a simple PERL script, right? But on Monday morning, at 8:00 am, no one wants to explain to the CEO why overnight trading in Tokyo and Hong Kong was down either. In many ways emergency fixes have more risk because they are usually developed under pressure, tested less, by pass approval levels and get deployed in whatever way seems quickest. There is too great a temptation (or expectation) to do what is expedient over what it right because the right way is the long way. Yet we all know the automated way is the right way and the fastest (and safest) way.
I remember sitting in a meeting room. Outside I could see two developers peering at a screen and nodding their heads up and down in an erratic and random manner. I asked the client what was happening and she explained they were deploying a release and “watching the script go by in case something bad happened.” If a deployment fails many release engineers will roll up their sleeves and start to unpick the changes and the rest will try to fix it on the fly and keep going. This is why deployments need their release engineers to be close during the process. But what are they going to do if the deployment stops (or worse flags and error and doesn’t stop)? They have to find the problem and decide “fix forward” or “backout”? They have to then work out which is best and easiest, quickest, safest to do. If this was built right into the scripts from the beginning they could devote their energies to finding out why it failed and fixing that.
There is a sense that errors are inevitable. That something will always get missed and we can correct for that later. If we are agile and iterating in small increments the risks are low and the impact minimal. In our hearts we know that is bogus. One change to one configuration line in a server can stop it from executing and bring our entire system to a halt.
In the next post we will address these issues and look at what is needed to be effective and efficient in deploying software. As this series continues I would welcome your thoughts and experiences. You can add them right here.
Next week: why neither deployments nor target platforms unique.
|Kevin Parker is a 30 year industry veteran, holder of three technology patents and is VP of Worldwide Marketing at Serena Software. He speaks and writes on application development methodologies, business analysis, quality assurance techniques, governance, open source issues, and tool interoperability, from the mainframe to distributed platforms to the web and mobile and embedded systems.|
what about following a strict SDLC – if this process is rigorously followed you establish #1, correct for #2, avoid #3, relieve #4 and eliminate #5.
Great point Christopher: you are exactly right! The key word in your note is “strictly”. How can we follow the SDLC strictly without automation? It requires too much diligence on the part of the PMO and individual heroics on the part of everyone else if we try to adhere to the SDLC through checking boxes on forms and filling in cells on s spreadsheet. If we use automation to execute the steps of the SLDC we are much more likely to apply the constraints the SDLC expects. And, when we have to update the SDLC, we will get 100% compliance with the new process from the automation whereas humans take a while to adapt and adjust to the new methods.
[…] is the conclusion of a 7-part series. Read part 1, part 2, part 3, part 4, part 5 and part […]