Jonathan Thorpe Archive

Release Management bottleneckI spent the past few days at the Gartner Data Center Conference in Las Vegas.  While there, I got to show off Serena Release Manager v5, which was very well received.  It was great to see how people reacted to our way of visualizing releases.

The people I talked to at the show, for the most part, had no control over release processes.  I can only recall one person who didn’t believe their organization had release management challenges.

That got me thinking.  If others in an organization can see obvious release management issues (often significant), why aren’t they being addressed with more urgency in many organizations we talk to?  I’m guessing it’s because release management issues span multiple teams and that makes them hard to solve.  It’s easier to ignore the issues until a major problem arises, such as the Knight Capital incident.

If you decide to address your release management problems head on, how do you know your initiatives have an impact?

While CAMS is associated with DevOps, it is essential to helping improve your release management processes.  When demoing Serena Release Manager v5, I was easily able to demonstrate how we addressed all aspects of CAMS.

DevOps Drive-InDevOps is starting to attract mainstream attention, but there is still confusion about what DevOps is and what it takes to have a successful DevOps initiative. Some companies have tried to automate their way to DevOps success; many others are trying to stay true to the core DevOps CAMS (Culture, Automation, Measurement, Sharing) principles.  Unfortunately, most information available is related more to web 2.0 companies than more complex enterprise environments.

These stories are starting to emerge with more enterprise-friendly content and are being presented at DevOps-related conferences. Unfortunately, many organizations can’t afford to wait for lots of successful enterprise DevOps use cases to be published.  Although admittedly, they might not know that they don’t have time!

DevOps bubbleI’ve just spent an amazing week in London. I already had high hopes for this trip having been to DevOpsDays London earlier this year but I didn’t expect to be blown away by the events.

Attendees at both events were from all over Europe. I’m used to meeting people from different cultures and backgrounds at events in the USA but this was on a totally different scale.

As to be expected, culture and process were areas of pain that people are having a hard time dealing with. Tools are one of the easier things to bring into an environment. At DevOpsDays John Willis did an excellent presentation on Software Defined Networking. While this isn’t something I would mention, there was one thing he said that was really important. A lot of his talk was about tools and technology. John ended by reminding all of us, including himself, to continue to do CAMS not AMS. It’s too easy for those of us who are really passionate about technology to fall in love with tools and forget about culture.

There were many other great presentations and open space sessions at DevOpsDays. There was an excellent ignite session on the topic of Burnout in the industry. Based on what is coming from my Twitter feed, there is a lot of discussion about burnout and depression in the industry. It is healthy that this is being brought up for discussion. Although I didn’t get to attend the open space session on the topic, I’ll make sure I do at the next conference I attend that discusses it.

The next conference was Velocity. I’ve been to a few Velocity Conferences and this was the best yet. As someone who is extremely passionate about technology, some of the tricks and tips for getting extreme performance from websites and apps were extremely exciting. I’ve not played with web development for a while but I came away from the conference wanting to find reasons to try out some of the techniques demonstrated.

As much as the technology itself is exciting and important, time after time culture and process are highlighted in sessions, especially around accepting that failure will happen. I firmly believe that Serena’s Release Management solution can go a long way to help you tame your processes and help facilitate collaboration. Over the coming weeks we’ll be writing more about Serena’s process-centric view of release management, which can help you transform the way you work.

Finally, there were a couple of sessions on DevOps and continuous delivery in financial companies. I’ll write about those once I have a chance to review the slides and watch the sessions again. They were very exciting sessions and are great examples for Serena customers who would really like to change the way they work in risk averse environments but don’t know where to start.

devopsdays-london-logoDevOpsDays London is over and, as usual, there were some great sessions. Exposing fragility in systems was once again a topic that was discussed.

If you follow the DevOps movement at all, you will no doubt have heard of chaos monkey at Netflix numerous times. For those who have not heard about it, chaos monkey runs in all environments, including production environments, killing processes in order to test how the environment responds to failure. This has led to a loosely coupled, fault tolerant system.

While most companies aren’t able to inject failures on a scale that Netflix does, that doesn’t mean that others aren’t embracing this approach.

There was a presentation on how Pager Duty has “Failure Friday,”  where they introduce failures into their system. This allows them to test not only how resilient their software is but also how good their monitoring, logging and processes are.

Are you able to inject failure into your systems and respond to them without your application going down in flames? Whatever the answer is, it might be a good idea to have your own Failure Friday and learn how to cope with failure.

Tags: DevOps

PDX DevOpsDays logoLast week I was at DevOpsDays Portland.  As expected, the open space topics were really interesting.  Two in particular stood out for me.  The first was about fitting DevOps into your organization.  DevOps teams seemed to be the way people were going.  If you have been to other events with prominent figures of the DevOps community present, you may have noticed how the mention of “DevOps teams” can spur heated debates.

I believe organizations are opting for DevOps teams because it side steps the cultural aspect of DevOps.  I’m torn on this because this could lead to one more silo that will make matters worse.  However, there is a valid case for DevOps teams, as discussed by Jez Humble.  I got the sense that there will be silos built that hopefully will be torn down and the culture of DevOps fully embraced.

The other open space session that got my attention was about DevOps in the enterprise. There was a lot of discussion around tools.  Participants realized that process and being able to model processes in tools are important.  I don’t go to DevOpsDays events to pitch Serena products, but I was asked to describe what Serena does for managing DevOps processes.  At Serena, we take a process-centric approach to release management.  We understand that processes in the enterprise are complex, need to integrate into many different systems and need to provide robust auditing and reporting capabilities.  The fact that Serena has been helping model processes for many years in some of the most complex environments with heavy regulatory requirements seemed to resonate.  Add to that Serena’s investment in release management and DevOps and I believe we have an exceptionally compelling solution.

Serena has a monthly DevOps Drive-In webcast series.  This month on November 21 at 8am PT we have Richard Michaels from Eaton Vance discussing release management at Eaton Vance and how Serena’s solutions helped tame their release management process.  You can register for the webcast here.

Last week I attended FlowCon in San Francisco. Unlike DevOpsDays and Velocity conferences that I have been to FlowCon had an enterprise focus. Early on in the conference there was a presentation by a typical DevOps poster child. I had a sinking feeling, was I just going to sit through another presentation about all of the cool stuff that is done in Silicon Valley does but doesn’t resonate in the enterprise?conveyorbelt

The good news is that this wasn’t the typical story that I have heard so many times. While a lot of time was spent on uptime and scalability, the presentation covered segregation of duties. That’s not something I had heard before and I got a bit excited.

This online company has compliance requirements, just like banks, insurance companies and other verticals that are interested in DevOps and Continuous Deployment but has doubts that it can work in their environments.

According to the presentation the company is able to do Continuous Deployment because they don’t treat all servers in the same way. One of the benefits of service oriented architecture is that services requiring different levels of control and compliance can be handled separately to services that have no such compliance issues. Services that need to be handled in a more restricted manner have restricted access and changes aren’t pushed out automatically. Services that don’t need to be so strictly controlled can be updated many times per day using Continuous Deployment.

This approach was startlingly obvious once I heard it. If you are reading this and would like to take steps on the road to Continuous Deployment or Continuous Delivery and you thought you couldn’t due to compliance issues, could you do something similar? You may not have applications and services that have a service oriented architecture but perhaps it is possible to isolate parts of your codebase so that you can make changes quickly with minimum fuss in some parts of your application.

Continuous Deployment doesn’t have to be all or nothing for your application. If you can deploy quickly to parts of your application and it adds value to your customers then why not give it a try?

For more about continuous delivery, register for our next DevOps Drive-In Webcast, “Enterprise DevOps: Implementing Continuous Delivery with Serena Release Manager.”

volcano cloudAn excellent blog post on the events that led to Knight Capital’s bankruptcy in just 45 minutes was forwarded to me.  Most of us in the industry have heard the story.  It has been used either by vendors as an example of why tools and processes are critical or by others as an example of how things can go very wrong.

With the release of the SEC report, we now have more information than we had in the past.  Looking back on my career, I can see how some of the events that occurred could quite easily have happened to me or other people that I know.  Fine examples are:

  1. One server in a cluster not being updated.
  2. Developers reusing a flag that was no longer thought to be in use.
  3. Dead code coming back to life.
  4. Unusual messages from an application being ignored.

As I work for Serena and I’m writing about this, it shouldn’t be a surprise that we have tools that can address some of the issues that led up to Knight Capital’s bankruptcy.  Serena Release Automation (SRA) can be configured so that all servers in an environment are modeled in SRA, ensuring that code is deployed to all servers.  Serena Release Control can model the processes and approvals necessary to ensure that all turnovers have been completed and so on.

The clear message to me, though, is that tools alone can’t solve all release problems. At Velocity NYC, I attended a great keynote by Zane Lackey of Etsy and Dan Kaminsky, a well-respected security researcher.  They talked about many aspects of security, including zombie code, which is code that you long thought dead in your codebase but is still accessible.  There is a lot of old code that is still active, just waiting for something to trigger it.

As pointed out by Zane, the good news is that for web apps, detecting zombie code can be quite easy and done by analyzing logs that you already have.  Plus, once you have good release management processes and tools in place, including deployment automation, it is relatively easy to include such checks into your automated release process.

How does this relate to the Knight Capital incident?  Engineers at Knight Capital repurposed an existing flag.  This is something I would not suggest doing.  A new build that used the repurposed flag was installed to a cluster of servers.  However, one server was missed. This led to all servers, except one, working as expected.  After some troubleshooting, the code on the working servers was rolled back to the previous build.  While this led to a consistent set of servers, the repurposed flag was still set, resulting in code that should not be executing to be executed.  The rest is history.

There are many, many lessons that can be learned from this (not leaving unused code in your codebase being one of them).  Another is that an important, robust release process is a necessary part of keeping your business healthy.

DOD-VancouverI’m at the first ever DevOpsDays Vancouver, Canada. The event came about from a casual conversation at DevOpsDays Austin and it’s been a fantastic success.

The sessions have been great. I really enjoyed the session from Hootsuite on how they are able to deliver working code faster by using vagrant as part of their tool-chain.

Brian Johnson delivered a great ignite session that was basically a warning not to let DevOps go the way of ITIL because the intentions of ITIL were very close to DevOps before it changed over time. I’m not going to attempt to deliver Brian’s message. The full length version of the presentation that Brian delivered at DevOpsDays Atlanta was amazing and is well worth watching.  It can be viewed here.

The open space sessions were very informative as usual. I proposed a space on doing DevOps in a Microsoft environment. The session had surprisingly good turnout and proved to me that the Microsoft ecosystem is still alive and kicking in the Vancouver tech scene. As good as the Microsoft developer ecosystem is, currently people have to rely on many other solutions to fill gaps in the Microsoft tool-chain. Unlike conferences in the U.S. that I have attended that have a very positive impression of PowerShell as the scripting language of Windows, opinions were much more negative here. It is clear though that in many cross platform open source projects on the DevOps space, Windows is still an afterthought.

For the first time at a DevOps event this year there was no mention of “The Phoenix Project.” DevOps culture conversations are certainly easier and more fun when you can talk about Bill, Brent, Erik and others.

I’m looking forward to next year and also hopefully a few DevOpsDays events elsewhere in Canada.

Tags: DevOps

Kanban boardRecently I attended a workshop hosted by Dominica Degrandis on using Kanban in an organization working in a DevOps-like manner. The workshop was full of practical advice not only on why Kanban can help you but also on how to be successful in your implementation. Attendance was good; people came from the east coast to attend what I thought was a local workshop.

As we discussed things that we had each tried in our organization, it became apparent that some of us had worked in a stealth Kanban-like manner. In my case it was monitoring the throughput of tasks and making sure there was slack in the system so that my team could respond better than if we optimized for efficiency.

I was most looking forward to Dominica’s DevOps game and wondered what insights playing the game would lead to. In the game there is a Kanban board with columns for exploration, dev, QA and ops tasks. There is also a lane for handling tasks that needed to be expedited. Without giving the game away, money was earned for completing tasks. The game wonderfully illustrated how focusing on features and ignoring technical debt was harmful and also showed how optimizing for throughput rather than efficiency yields better results.

Overall, the workshop and the game illustrated the value of thinking holistically rather than as individual units in an organization.  It is a more effective way to work. Doing the opposite can result in decisions that are “locally brilliant and globally stupid.” I can’t take credit for that wonderful phrase.  I got it from Glenn O’Donnell from Forrester, but it is so very fitting for both decisions I have made in the past and ones I have seen others make.

I’m not saying that breaking down silos and thinking in terms of the entire system is easy.  It most certainly isn’t, but the game simulates the benefits in a way that people can relate.

I’m looking forward to hearing Dominica speak at Flowcon on November 1st.

velocity-igniteThis is my first time attending the tutorial day at a Velocity conference. It’s a small world as I’ve already met someone I met at DevOpsDays Atlanta and a couple of people I met at a Kanban for DevOps workshop back in Santa Clara a few weeks ago.

The first session of the day covered using Jenkins in operations. I’m a big believer of using the right tool for the job so alarm bells were ringing in my head when I read the session description. This meant that it was a must see session for me.

The session didn’t disappoint. It was a good session but I didn’t agree with the use cases. It was a great example of “when the only tool you have is a hammer everything looks like a nail.”  Jenkins excels in managing builds. Look at the UI how often is build referred to? There are better solutions for ops runbook automation.

The next session I went to was about using Sensu for painless metrics and monitoring in the cloud. I was very impressed with the preparation. A VM was provided for people to use when working through the examples.

The afternoon sessions were informative, although more in line with regular conference sessions than tutorials.

The day wrapped up with ignite sessions. Somehow I ended up presenting and had a lot of fun. The great thing for the audience is that the range of presentation topics varies wildly and yesterday was no exception. As my ignite was the least serious of the proposals I got to go last and presented Game of Thrones: DevOps Edition. Fortunately the audience enjoyed the topic and there were a fair few Game of Thrones fans in the audience. I’ll admit that presenting ignite sessions is becoming something I am starting to enjoy.

If you enjoy getting information in an entertaining format then I encourage you to attend our October DevOps Drive-In webcast which is a DevOps panel starring Damon Edwards from DTO Solutions and Dave West from Tasktop. Damon and Dave participated in an extremely entertaining panel at the Serena xChange13 user conference that was very well received.  You can see the webcast here.