Fear of bugs is worse than the bugs themselves!
Bugs cause fear and fear causes bugs
What happens when fear(some may call it being safe) plays a significant role in your software delivery strategy?
Talking of software delivery, let’s first revisit the DORA metrics, which measure software delivery performance.
Deployment Frequency (DF): How often an organization successfully releases to production.
Lead Time for Changes (LT): The time it takes a commit to get into production.
Time to Restore Service (TRS): The time it takes for the team to recover from a failure in production.
Change Failure Rate (CFR): The percentage of deployments causing a failure in production.
When management by fear becomes a de-facto software delivery strategy, all your software delivery measures are affected negatively.
Let’s see how!
Fear causes delays:
Typically, fear stems from failure. Moreover, the bigger the visibility of the failure, the worse the fear. I believe that is the very reason why processes related to security kick in first. Security failures tend to be the most harmful and most visible.
Many companies slow things down once an incident happens, intentionally or unintentionally. Because of fear, one of the following happens.
Introduction of New Processes:
New processes are introduced to slow things down. Here are a few examples
Lengthy change approval processes require approval by PMO, CAB, etc. Lucky if you are unaware of these acronyms, let’s keep it like that.
Issue life cycle processes do not allow tickets to move easily unless they pass a specific prerequisite level. It’s like levels in games: pass each level sequentially to get to the rewarding last level. The change management process may look like the following once companies become process-heavy.
Manual compliance checks require stakeholders to hop through multiple checklists manually.
Bad Behavior
These processes cultivate a behavior where extreme ownership takes a backseat. Sticking to process is considered safer than innovation or changing the rules of engagement because when you stick to processes, and something fails, it’s not you; it’s the process. Such behavior creates additional processes.
Another example of bad behavior is playing volleyball with change requests and passing the ball on the other court. This behavior creates massive delays because change requests spend most of the time waiting in a queue somewhere!
Introduction of Quality Gates
Quality gates are automated or manual gatekeepers to decide whether a change makes it to a release or not. Many of these go against the DevOps mindset of automation.
These manual gates come in all kinds of acronyms like Change Advisory Board (CAB), Schema Review Board (SRB), Architecture Review Board (ARB), Security Architecture Review Board (SARB), and Security Compliance Board (SCB).
These gates delay delivery without adding much value (here I go, waging a public war)
Delays have consequences:
Batching
Every time where there are delays, developers try to batch work. Imagine a CI/CD pipeline where the build takes 80 minutes. In such cases, developers benefit if they batch multiple items together to release them in one build.
Or when the release cycle is intentionally made slower by releasing only, let’s say, once a month. As a result, many unrelated changes are batched together. Let’s see what happens because of that.
Work in Progress(WIP) increases, developers work on many items in parallel, and many of them may sit in the pipeline in a “Ready for Release” state. These items may be 99% done but not released yet, which means work done has not yet made any impact.
Bugs are introduced to code because of context switching, delivery pressure, integration delays, etc.
Deployment Frequency is lowered either as a policy or as a function of all the delays.
Mean Time to Resolve increases because once multiple changes are batched, troubleshooting becomes more challenging, and so does deployment revert. Also, as your deployment frequency reduces, so does your expertise at it, making time to resolve even worse when you need to deploy a fix.
Does this mean all quality gates are bad? Not really. Many automated tools like Sonar, Snyke, and CodeScene do a great job of introducing automatic quality gates. Use them carefully, as they too can introduce delays in the development workflow. Maybe treat the pursuit of quality as a cultural trait instead of finding technology or a process that layers existing ways of working.
In conclusion, your fear may be real, but don’t let fear drive your software delivery strategy.
has defined courage very well in XP Explained.Courage is effective action in the face of fear. -- Kent Beck, XP Explained 2ed
When in fear, measure twice and cut once. The actions you take in the face of fear must be effective and measure their impact on DORA metrics. Stop thinking of velocity and quality as a dichotomy. They do not have to be at odds with each other.
I would go deeper: software delivery strategy is more and more driven by fear of failure. It really doesn't matter if it is a fear of bugs, defects, missing deadlines, product metrics etc. We need a room for failure to learn and drive the change. Instead of that we get (or set) safety checks or bureaucracy bu...t in form of processes i that keeps us in the comfort zone that cause only lack of progress and stagnation.
Defect vs Bug