Welcome to the short series of articles explaining End to End process of adopting agile in a software organization.
It is not just about presenting the theory behind agile in a novel style. Rather, the intent is to focus on the entirely practical aspects of agile process adoption by any software organization.
What you will learn in this free ‘Agile Adoption’ Course:
Today, let’s start with part 1 – Reasons for Agile adoption.
While the agile manifesto and principles look extremely fascinating and practical on paper, it starts appearing equally ineffectual to produce a balance among project’s scope, time, cost, quality, and risks when materialized in a certain organizational culture, structure and environment.
Enough has been said and written for praising agile. It has been criticized equally. If we just go through the formal research studies related to the pros and cons of agile, every other scholar on the planet seems to be working on the same.
Now is the time, I believe, to make it work.
Throughout the series, we shall respect the diversity in the organizational dynamics to make Agile work for the organization productively rather than driving all other organizational components starting from its vision to purchasing the hardware for the sake of implementing agile.
Remember that development process is one of the tools to run an organization just like other tools. We cannot allow one tool to dictate and others to follow. Rather, we have to make them compatible and complement each other in the larger interest of the organization.
The two extremes of Agile adoption approach are:
1) Either philosophize the concept to an extent that makes it impossible for the process engineers to materialize the concept objectively or
2) To ritualize it enough to focus on the process more than the end goal.
We shall be discussing the approaches that lie between these two extremes and as much in the middle-of-the-road as possible.
What You Will Learn:
Agile adoption has gone through different phases since it was proposed. These phases are termed as “waves of agile adoption” by agile practitioners.
In the first wave, it was adopted by small high-tech companies which needed a shorter time to market and could not afford to sustain for too long without getting the business value out of their products.
So, the concept of MVP (Minimum Viable Product) emerged among such companies needing more for less. The development and refinement of the agile framework by smaller companies inspired medium and large sized companies to go for Agile methods in order to cut time-to-market and better customer relationship management.
Most of the companies were successful in adopting agile and had a positive impact on their businesses. That led to our current third wave of agile adoption with the left out companies trying to board the bandwagon of Agile community.
It also includes 30% of the Agile community which has been practicing Agile for less than 2 years, according to 2014 statistics released by PMI. Another survey shows that it is even more popular among smaller and medium-sized software companies with the adoption rate of almost 90%.
It has given rise to many issues since these wannabees are not as certain about their motivation behind moving to agile as the early and follow-up adopters were. So, it’s not only difficult for them to choose the right approach for agile adoption but also to evaluate its impact.
The reasons for adopting agile are the blend of common as well as unique issues each organization faces. Once you know the problem, you can plan how agile addresses them and can determine the process’ key performance indicators based on these reasons. At large, Agile has been a pretty successful process with 51% project success rate which is more than any other software development process.
Although it has been observed to be more successful in some particular organizational designs, I don’t discourage any software company to explore agile because there has been no empirical evidence of Agile’s incompatibility with any of the organizational variables. My recommendation is to give it a shot for sure but not just to follow the trend blindly and adopt agile under peer pressure. Rather, create your own problem statement that is going to be addressed by the Agile process.
Beware that if your management philosophy is at odds with the core Agile values, don’t try to make them work together without transforming the top management beliefs since this is the most cited reason for Agile projects failure i.e. 13%.
If you ask me a single reason for adopting Agile, it is certainly its proven success in the software industry recently. The infographics below clearly show that it is the most rapidly improving model among its competitors because of its flexibility and self-improvement capability.
Agile Projects Outcomes:
V & V Projects Outcome:
Waterfall Projects Outcomes:
But as I discussed, the organizations have their own reasons for adopting agile.
I have broadly classified the most commonly found reasons among different software organizations which adopted Agile successfully as a need rather than a fashion. When I say “Fashion”, it’s because the percentage of Agile companies have increased from 34% to 75% within the period of 2010 to 2013.
“Excellence is not an act. It’s a habit”, said Aristotle. In any discipline, consistency is the key to success. Odd superhuman performances do not place you anywhere. That’s why there is no appraisal for CMMI level 1 because it requires individual heroic efforts for occasional achievements.
Every startup relies on the individual heroics initially but to reinforce the same performance and attain the excellence, you need a right process. This is the single major argument for emphasizing on adopting the “right” process. Unfortunately, there is no “one size fits all” process available that can be universally applied to any organization for any project. But fortunately, you can develop one for yourself.
The dedicated resources and time required to develop a highly structured and complex process definition and continuously improving it forever is not feasible for many companies. So, they are compelled to overlook an important dimension of a production business i.e. process and relying on Adhoc practices. On average, large IT projects run 45% over budget and 7% of the time, while delivering 56% less value than predicted.
This leaves Adhocism as the direct competitor to Agility for such organizations. With respect to simplicity, flexibility, and inexpensiveness, it’s probably the next best thing to ad-hocism and certainly far more efficient. Agile keeps its principles abstract enough to be materialized in a form that best suits the needs of an organization as simple as possible. You can just take an inspiration from other organizations or projects following Agile and pragmatically evaluate what works in your specific setting.
We were jubilant when we were appraised for CMMI level 2 in 2006. Among more than thousand software companies in the country, we were among the only ten to acquire the status. We looked proud to share the achievement with our clients and prospects as if we had come across a prominent marketing tool.
It seemed all good before we realized that there’s something killing the organizational productivity as well as creativity considering its potential. We were turning into a dogmatic organization like armed forces. There was an unnecessary red-tapism in the flow of information and completion of tasks just because the focus was more on the designated roles than the tasks. People were not willing to go beyond their roles and were religiously following the organizational structure.
I am indecisive about whether the disciplined processes were causing the bureaucratic culture or vice versa but the two are meant to be as I talked to the other organizations aiming for highly structured and established processes. Bureaucracy doesn’t give you much room for experimentation, risk-taking and rectifying your errors.
For creative software development, you need a right balance between discipline and flexibility. The approach has to be more democratic than bureaucratic. As they say “Bureaucracy is a giant mechanism operated by pygmies”, high stature software developers do not like it at all. They were becoming rebellious with the defined practices which were becoming more formal with our notion of continuous process improvement and were shifting to adhocism.
It was finally decided that Agile is the only way to save them from bureaucratic processes, other than going entirely Adhoc. PMI ranks increased productivity as a fourth top reason for adopting Agile.
Almost all the software process models have transformed themselves to incorporate the change management as they realized that inevitability of changes and scope creeping in the software. But this is where the problem starts. All the traditional models introduce change management when the change becomes inescapable. But originally, they try to resist and escape the changes. The change management procedures are invoked as a last resort.
Agile with its manifesto “Responding to change over following a plan” introduced a philosophy that answered the problem of changes with using it as a primary tool to develop software rather than treating it as a constraint. It believes that software development is naturally evolutionary and there is no better way to build a software than to continuously respond to changes without trying to evade them. The software is meant to resolve a specific problem of masses or for a particular community.
There are often multiple solutions to the problem and it is not humanly possible to jump to the best solution unless there is an off-the-shelf solution available and your target is just to imitate. Here, Darwin’s concept of “survival of the fittest” comes in and the best-attempted solution prevails. PMI ranks change management as the second top reason for adopting Agile.
One of the distinct characteristics of software products is its dynamism and volatility. There’s a continuous stream of innovative ideas in the market with previous ideas and solutions becoming obsolete at a similar pace. So, we cannot wait forever to build the right product before taking it to the market.
The phase that transformed our whole process was when our traditional process was not able to get along with the frequent market demands. Every other week, we had to ship the products with changes. Planning and execution within the defined process had become unmanageable. Gradually, we had to forego our practices one after the other because of the market pressures. Top management was not willing to compromise on the client deliverables just for the sake of following a particular process. This created a vacuum that had to be filled in by another process that ensures the shorter time to market and ultimately the early returns. Therefore, you have to deliver it when the market needs it.
If you don’t reach the market early enough, you cannot convince them to buy your product just because you built it using ultra-sophisticated processes. Agile encourages you to build “Minimum viable product” for market delivery. And it has been very successful with 73% of the Agile adopters has verified that the process reduced their time-to-market significantly.
A significant dilemma of the software companies following traditional processes is that they never know before the completion of the project that they are building a “right” product. At max, you can know at the end of the iteration if an iterative model is being used. The reason being that it is only the customer who can validate the product. This is why I was in favor of the Prototype model the most before Agile made its way. We used to build a simulation of our proposed solution for the client to validate the product before building it. We used to call our prototype as “proof of concept”.
But, it proved to be the over-simplified solution of an extremely complex problem of a gap between customer’s expectations and our offering. It was not just about the challenge of creating a representative prototype but also the customer’s own indecision and fuzziness about his actual requirements at the start of the project. Then, if it is a product for mass usage rather than a customized project, the situation gets more complex.
The only way to cope up with this is to engage the customer(s) throughout the development process in order to collaboratively build the final solution. Agile manifesto is very clear about the focus on customer collaboration more than following a contract. PMI ranks better alignment of business and software development as the third top reason for adopting Agile.
Project plan, Gantt charts, Project management reports, test plans and the release plans were my favorite artifacts in the younger days. It was based on a traditional belief that well planned is half done. My belief is still intact. However, my definition of “well planned” has evolved over time. The project manager used to decide who will do what and when, based on the input and estimations from the stakeholders. Once the plan was put in place, team members had to consider it as a bible.
While it was meant for the accountability of team members, it was found that they use it as a tool to run away from the responsibility of product’s success or failure. All the team had to do was to follow the plan. They did not own the product.
It was somehow obligatory for us to shift the ownership of the product towards the team to make them accountable as well as to raise their morale. Agile holds the teams accountable for the collective goals rather than encouraging people to work in silos. It encourages the cross-functional tasks by mutual assistance. The team members with a limited or different level of skill set do not feel left out and are equally determined to contribute towards the team goals. I suggest replacing the individuals’ happiness index with team morale which is a major factor in high-performance reinforcement among Agile teams.
The sample representation of team morale is below:
Although I have used my personal experiences with the traditional processes that pushed us towards Agile, the purpose was not to get a praise at all for thinking smartly. Rather, I validated all the issues from the Agile practitioners and the process engineer of other organizations.
I have used my organization’s experiences just as a case study. Some of the issues were specific to our organization which has not been discussed in this article. But the motives I have discussed for Agile adoption are very generic in nature. If you have been using traditional approaches only, there is every chance you are facing these issues.
I would suggest taking a final decision only after discovering that you are coming across these glitches in some form. That would help you to set your direction and assess the success of specific Agile practices in your organization. Else, you can be caught in two minds at any stage of Agile adoption process whether Agile is better than your previous process or not.
Author: This series of articles is written by STH author Umair Khalid. He is having 10+ years of experience working in Software development and testing field and currently working as a Quality Assurance Manager in eDev Technologies. He is having hands-on experience as a SCRUM Master and also switching software development methodology from traditional CMMI driven processes to Agile.
Now, if you are certain and convinced that you need Agile, please wait for our next article “Is Agile feasible for you?” to assess if you have what it takes to adopt Agile in your organization.