“We are Agile. We don’t need to follow a process”, declared Chris.
Tina was aghast. “How do you think we deliver solutions, Chris?”
“What do you mean? We do Agile. We follow Scrum. We don’t follow processes like the Waterfall guys do. We neither need a whole lot of planning nor documentation. We don’t do BDUF (big design upfront). We love to code and that’s what we do”, responded Chris.
He would have continued, but Tina interrupted it.
“Ok. Let’s get this straight – you think if you follow the Scrum practices to develop software, you are agile?”, she asked. She decided not to pursue the “do Agile” line of questioning.
“Yes”, responded Chris.
“And you think if you follow the Scrum practices, you are not following process?”, she asked.
“Yes”, asserted Chris.
“And you think neither planning nor documentation is essential?”, Tina pushed along.
“Yes”, said Chris.
“Fair enough”, said Tina. “Let’s look up the definition of process, shall we?”
Without waiting for a response she typed “process definition” in her browser and in 0.68 seconds Google responded with 606 million results.
“A series of actions or steps taken in order to achieve a particular end”, she read aloud. “I think this applies to your development efforts. I mean, you are going through a series of steps to develop your software, aren’t you?”
“Yes”, said Chris unsure of where this was going.
“So Scrum does have a process. It is just different from the other development methodologies”, said Tina. “At the core all development follows a similar process – you need to define what you want to build, you need to design it, you need to build it and deploy it into production. It’s just the how that differs.”
Chris looked thoughtful. He couldn’t disagree.
“Since we have that out of the way”, she continued, “let’s talk about planning. How do stories get created? How does one decide how many sprints are needed? How can the support team help prevent disruptions if there is no documentation?”
Chris knew when he was cornered. He knew the Product Owner and the Scrum Master met weekly to groom the backlog and once stories were broken down, the entire team would sit down biweekly to estimate them. He also knew that releases were planned. He also knew on certain projects, they dedicated the first sprint to figure things out at a high level. “Envision” was the word that Rob had used. He remembered Rob saying, “We envision the requirements, architecture, design in that first sprint. It helped with regulatory projects.”
“You know, it might be a great idea to map out your agile value stream”, continued Tina.
“Value… what?”, sputtered Chris.
“Value Stream”, said Tina. “It’s just a fancy word used by businesses to describe the process that is used to create and deliver physical goods, software, solutions or services. A pictorial representation of a value stream is called value stream mapping. The great thing about a value stream is that you can see both the flow of software from idea to delivery along with all the feedback loops.”
“How will that help?”, asked Chris.
“A value stream map helps you visualize how a feature or a story makes its way from start to finish. It also shows how long an average journey takes. It can show you where it waits and where it goes fast. Moreover, it helps visualize what information is needed at each step of the process”, answered Tina.
“What will I do with that information?”. Chris pressed on. He was never fond of these business terms. He eyed them all with suspicion.
“I will”, responded Tina. “But first tell me, what determines how long it would take to get work done?”
“Why do we care about that?”, asked Chris. “We onboard what we can do within our sprints and try to deliver working software in that time frame. We’ll get features done when it gets done. ” He was beginning to be convinced that all this value mumbo jumbo is used to distract him from getting real work done – code.
“Unfortunately business does not work that way”, said Tina. She continued, “Customers need to know when will we be able deliver solutions to them and what it will cost them. That’s reality. Moreover, if you have to provision hardware, work with vendors or with a number of teams you now add more complexity to the delivery process. That’s the reason why agile methodologies insist on having a self-sufficient team. The moment your team does not have all the skills needed, you are now dependent on factors outside your control. These create risks to delivery and if not managed well, it can extend the duration and potentially costs. In fact, they can even derail your sprint or release goals.”
Chris leaned back thinking about what Tina was saying. It was true. In fact, their team did not plan to onboard work into their sprint until the preceding dependent work was completed. He never bothered to find out how long it took the feature to be delivered end-to-end. It did not seem necessary.
“Well”, he started, “the time it takes to develop a feature depends on how long we work on it and the amount of time it takes for others to deliver their work to us.”
“Excellent”, said Tina. “You are on the right track. The time it takes to deliver a feature is a function of the actual time you and other teams spend working on it and the time work waits. Why do you think work would wait?”
“Handoffs”, responded Chris immediately. No matter how well one tried to manage it, work always waited for people when there were handoffs.
Tina wrote this down
“Now you understand why the thought leaders of the agile/lean community insist on self-sufficient teams. Self-sufficient teams eliminate hand-offs. In such cases, the total duration is only a function of the actual effort and any wait within your team. That’s totally under your control”, said Tina with a flourish. “Traditionally, people have always worked to improve ‘estimates’. While important, they neglect to do anything about the handoffs which is where you get the biggest improvement to speed of delivery. If one were to drill down deep enough, they would see that every prescribed agile ritual is geared to deliver what customers value the most, eliminate handoffs and speed up the feedback loop.”
Chris blinked.
“Don’t get me wrong. There are other benefits that you accrue. But the primary improvement to speed of delivery comes from eliminating handoffs. Duration improves, quality improves since all work is contained within the team and we are not dependent on communication or lack thereof outside the team. More importantly, the team delivers what’s most important to the customer.”, said Tina. She continued, “For example, the intent of the daily Scrum is not a status meeting which is what it often turns out to be. It is a quick way to identify if any work is stuck and needs help. Without discipline, this ritual can quickly lose its value. Instead of asking individuals in the team, ‘What did you do yesterday?’, ‘What are you planning to do today?’, ‘Are you facing any bottlenecks?’, try looking at work in its various steps, identify how many days has it been waiting in each step and tackle the ones in there the longest. Shift the focus away from people to work. And that’s where the value steam map can help”.
“Wait a minute”, said Chris. “What about the other rituals? How do they help reduce the duration?”
“Well, let’s look at the retrospective ritual. What do you do during retrospectives?”, asked Tina.
“Our retrospectives and sprint planning are combined together”, answered Chris. “We look at the velocity if it is more or less close to our historical one, we really do not spend a whole lot of time. We talk about what worked well and what did not work well. Then we move on the the planning.”
“So what happens to the ones that the team says did not work well? Are there any action items to dive into the root cause?”, asked Tina.
Chris thought about this. It was true they measured velocity of the team and used burn-charts to track sprint commitments. But often, they would swap out one work for the other, or inject other work into the current sprint. While they tracked their velocity number, they did not track the value of the work delivered. Since no one was screaming about the work delivered they just assumed value was a given. When they were stuck, it was usually because requirements were not clear or they could not envision what was needed based on description. They then had to wait for the Product Owner – usually for about an hour or so but sometimes can be a day. But he did not have the data to show if this area needed the most improvement. He also knew of other teams who could not finish their sprint commitments and unfinished work was pushed into the next sprint.
While the team recorded what did not go well, as far as he knew there was no action item or any improvement action items. When there was a major issue, management got involved and they’d go through figuring out what went wrong, but at other times they did not do any of that stuff.
Chris was beginning to get a sense that there was something more to Agile than just following the prescribed practices.
Tina was content letting him mull through what she said. “You say we can map our Agile process?”, asked Chris. He winced using Agile and Process in the same sentence. “It will help us figure out where our bottlenecks are so that we can eliminate it and improve our delivery speed?”, he continued.
“Yes”, said Tina. “You see, it was never about Waterfall or Scrum or TDD or Kanban or RUP or any other methodology. It was always about identifying bottlenecks to the flow of work and taking steps to eliminate it. People just went about different ways of achieving this based on a number of factors: transparency, respect for people, need for control, trust, organization culture, knowledge, education, their core values”, answered Tina.
Chris shuddered as he remembered his waterfall days. “All right, let’s get this value stream thing done”, he said. He was now curious to learn how this value stream mapping will help him and his team.
“In order to create the value stream map, it helps to ‘walk the process'”, said Tina walking to the whiteboard and picking up the marker. “What I mean is that let’s assume we are the user story. As a story how am I created?”, she asked. Chris stared at her wondering what he had gotten himself into. Walk the process – seriously!
“John, our product owner, creates them”, he offered.
Tina wrote down “Create User Story” on the board. “What happens next?”, she asked.
And they went back and forth with Tina asking the questions and Chris providing answers to what he thought was the process. Tina kept the value stream map at a high level. After about 10 minutes, this was the result.
“That was easy”, said Chris.
“We’ve only just begun”, said Tina. “We could go a level deeper, but for the sake of understanding let’s keep it at this level. On to the next step. What we need to do, is on average tell me how much time it takes to complete each step. Do not include any wait times. I just want an estimate on the work time.”
She wrote “Value Time” on the to the extreme left. She guided Chris in determining the estimate of the value time for each step. Here is what it looked like after 20 minutes.
“Now let’s add up the numbers”, said Tina. “Don’t add the ‘Fix issues if test fails’ yet”, instructed Tina.
“1583 minutes or 3.4 days”, said Chris.
“The next step is to now identify the wait times”, said Tina. And just like before Tina guided Chris to arrive at an estimate.
“17 days”, said Chris without being prompted this time.
“Do you see, it takes 17 days in order for about 3.5 days of work to be completed”, said Tina. “And we did not even consider any wait time within each step. For example, if you had any questions around the feature as you began developing the story, you would have to wait for the Product Owner to answer your questions. If the Product Owner is available, then there is no wait time. But if the Product Owner is not available, then that story would have to wait. What would you have done when such a situation arises?”
“Oh, that happens all the time”, said Chris. “If we can’t make assumptions based on past experiences, we just put that work aside and grab something else to do until we have our clarifications.”
“Now you are context switching, and there is a cost to it”, said Tina. “You now have to shut down that project and maybe open another one. You may also have to switch to solving another problem. The more work you have on your plate, the chances are you are context switching more often. This often extends out the duration of all the work that you are context switching on.”
“So should the goal be to deliver 3.5 days of work in 3.5 days?”, asked Chris frowning. That would be very difficult to achieve. We’d have to hire a lot more people to be able to achieve this.
“It depends”, answered Tina. “There is a significant cost to doing so. You also have to remember that we just estimated these numbers. We don’t yet know if we have a problem. Maybe it’s ok for 3.5 days of work to be done in 17 days. Then again, maybe it’s not. We are now in a position to hypothesize. For example, our hypothesis could be, ‘our development team is taking too long to complete work.’ We need to gather some data to back up this hypothesis.”
“What kind of data”, asked Chris. He was beginning to get that sinking feeling again. Tina was probably giving him more work. It’s hard enough to meet sprint commitments. He did not want more work.
“Cycle Time and Lead Time”, answered Tina. Seeing the question mark expression on Chris’ face Tina continued, “Cycle time is the time between start of work to completion. In order words, it is the difference between the completion timestamp to the start timestamp. Lead time on the other hand is the elapsed time between when a request was made and when it was received by the customer. Customers does not see Cycle Time, they only see Lead Time.”
Tina asked, “We already use an electronic board to track our work, don’t we?” Chris nodded. She continued, “What are your column headings on your board?”
“Sprint backlog, In progress, Testing, Testing failed, Done”, answered Chris. “The sprint backlog contains all work that we onboard into the sprint.”
“Excellent!”, exclaimed Tina. “Is there a way to record the time on each card when it is added to the “in progress” column. Similarly, is there a way to timestamp the card when it gets added to every other column in your board?”
“I am sure we can do that”, said Chris. “We do not have to do anything, the software we use to track our work does it automatically.”
“Maybe we can do some retroactive analysis then. We may already have the data to validate or disprove our hypothesis on value time and wait times”, said Tina. “The value stream we’ve documented so far just shows the forward flow; i.e. flow of code through the process steps. We have not yet delved into the information flow.”
“Information flow?”, inquired Chris.
“You see value stream mapping is a Lean tool that enables you to “see the process” as it currently exists. When used in a manufacturing environment we include Material Flow, Inventory, Buffer Stock, Material Transport, IT systems and information flow in the value stream”, said Tina. “But we are not a manufacturing company. Hence, we need to adapt it to our needs. We’ve talked about the flow of code, but we have not touched on the information flow which is critical.”
“How?”, asked Chris.
“Think of Scrum as a batch process. Your batch size is dependent on the length of your sprint. You try to take on only so much work that you can complete within your sprint. As work moves from the left to the right, your information needs to move from right to the left as a feedback loop. The smaller your feedback loop the faster you can make adjustments and the better your quality will be. The second, and probably the most important point for software engineering is that you need to have the right information available at the right time before you begin coding. Without this, teams waste a lot of time figuring out the requirements after coding has begun”, said Tina.
“It happens a lot”, said Chris. “We begin coding and then figure out the requirements along the way.”
“I bet a lot of these items come back as bug fixes once tested”, asked Tina. Seeing Chris nod, she continued, “That is expensive. More importantly, the organization has lost capacity. The time spent in fixing bugs could have been better utilized in developing additional features or retiring technical debt. This waste of time could have been prevented by having the requirements nailed down. After all we are not talking of requirements that will be built months down the road. We are talking of requirements that we are about to start building.”
“Happens all the time”, said a voice. So focused was their conversation that they failed to notice when Matt arrived. “Sorry to interrupt”, he said. “I could not resist stopping by once I saw the value stream map. Simple, but effective. Tina, I am about to start working on a project where we need to comply with a few regulations. I would love to have something similar done for that. I think it will help.”
“Thanks”, said Matt. “I’ll set aside some time for this conversation. Got to run. Thanks.”
“Chris, look at the time – it flies when one is having fun. Do you think this conversation was useful?”, asked Tina.
“Absolutely”, said Chris. “Agile is lot more than just following the prescribed rules. I am going to read up on this value stream thing and see where else within our team I can apply.”
“Great!”, said Tina. “I’ve got an assignment for you then. First send me the six months of data on your team’s work. Specifically, for every work type send me what date it was changed to each stage in your workflow. I can help with some data analysis to figure out your team’s cycle time. Second – take the value stream map as we’ve drawn it and share it with the team. Validate if we’ve mapped the flow accurately and adjust where necessary. Then find out the information required at each step to minimize the wait times. Also, think about this – what impact will you achieve, if the team sets a goal to minimize the wait time at each step? Is there a better alternative?”