In this interview with AgWorld, a company that undertook the Agile Testing course that I present through my company About Agile, we learn about their journey to Agile Testing. In the six months since the course they have applied the ideas and reaped massive benefits from doing so.
I presented the course in a coaching style which they responded to positively and subsequently they really picked up the ball and ran with it afterwards. They did all the work to achieve the benefits and recognize that the journey is not complete either.
The full interview transcript is reproduced below. I can give this course to your company or combine it with a coaching engagement for even better effect. Use the contact details on the About Agile website to get in touch to find out more. I also run the course in a public setting if you prefer to use that avenue. A schedule appears on the About Agile website for that as well.
Here we have an interview with two participants to About Agile’s Agile Testing course. Stephen Baldry is Chief Technical Officer at AgWorld and Michael Holmes is QA and Testing Team Lead at AgWorld.
The interview was conducted over Google Hangout whilst I (Nick) was in Croatia on a break, and Stephen and Michael are in Perth. We find that the central theme is that Agile Testing is a collaborative and whole of team effort and this has been the main benefit of the course.
a 50% reduction in bug count
The course is not a Scrum or a Fundamentals of Agile course, rather it overlays the Agile principles over testing activities and the biggest outcome is better communication and sharing of responsibilities. The course does delve in details like types of testing, the testing quadrants and Exploratory Testing and Testing For Requirements. We don’t talk so much about these important facets in the course during the interview. Read on to find out more.
Nick: Hi Guys, lets do a freeform chat and just start with the results you got in the last 6 months since you did the course. For instance reductions in bug counts and also that you had the first release without any known defects.
Stephen: Michael might want to take this answer and you’re closer to the QA process.
Michael: Sure, I posted recently on LinkedIn a major reduction in defect count and finding the issues earlier rather than later. Weekly builds have helped, but also (more so) having a tester involved from the outset (i.e. doing Scrum rather than ScrumBut). We are finding the issues before they become issues.
Nick: Any specific examples you could give?
Michael: There have been many none of which spring to mind at the moment, it’s more rather a continuous feeling of getting requirements right for all the stories we take on.
Nick: Essentially testing the requirements, is that right?
Michael: Yes, that’s right
Nick: Be great to get a specific anecdote :)
Stephen: It’s been something that we have been observing more externally (i.e. when the product goes to production). Let’s expand a bit more, a year ago QA was discovering requirements when it was time to start testing. Now QA is discovering requirements when we are planning development. Beginning rather than at the end.
Nick: So you have been keeping track of this via metrics and stats have you not?
Stephen: Yes we have year on year stats for bug counts. So year on year we have a 50% reduction in bug count compared with the previous twelve months. And as we said earlier, this is the first major release we have released with zero known defects. This is a massive change for us, this has never occurred before.
customers are taking the time out to give us the bouquets rather than the brick bats
Nick: How did everyone feel about that?
Stephen: Ahhh, we didn’t care (we all laugh in unison)
Nick: Ah, it’s a bit of a nonchalance there…. It’s just what we do now
All: Agreement :)
Stephen: We have come to expect it now.
Nick: That’s great to hear, so drilling down a bit more on that, what sort of effects has that had on the team and has it had any effects on a company wide level?
Stephen: We’ve had contact with customers both via telephone and email, where customers have told us specifically that what we’ve been doing with the product in the last 12 months has been amazing. The quality has gone up so much and it’s really saving them a lot of time, and they wanted to commend us on what we’ve done in the last 12 months and a lot of that has been around improved quality. It has flowed right from the development team to the support team to the sales team and all the way through to customers.
Michael: It’s great that our customers are actually cognizant of that and actually the taking the time out to give us the bouquets rather than the brick bats.
Nick: That’s got to be a good morale booster?
Stephen: Yes, and it’s completely unprompted. For example, our CEO got a message on his phone where the customer said: “You don’t need to phone me back, it’s nothing urgent – just wanting to tell you that what you guys have done with the product lately has been amazing, a big increase in quality.” It’s really good to get that kind of feedback from customers.
Nick: Yeah that’s great, that’s the best kind of feedback – the feedback that you don’t ask for. How has it helped with the team dynamics?
Michael: I’ve heard back from the developers that it helps them get on with doing it. Because we find the problems earlier, it’s still fresh in their mind. They also don’t need to second guess themselves as to what was I thinking when I did that bit of code. They can just go in the make the fix. It makes the turnaround time on detected issues much faster because of the freshness (Nick: It can take 24 times longer to fix a bug after the fact). Because the turnaround is fast they are much more efficient and deliver much more with better quality.
If your sending a rocket to the moon, if you’re off by a small degree at the launch point, you are totally going to miss.
Nick: Certainly, still having a developer hat at times, achieving milestones quickly and reliably is a great confidence booster.
Stephen: We have made plenty of improvements over the last year, but there is still plenty we can still improve upon. We haven’t yet reached our objectives with all of this, but we have seen significant improvements and we’ve seen the benefits of making these changes. Including QA earlier in the process, we clearly are gaining the benefits from that, we feel there are other things we can improve as well. (Nick: Good improvement culture)
Nick: Please tell us more of what you’d like to improve in the coming year?
Stephen: Not specifically pertaining to QA, but more Agile process like the Scrum Process and better prioritization. Part of the analogy that you covered in the course which showed making a cake whereby traditional software development if you don’t get to complete things you get holes in the cake. Whereas in Agile development the layers that you have done are fully functional perfect layers of the cake. Whilst we have a good agile process the prioritization of issues within the sprints is an issue, we sometimes still end of up a cake with holes in it at the end of a sprint (Nick: Probably requiring a hardening sprint) Note: The Cake Analogy is courtesy of Ilan Goldstein.
Nick: Get a less holey cake?
Stephen: And developers still like to cherry pick issues, so we tend to end up with a holey cake.
Nick: Yes this is a common issue, i.e. thinking of the what I like first at the expense of the product overall (hard one to manage especially if innovation is what you want)
Stephen: So we still also want to understand the problem. This can still be disconnected from the development team, the people who do they initial gathering of requirements are not part of the development team. Not necessarily a problem, but if you can’t articulate what the problem is to the development team then you have a gap. We would like to see that gap close.
Nick: We are getting in the Product Ownership area and Agile Business Analysis area and making that more collaborative. I still tend to think that Agile Testing overlaps quite a bit with techniques like Impact Mapping and Story Mapping are applicable in Agile Testing as well as in Business Analysis.
Nick: One of the quotes I had back from the feedback from the initial delivery was from Tristan which was quite good and I wonder if it resonates with you? He said: ”I got a lot out of this course (Agile Testing). It has changed the way I work subtly but the effect has been dramatic. I feel quite a few things have crystallized in the week following as the information is sinking in” from: http://www.aboutagile.com/testimonials/
Does that resonate with you?
Stephen: I think that’s right. The changes that we’ve made are kind of subtle but the the impacts are much larger than the changes would indicate.
Michael: If you were to make an analogy. If your sending a rocket to the moon, if you’re off by a small degree at the launch point, you are totally going to miss. Whereas at the start if your accuracy is correct you are more likely to arrive on the target and successfully land on the moon.
Stephen: One of the things that I have noticed that is kind of small and subtle is just having QA involved early and continually through the process has often led to discovering missed requirements.
Nick: Discovering Gaps?
Stephen: Yes and that’s often because of the different angle QA is looking at it I think. Because of that they are asking questions that developers aren’t asking and we are discovering it early on. We add those requirements as we go (Nick: still fulfilling the sprint goal, not going out of scope so to speak).
Nick: During the course you mentioned that errant requirements were missed, do you think they are being captured as well.
Stephen: Not yet I think, I was was referring to earlier this is about understanding the problem. Gaps in understanding between those building and those asking. We can only have a subset of the team understanding the problem and that subset need to articulate that to the other team members. This is our next place of improvement.
Michael: We had a coaching session with our organisational coach and it ties back to the trust that develops between people. You need to believe what the other person is saying and not doing something stupid but doing that intelligently. I’m not asking this to be done because I say so, but because it makes sense.
you don’t know it because if you knew it you’d be doing it
Nick: So you know the Why?
Stephen: Yes, the session was the speed to trust. What sometimes happens is that we may do a lot of work in requirements gathering then trying to articulate those requirements we can say ‘You need to build it like this, just trust me – I’ve done all the work to figure it out and you need to do it like this.’ That actually doesn’t work because we are knowledge workers and we need to understand the rationale behind things and we need to trust that the right methodology and process has been used and if that is not transparent then it’s hard for us to trust that. We had a workshop to promote transparency in the process and the outcome of what has been learnt is that this aligns really well with the requirements and the team understands this. That speed to trust increases because the process of how they were gathered is transparent. (Nick: Specification by Example is deliberate in this respect and well as Deliberate Discovery)
Nick: Exploring a little bit more around the testing side of things. Michael are there any specific areas from the course you may have felt were immediately applicable?
Michael: Just this week, we’ve had feedback from developers that we hit a problem and the feedback was that the tests around the work we were doing can improve and be doing that better. We as ‘developers’ are not bashing QA anymore, as we have all these other lines of defense that come into play.
Stephen: So what I think Michael is trying to say is that the course helps us better recognise that QA is everyone’s job and that it’s everyone’s responsibility and that we just don’t hand it over to QA. They understand not doing this is breaking down the QA process and they now understand they have a part in this as well.
Nick: That’s cool, do you find yourselves doing more things like triads, increasing T-Shaped skill sets?
Stephen: To a small degree. A lot of this is the change in mindset. So changing from testing is only QA’s problem. Previously, if a bug appeared it was because QA didn’t test it properly. Now it’s everyone’s responsibility and this is the first step? That’s the biggest benefit in my mind, the change in mindset and the accountability for all of the process and not ‘just their part.’
It’s like a lot of things, sometimes you can have a course and you say to yourself – we know this why are you telling us this. We don’t need to be in this course. What you discover is that you don’t know it. It’s like common sense and you do know it, but you don’t know it because if you knew it you’d be doing it. That is what I found this course is that there is a lot of things in it that are from our experience was like we already know this stuff but then it becomes apparent that you don’t know it because you aren’t doing it.
Michael: Everybody has their own take and perspective on it, but here we are as a group have the same perspective and the same picture, so everyone realizes the gaps that exist between each other and at the same time those gaps get filled.
Nick: Once the gaps are apparent you can then deal with them?
Stephen: Yeah, and also agreeing on a common vocabulary and common ideas because you had common training just made it easier. Therefore when we talk about X we all know what X is. This is also helping us to be more effective.
Nick: That is a common statement about Common Sense. It’s very easily mentioned and agreed but rarely practised in a coherent way. I read a quote this week – ‘It’s common sense to treat adults are adult human beings but in fact it’s rarely practiced’
Thanks guys for sharing your learning’s since taking the Agile Testing course. Good luck and all the best in your ongoing journey. I’m sure there will be more tangible and intangible benefits to AgWorld’s bottom line.
Stephen: Thanks Nick, no problem. Thanks for staying up at 1am to do the interview.
Nick: That’s not a problem, not a problem at all. Once again, I’m happy that you’ve received positive outcomes from taking the course as well and achieving the results to your bottom line. Those intangibles you get from the customer are going to result in tangibles to the bottom line.
Stephen: Yes, I think it has definitely been contributing to changing results and changing the bottom line and changing customer feedback. I definitely think that the changes we have made, some of which are directly attributable to the course are having definite effects in our efficiency and effectiveness and our customers perception of us as a company.
Nick: Once again thanks!