While working in various IT projects I realized that regardless of all these ‘lean’ product development movements, many companies still encounter major issues in terms of product validation. In other words - it is hard for people to understand if the product they develop is what the market desires.
To help you with your validation efforts, I’d like to give you an inside look into the day-by-day product validation process for a live product that we have worked out with one of our clients. Let’s start by answering the fundamental question:
What is product validation?
There are many approaches to what’s called product validation. Try to google the phrase and you will find that there is more than one way to define it. My short explanation would be that product validation is a process aimed to answer the question: is my product something that people need? If you are building a product, it’s a good practice to verify its value each time you release a new version.
One of the most famous frameworks in this field was presented in The Lean Startup book by Eric Ries (you can find it on our list of Must-Read Books for CTOs). He suggests to follow these steps:
Build an experiment to test if people need what you are trying to sell them.
Measure what is the result of your experiment.
Learn your lesson and then iterate by building your next experiment.
What I like about this approach is that Eric reveals that in order to gain value out of the cycle, one should plan it in the reverse order. First you need to know what you want to learn about your users. After that you should think of how you will measure it. Only at the end you may consider how to build the fastest and cheapest experiment to gain that knowledge.
An inside look at our team’s product validation process
I personally think that product validation is one of the greatest challenges in the whole workflow. In the project I currently work at, we encountered the same challenge. However, we decided to structure a process that would support us in doing the validation. We don’t call our approach the lean startup framework, although ultimately it does include the 3 aforementioned steps: build, measure, learn.
What does our validation process look like?
I. Pre-development phase - “how to measure?”
1. The PO adds his idea of validation to the user story. (Not sure what a PO does? Read our article about Product Owner responsibilities.) Every time our Product Owner adds a new user story to the product backlog he is supposed to fill the field titled “how to measure?”. This approach makes him analyze prior to any further discussion how to recognize if the change we do is something that our user needs and if we possess sufficient tools to allow user story validation. See the snapshot of our user story template below. This is the “planning” part of our framework.
2. The validation idea is discussed with other people during the weekly meeting. We try to have a few important people on the meeting: the Product Owner, UX designer, Scrum Master and the client’s representative (optionally). The way we run this session is very easy: the PO presents his idea for validation and other people discuss it and suggest improvements where applicable. This way, the PO receives a wide range of opinions regarding the case and makes more informed decisions.
II. Development phase.
No validation work is done during this stage. This is the “build” part of the lean startup framework.
III. Post-development phase - product validation.
After the new product increment is deployed to production, our validation process starts. Some of it happens ad-hoc - our data analysts start producing stats. However, the main point of the process is a weekly meeting during which the Product Owner, UX designer, Scrum Master and optionally the client’s representative discuss what has been released to our “live” environment and what results it brought us. In other words, we validate whether the new product features had a positive, neutral or negative impact on end users. For this purpose we use Google Analytics, a heatmap tool and other data analysis tools. After we collect the necessary data and validate new features we make decisions regarding future development plans. We use a Jira board to facilitate the meeting; see below. We take items that are in the “To verify” column and try to move them to “Verified”, which means that we have actually validated our feature. This is the “measure” and the “learn” part of the lean startup framework.
We iterate these practices and the cycle starts from the beginning every next development sprint. To summarize, the validation workflow is presented on the graph below.
What tools do we use to visualize product validation?
Aside from JIRA mentioned in the example above, we use one additional tool to visualize product validation status - the impact map.
Impact mapping is a technique created by Gojko Adzic to support a process to achieve business goals.
(We were quite excited to see Gojko Adzic himself retweet this article soon after it was published.)
The idea is simple - you define a business goal e.g. “Sell our product to a million clients within 3 months”. Gojko suggests to think about impact mapping as if it was an exercise to navigate the map from point A to point B. In this case, point A would be the place where we are at right now and point B is the goal we would like to reach. If we look at the map, there are usually many ways to reach the same goal. The idea is to hit the road and check if all the roads are passable. It might turn out that some are actually closed, or there may be roads that don’t exist just yet.
Using impact mapping terms, you are supposed to define 4 elements:
Goal - decide what would you like to achieve from the business perspective. Maybe you would like to reach a certain number of sales in a given period of time, or maybe you prefer to focus on the number of new registered users. Whatever that is, try to define it as precisely as possible.
Actors - think about various groups of people who can help you achieve the goal. These can be your segments of users, your employees or anyone else.
Impacts - at this stage, you are suppose to think about the things that your actors can do to help you achieve you the goal.
Deliverables - try to define precise product features that will help actors bring their impacts.
An example could look like this:
If creating a deliverable drives actors to bring the impact which supports our main goal, we validate that deliverable positively and using that information we plan our future work. If a deliverable doesn’t bring any positive change, we validate it negatively, which is also a significant part of our learning.
What does our impact map look like?
The central part of the impact map (darker blue) presents the main goal we set as a team. It can any sort of business goal, e.g. 100k new users registered in our application in the next 3 months. It should be something precise and easy to measure at the end of the specified period.
On the borders of our impact map we presented deliverables (yellow, red, black and green) that we would like to offer to the end users in order to support our main goal, e.g. a free gift for everyone who registers within the specified period. In the agile world, deliverables would be called user stories or product backlog items. The colors of the deliverables carry additional meaning:
yellow - for all deliverables that were added to the impact map and are in the pre-development stage.
black - the deliverable is either in the development or post-development stage but prior to the validation stage.
green - the deliverable was validated positively.
red - the deliverable was validated negatively.
In our case, the impact map is updated weekly and is overseen by the Scrum Master, although the Product Owner or anyone with the appropriate information can make sure the map stays relevant.
Based on the information we have visualized on the impact map, we make data-driven decisions about the future development of the product.
To recap, here’s a few reasons why impact mapping for product validation is probably worth your time:
creating the roadmap helps the team remain focused on the business goals, i.e. the actions that actually create revenue
it allows you to see the whole landscape of all possible routes towards your business goal - including faster, easier, cost-effective routes you may have missed
impact mapping opens you up to more creative decisionmaking: you can look at whole sections of the map and say e.g. “the left side of the map has not been effective, let’s try something from another area” - there’s always a different section of the map you can focus on
thinking with the map helps you stay motivated: “it’s not that our efforts are not working, only the efforts in this particular section seem to be ineffective”
Hopefully this post will help you organize your product validation process within your team. Good luck with it and let us know how it goes!
If you're looking to learn more about effective software development processes, we have a few articles you might enjoy: