Product managers de-risk their products by constantly learning — cost of the different learning methods is critical to consider
Every day in product management is a day of learning new problems and finding ways to solve them.
The learning process is comprised of discussions with the team, executives, customers and the research of competitors. Different learning methodologies will have different costs and risks. Some product managers misuse methods and take on large risks or high costs with the outcome of “we learned that this does not work” positioned as a success. There is another name for that — its called “failure”.
Product manager’s job is to succeed, even if they need to cost-effectively fail some tests on the path to it.
Failure is a part of the learning process but its costs have to be controlled.
Any product or idea needs to be de-risked and move from ambiguity towards certainty of outcome — move on the x-axis below from left to right.
In order to move to the right, you have to accumulate knowledge and data — you have to learn.
The two ways to learn are:
- Learn yourself — experiment
- Learn from others — see what competitors did
Learning from others is exceptionally cost effective — you can research yourself or even hire a research firm to do it for you.
When you need to learn yourself, costs start to play a much bigger role and you have to align to the benefit like in any other decision you make. This is a pure prioritization exercise of knowledge gained vs. cost invested.
Company or product stages can be looked at from the perspective of how many customers do they have and how much budget is available.
For example, a startup, will have very low earnings (hence, a startup) and should have a very specific segment it is going after at first.
In the box below, a startup will be in the first square.
A large company like Amazon, whose Amazon.com store is very established with hundreds of millions of users of various segments May operate on the outer square when considering new product.
A startup founder (or a product manager of a new product in an established company), should have a very clear idea who are the customers it is going after first (how else would they have thought of the idea if not to solve a problem for a specific segment).
Because the founders / product managers will be usually close to the problem, their knowledge will be high (maybe experienced the problem themselves or know the segment very well).
As they come with knowledge, their costs of knowledge and learning are low (already paid for). This matches up to their actual funding available — ie a small budget. If they are bootstrapped (meaning, investing their own money with no VC funds) they cannot afford to invest in testing, but in building.
This applies to a product manager of a new product in an established company as well. Their budget will almost always match their product’s earnings. Little to no earning, small budget.
Example: Facebook
Facebook launched when Mark Zuckerberg was 19 and focused on students in universities and high schools as a go-to-market strategy. Because the first employees were very close to the target customers (students), efforts were on building the product, not testing it. The first employees were students or fresh graduates themselves — they were the customers and needed a little testing.
At the beginning of a product’s life, focus on earning (building) not learning (testing).
Example: Google
Larry Page and Sergey Brin started a project to better organize internet search results on the Stanford network. Their breakthrough of creating PageRank (how many pages link to a page as a method of voting to sort search results) was revolutionary.
Instead of testing and learning, they focused on building and acquiring customers earn. They licensed their solution to existing search engines and by 2000, licensed it to the largest search engine at the time, Yahoo.
Like Facebook, Google also focused on building and capturing obvious customers before starting to actually test.
A starting out product does not have fail-fast-fail-often — only succeed or die
After initial product success, a product now has customers and some earning power. To grow to a new customer base, assuming one cannot copy a competitor, the product manager has to test.
Similar to how VCs do it, the product managers have to run fast, cost-effective tests to find the next move — testing the waters. A VC has a diverse portfolio of investments, a product manager has a diverse portfolio of ideas to test.
Because the tests are fast, and because you cannot afford large ones, you will not have statistically significant results. You will probably conduct quick tests, probably in adjacency moves to your already successful product. Each test is an investment in an idea (just like a VC does) and the odds of it working are slim to none.
Common methods will be customer interviews, surveys, running dummy ads on Facebook and seeing click throughs and others.
The goal is to find the next step, not to create an exhaustive list of winners.
Once you find a potential winner (a potential next step), the startup phase starts all over again for that product (build and earn vs. test and learn).
At this point, fail fast often means rapidly kill ideas — In the past, have used very limited interviews and surveys of roughly 10 customers to discredit an idea. Yes, you run the risk of false negative, but you cannot afford to do more.
Now you have a few winners in the bag and you are earning money. You have a budget and the means (customers) to test in a much more meaningful way.
Because you have a larger budget you have more choice — you can continue to deploy the above tests, or you can use A/B testing.
The decision on which method to use depends on your goal:
- To optimize an existing product — use A/B testing as it allows quantification with statistically significant results. A/B tests are best for single variable testing, assuming you can isolate it.
- To de-risk an idea— use the high level approach described before (surveys, interviews, fake ads) or run a “fake door” experiment
In Fake door experiment you deploy a feature to a beta group with minimum to no functionality to prove there is some appetite — you are testing a much broader set of variables here so A/B test is not appropriate as you will not be able to isolate the reason for success/failure and you are not trying to — you are simply de-risking.
If you make the mistake of using A/B testing to do de-risking, you are building twice and achieving accuracy where you don’t need it. You are burning money and time.
It is better to be generally right than precisely wrong
At this stage, fail-fast-fail-often means you tested hypotheses for your existing products in A/B testing and failed them or you ran de-risking activities to kill ideas or promote them.
Fail fast fail often does NOT mean building products and seeing if they succeed. That is very expensive when other testing alternatives exist.
The most successful tech companies in the world run thousands of A/B tests a month. They can do it because they 1) can afford it and 2) they are testing incremental improvements on an already successful product.
If an MVP is built as a test (ie not necessarily viable), it can only be done when you can afford it.
When the big companies do it, it is still cost effective for them, but it looks much bigger from the outside.
Example: Google Glass — ~$1B test
In the absence of market precedence, Google had to experiment in the market with Google Glass to find out if it works (de-risk).
For Google, this was a reasonable test. It invested around $1B in this test and lost all of it when it failed. $1B is less than 1% of Google’s annual revenue in 2017 and 6% of their entire R&D budget at the time.
Meaning, if you choose a moonshot idea and find your only way to de-risk it is actually fully build it and launch it, you need to have a Google size budget. Otherwise, it is just an idea and you need to advance your budgets and succeed until you can effectively de-risk it.
You literally need to earn your right to learn.
