Your users are lying to you!
Not intentionally. Not to hurt you. But they are.
The way you validate your idea or ask for feedback has a lot to do with it. Users would hate to make you feel bad about your idea, so they often decide to hold back on their criticism instead.
If you just avoid mentioning your idea, you automatically start asking better questions. Doing this is the easiest (and biggest) improvement you can make to your customer conversations.
That is one of the many valuable lessons from The Mom Test by Rob Fitzpatrick (It’s called The Mom Test because it leads to questions that even your mom can’t lie to you about).
Here are my notes from the fantastic read.
For context, I have conducted user research in the past and am now the co-founder and CEO of a startup in the UX Research space.
Core rules:
- Talk about their life instead of your idea
- Ask about specifics in the past instead of generics or opinions about the future
- Talk less and listen more.
Bad Questions
- Do you think it's a good idea?
Opinions are worthless
- Would you buy a product which did X?
1. You’re asking for opinions and hypotheticals from overly optimistic people who want to make you happy. The answer to a question like this is almost always “yes”, which makes it worthless.
2. Better: Ask how they currently solve X and how much it costs them to do so. And how much time it takes. Ask them to talk you through what happened the last time X came up. If they haven’t solved the problem, ask why not. Have they tried searching for solutions and found them wanting? Or do they not even care enough to have Googled for it?
3. Rule of thumb: Anything involving the future is an over-optimistic lie.
- How much would you pay for X?
Just like the others, fix it by asking about their life as it already is. How much does the problem cost them? How much do they currently pay to solve it? How big is the budget they’ve allocated? I hope you’re noticing a trend here.
- Would you pay X for a product which did Y?
1. Let’s fix it: As always, ask about what they currently do now, not what they believe they might do in the future. Common wisdom is that you price your product in terms of value to the customer rather than cost to you. That's true. And you can't quantify the value received without prodding their financial worldview.
2. Another way to fix it, if you’re far enough along, is to literally ask for money. If you have the deposit or pre-order in hand, you know they were telling the truth.
Good Questions
Say you’re building an app to help construction companies manage their suppliers. You might ask them to show you how they currently do it. Talk about which parts they love and hate. Ask which other tools and processes they tried before settling on this one. Are they actively searching for a replacement? If so, what’s the sticking point? If not, why not? Where are they losing money with their current tools? Is there a budget for better ones? Now, take all that information and decide for yourself whether it’s a good idea.
- What would your dream product do?
1. Sort-of-okay question, but only if you ask good follow-ups. Otherwise it’s a bad question.
2. The value comes from understanding why they want these features. You don’t want to just collect feature requests. You aren’t building the product by committee. But the motivations and constraints behind those requests are critical.
3. People know what their problems are, but they don’t know how to solve those problems.
- Why do you bother?
1. Good question
2. Rule of thumb: You're shooting blind until you understand their goals.
- What are the implications of that?
1. Good question. This distinguishes between I-will-pay-to-solve-that problems and thats-kind-of-annoying-but-I-can-deal-with-it “problems”.
2. Some problems have big, costly implications. Others exist but don’t actually matter. It behooves you to find out which is which. It also gives you a good pricing signal.
3. Rule of thumb: Some problems don’t actually matter.
- Talk me through the last time that happened.
1. Being walked through their full workflow answers many questions in one fell swoop: how do they spend their days, what tools do they use, and who do they talk to? What are the constraints of their day and life? How does your product fit into that day? Which other tools, products, software, and tasks does your product need to integrate with?
2. Rule of thumb: Watching someone do a task will show you where the problems and inefficiencies really are, not where the customer thinks they are.
- What else have you tried?
1. Good question. What are they using now? How much does it cost and what do they love and hate about it? How much would those fixes be worth and how big of a pain would it be for them to switch to a new solution?
2. In the abstract, it’s something he would “definitely” pay to solve. Once we got specific, he didn't even care enough to search for a solution (which do exist, incidentally).
3. If they haven't looked for ways of solving it already, they're not going to look for (or buy) yours.
- How are you dealing with it now?
1. Good question. Beyond workflow information, this gives you a price anchor. If they’re paying £100/month for a duct-tape workaround, you know which ballpark you're playing in. On the other hand, they may have spent £120,000 this year on agency fees to maintain a site you're replacing. If that's the case, you don't want to be having the £100 conversation.
2. Sometimes, both of the above will be happening simultaneously and you get to choose how you present yourself. Do you want to be a replacement for the web app at a yearly value of £1.2k or for the agency at 100x that?
3. Rule of thumb: While it’s rare for someone to tell you precisely what they’ll pay you, they’ll often show you what it’s worth to them.
- Where does the money come from?
1. In a B2B context it’s a must-ask. It leads to a conversation about whose budget the purchase will come from and who else within their company holds the power to torpedo the deal.
2. Often, you'll find yourself talking to someone other than the budget owner. Your future pitches will hit unseen snags unless you learn who else matters and what they care about. This knowledge of their purchasing process will eventually turn into a repeatable sales roadmap.
- Who else should I talk to?
1. Yes! End every conversation like this. Lining up the first few conversations can be challenging, but if you’re onto something interesting and treating people well, your leads will quickly multiply via intros.
2. If someone doesn’t want to make intros, that’s cool too. Just leave them be. You’ve learned that you’re either screwing up the meeting (probably by being too formal, pitchy, or clingy) or they don’t actually care about the problem you’re solving. Take anything nice they say with an extra grain of salt.
- Is there anything else I should have asked?
Asking this question gives them a chance to politely “fix” your line of questioning. And they will!
Other Important Lessons
- With the exception of industry experts who have built very similar businesses, opinions are worthless. You want facts and commitments, not compliments.
- When you hear a request, it’s your job to understand the motivations which led to it. You do that by digging around the question to find the root cause. Why do they bother doing it this way? Why do they want the feature? How are they currently coping without the feature? Dig.
- You’re searching for the truth, not trying to be right. And you want to do it as quickly and cheaply as possible. Learning that your beliefs are wrong is frustrating, but it’s progress. It’s bringing you ever closer to the truth of a real problem and a good market.
1. The worst thing you can do is ignore the bad news while searching for some tiny grain of validation to celebrate. You want the truth, not a gold star.
2. Lukewarm response is data -> they don’t care
3. If they’re still engaged in the conversation, it’s worth asking a couple follow-up questions to understand the nature of their apathy. Is the “problem” not actually that big of a deal? Are they fundamentally different from your ideal customers? Do they not care about the specific implementation? Are they worn out and skeptical from hearing too many pitches, like cafe owners in the aftermath of Groupon? Are they just plain tired today?
4. Rule of thumb: There’s more reliable information in a “meh” than a “Wow!” You can’t build a business on a lukewarm response.
- Let’s say we suspect that teachers from the poorest schools are completely overloaded, and that our tools would save them time so they could better educate their students. We go talk to them and confirm that yes, they are completely overloaded. We then spend weeks with them, figuring out exactly what their dream tool would do. Unfortunately, we've missed the elephant, which is that the poorest schools may not have the budgets available to pay us what we need. We're liable to spend a huge amount of time exploring a real and urgent problem, only to hop into the deadpool due to our customer's budgeting issues.
1. Successful startups tend to depend on multiple failure points. In this case it is both the needs of the teachers and the ability of the schools to pay us. If any of these conditions doesn't exist, we have to significantly overhaul our idea. It's tempting to obsess over the most interesting of several failure points and ignore the others. It's a great way to miss important questions.
Validation
Once we've learned the key facts about our industry and customers, it’s time to zoom in again and start revealing our idea and showing some product. The bad news is that this invites nefarious compliments. The good news is that since we have the beginnings of a product, we're now in a position to cut through the false positives by asking for commitments.
Commitment — They are showing they’re serious by giving up something they value such as time, reputation, or money.
Advancement — They are moving to the next step of your real-world funnel and getting closer to a sale.
- Commitment
A time commitment could include:
1. Clear next meeting with known goals
2. Sitting down to give feedback on wireframes
3. Using a trial themselves for a non-trivial period
Reputation risk:
1. Intro to peers or team
2. Intro to a decision maker (boss, spouse, lawyer)
3. Giving a public testimonial or case study
Financial:
1. Letter of intent (non-legal but gentlemanly agreement to purchase) Pre-order
2. Deposit
If I was very early stage, I might ask for an introduction to his boss or tech team or the budget-holder so I can make sure I fully understand their needs.
If I was slightly further through development, I might push him to agree to be one of our alpha users and to roll it out to some subset of their team and be a case study for launch. If I’ve hit on a real problem, he’ll jump at the chance to start making progress today and get early access.
Know the Next Steps
- When can you come back to talk to the rest of the team?
1. Good meeting. Bingo! If you are selling anything to companies, you’re going to have to talk to multiple people. If they won’t introduce you, then it’s a sure bet you’re at a dead end.
2. Enterprise sales is tedious, but one of the perks is that you can get really accurate signals like this one quite early in the process. Building consumer products is a lot murkier since the customer conversation process doesn’t mimic the purchase process.
Feedback
- If the feedback you're getting from your customer feedback software is absurdly inconsistent. If they run twenty conversations, they end up with twenty different must-have features and twenty separate must-solve problems. The more people they talk to, the more confused they get. What’s going on here? Their customer segment was incredibly broad
1. You have too-broad of a segment and are talking to everyone
2. You have multiple customer segments and missed some of them
3. You are selling to businesses with a complicated buying process and have overlooked some of the stakeholders
Process
- When all the customer learning is stuck in someone’s head instead of being disseminated to the rest of the team, you’ve got a learning bottleneck. Avoid creating (or being) the bottleneck. To do that, the customer and learning has to be shared with the entire founding team, promptly and faithfully. That relies on good notes plus a bit of pre- and post-meeting work.
The most extreme way to bottleneck is to go to the meetings alone and take crappy notes which you don’t review with your team. At that point, your head has become the ultimate repository of customer truth and everyone just has to do what you say.
- Prep questions to unearth hidden risks:
1. If this company were to fail, why would it have happened?
2. What would have to be true for this to be a huge success?
- Any strong emotion is worth writing down.
- Google Docs spreadsheets and Evernote are both great for team sharing, search, and retrieval. Spreadsheets are wonderfully sortable if you write your key signals in their own column. The downside is that it comes across as rude to take notes directly on a computer during a meeting, so you add a bit of mandatory post-meeting work to transfer your notes. I always found this a bit annoying, but it’s way better than losing your learning in an off- topic notebook.
When I first started, I would make audio recordings, but that suffered from the same problem as notebooks. I'd end up with a ton of content and no real way to sort it or find the bit I wanted, so I ended up ignoring the recordings.
- The process before a batch of conversations:
1. If you haven’t yet, choose a focused, findable segment
2. With your team, decide your big 3 learning goals
3. If relevant, decide on ideal next steps and commitment
4. If conversations are the right tool, figure out who to talk to Create a series of best guesses about what the person cares about If a question could be answered via desk research, do that first
Jobs-to-be-Done Framework
Source: The Product Manager’s Guide to Interviewing Customers, Abhishek Chakravarty
Source: Understanding the Job, University of Phoenix
As (Late) Clayton Christensen said — “To get to the right answers you should be asking: What job would consumers want to hire a product to do?”
The basic premise of the JTBD framework is deceptively simple ****— understand what Jobs your customers are trying to get done, and then give them a product that helps them do that job better. The more pressing/ important the job and the easier your product makes it for your customer to do that job—the better the chances of your product to be successful.
The Customer doesn’t care about your product
We need to constantly remind ourselves that customers don’t care about your product (or any product for that matter). They care about themselves, and they care about figuring out how to do the things they need to do — better, faster, easier. That’s all.
The goal of interviewing your customer is
- To learn about what is it that they are trying to get done?
- Understand the pains and challenges they experience when doing these jobs?
- Understand what would help them do their job faster, easier better?
Quick video that explains this concept well:
.svg)
