I’ve heard about an industry trend from other professional software engineers that I want to talk about in this article. Sometimes, product managers (PMs) prefer “talking to users” (user research) over the work of identifying and documenting features and bugfixes for the product.
I guess that’s because, as you probably know, writing down work items (“tickets”) is — in fact — work. It’s boring, and it sucks, but it helps the engineering team succeed. However, “hanging out” with your users on long video calls doesn’t exactly “feel like work,” because — well — it’s not.
(In fairness to full-time user researchers, who typically do things like time how long it takes users to complete certain tasks and may use heatmaps to quantify cursor positioning, most “user research” by PMs appears to be just a lightly-structured conversation, from what I’ve heard in the rumor mill.)
Typically, when the team doesn’t have full-time, qualified user researchers, then the PMs and/or product designers step in to fill that role. That would seem to make sense, until the engineers are waiting around for tickets and needing to do the job of those PMs and designers in writing up the tickets.
“User research” sure sounds like an important, work-related activity, right?
But user research doesn’t just take time away from supporting engineering teams that almost universally don’t have enough coders; it’s not actually useful in the first place. This was a sentiment that Steve Jobs agreed with:
“People don’t know what they want until you show it to them. That’s why I never rely on market research. Our task is to read things that are not yet on the page.” — Steve Jobs
Here’s why I wish PMs would spend their “user research” time writing up tickets for the engineering team instead.
What Do Product Managers Get Paid For?
In a resource-constrained team, especially one that doesn’t have enough software engineers to keep the product roadmap moving forward as planned, someone has to pick up the slack and write the ******* tickets.
Whose job is it on our team to write the tickets? In most teams that are lucky enough to have a product manager, that’s the PM’s main role.
But many PMs apparently take the opposite philosophy to heart, the sentiment that “Engineers should write the tickets.” Wait, what?
According to people with experience on teams that work this way, the PMs’ “responsibility is just to frame the problem or opportunity”— not write it up as a ticket. Apparently, the PMs are too busy talking to users and meeting with stakeholders to write up the tickets, from what I’ve read on the topic.
Sure, if you have enough software engineers — especially surplus engineers compared to the product roadmap, which I’ve never seen but apparently happens at some companies especially but not exclusively in the EU — then the engineers will often write better tickets than PMs.
In that case, sure, fire the PM(s), and have the engineers write the tickets. Engineers may hate writing tickets, but it’s a more efficient use of the budget to pay for more coders and less middle managers.
If that’s the case, why would you keep the PM(s), even though the engineers are writing the tickets? What the **** are they going to do all day?
User research! (Duh!)
Uh, OK, so how does that typically go? Well, “user research” seems to mean an informal conversation lasting 30–60 minutes and covering these topics, which I’ve selected from a Hotjar article about qualitative user interviews:
1. What’s your first reaction to this prototype?
2. What do you like and dislike about this prototype?
3. How would you improve this design?
4. Which features are most important to you in a product like this? What do you like and dislike about this product?
5. Does this product have the features you need? Is there any additional functionality you need?
6. How do you feel this product compares to similar products on the market?
7. On a scale of 1–10, how likely are you to buy this product?
8. On a scale of 1–10, how easy was it for you to complete this specific set task?
9. On a scale of 1–10, how likely are you to recommend us to a friend or colleague? Why did you give that rating?
10. What would make your experience better?
Do you see how users (whether prospective or current) are going to lie — without meaning to — just by answering these questions confidently?
Well, if you don’t already see it, let me walk you through the research of overconfidence to illustrate why these qualitative user interviews are mostly a waste of time, especially compared to PMs writing up tickets.
Overconfidence Is Why User Interviews Are Useless
Don’t get me wrong, I’m the biggest fan of customer experience that you’ll ever meet in a software engineer, which is why I’ve focused my entire career on UI / UX (user interface / user experience) product engineering.
But there are simply too many universal truths to software development and consumer preferences that your users will never mention during informal user research conversations, even though these are what matter:
- Websites and apps should be fast and performant.
- Websites and apps should respond to user input.
- Websites and apps should be clear and easy-to-use.
- Websites and apps shouldn’t have bugs.
- Websites and apps should work on all screen sizes.
I guarantee that users care more about these 5 items than the previous list of 10 questions, but these items aren’t listed in the previous resource about qualitative user interviews. Shouldn’t fixing bugs always be #1?
Qualitative user research along the lines of the questions I cited in the previous section isn’t especially useful, because people default to being overconfident about their predictions in life.
“Talking to users” prompts users to give confident — but wrong — answers about a proposed user interface (UI) design and user experience (UX).
That’s not a great idea.
After all, users are typically not software engineers, product managers, or product designers, so why would they know **** about UI / UX?
- Do you think users, who are “wowed” just to have the chance to participate in 1-on-1 user research, are going to report a meticulous list of all the bugs they’ve found in your software product?
- Do you think users’ suggestions, which often have no place in the next 12+ months of a product roadmap, are actually “good ideas”?
- Do you think users are able to accurately predict their future behavior when it comes to a software product’s “great new design”?
It’s just overwhelmingly clear from both research and personal experience that “traditional user research” is flawed — users aren’t technologists.
Let’s go to the evidence based on failed company experiments (New Coke) and scientific research into why people confidently make predictions, even though those predictions typically turn out to be false.
“Traditional consumer research is just as likely to unearth falsehoods as it is truths. In fact, behavioural science has proven just how bad humans are at understanding why we do what we do, and has shown that most of the time consumers either don’t know what they want.” — Adam Cleaver on WRAC
This isn’t new information, with Wikipedia citing research papers from 2002 and 2008 as the key sources for the “overconfidence effect”:
“The overconfidence effect is a well-established bias in which a person’s subjective confidence in their judgments is reliably greater than the objective accuracy of those judgments, especially when confidence is relatively high.[1][2]” — Wikipedia
The fact that users are confidently lying to you about what they want, but aren’t aware of that fact, is related to the Dunning–Kruger effect, or the idea that every driver rates themselves as “better than average.”
Your users simply aren’t going to embarrass themselves — or “waste your time” — by saying that they don’t know. So they’ll make a confident statement, but it’s most often just an uninformed guess of no real value.
That invalidates the concept of asking people what they want to see in your software, which may explain why iPhone has 1.46 billion active users.
“Myth #21: People can tell you what they want
Many organizations still rely on asking people what changes they’d like to see in their website or service, neglecting historical research failures like the New Coke or the Aeron chair.
When asking people, you have to be aware that people make confident but false predictions about their future behavior, especially when presented with a new and unfamiliar design.” — UXMyths.com
When a company gives you a private invitation to show you their “exciting, new design” for their software product, do you think most people will say, “Yeah, this sucks ***. Stop changing things for no reason!” No chance!
But, when you’re a daily user of a software product, and they change how the entire user experience for no reason, most people will say just that!
“Suffice it to say to NI (and everybody else in the computer industry) — PLEASE STOP CHANGING THINGS FOR NO REASON AT ALL. STOP USING OPEN SOURCE LIBRARIES THAT DON’T WORK. STOP ADDING THEFT PROTECTION THAT NEGATIVELY IMPACTS YOUR HONEST USERS.” — bergsten, writing on the Native Instruments (NI) forums
And that brings us back to the (infamous?) Steve Jobs quote that I opened this article with and will repeat here for reference. Here’s the full quote:
“Some people say, “Give the customers what they want.” But that’s not my approach. Our job is to figure out what they’re going to want before they do. I think Henry Ford once said, “If I’d asked customers what they wanted, they would have told me, ‘A faster horse!’” People don’t know what they want until you show it to them. That’s why I never rely on market research. Our task is to read things that are not yet on the page.” — Steve Jobs
Steve Jobs and Henry Ford both had it exactly correct: consumers often have no idea what they want, at least until your new product becomes popular on social media because of its outstanding user experience.
That’s why PMs need to get back to writing tickets! The goal of software development should be a performant, bug-free experience, not “pivoting” in a new direction every 6–12 months because of “quick chats with users.”
Leave the user research to professional user researchers who will help your team optimize your app so that your users have an even better user experience, through reducing the time it takes users to complete tasks.
Oh , and please prioritize fixing bugs. There’s nothing worse you can do for your brand than releasing buggy software and then not fixing those bugs for months or years. Clearly, squashing bugs is always priority #1.
Wrapping Up: Steve Jobs & Henry Ford Were Right
To wrap up, let’s take the hypothetical example of an understaffed, undercapitalized software engineering team at a video game company that only has a PC app but wants to release on Xbox, Playstation, and Switch.
In this case, users are gamers, so we can assume they likely have at least 1 console lying around somewhere in the house, not to mention the fact that they have mobile phones and wouldn’t mind a phone version of the game.
If you talk to them, your users will probably say they want console ports, and they’d say they could imagine that they would use it more than they currently use the PC app, especially if you ask them leading questions: “Would you use a console port more than our current PC game?”
But then, if you deliver that exact thing to them in this hypothetical example, it’s entirely possible you’ll only see marginal migration from your PC game to your consoles.
Why?
Your users are already using the PC game! They know it, they trust it, and they would prefer the bugs get fixed there than switching over to consoles. That’s a real possibility for actual user behavior, and it’s exactly the opposite of what most people will say during freeform user research “quick chats.” After all, your console ports cost money to purchase, right?
So, yeah, ports sound like a great idea, but your users are going to be awful at predicting their own future behavior accurately, like everyone else!
As Nate Silver explains in the summary of his book The Signal and the Noise, “Both experts and laypeople mistake more confident predictions for more accurate ones. But overconfidence is often the reason for failure.”
Since your PMs, product designers, and executive team are like anyone else, they’re going to assume that those “confident statements” from your user research are also “accurate statements” about users’ future behavior.
They’re also going to succumb to their own Dunning–Kruger effect, in that they’ll assume that they’re “better than average” at their own jobs, despite any evidence necessarily supporting that conclusion.
Because of their own overconfidence in their skills, your non-professional user researchers (PMs, product designers, executives, etc.) are going to — almost always — take the users’ statements at face value, without understanding that what people say and what people do are different.
Everyone is excessively confident in their own abilities, whether to evaluate user research as stakeholders or to predict their own future behavior as users participating in user research.
Performing qualitative user interviews as your sole method of user research, without quantitative usability testing (observing and timing your users as they complete tasks in your product) is garbage in, garbage out.
But user interviews “look like work” and area lot more fun then what PMs are actually supposed to do, which is identify requirements for the product in collaboration with stakeholders and designers, then write up tickets.
- PMs: Get the **** back to work, and leave your users alone. (They have no idea that they’re full of ****, and they won’t appreciate hearing it!)
- Executives: Hire separate, professional user researchers with the sole mission of reducing the time it takes users to complete tasks.
- Software engineers: Skip listening to user interviews, given that the content of those interviews isn’t actually going to be useful at all.
Engineers don’t want to waste time on activities that don’t add value; they don’t want endless, last-minute (“agile”?) changes to the product roadmap from user research; and they don’t want to write up most of the tickets.
That’s why I agree with Steve Jobs and Henry Fords!
You can’t trust qualitative consumer research (“What do you want?”), only sales (revenue) and quantitatively timing real users completing real tasks.
