Let's talk about cake for a minute. You're probably wondering what on earth cake could have to do to do with testing your brainstormed ideas. Bear with us — we promise it will make sense soon.

Pretend, if you will, that you’re throwing a dinner party. You have a dozen guests, and one of them is vegan. When thinking about what to serve for dessert, you scour dozens of food blogs and pore over stacks of cookbooks. Finally, you end up with a carefully-curated list of 20 delicious cake possibilities. You can’t decide which idea is best, so you run the list through a cake-ranking app and narrow your options down to two: a double chocolate layer cake and an eggless carrot cake, for the vegan guest.

You then go out and buy the finest, freshest ingredients for your cakes. You follow the recipes to the letter. Your confections turn out picture-perfect.

When it comes time for dessert at your dinner party, you are surprised when most of your guests choose the carrot cake. You would have bet the farm that most of them would have gone with the chocolate. “Oh well,” you conclude, “I guess carrot cake is always the best dessert to serve at my dinner parties.”

Unfortunately, your conclusion is based on a false premise (my guests want cake), and is therefore entirely wrong. You see, unbeknownst to you, some of your guests are allergic to chocolate, some are diabetic, and the rest of them are on a diet. So when you gave them a choice between two cakes, they chose the one that seemed the least opposed to their needs, when what they really needed was a low calorie sweet finish to the meal that didn’t contain refined sugar.

But wait, you’re probably wondering -- why are we talking about cake? What could this possibly have to do with with product ideas? Well, in a way -- everything. Just as your customers hire your product to do a job -- say, drill a quarter inch hole -- when you were making your cakes, you forgot to find out what job your guests we're hiring your dessert to do. (Provide a sweet finish to the meal without sugar and fat.) Of course you meant well. You brainstormed a lot of great cake ideas. Then you used a filter to narrow the ideas down. Finally, you gave your guests a choice between what you thought were the two best ideas.

At its core, this is what happens when you concept test one of your hundreds of ideas in order to find your magical product unicorn — that One Big Idea that customers will find so perfect and irresistible that they will adopt it in droves.  In the case of the dinner party, you thought everyone loved the carrot cake because that’s what most people chose, but what you didn’t realize, because you didn’t approach dessert from a needs-based perspective, is that none of your guests got what they wanted or needed at all. They didn’t want sugary cake in the first place. Had you ascertained your guests’ needs first, you might have offered them a fruit and cheese plate.

So too, when you trust your ideas to a ranking process to test product ideas with potential customers without first ascertaining their needs, you’ll never know if your product is actually something they would buy in the real world.

Ascertaining the Needs of the Customer

Before we interrogate the fundamental assumption  that your ranking process will magically reveal what your customers most desire, let’s first, briefly, understand a bit more about a handful of the more common ranking methods.

• Forced choice scaling: When presenting potential customers with ideas, this technique does not allow for a “no opinion” or “undecided” response, but instead “forces” the respondent to make a choice
• Paired comparisons: Like your dessert cakes, this method forces participants to choose which feature they like best out of the choices they are given
• Conjoint analysis: Wikipedia defines this method as “A controlled set of potential products or services is shown to respondents and by analyzing how they make preferences between these products, the implicit valuation of the individual elements making up the product or service can be determined.” In other words, you offer your customers a choice  observe how they choose and place value on what you’ve offered.
• Surveys or focus groups: Customers are asked a series of targeted questions relating to their product preferences

Well. That all sounds very technical and scientific, so .. it must be valid, right?

Actually, just because it sounds reassuringly “science-y” and important doesn’t mean that it’s a useful method to gather the information you need — especially when the premise is faulty. It's human nature to place our faith in time-honored systems, and it's reassuring to believe that this system  can reliably predict success. Unfortunately, though, this faith is misplaced. Offloading the process of choosing the “best” idea from  hundreds of faulty ones onto a trusted filtering and ranking process will not, ultimately, prevent that idea  from failing in the marketplace

Let’s break down the faulty assumptions that drive these methods, and learn why they aren’t actually your silver bullet to product success (or even likely to lead to any success at all):

• The primary and most compelling reason these methods aren’t useful is because there is a high probability that the best solution isn’t even among the concepts being tested. Think back to those dessert cakes. Sure, carrot cake was the preferred choice — but that’s because the best choice wasn’t even offered.  Simply put, the best idea out of 100 bad ones is not inherently a good idea. And while you're busy testing these second- or third-best solutions, someone is busy cooking a killer carrot cake that everyone is going to want.
• None of these testing methods determine whether or not your idea is better than what is already out there. Perhaps your potential customers already have a product that fills their needs, and does it well. Sure, they’ll pick one of your ideas as the best out of what is offered, but if they already use something better suited to their needs, they aren’t going to buy your product.
• There’s a decent chance your potential customers don’t fully understand how the technology or features you’re presenting to them are actually related to their needs. And if you don’t understand their unmet needs, there’s no way you can convey value to them.
• None of these methods offer a consistent way to rate value. With no consistency, there is no reliable way to ascertain preference.

Of course, even if there were a consistent way to rate value or make sure your customers understood the value of your idea, there’s still that one pesky problem: If you're asking people to rate a selection of features you've brainstormed, odds are you aren't even offering them the best solution for their needs.

These same problems apply to more seemingly personal, less esoteric methods such as surveys or focus groups, only with the extra added wrinkle that respondents lack a consistent language with which to convey their thoughts.

Okay, so ... brainstorming from an ideas-first approach is a no-go, and traditional means of filtering out these ideas to arrive at a winner are similarly flawed. So, you’re thinking - why not just go ask the customers what they want? If you don't know why this is a terrible idea (and don't feel bad, most people don't) then you might want to check out our Feedback Rules course to learn all about the feedback gathering techniques you should avoid like the plague -- and the ones you should embrace with your entire body and soul.