You have an idea for a B2C product, for a mobile app even. But you're worried about bugs & bad reviews on app store. So, you only release it for a very limited audience. That's probably wrong...

Executive Summary

• B2C (business-to-consumer) products may require a large test audience to validate your idea
  • Well, at least a few hundred users
• Test audience should be large enough to meet your KPIs
  • Is it even mathematically possible to target 20% retention rate, if you audience is limited to 20 users?
• Don't fear bad reviews for a mobile app
  • You need a huge number of installs to get any reviews
• This article is part of the Product Management in the Open Series

Intro

As I described previously I came up with an idea, a value hypothesis, KPIs. Now it was time to confirm or discard that hypothesis. There are numerous ways of doing that, but I opted out for a mobile app as an MVP. That would be something tangible users could play with, while I would be gathering data on their usage.

However, mobile app development has certain barriers to entry. Unlike a website which you can just launch and make available to the whole world, a mobile app has to go through an approval process before it appears on an app store. All that sounded daunting when I quickly wanted to validate my idea. I was also worried about negative reviews because of bugs or less than ideal user experience. After all, mobile users are a few taps away from giving your app a 1 star rating. And that could really kill an early app idea, if people immediately see poor reviews. Or... so I thought at that time.

Luckily, I had a dozen of friends who were were also into food & drinks (my target group). So I had an audience I could test my app with.

As a result, I embarked on scoping features, content, platform and building an actual MVP (I’ll talk about it in a upcoming blog post). I also figured out a way to have a private distribution for both iOS and Android. (Unfortunately, my friends didn’t use a single platform, so I had to target both of them). Anyway, after some time, I had an MVP for my limited audience.

Field Testing

And I even organised a field testing session among a group of foodies. We spent a night wandering around town in search of the best wine and craft beer thanks to Best in Town.

Happy users

And their phones... and beer

The idea and the app got a positive reception and it was great to observe people using your app.

I even got a few e-mails after from a few friends of friends. They gave me detailed feedback on the app and some new ideas.

But Do You Get Enough Data to Validate Hypothesis?

But as the time went on my analytics system (Google Analytics at the time) registered very little activity. At that time I managed to sign up around 20+ users to the app. And if we look at the KPI we have set for validating the hypothesis it was a Monthly Retention. The hypothesis would be validated, if we achieved a 20% monthly retention rate.

Okay, so we had around 20 people and 20% of them would be what 4 persons? That hardly looks good from the point of view of statistical significance.

Your test audience should be large enough to make your KPIs statistically significant.

Issues with Releasing B2C Mobile Apps to Small Audience

So this leads to the following points when it comes to releasing an B2C product to really small audiences. By the way, you may still have a limited launch - think of when Gmail was by invitation only. But probably, they released it to more than 20 people.

• You won’t get enough quantitative data to validate your hypothesis. B2C (business-to-consumer) apps require scale. As we said before, the average monthly retention is about 20-35%. Even if your app in its first iteration does produce the 20% monthly retention you’ll need a few hundred users to ensure the number is significant enough. It is also highly likely that you may need a few iterations before your app hits any reasonable retention rates. And that requires a constant stream of users coming to the app. Think about if your first monthly retention rate is 10%, you’ll lose 90% of your audience. And for your next experiment you’ll have to acquire a new user base.
• You accidentally exclude one of your target audiences. In the Product Canvas I set that we had two target audiences - locals and tourists. But by releasing an app to my friends, I totally excluded the tourists group out of my tests.
• Friend bias. If you release an app to friends and friends of friends then you may get positive feedback at first. And people will be generally impressed with your work. But that does not mean they will use your app in the future or they have the problem you’re trying to solve. And yeah, if you have a small audience you shouldn’t get too excited about sudden upticks you spot in your analytics system. It could be that your friends are just talking about you and showing your app to your other mates. And… your other mates won’t be able to get an app, because it’s not an an app store.
• You also limit your chances to present an app to random people. In case, of Best in Town, it was difficult to sign up random people. I would often talk to people recently opened their own cafe. I could’ve shown them Best in Town, but it would be a painful process of getting them onto the app during to the limitation of restricted audiences on iOS & Android.

Qualitative Analysis is Still Possible After General Release

Some people may say ‘But what about the benefits of qualitative analysis’? It is useful indeed, but releasing an app for a general audience, does not exclude qualitative analysis.

I launched the app among the people I knew. We also did field testing together. Of course, it gave me lots of insightful information. I concur it was great to see how people were using the app. That really helped to spot various usability issues.

However, releasing an app to everyone does not exclude qualitative analysis. You can still have sessions in person with users to see how they use an app, while at the same time you gather quantitative data from all other users you acquired through app stores.

False Fear of Bad Reviews

Jumping a bit forward the app is now on the market - both on Apple App Store and Google Play Store. There are close to 1,000 downloads in total. And there haven’t been any rating below 4 stars. There is the data that suggests that a ratio of rating per download can vary from 1 to 75 to 1 to 600 (1 rating per 600 downloads.

So, clearly there was nothing to fear.

Going Forward

It’s not enough to have 20 installs if you are working on a mass market B2C app. So, I decided to release the app on app stores and spent money on advertising to acquire users. I’ll talk about that in the future posts.