Don’t benchmark to failure

Flurry released data this week on the average retention and frequency of use for mobile applications they track, which presumably includes everyone from Zynga to CBS to Yahoo. That post is now being used to assert a myriad of things, from “communication apps having great retention” to photo/video having “terrible stats.” I think it’s important that founders realize what this data is before they start to steer their business based on this information.

Most products fail. 
Most products, especially in mobile, don’t make it. They never get the attention they perhaps deserve, make enough money to be sustainable or enough of an audience to last. Particularly if they have network effects, such as a social service like Instagram or marketplace like Ebay, they have a harder time in the early days. Those same characteristics make network effect businesses very powerful as they grow, but it does make them hard to get started.

Just take a personal case. If I take the frequency and 30 day retention curves of the last few products I’ve worked on and have statistics for, I would never average them out. There are a small set of successful products, and lots of products (some experiments) that just didn’t work out. 

An average of the market, is an average of failure
No doubt in Flurry’s data there are some wildly successful companies, but on average we’re talking about failure. So that means any data needs to be taken with a very hefty grain of salt. Or, to put it visually…

This is not valuable.

Startups are about creating outliers. A good benchmark is not the average of failure, it’s about what it is going to take to grow rapidly, have a sustainable long term business, and build a product people care about. 

Startups are starved for good benchmark data, and when you are thirsty it’s easy to overlook where the source of the water came from. Let’s just keep in mind that this is an average of all the products that are pushing data in Flurry.

This further highlights the importance of sharing good data to help us grow companies properly. I love that Flurry took the time to try and break down stats by category of product, and I hope they (and Andrew) find new ways of carving the data to better inform those developing product. 

Previous Next

8 notes

  1. dshan reblogged this from nabeel and added:
    Great discussion of how misinterpreting the underlying data can lead to dangerous conclusions for founders. Startups...
  2. nabeel posted this