Not all Users are Alike in a Lean Startup

A lean startup measures user behaviour in order to test some hypotheses and accquire validated learning. However, not all users should be taken into account when validating your hypotheses.  At least that's what we have concluded from our measurements with browseye. According to this learning, we discuss in this post which users you should be working with.

New vs. Old Users

Gotcha: When you deploy a new product version you should ignore the behaviour of the user base from previous versions (i.e., the old users). Old users will invalidate your learning!


Eric Ries mentions in his Lean Startup book that:

when you deploy a new product version the users you want to be measuring are the ones who have not been in contact with previous product versions, that is, the new users.

The reason why you want to work with new users is to have the same initial conditions before and after deploying the new product version. If you test the new product version with users who have been exposed to the previous version their knowledge of that version may influence their behaviour and bias your results.

Versions A and B of your product should always be compared with their corresponding new users. A new user is a user that starts using the app for the first time, and is represented by a "new user" arrow in the figure.

Also, if you apply the Scientific Method (which you should, since you are a Lean Startup) you have to use Statistical Hypothesis Testing as a tool to reliably validate or reject a hypothesis (read more about this on this post). This tool usually makes the assumption that you work with new users (read about the Independence Assumption on this post).

Recommendation: Always compare the behaviour of new users who use a version A of your product with new users who use a version B of your product.

The same recommendation applies to simultaneous A/B split testing.

OK, this post was quite straightforward, but we haven't covered everything there is to learn about users in a lean startup. The next post is less straightforward and a bit scary, it talks about zombie users.