The guiding light to a good experiment is the hypothesis. It is the second step to setting up an experiment, right after establishing your risky assumption. If you can't articulate a hypothesis, then you don't have an experiment.
The most straightforward and clear way to write a hypothesis is the if / then format. Let's break it down.
The if is an isolated change in the application that will lead to a desired behavior or outcome and should be coupled with your assumption. It can be as simple as better feedback, or as complex as adding a feature. I try to write these as solutions. From there, you should be able to infer the desire or needs of the user - or better yet, a problem they are facing.
- If we add a success message after...
- If users can chat with one-another in the application...
- If users can challenge one another...
- If there is transparency in who is rating the products...
- If we add hundreds of new products...
- If we make our app responsive and remove the landing page...
The then is focused on our desired behavior or outcome and should be measurable. Avoid desired outcomes that deal with emotion. Although you could measure how happy a person is, it is generally difficult to rely on.
- Then we will get more sign ups.
- Then users will share the app with each other.
- Then more users will participate in the forum.
- Then the users will have more fun.
- Then the users wont be so annoyed.
A well written hypothesis should help you understand from a high level how you are going to add value to the user, and how adding that value is going to result in a desired and measurable outcome.