It’s not uncommon to get excited about running your experiment and rush into it. This is one of the most typical pitfalls we see in running flights. The decision what to test lays the foundation for all other parts of a successful test setup.
Hence, we recommend spending enough time on this decision to avoid disappointment and frustration later on.
Ideally, you should be able to state your objective clearly in a manner similar to the following:
We are trying to decide x . To help us make this decision, we are going to test y (bonus: and analyze the data points: z). After analyzing z, we will have collected meaningful insights about x and feel confident in moving forward.
Very often, users come to us with a long list of insights they want to acquire. While many of those insights can be acquired through Horizon, we generally advise against trying to answer more than one question in one test. In our experience, tests that try to collect data on multiple questions tend to be more complicated and susceptible to unintended biases.
If you need to collect data on a variety of questions, we recommend several sequential tests. Rank the questions you would like to collect data on by how critical their implications are for your project (with the focus on those that could immediately doom your project if the data showed evidence against your hypothesis). If multiple questions feel like deal-breakers, begin with the ones that seem the most risky/the ones you are the most scared & pessimistic about. As humans, we have a tendency to begin with the topics we are optimistic about. As a result of that, projects can go on for a long while before the critical hypotheses everyone was worried about finally gets tested. Testing them early, allows you to kill the project before wasting resources on the easier parts.