Here was my prescription for this situation. I asked the team to consider creating what I call a sandbox for experimentation. The sandbox is an area of the product where the following rules are strictly enforced:
1. Any team can create a true split-test experiment that affects only the sandboxed parts of the product, however:
2. One team must see the whole experiment through end-to-end.
3. No experiment can run longer than a specified amount of time (usually a few weeks).
4. No experiment can affect more than a specified number of customers (usually expressed as a % of total).
5. Every experiment has to be evaluated based on a single standard report of 5-10 (no more) key metrics.
6. Any team that creates an experiment must monitor the metrics and customer reactions (support calls, forum threads, etc) while the experiment is in-progress, and abort if something catastrophic happens.
(Link: Lessons Learned: Innovation inside the box)
Lessons Learned: Innovation inside the box
Advertisement


October 12, 2009

No comments yet... Be the first to leave a reply!