Cheap Products

Hi

Vists x Conversion Rate x Lift x Order Value

We do some straight-forward arithmetic.
This is how much raw GMS we would expect to make. Etsy takes a small slice of this.
The idea is to take on projects that at least pay for your salary, and to weight projects against each other based on their projected impact.
What kind of lifts are reasonable? It really depends, but the most spectacular lifts we've seen is 2%.
We've actually achieved 5% lifts in mobile web experiments, but that's probalby beacuse our mobile web experience was so embarassingly poor to begin with.
Your specifics of your metrics will obviously vary, but you can get a simply back-of-the-napkin model most things this way.

Empty Search

And here's the thing, we couldn't tell why it failed
If you change everything, you've learned nothing.
It's virtually impossible to tell which of the dozens of differences, or combinations of differences, yielded this result.
There are ways to deal. You end up doing a lot of triage experiments
Where you isolate individual changes and test them against each other
But those are time consuming and soul crushing. 50 steps forward, 20 steps back. IF you're lucky.
The more granular you can make your changes, the finer the feedback that you'll get. Which brings me to the subject of gaining confidence