Over eight months and one marketplace app client, we ran 40 store-listing A/B tests — icons, screenshots, preview video thumbnails, short and long descriptions, all tested through the platforms’ native experiment tools with traffic split evenly and each test run until it reached statistical significance, usually seven to fourteen days depending on install volume. The icon won more of those forty tests, measured by lift in conversion-to-install, than every other element combined.
How we actually tested
Each test isolated a single variable against the current control, never multiple changes at once — tempting as it is to ship a “refreshed” icon, new screenshots and new copy together, doing that makes it impossible to know which change actually moved the number. We required a minimum sample of roughly 1,000 store visits per variant before calling a result, and anything under a 95% confidence threshold got re-run rather than shipped on a hunch.
Four things that held up across nearly every test
- A single bold silhouette beat a busier, more “informative” icon, every time we tested them head to head. Icons trying to communicate two or three features at once — a small badge, a secondary symbol, extra text — consistently underperformed a simpler mark, even when the busier version tested well internally with the team. What reads as informative up close reads as noise at thumbnail size.
- Color contrast against the store’s background mattered more than brand color fidelity. One icon “won” its test purely because it was the only warm color on a screen otherwise dominated by blue and white app icons — a result that had nothing to do with the icon’s inherent quality and everything to do with what it was sitting next to. We started testing icons against live category search results instead of in isolation after this.
- Text on the icon almost always lost. Any legible word reduced conversion versus a text-free version in 11 of 13 tests where we tried it.
- Human faces outperformed abstract marks when the app itself involved a service delivered by a person — a small but consistent lift for anything hospitality- or coaching-adjacent, absent for utility apps, where an abstract mark still won.
The exception that matters more than the rule
Two tests broke the “no text” rule, and both were a single large numeral — a “2” in one case, referencing the app’s second major version, styled as the entire icon. Both outperformed the icon-only control. Our read: a numeral doesn’t get processed as “text to read,” it gets processed as a shape, the same way the rest of the icon does — so it doesn’t add the cognitive load a word does.
The more important exception is structural, not visual: the icon that won in isolated A/B tests wasn’t always the icon that performed best inside a full store listing, once real screenshots, ratings and a category ranking were sitting next to it. Context changed the winner in roughly a quarter of cases. That’s the actual argument for continuous testing over picking a “best” icon and moving on — the winner isn’t fixed, because the competitive set around it never stops changing.
Where we tell clients to spend the first budget
Screenshots and descriptions still move install rate, and we test them continuously. But if a client only has budget for one test this quarter, we tell them to test the icon first. It’s the smallest asset on the page and it moved the number more reliably than anything else we measured.