Marketing Stats: Confidence Intervals for Mobile Device Performance
We recently conducted a statistically rigorous analysis looking at the old questions of whether on average there is a difference in average order value (AOV), click volume and conversion rate (CR) across different types of devices. Due to our testing design, inference from this study applies—but perhaps is not limited— to the online retail community for the period of Q4 2012, excluding the peak of the holiday season.
The data are metrics by device class— desktop computer, iPad, smartphone, and “other” tablet— for a random subset of online retailers measured over a random series of days prior to the holiday season in 2012.
Since the Apple iPad drove roughly 88% of tablet traffic, we consider iPad performance apart from other tablets such as the Amazon Kindle Fire and Google’s Nexus tablets. The study design is a balanced randomized complete block design, with device as the treatment factor, day-of-week as a blocking factor, and client as random subject.
The model incorporates a device-client interaction. Complex interactions may exist between device and the type of product or service sold (i.e. a user is a lot less likely to design and purchase a custom t-shirt on a small-scale device, than he is to purchase a textbook). Moreover, many of our clients have optimized their mobile user experience through mobile sites and smartphone/tablet apps. We also control for weekly cyclical effects that influence purchase patterns.
We recognize that, particularly for companies with significant brick and mortar operations, softer success metrics like “Get Directions” and click-to-call functionality should be incorporated to understand the full value of mobile traffic. Those metrics are not included here.
Generally Applicable Results
- Both the fixed device effect and variance among clients are highly significant at the 1% level with respect to mean daily AOV, click volume, and odds of conversion. This means relative device performance is consistent no matter who you are.
- There is a significant client-device interaction affecting smartphone and “other” tablet click volume, AOV and CR. However, the interaction does not significantly influence desktop or iPad traffic. We suspect, that this is due to differences in site design, and use of mobile sites/smartphone apps optimized for small-scale devices. Sites customized for mobile use have special practical importance with respect to CR; our study suggests that your odds of conversion from a smartphone or “other” tablet increase by 21% to 28% when you target and optimize for small displays.
- At the 5% experiment-wise level of significance, Dunnett’s multiple comparison of each device with desktop as the control indicates that, on average, iPad and smartphone AOV is not significantly lower than that of desktop computers. Only “other” tablets (other than the iPad) have a significantly lower—by somewhere between 12% and 61%— AOV by comparison.
- With respect to click volume, all three device classes underperformed desktop computers by a huge margin as expected: click volume was 94.0 – 95.5% lower for the iPad, 95.0 – 96.5% lower for smartphones, and 99.1 – 99.4% lower for “other” tablets at the time.
- iPads best desktop computers with respect to CR by somewhere between 7% and 11%. Smartphones and “other” tablets significantly underperform desktops by 66 – 68%, and 49 – 56% respectively.
Where’s the Opportunity?
So what can you do to improve traffic and boost your sales dollars? The difference in click volume and conversion rate between desktops and iPads may solely be due to the fact that iPad users are fewer in number and still tend to be more affluent, hence these users generate much less traffic but spend more readily than your average desktop user. Moreover, since iPads usually require no special treatment with respect to how you serve your site, there is not much to be done to improve iPad traffic, except to bid up your ads.
Smartphone AOV already keeps pace with desktops and iPads, while click volume is marginally lower for smartphones relative to the iPad. Conversion rate for phones lags considerably, but appears sensitive to site design optimized for small-scale devices. Plus, there are plenty of users: approximately half of American adults own a smartphone, compared to only about 10% who own an iPad. So, if you aren’t doing so already, consider optimizing for small displays to boost your phone conversion rate.
The Android tablet market is growing, and delivering ever more affordable alternatives to the iPad. We also have Blackberry and Windows 8 devices in the mix. There may be a lot of opportunity in this new space. That said, we only have to look at how wide the AOV confidence interval is for “other” tablets to conclude that first we need to break this class of tablet down by individual device before we can make any meaningful inference with broader implications.