Are there any good strategies for A/B testing on mobile devices?
ab-testing, frameworks, mobile, optimization, testing
Solution
A/B testing on mobile is different from website in two main points:
- A website is always connected to the internet. A native mobile app is like a Windows app in the 90's: it does not always have a reliable internet connection
- The web is much more mature than the mobile. Modern web A/B testing tools can completely control all the elements on a web page.
The basic thing to do is to create a client-server architecture:
Tag manually the elements to test in your app
You will tag manually the element that you would like to A/B test and your framework will ask to the server which element to show. (The variations are stored on the server side). For example, the color of the button can be retrieved from the server and set at runtime when the button is displayed. It is not completely true that native mobile interfaces are static. They can be modified at runtime programmatically.
Send events to your server
The second step is to send an event to your server when the variation has been viewed and when the conversion succeeds.
Potential issues
By using this trivial architecture you will have the following problems:
Always show something
When the button should be displayed, the mobile device might not have an internet connection at all. You still need to show something to your user event if the internet connection is not there. The solution is to display a default value.
Store events
When a goal is performed (for example a click on a button) and if the mobile device is offline, the event will be dropped and you will end with wrong statistics. The solution is to save your events locally and to send them to the server later (when the internet connection is back).
Never wait for a variation
If you ask a variation from the server only when you need it, you application will be really slow (which is not acceptable). You need to retrieve your variation data as soon as possible in your application.
You mobile framework should be built to never block the application. That's a golden rule on mobile.
Save variation attributions
The server should save which variation has been assigned to a device to get reliable results.
Technology stack
Operating system/programming language
You can choose any technology stack. Just make sure that your server side can handle the load. If your mobile apps have many users, you will get many requests on your server. You might need to have a load balancer between your web servers. If you choose non-free technologies, watch out for licensing cost as high load means high cost.
Database
You might also have millions to database records. Generating reports might not be just easy as doing a "SELECT COUNT(*)" from you database table for performance issues. Be careful if you choose non-free technologies.
Disclaimer: I am the CTO of Arise.io
Problem
Based on what I know, A/B testing frameworks are for working with websites. For example, you want to know which of blue/red color of "Buy" button will encourage user to click it. Since website information of user interface comes from server, and browsers display it, A/B testing is possible. However, since the user interface of mobile applications are native and static, it's not possible to implement A/B testing in the same manner (if I'm not mistaken). So, my question is how to implement an A/B testing framework from the database/services all the way through to a native mobile application. Any suggestion would be appreciated. Thanks.