Mastering Product Page Optimization and iOS A/B Testing for App Store Success

In the fiercely competitive world of app marketing, every element on your App Store product page can impact your app’s success. From eye-catching icons to persuasive screenshots, optimizing these assets is critical for attracting users and driving downloads. This is where Product Page Optimization (PPO) and A/B testing come into play, giving you a data-backed edge to fine-tune your app’s appeal. With Apple’s PPO tool, you can experiment with different visuals, messages, and layouts, identifying what truly resonates with your audience and converts visitors into loyal users. Ready to turn your product page into a conversion powerhouse? Let’s dive into the essentials of mastering iOS A/B testing and product page optimization!

Table of Contents

• What is Product Page Optimization and How Does it Work?
• What Can You Test in the App Store?
• How to set up a test?
• Analyzing the Performance of Your Tests
• iOS Testing Glossary
• Differences Between CPP (Custom Product Page) and PPO (Product Page Optimization)
• Difference with Google’s Store Listing Experiments
• Best Practices for Product Page Optimization
• Final Thoughts

What is Product Page Optimization and How Does it Work?

A/B testing, also known as split testing, entails creating multiple variations (A and B) of a particular app element, like screenshots or icons. These variations are then tested against each other, with users randomly assigned to see one version. The interactions are analyzed to identify which version performs better in terms of conversions.

Apple Product Page Optimization (PPO) is a native A/B testing tool that allows you to test your product page in the App Store Connect, and compare its performance against variants where some of your creative assets have been changed.

The iOS A/B testing feature allows you to experiment with up to 3 different variants (or treatments, using the App Store Connect specific language) and run each test for up to 90 days. These treatments will be shown only to a portion of your audience, which you will define. This way, you control how best to allocate your efforts. Keep in mind that you cannot run more than one test at a time.

Product page optimizations will provide you with valuable insights, such as: what kind of assets and value propositions resonate best with your visitors, what kind of content drives the most conversions in different markets, and whether seasonal content has a relevant impact on your app, among other things. Tests help you better understand the expectations of your audience so you can adjust your app page accordingly.

The main goal of iOS A/B testing is to improve your product page and main message by making data-based decisions. If you detect what works better through testing, you are more likely to see better performance and increased CVR from your app moving forward. If you implement new assets without testing, then you are only guessing. Product page optimization allows you to move forward with certainty.

Note: PPO was released with iOS 15, meaning only store visitors with iOS 15 or later on their device will see A/B test treatments.

>What Can You Test in the App Store?

• Icons: The app icon is the most visible and important marketing asset for the App Store. It is especially important for browse placements since it is the only visual element displayed throughout the browse results (e.g., Top Charts, Category tabs, Today, Apps, etc.)(In order to test different icons, all variants of your icon must be included in the binary of your published app and must be built using Xcode 13 or later. After submitting the version and adding the icons, you’ll have the option to pick them using the PPO. You’ll see a new tab for “App Icon” – click on it and choose your variants).
  

• Screenshots: The app’s screenshots are the most prominent asset, taking up most of the product page’s real estate. Make sure to include your main features and unique selling points in the first three screenshots.the elements to be tested here can be many; the text captions, the style and the background, the messaging strategy and storytelling, the graphic composition and layout of the image, color palette, fonts (if applicable, since brands will likely have their font defined), the features of the app shown and the order in which they are shown, etc. Please make sure to follow the iOS minimalistic and clean design and show the app’s UI to avoid getting rejected.
• Preview videos: The elements to test are the same as in the screenshots. For the video to be approved, it is important to follow the App Store guidelines and video specifications. For example, the video should be 15-30 seconds long, under 500 MB, and must match the app’s real user interface (don’t include actual people interacting with the device, such as fingers tapping the screen).Tip: App previews play with the sound muted by default. Therefore, adding a compelling copy to the video would help convey the message.

Testing the app’s metadata is currently not an option.

How to set up a test?

1. Log into your App Store Connect account
2. From “My Apps,” select the app you want to conduct the A/B test for
3. Select Product Page Optimization from the Growth & Marketing menu on the left
   
4. Click on the “+” on the header. If you don’t see the “+” button, you likely have a test already running or in draft. You can only have one active or draft test at a time
   
5. Choose a reference name for the test. It can be anything you like. Our suggestion is to write something that accurately describes the purpose of the test, and that you can easily identify (ex. “White background Screenshots test”)
6. Choose the number of variations (“treatments”) you want to test
7. Select the traffic proportion to which you will show the test. If you are not sure, you can always divide evenly, so if you’re only testing one treatment, you can divide by 50% to keep the test balanced.
   
8. Pick a language for localization purposes if this applies to your app and your test. Keep in mind that your test may take longer to reach meaningful results, depending on the localizations you’ve selected
9. Click Create Test
10. On each Treatment, delete the default assets (which are copied from the default product page) and upload the new assets you want to test. You will need assets in different sizes: 5.5, 6.5, 6.7, and also iPad sizes
    
11. Make sure your assets are shown in the proper order. You can rearrange them if not so
12. Click on Start Test
13. Click on Add for Review (assets need to be reviewed by Apple before the test can start)
14. Click on Submit to App Review (make sure you submit otherwise the test will not start).

Once done, and if no errors are detected during the review, the test should start automatically since iOS A/B testing is not dependent on version release.

Approval from Apple usually takes from 12 to 24 hours.

Analyzing the Performance of Your Tests

List of things to keep in mind when evaluating the performance of the test:

• Once the test is collecting data, you can check its performance by clicking on “View in Analytics” from the product page optimization menu.
• The main metric to measure is CVR (conversion rate), which determines which treatment converts more users.
• How long should you keep a test running? Until reasonable results are obtained with a high confidence level, which will depend on the volume of traffic the app receives. As a general rule, wait at least two weeks, or even a month, if the results are not conclusive.
• Once your test shows a trend, determine if it is statistically significant. If significant, pick the treatment that performs better and apply it. If results are inconclusive, keep running the test and gathering more data, or end the test and keep your original assets.
• Before applying a treatment or stopping your test, we recommend waiting until at least one treatment is declared to be performing better or worse than the baseline, with at least 90% confidence.

Archer Pride Month – Icon A/B Testing:

Current:

Treatment A:

Test Results:

iOS Testing Glossary

Unique Impressions: The number of unique users running iOS 15, or later, that viewed the variant on the App Store during the testing period. A Unique Impression is counted when a customer views your app on the Today, Games, Apps, or Search tabs on the App Store, or on your product page.

Conversion Rate (CVR): The estimated percentage of users who download the app after seeing this variant of the product page.
Improvement: The estimated relative difference between this variant and the baseline. For example, if the CVR grew from 30% to 33%, the improvement rate would be 10%.

Confidence: The probability that the data gathered in a test suggests that two variants are performing differently. The confidence level depends on the statistical significance of the test results. If the differences between variations are small or if there is a high degree of overlap in performance metrics, it may lead to a lower confidence level.

Apple calculates the confidence level based on a statistical analysis of the test results. While the exact methodology used by Apple is not publicly disclosed, it likely involves techniques commonly employed in A/B testing and statistical hypothesis testing.

Differences Between CPP (Custom Product Page) and PPO (Product Page Optimization)

• PPO should not be confused with CPP (Custom product page). They are two different features that were both released with the launch of iOS 15.
• While PPO is an iOS A/B testing tool, CPPs allow you to create a copy of your product page with different assets to be used as landing pages for your paid campaigns, especially ASA (Apple Search Ads).
• While you can run only 1 PPO at a time, you can create up to 35 custom product pages per app.
• The goals of each feature are different. CPPs aim to customize the user journey according to the keywords/ads they come from, thus increasing the conversion rate. PPOs, on the other hand, aim to test different visual assets and layouts to improve your main product page and boost CVR on the App Store.
• The traffic sources for both are also different. CPPs can’t be found through organic search results; they can only be accessed through unique URLs (they will be featured in search results if the app is doing Apple Search Ads). PPOs, on the other hand, share the visitors with your main product page, splitting them for testing purposes. Users from all the different traffic sources will see the test: organic, paid, referrals, and browse traffic.

For more information and tips on how to optimize your iOS store listing, read our full guide.

Difference with Google’s Store Listing Experiments

Google also offers its own A/B testing tool (Store Listing Experiments), but there are a number of differences between them and PPOs:

• The main difference is that only creative assets can be tested on iOS. On Google Play, however, it is also possible to test short and long descriptions from your metadata.
• When testing an icon, PPO requires that you include the icon in the binary of your app. On Android, you don’t need to prepare a new version each time you want to test a new icon. The icon can be uploaded as a file, like the test of the assets on Android.
• You can have up to 5 tests on Android, while you can only have 1 on iOS.
• Also, the maximum duration of the tests differs. PPOs can run for up to 90 days, while A/B testing on Google Play can run unlimitedly until you stop the test.
• On Android, the test is only visible for first-time installers, unlike iOS.

Best Practices for Product Page Optimization

• Don’t test for the sake of testing. Define a hypothesis you wish to challenge with your test, this will help you obtain clearer insights. For instance: “Including review snippets (social proof) in the screenshots, will increase the users’ sense of security and will lead to a higher engagement rate”, or “Adding popular characters to the screenshots will increase familiarity and engagement while also increasing retention rate”. The hypothesis will be verified or denied by the test results, and you’ll be able to make informed decisions about your product page.
• Designing a test is a creative process, and you might be inspired to include many different variations. Should you change the icon, and also the background color, and the text? While you can make as many changes as you prefer, we recommend testing one asset at a time. This way, results will clearly apply to the one element you are testing. If you test many elements at once, you will not be 100% sure which of those elements is responsible for the better performance.
• Make sure to allow sufficient time for your test to collect enough data. As mentioned, at least two weeks is recommended since user activity can differ depending on the days of the week. But the more time you allow your test to run, the more confident you will be about your results. However, not overly long, as testing for a long period of time (2 months) can also yield inconclusive results: over time, extreme performance results can naturally regress to the mean. This means that any early differences observed between variations might diminish over time, making it difficult to determine if there was ever a true difference or if it was just an anomaly. And, naturally, seasons, holidays, or events that significantly impact user behavior need to be taken into account. Ultimately, the measure of what “sufficient time” is will depend on the volume of traffic your app gets.
• Try to avoid A/B testing while running In-app events. The events might interfere with the results, as some users will see the event card first on the results page or on the product page.
• Consider timing when planning your tests and other marketing efforts. Be mindful of marketing actions like paid campaigns, which can influence user behavior and distort test results.
• Avoid testing during holiday seasons if your assets are not relevant in that particular context, as they can impact traffic and result in unreliable data. Naturally, if you’re adding seasonal assets to your product page you might want to test them (for instance, different screenshot treatments during Christmas).

Final Thoughts

Mastering product page optimization and iOS A/B testing is crucial for achieving success in the App Store. By leveraging tools like Apple Product Page Optimization, you can make data-driven decisions to enhance your app’s visibility and conversion rates. Whether you’re testing icons, screenshots, or preview videos, each experiment offers valuable insights into what resonates with your audience.

For expert guidance and to maximize your app’s potential, contact yellowHEAD today. Our team of specialists is ready to help you navigate the complexities of App Store optimization and drive your app’s success. Reach out to us and let’s elevate your app’s performance together.