What are Creative Sets?
When Apple introduced its Search Ads product, allowing marketers to directly advertise on the App Store search results page, it also left a wide gap in its product.
Compared to other advertising products, such as Google AdWords and Facebook mobile install campaigns, Apple didn’t allow users to test their ad creatives. Knowing that user acquisition budgets usually pour towards ads with the highest return on ad spend (ROAS), Apple decided to allow marketers and user acquisition teams the option to test their Search Ads creatives.
They called the testing solution “Creative Sets.”
Creative Sets are collections of app screenshots and previews that you upload to App Store Connect. After setting up your app and assets on App Store Connect, you can create and link Creative Sets to ad groups through the Application Programming Interface (API) or through the Apple Search Ads User Interface (UI).
Creative Sets enable you to test ad variations in groups and optimize for different keywords, devices, and display sizes.
These sets can be applied to change the screenshots used for a specific keyword (or keyword group). For example when a food delivery app is targeting the term “burger,” it can use screenshots of scrumptious burgers, and when targeting “sandwiches” it can use images of mouth-watering sandwiches.
What Apple Search Ads Creative Sets Are and What They Are Not
Search Ads Creative Sets are not actually an a/b testing solution for App Store creatives. Instead, they are Apple’s attempt to allow Search Ads users to dynamically serve different creatives in different ad groups. It also optimizes the tap-through and install rate by matching assets and messaging to users based on keywords.
Similar to how Facebook, Google, and other mobile ad platforms provide advertisers with testing tools to optimize their ad creatives, Apple created a great tool for optimizing ROAS.
As with any tool, it was designed to do a very specific job (optimizing ROAS for Search Ads), and utilizing it to perform other jobs can be tricky at best and dangerous for App Store Key Performance Indicators at worst.
The Risk of Testing Creative Assets on Creative Sets and Implementing Them on the App Store
One of the most common mistakes we see on the App Store is marketers utilizing Creative Sets to test creatives, using these creatives to update their app store page, and then expecting an uplift in conversion rates for browse, search, and referral (paid) traffic.
In many cases, doing so will lead to an increase in performance for Search Ads but a decrease in performance for other channels, including organic search. Let’s examine why this is happening.
Reason 1: The User Discovery Journey is Different
Users in the App Store discover apps through three main journeys. Let’s analyze the mindset users have throughout each journey and the creatives they are exposed to in different stages of that journey.
App Store Browse Journey
- The journey
Users go into the App Store and find apps that are featured either on the Today tab or the Games/Apps tabs. Users also browse the Top Charts, either for general apps/games or overall categories.
- The mindset
Users in this journey have a very low intent mindset. They aren’t looking for anything specific, but are browsing the store opportunistically for great apps and games to download.
- The creatives
In most instances, including the Top Charts, users are exposed to the app’s icon, title, and subtitle. In some situations, they are also exposed to the product page screenshots.
Referral Journey
- The journey
Users land directly on a product page (either from a paid ad or any other type of link, for example, from a brand’s mobile website).
- The mindset
Users in this group have a very high intent to download the specific app/game they were exposed to. An ad convinced them to tap on it and explore further.
- The creatives
Referral users are exposed to the ad creative (usually on other platforms, such as Facebook, Google, or a different app/game through in-app advertising). They have then been exposed to the App Store product page where most users will never go past the “First Impression” or the above-the-fold area of the product page (which includes the icon, title, subtitle, first one or two screenshots, and (if it exists) an autoplaying video).
Search Journey
- The journey
Users use the App Store search tab for either brand-related terms (such as “McDonald’s”) or intent-related terms (such as “burgers”). In most cases, these users will also be presented with an ad. Some users will tap on the ad: this group is categorized as paid search. Some users will tap on an organic search listing and be categorized as organic search.
- The mindset
Users in this group have a mid-high intent mindset. They are either looking for a specific app they heard about from a different source (word-of-mouth, seeing an ad, reading content about the app, etc.) or are looking in general terms, as they know what type of app/game they’re looking for but don’t know which one specifically.
- The creatives
Search users are exposed to the app icon, title, subtitle, and the first one to three screenshots from the product page gallery (that may include an autoplaying video). If they choose to install directly from the search results, they won’t be exposed to further creatives; but, if they choose to tap through to the full product page, they’ll be exposed to the entire product page before they make the decision whether to install.
So, using Creative Sets, you’ll be optimizing for a particular group of users—paid search traffic. You won’t be optimizing your creatives for the other groups of users, such as browse, referral, and organic search. It’s merely a different journey, and the creatives you’re optimizing (the screenshots that appear within a search ad) would be the right ones to attract that paid search audience. As the mindset of these users is different (shopping for apps/games vs. looking for a variety of options), the creatives that perform better will, by definition, be nonoptimal to the other groups of users.
For these groups, you need to use other messaging to fit their mindset and motivations, or at least messaging that doesn’t deter them from installing.
Lastly, in most cases for non-paid search traffic, creatives should work together with the screenshots to tell one cohesive story. By updating the winning screenshots from the Creative Set test, you could actually hurt the performance of these other traffic sources by creating a detachment between the screenshots and the other elements that work together.
For example, it could be that for browse traffic, one of the most important dynamics of the page is that users see a character in the icon, get intrigued, and explore the screenshot gallery to find more assets relating to that character which excite them and drive them to install.
Let’s assume that a Creative Sets test showed that a certain screenshot set has won, and you updated your product page with it. These winning screenshots lack that character (because it was less important for paid search traffic), so the conversion rate for browse traffic (that could represent 30-40% of all traffic) drops.
Reason 2: The Test Audience and Method Are Skewed
Another factor to consider is that Apple won’t randomly serve the Creative Set to 50% of the traffic at any given time, and the percent of the traffic that each creative gets will change over time. This is because the solution is designed to be more of a dynamic optimization and personalization tool than an a/b testing tool.
When traffic isn’t being randomly exposed to each variant of the Creative Set test at any given time, results can be dramatically skewed as they don’t follow the principle of using a randomized sample.
This will render results invalid in most cases, and once the results are implemented, you’ll experience a very different KPI profile (conversion rates, tap through rates) as the winning variant will be shown to 100% of the paid search traffic all the time.
In Conclusion: Why Should You Use Creative Sets?
So, to recap, Creative Sets are a great way to optimize ROAS for paid Search Ads. But using them to test App Store screenshots and expecting an uplift in conversion rates for the other channels is risky, as it can drive a decrease in performance overall.
Given that, in order to run an effective Search Ad test, you need to invest enough in the ad to drive a sufficient sample size. Since Search Ads have an extremely high cost per install, we recommend reconsidering.
Although Search Ads might be a good signal for what messaging and creatives will work well with mid-high intent search users in general, it will most likely create problems elsewhere (in the form of lower conversion rates for referral, organic search, and browse) which could be extremely costly.
