Sometimes game and app marketers face an extremely challenging task: preparing marketing materials for an upcoming launch. As the preparation starts before users are exposed to the new product in scale, it’s extremely challenging to understand which marketing messaging strategy will be the most effective for your ASO work. The last challenge is the fact that with a mobile product some of the most important marketing tools are app store creative assets. As users can’t install a mobile app outside of the App Store or the Google Play store, all of the potential users will be exposed to these messages. But what works best?
The results speak for themselves. We at Storemaven saw thousands of app store tests that were implemented prior to launch. Many times, we pause to benchmark app store conversion rates for newly-launched apps and compare two groups.
The first group consists of app store pages that were extensively optimized prior to launch by utilizing app store a/b testing. The second group consists of new products that were first launched and only thereafter began a methodical optimization process.
The results? Those who optimized before the launch converted, on average, 90% better than those who launched with an unoptimized version of their app store page. It’s easy to grasp this difference—those who didn’t test simply used creative assets and messages that reflected their very initial hypothesis for what will drive users to install. Those who optimized were already incorporating several rounds of learnings about their users into their app store marketing assets.
The Implications of not testing before launch
The consequences of not optimizing your app store page before launch are two-pronged.
- Organic Hit – Both the App Store and the Google Play store use first-time install conversion rates as a metric that influences their algorithm as it decides which apps to highlight to users, be it through the Top Charts or through search. Launching with subpar conversion rates makes optimizing for discoverability of your app an uphill battle.
- Paid Hit – As users can’t install mobile apps outside of the store through a paid ad, the paid user journey goes from an ad to the app store to an install. Conversions rate on that second step of landing on an app store page significantly influences the cost per install.
At the end of the day, you’re not the only one launching and marketing a new app in your category. Your competitors who test will enjoy better user acquisition (UA) and faster growth rates. The App Store and Google Play are full of beautiful products that never managed to achieve traction and were eventually abandoned by their developers as they didn’t turn out to be the success they had wished for.
In our view, testing your app store page prior to launch and gathering information about it are critical in driving users towards the GET or INSTALL button and should be an inherent part of launching on the app stores.
How to Test App Store Pages Before You Launch
Let’s start by defining the key terms:
App Store Concept Testing
App store concept testing is the process of quickly collecting market feedback about initial app ideas and concepts in order to validate market demand before launching. Additionally, app store concept testing is used to refine the development process, test various features, and test how users respond to the concept around these features as they consider installing.
App Store Prelaunch Testing
App store pre-launch testing is the process of refining an app’s creative assets prior to launch in order to maximize conversion rates for the most successful launch. Instead of learning about how users respond to different concepts, the goal is to learn how to best convey the developed product to your audience.
How to Prepare for App Store and Google Play Product Launch: A Mental Framework
We love mental models. They help us organize our thoughts around a complex topic so we can methodically approach a problem and solve it in an optimal way (yes, we optimize way too many things).
We divide the process of launching a new mobile app into two distinct phases:
How testing app store pages before launch works
One of the clear advantages of running an app store test using a platform such as Storemaven and using a separate testing environment is you can test your product before it’s even live.
The process of testing prior to launch goes through four stages:
- Defining an audience: carefully analyzing the audience you’ll be targeting, both in demographics and geography, and creating a targeting strategy that is aimed to bring these users to the test.
- Creating test ads: creating ads for the future app which call for users to install it and running campaigns on the channel of your choice (depending on your audience definition), such as Facebook.
- Creating app store product page variants: create replicated app store pages that users will land on after tapping on the ad using a platform such as StoreMaven.
- Create an ‘ending’: as the app isn’t live yet, you have two options for users who tap on the GET or INSTALL button. First, you can create a landing page announcing that the app isn’t live yet and offer the chance to sign up to get updates as to when it’ll be launched. Second, you can create a post-install survey which gives you a unique opportunity to ask this highly-qualified group of users additional questions to learn more about them so you can tailor your development and marketing preparation to their unique needs.
Here’s a section from an older article of ours, highly worth reading. It gives an even deeper look into concept testing, and the steps needed to follow to master it.
How to Approach Concept Testing
Before even approaching the actual test planning itself, it’s worth pausing and thinking about the questions you would like these tests to answer. Here are a few common questions:
- Is it worth adding a new franchise or a new app in an existing franchise?
- What is the best combination of intellectual property (IP) and game mechanics? What is the best combination of app features given the app’s concept/theme?
- What app concept will have the largest reach?
- What is the new app or game’s potential return on investment (ROI)?
- How do different demographics and interest groups respond to each concept?
- How does your existing network respond to this concept?
If we bring it into the real world, many major developers own multiple apps in the same theme or franchise. A major question that these developers have is whether they should come up with an entirely novel concept and expand to a new category or stick with what’s proven successful and use their current franchise(s) or brands.
For example, Supercell currently has two games in their successful “Clash” franchise—Clash of Clans and Clash Royale. Their situation would be an ideal case for concept testing, as it would help them determine if they should add a third title to this franchise or develop an entirely unrelated game that would enable them to reach a new market.
As you can see, the business questions you choose to answer lay the foundation for your entire concept test strategy.
Let’s dive into the steps of running the concept test process:
Phase 1: Determine the audience and the different concepts to test
First, you should define your different audiences. You can segment them by factors such as:
- Age (e.g., 18-24, 25-35, 35-44, etc.)
- Interests (e.g., Game of Thrones, casual games, etc.)
- Gender
- Location
At this stage, you can begin to identify the target audience and create a user persona. A major benefit of defining your audiences early on is that it enables you to understand if it’s an audience worth targeting. For example, if you were deciding whether you should build a tower defense game using an established IP (e.g., Game of Thrones), you’d first want to look at the reach on Facebook (i.e., how big the audience is for people interested in those two genres).
Continuing with the IP example, there would be two audiences you should look at:
1) the IP (in this case, Game of Thrones) audience and
2) the broader tower defense genre-loving audience.
The goal is to determine how attractive this concept would be among people who are fans of multiple genres. If these two audiences complement each other and have a wide reach, then you can be more certain it’s a good route and can begin concept testing. If the overall reach is low, then you might want to reconsider this concept.
As we’ll discuss in the analysis section, the goal is to find an idea that will appeal to the largest audience with the lowest cost per install (CPI). The metrics used to determine this is ad reach, overall conversion rate (CVR), and (of course) CPI.
Once you segment your audience, you should then identify the specific concepts (e.g., different game mechanics, app functionality, storylines, themes, etc.) you want to test. These concepts should be distinct enough so that when a “winner” is found, it cannot be confused with another concept.
Here’s an example of an initial concept test for a new game. The defined audience segments are directed to variations that showcase different game mechanics (in this example, match-3, interactive story, role-playing game (RPG) or tower defense). This first test is crucial in determining the type of game that will appeal to the widest number of potential users, and the results will guide the rest of the concept testing process.
Phase 2: Design app store pages that convey the different concepts
After narrowing down the different concepts, we can move forward and design app store pages with creative assets and messaging that conveys the concept.
After the test variants are ready, it’s time to start running traffic to the test which will run until we reach a statistically significant result.
Phase 3: Analyze initial results
After the initial results come in, we’ll be able to see which concept drove the highest conversion rates. This, in combination with data from Facebook (or other channels you chose) regarding the audience reach and the CPI, will shed light on the concept with the most potential.
If we continue with our example, you can see that the tower defense game mechanic won because it had the highest CVR and lowest CPI. Based on the results of this first test, you can more accurately verify that resources will be invested in the correct place and that the game will be developed to match the preferences of the widest possible audience.
To further enhance your learning, we recommend adding a survey that users have the chance to complete after clicking “install” on the test app store page. The questions can range from specific (e.g., Which character do you like best? Which features interest you?) to more broad, behavioral questions (e.g. Have you spent money in a game before? How often do you play games?). These survey questions are invaluable to:
- determine the quality of the traffic segment,
- find the most beloved (or hated) characters,
- identify aspects or features that could drive the most monetization and ROI, and
- pinpoint the most preferred (or disliked) features.
Based on concept tests we’ve run with leading developers, we’ve seen completion rates of up to 70%.
Phase 4: Continue to narrow down the concept
Using the winner of the first test, each subsequent test will help you determine a different aspect or feature of the app. The hypotheses become more specific, and the audience is more representative of your real audience.
For example, the question driving the second test above is which theme will appeal to the most users: aquatic, futuristic, middle ages, or tropical?
The rest of the concept testing process helps you home in on the granular specifics of the app or game. For apps, it could be which features or branding style piques the most interest. For games, it could be the characters, game setting, or storylines that appeal to most users.
Each test also uncovers unique app store engagement data. Among different target audiences,
- How do their preferences vary?
- How many visitors scrolled through my gallery?
- Which screenshots had the most impact?
- How many visitors dropped after seeing the first impression (everything above the fold)?
But why are these questions important? They represent another crucial step in the product development process—you’re trying to understand which messages and assets drive more people to install and how best to convey the unique selling points of your app. This enables you to identify the most impactful marketing messages to highlight in your broader mobile marketing and UA campaigns once development is complete.
Overall, concept testing offers you a data-backed way to better hedge your bets and find a product opportunity with a high chance of success. As you can see above, the tower defense genre is already saturated with different themes. Without concept testing, we would’ve spent valuable time and resources developing yet another tower defense game without fully understanding if it would appeal to a wide audience or even which theme to use in order to stand out among the current competitors. Through the testing process, we identified an opportunity to develop an aquatic-themed tower defense game that found a niche—a profitable place in the mobile game ecosystem.
How to Nail IP-Based Mobile Games with Concept Testing
One of the biggest challenges in mobile game development is matching IP with the right type of game. In many cases, a publisher would acquire the IP rights for a certain franchise (for example, Avatar) and will be puzzled by the question: which game should I develop based on this IP?
In some cases, the IP license is only for certain types of games (or even one). In this case, a publisher would want to ensure they’re acquiring a high-potential license.
The cost of developing a game type that isn’t received well by IP fans is tremendous. Not only will massive development and marketing budgets go down the drain, but the cost of acquiring the license to use the IP would also be wasted.
As we see in app stores, developing games based on IP is one of the most successful business models for mobile game publishers; but, in our view, they should treat the development process extra carefully given the sunken cost of the IP license.
Using concept testing to understand whether a match-3 Avatar game would work better than a 5×5 Player v Player (PvP) Avatar game is a great way to reduce risk and maximize the likelihood of success.
Recap and What’s Next?
As we reviewed, the value of concept testing isn’t minor—it can make or break an app launch. If you’re in the business of marketing or developing new games or apps (and I hope you are because, if not, it’s pretty weird you read this far), consider incorporating app store testing into the game development process and using it as a lever to maximize the likelihood for the app’s future success.
After nailing your concept, it’s time to prepare for the soft and global launches and tune the marketing messages and assets that’ll lead to maximum conversion rates.
