How do I carry out Targeting-based QA?

In this article:

• Summary
• Examples
  
  • Using Custom Attributes for Targeting
  • Using Pilot Groups for Targeting
  • Using Version Numbers for Targeting

Summary

Sometimes the best way to QA is to experience a variant exactly like how an end-user would experience it. That way, you can verify experiences that might otherwise get missed in regular testing. This approach is also useful to check that the experiment does not introduce any lag and that the results are being collected and displayed as expected.

This approach works well for customers with larger QA teams where it might be challenging to distribute development builds or teach everyone how to use pairing tokens. This can be achieved by specifying targeting criteria. Using this approach in conjunction with verbose logging makes it very simple to troubleshoot experiments and feature flags. It is recommended that you use your Staging environment and set projects live on an internal build.

This method is currently our main recommended method for carrying out QA for web projects.When you set an A/B experiment live in your staging environment, you can QA the various variants by toggling the percentage allocation for each variant. Apologies for the inconvenience; however, we are currently working on new methods of facilitating the QA process. Please email us at support@apptimize.com if you have any questions.

Using Custom Attributes for Targeting

You can set any user characteristic that is available in your app as a Custom Attribute to target your user cohorts.

For example: You can use the email used to sign in, to set a Custom Attribute that tells you if a user is an employee of the company and target the experiment to be delivered to them.

Link to Documentation: Custom Attributes

Using Pilot Groups for Targeting

One of the most common ways customers use to target experiments to the internal team is using Pilot Groups. Groups can be created in Apptimize based on the values of a pilot targeting id that can be set in your app. The pilot targeting id can be any attribute in your app and can be set up very easily. Once the pilot targeting id is set in the code, we recommend creating a group for your QA team by uploading their ids to the dashboard. This way every time you want to send an experiment to only your internal team, you can easily do so by just adding the group to the targeting criteria.

We’ve added an extension to this with the addition of Preview Groups when setting up an experiment. With Preview Groups, you can add Pilot Groups and Pilot Users to see specified experiment variants before or during an experiment for QA purposes. There are two modes available for Preview Groups:

 Follow Experiment - Preview Groups set up to follow the experiment will not have access to their variants until the experiment is launched.

Launch Preview - For Preview Groups you want to give early variant access to, prior to an experiment launch date, you can select the Launch Preview mode. This mode will give users access right away!

Note that pilot groups are independent of targeting. Pilot users will always see your experiment or feature flag, regardless of whether they meet other specified targeting or version criteria.

Link to Product Documentation: Pilot Groups, Preview Variants, and Preview Groups.

Link to SDK Documentation: setPilotTargetingID function (iOS / Android).

Using Version Numbers for Targeting

One of the easiest ways to target only the QA team for an experiment even if you do not have a custom attribute or pilot targeting id set up in your code is to target based on app version. Usually the app version in QA is higher than that in the AppStore or PlayStore. So you can target the QA team for an experiment just by simply setting the targeting for app version to be greater than the current version of the app in the AppStore/PlayStore. Keep in mind that if you use this method, you must make sure to update the targeting criteria if an update to your app is released.