General Best Practices
We always recommend that customers create separate apps in Apptimize for their Staging and Production environments. This allows them to freely and extensively test out experiments and feature flags in Staging without worrying about any adverse effects on the end-user experience. This also keeps the Production Dashboard clean and manageable. You can find steps to Add a New Application in our Documentation here.
For teams that have the necessary development resources, we highly recommend always using the programmatic approach for QA along with using Preview Variants to test the variants.
It is recommended to make use of the tags and notes fields as much as possible and to integrate updating these fields into the day-to-day experimentation process. These fields can be used to store Jira ticket numbers and links to specs so that it is easier to track and reference the stories related to the experiment or feature flag.
It is recommended to use multiple devices to test out an experiment before releasing it to the end-users, especially for visual experiments. It is a good idea to use devices with different screen sizes to verify that the changes appear as expected on all screen sizes.
It is crucial to verify that participation events are reported where they are expected in the app and that any events that are important to measure the success of the experiment are recorded after the participation event. Apptimize results only show conversions for events that occured after the user has already participated in the experiment.
Ideally, you should always do all your QA in your Staging environment but if you were to do it in the production dashboard in Apptimize, remember to create a fresh copy of the experiment and archive the original one so that any results introduced during QA are not a part of the actual experiment results.
What are some things to look for during QA?
- Ensure participation for the experiment is recorded where you expect the user to be exposed to the experiment
- Check that your events are getting picked up by the Apptimize SDK
- Ensure participation gets recorded before the events relevant to the experiment
- Ensure metadata is received in time to deliver the experiment
Recommended Methods to QA
Check out our guide on various approaches to QA'ing your Apptimize projects here! As always, please don't hesitate to reach out to us at firstname.lastname@example.org if you have any specific requests or questions.