App Review Analysis

My time with Testlio allowed me the opportunity to collaborate with some of the most brilliant minds and largest enterprise-level companies in the industry today.

One project, my App Review Analysis endeavor, stood out as a great example of my work. Using a process and resources I created, I was able to identify and report on what was most important to customers with supporting facts and data. This work allowed me to pioneer a successful IAP/subscription and payments testing strategy for the client that is still in use today.

Setup & context

I identified a pressing need for user sentiment analysis, then proactively created an easily-maintainable resource to track/report trends and uncover high priority issues faced by users for reproduction and escalation.

This mapping of user sentiment to QA coverage via contextual/user-centered design principles was done based on the growing negative sentiment I was seeing in one of my client's App and Play Store reviews.

To track reviews, my team first worked together to integrate app reviews into our Slack workflow so we would be alerted each time a new review 3 stars or lower was posted to the App Store or Play Store.

Next I established the master doc for tracking and reporting each review with a set ontology structure to easily group similar sentiments into identifiable affected features. I collected, labeled, annotated, and analyzed review data to understand context.

Finally, I created a month-by-month reporting structure with reliable, timely, and actionable reports to keep the client abreast of trends while escalating high priority concerns immediately.

The challenge & problem PROCESS

I faced a number of hurdles along the way, including review integration issues (duplicates, filtering, etc.), limited time/resources, and a learning curve to figure out what mattered most to the clients.

By iteratively improving the process via feedback and trial-and-error, I was able to build a fully-functional app analysis and reporting flow with established documentation and templates to keep maintenance lightweight.

Launch, impact, results

I was overwhelmed by the positive response to the new analysis process. From April 2024 to January 2026, I delivered impactful monthly, quarterly, and yearly metrics-oriented reports with linked JIRA defects based on deep analyses of app review trends that were immediately and enthusiastically picked up by the client.

Despite some technical hiccups, I was able to track over 400 user reviews; deliver 50+ actionable, high/critical priority issues that saved the company over $200k in ongoing revenue; and even had my work cited by the client's VP of Engineering during QBRs and planning sessions.

Click the spreadsheet image above to open a Proton Sheet copy of the anonymized version of my work. All identifying client information has been removed to protect their privacy.