Picking GEMS from analytical events

Problem

An important aspect of any product's life cycle involves collecting user feedback, which can be categorized into two main types:

  • Explicit feedback: This includes direct user reviews, requests, and feedback provided openly.
  • Implicit feedback: This is related to raw data sourced from analytical events, logs, market trends, and other sources, often requiring careful interpretation for meaningful insights.

The handling of implicit feedback often becomes difficult due to poor inter-departmental communication within companies, particularly concerning analytics. There's a frequent disconnect where analysts may struggle with implementing events in code, while developers may lack comprehension of business processes and data analysis. This results in either an inadequate number of events being tracked or an overwhelming volume of user actions being monitored.

In our own experiences, we encountered similar challenges. After numerous projects, we developed an iterative framework named G.E.M.S. This framework allows for the precise selection of essential analytical events while ensuring they align with business requirements and needs.

Picking GEMS

The framework is simple for understanding in theory but can be more complex to implement in real projects, as it necessitates analyzing data across various roles and levels of abstraction. Fundamentally, it has four steps that can be iteratively repeated to refine its outcomes.

1 - Goals

In this phase, we declare the objectives we aim to accomplish with our product. Gathering data becomes purposeless if it doesn't contribute to these goals. Simultaneously, it's crucial to embrace all our objectives here to ensure comprehensive coverage in our analytics efforts.

2 - Expectations

Prior to commencing data collection, let's prioritize formulating suitable questions that define how data can best reflect the attainment of our goals. These queries don't necessarily represent final metrics; instead, they offer a high-level understanding, signifying the information required to navigate in the right direction. During this stage, the focus should be only on strategizing how to achieve the goals declared in the preceding step.

3 - Metrics

Metrics represent the foundational raw data from which we derive information. In this phase, we define the analytical events themselves and determine how they will be integrated into the code. These events ought to provide the minimum essential data required to address the questions posed in the preceding step. While executing this stage, the accent should be placed on identifying only the necessary data rather than collecting everything that might be potentially useful in the future.

4 - Statements

This step is the statistical implementation of the previous phase. Here, you craft the ultimate visual representations within your analytical tool. The main objective is to ensure that these visualizations accurately depict the progress toward our business goals, offering clear insights into their dynamics and interrelationships.

How to

  1. You can use any whiteboard tool, be it physical or virtual, to implement this framework. Divide it into four columns as per the steps outlined earlier and place cards in each column with corresponding explanations.
  2. While progressing through the steps, it's crucial to maintain the proper level of abstraction and only address the current question. There's a tendency to shift between adjacent steps, such as veering into event design instead of focusing on defining the essential questions required to achieve your goals.
  3. For improved understanding, consider linking cards between columns to visualize all possible pathways.
  4. The cyclical nature of this approach enables you to assess whether the final conclusions effectively reflect the level of goal completion. If they don't, this process helps identify which pathway might be incorrect and in need of improvement.

Benefits

What insights does the framework offer us?

Primarily, it grants us a full understanding of why we're gathering analytical data. This understanding is crucial across all levels:

  • Business stakeholders can precisely articulate their goals, enabling others to devise the most effective strategies to achieve them.
  • Analysts gain a clear roadmap for measuring goal progress and can curate only the necessary events for engineers, minimizing the flood of exhaustive raw data.
  • Engineers implement only the truly important events in appropriate contexts.

The secondary advantage lies in the reduction of events, keeping code clean and preserving limits within our analytics service.

Most significantly, our analytics framework is purpose-built to serve our business goals. It is designed before implementation, ensuring that the analytics efforts are aligned with the objectives at hand.

Example

We operate a sandbox app named Tiamat with the primary aim of engaging users with cellular automatas. The app provides a wide set of tools for experimenting with these model, intending for users to explore the breadth of the theme.

To track user engagement, it's crucial to understand the usage of each tool. While capturing every user action through events is an option, it risks to get out of service limits and creating mess in the raw data. Hence, to meet our expectations, we must focus only on events marking exactly the tool usage.

By measuring the frequency of tool usage per user, we can assess which tools are mainly utilized. This usage rate often means popularity, ease of understanding, or accessibility via the app's user interface.

Upon analyzing the gathered data, we discovered that tools positioned deeper in the user interface aren't frequently utilized by the majority of users. To align with our objectives, a UX redesign is imperative, and subsequent analysis becomes necessary to ensure our goals are met.

George Ostrobrod, 2023