A Lesson in Implementing Product Analytics

With every company and every product, you’re told data is king. If you’re not measuring it, you can’t see it. By this logic, product analytics should be top of every founder’s list of priorities. Yet, implementing bulletproof analytics is often neglected.

The truth is that it’s hard. There is no clear solution, nor a best-practise guidebook on how to implement analytics for your startup’s product.

At Compose, we’ve been around the block with analytics. We’ve built SQL-based dashboards in Metabase, used new tools like Mixpanel, then moved to Google Analytics, all before realising none of these solutions met our needs.

In a final push, we went on a mission to finally reach the holy grail of perfect product analytics. To reach this goal, there were 3 questions we needed to answer:

1. What is the objective of our product analytics? What questions should they answer?
2. Where should I visualise and analyse my data?
3. What should I track and how do I implement it?

In the past, we had tried — and often failed — to answer these questions. But this time we finally came to a clear solution:

1. As an early-stage startup, we don’t need to become data scientists. Our analytics should be simple and track high-level metrics and trends.
2. To visualise our data, we used Segment to send our user data to June (and other destinations) this is the easiest, out-of-the-box solution.
3. To confidently visualise all our key metrics, we implemented and documented tracking of users’ in-app behaviour so we had full coverage of user events and associated properties.

We’ve learnt a lot from this project, a lot of the time by failing. We want to share these mistakes and what we learnt so you’ll be able to better implement your own product analytics.

Identifying objectives

When I started our analytics project, our CEO, Rami, asked for one clear output:

Bulletproof analytics that give us 20/20 vision on our product and users.

This is an objective every company should strive for. However, 20/20 vision is a broad goal. It’s also a goal which requires definite milestones, sub-goals and clear definitions.

Yet, it was at this very first hurdle which I failed. I failed to challenge the definition of 20/20 vision. I failed to break it down and took it literally to mean: we should be able to answer any product question we can possibly contrive, no matter how complex.

This added unnecessary complexity, particularly at the visualisation stage.

Instead, I should have — and you should — start with the following question:

What do we need to actually know about our product and users?

In answering this question, you should be very clear about the approach you’re taking with product analytics. Do you only want a high-level overview? Or, do you need deep dives into specific product questions? For example, your team may want to ask:

• Is our user base growing over time?
• What are the trends in new-user activation?
• Is our latest feature performing well and increasing overall engagement?

If I had asked myself what metrics we were really interested in from the beginning, I would have realised something very important.

Top tip #1: as an early-stage startup, you do not need to be doing fancy logistic regressions to predict retention, nor build complex user flows and dashboards with every detail on every user.

Fancy statistics can be useful, but only once you have all the fundamental insights of high-level metrics across all stages of the funnel, from acquisition through to retention.

In fact, as an early- (or even mid-) stage company, it’s often better to be data-informed rather than data-driven when making product decisions. Data is always useful, but intuition and qualitative feedback from speaking to users will provide the most value and insight.

With this hindsight, we took the objective of 20/20 vision too literally such that visualising our data and building dashboards became overly complicated. Instead, when starting your analytics project, it’s key to have a clear set of goals and objectives in mind to guide how you approach data collection and analysis.

Where should I visualise and analyse my data?

The next big question to ask when setting up product analytics is:

Where should I send, store and visualise my data?

This question is far less prescriptive and can be a minefield, especially for non-engineers. You’ll be hit with data warehouses, customer data platforms, all-in-one solutions, and auto-tracking tools. Yet none are straightforward, nor always suitable to your specific use case.

For brevity, I’ll avoid digging into data architecture best practise, but rather give an overview of what we found was the best solution for flexibility and control of product data.

Compose is built on Google Firebase (an out-of-the-box backend infrastructure). With Firebase, Google Analytics 4 (GA4) is effectively baked in. We therefore gravitated to GA4 which we hooked up under the assumption that it would be a quick and uncomplicated success.

This was a mistake. We failed to fully understand GA4’s use case, its limitations and, again, how it would fit into our mostly undefined objectives for Compose product analytics.

Google’s pitch for GA4 is that it has auto-tracking built in, thus reducing event-tracking set-up costs. This means GA4 is better suited to website analytics rather than SaaS tools like Compose.

Unfortunately, this prescribed tracking became a constraint.

Top tip #2: avoid auto-tracking tools. With these tools, you have to mould the tool to your code and work within its constraints rather than being able to mould your code to work best for the tool.

‍

Storing and sending data

Having learnt the hard way, we were given a second chance to assess how and where to send our usage data. This time around, we decided to use Segment to send our data to June. There are two clear learnings from this decision:

1. Use a Customer Data Platform such as Segment.
2. Don’t rush into deciding on one tool to use. Start with a simple, out-of-the-box solution before expanding out.

Top tip #3: use Segment, it’s a game-changer. Segment is a Customer Data Platform which collects your user data and sends it to multiple destinations. This allows for three huge wins:

• Flexibility to switch up which tool you use to visualise data.
• Opportunity to use a data warehouse if you want a longer-term storage solution.
• Visualise the same data in multiple locations simultaneously.

Below is a visualisation of how this data set-up could look. In this example, Segment is sending data to a) June to visualise high-level metrics; b) Amplitude for deeper product questions, and c) Google BigQuery as a warehouse for storing historical data for a later date.

The second learning is equally salient and is uncovered by the use of a Customer Data Platform.

Segment withdraws all the initial pressure to choose the perfect platform for data visualisation by opening the door to using multiple analytics tools simultaneously.

Data Visualisation

Now, you have the freedom to make a more informed decision on how to visualise your data.

Once again, this decision should be driven by the objectives for your product analytics.

Given our stage at Compose, detailed SQL queries and deep product analyses are unnecessary.

Therefore, we chose the easiest solution of them all, June.

Top tip #4: get up and running instantly with June. With June, we get instant, out-of-the-box insights for key metrics such as user retention, feature engagement and activation rates meaning no wasted set-up cost or inaccurate analysis. We have clear visibility across the funnel.

This solution has been excellent for us, but with our data being piped through Segment, we still have the flexibility for deeper analysis down the line in a tool such as Amplitude as we grow.

When you’re choosing which tool(s) could be best for you, consider the following:

1. Out-of-the-box solutions for key high-level metrics
2. Deep dive into the data and answer any questions we have
3. Build a custom dashboard of core KPIs
4. Build reports intuitively and quickly
5. Bring data from multiple streams (e.g., website and app) into one tool
6. Real-time data visibility

You’ll likely find no one tool gives you all the answers so don’t be afraid to use multiple tools. For early-stage startups, most analytics tools have comprehensive free tiers meaning you’re only faced with the set-up cost which, when using Segment, is minimal.

Now, with clear objectives for product analytics, and a clear set-up of how and where data is to be stored and sent, we can turn to actually tracking and implementing user events. This is where we succeeded the most.

Deciding what to track and how to implement it

The third and final question you should be asking is:

What do I need to track to answer the questions and objectives I have of my data?

If your objective is to simply track the high-level trends of your North Star metrics, then it’s likely unnecessary to track every plausible edge case, event and property.

Yet, this is exactly what we did.

If there was something a user could do in our app, we wanted to know about it and track it.

Our pursuit of 20/20 vision became a blessing in disguise. Because we wanted to ensure we left no stone unturned, it meant we dug deep into the old code of our existing analytics. It was this audit which revealed the huge amount of tech debt we’d taken on.

We uncovered a number of bugs and missing events, meaning we had essentially been blind to in-app behaviour. As event tracking is not retrospective, any untracked event holds the opportunity cost of valuable data and insight.

Top tip #5: do not incur too much tech debt when implementing event tracking.

While it may not feel necessary to add all user events and properties now, these may soon become more important. It is worth spending some additional up-front time to ensure you have full visibility on what you’re tracking, even if you don’t need to be digging into that data just yet.

To implement this effectively, it’s key to have a product or growth team member lead the analytics implementation with an engineer to direct what should be implemented and communicate and record how the implementations were made.

Top tip #6: keep track of what you’re recording in a database to give your whole team visibility.

We built an Event Specification database which allows everyone on the team, especially those coming fresh to analytics for the first time, to know what each event refers to and what each property means and records. This is a valuable resource to refer back to.

Top tip #7: finally, NEVER ship a new feature without analytics.

This seems obvious, but analytics can often be an afterthought. You want to ship new features as quickly as possible to users and adding in events can sometimes be tedious. Yet, without a view on usage, you won’t be able to tell whether a feature is being adopted and well received.

To mitigate this occurring, we’ve built analytics into our deployment process. Analytics is now a dependency for a feature to ship. Product and growth will use the new feature in a staging environment to experiment with how the feature is used before identifying what we want to understand about usage. We’ll then map and implement the event-tracking requirements.

Key Learnings

Product analytics are necessary for every startup, but it’s easy to be overwhelmed. There are so many product questions you could ask, so many tools you could use and so many events you could track.

We went through this pain so you don’t have to. So, to recap, here are the 7 key takeaways:

1. Be clear on your data objectives. You don’t need overkill statistical analysis early on.
2. Avoid auto-tracking tools. Be the master of your own data so you’re only analysing what you want to be tracking.
3. Use Segment to give yourself flexibility on where you can send your data for analysis.
4. Use June as a quick, out-of-the-box solution to minimise set-up costs and get high-level insights straight away.
5. Don’t incur too much tech debt. Track all the key user events from the get-go.
6. Record what events you’ve implemented in an Event Specification database to give your whole team visibility on what the analytics mean.
7. Never ship a feature without analytics; your deployment process should have analytics as a dependency!

I hope this has been useful to help understand the process of implementing product analytics.

Please let me know if you have any thoughts on how this could be improved and what’s worked well for you in the past too!

‍

‍