Converting billions of streams into actionable insights with a new data & analytics platform

In addition to financial data, Merlin receives non-financial usage trend information across DSPs worldwide. Combining and analysing this data is of immense value to Merlin's members, but the question is: how do you collect, structure, and combine the data from over one billion streams, royalty and trends inputs each day? And how do you derive actionable insights from it?

Merlin enlisted our help to design a data platform where all this data converges. A distinctive challenge in the design and development was the enormous volume of streaming data that needed to be collected and structured. Another challenge was the old system that used different cloud providers.

Forty sources had to be connected to the new system, with Spotify being by far the largest. To illustrate: for Spotify alone, nearly 1 billion rows of data are processed daily.

Approach

For this project, we worked in two phases.

1. Discovery phase

We started with a Discovery Phase, gathering all technical and business requirements from the involved stakeholders. The deliverables included a series of concrete recommendations and an architecture with a tool stack. In this case, we opted for a data lake architecture. The open nature of a data lake aligned well with the existing platform, reducing risks.

As seen in the image below, both the analytical data flow and the financial data flow were brought together on one platform.

2. Implementation phase

As part of the migration, many pipelines had to be migrated from the cloud platform and orchestrator. We also connected numerous new sources and transferred old data. In practice, challenges arose: a data model that changed over time or a pipeline that practically burst at the seams. Along the way, we addressed and resolved these challenges in collaboration with the stakeholders involved.

Platforms like these, processing terabytes of data daily, benefit significantly from a robust setup. Tool selection plays a crucial role, as does a good setup of Continuous Integration/Continuous Deployment (CI/CD). Additionally, having high test coverage of functional code and running automated quality tests across all data within the ETL/ELT process is essential.

Over nine months, our team, in close collaboration with Merlin, worked on implementing the new platform. The first phase involved implementing the platform using Infrastructure as Code (IaC) and deploying Dremio, the query engine. We then integrated the data pipelines and established the data layers within the data lakehouse. The pipelines ran on Python code within Airflow, and the tables were modularly set up using DBT and SQL.

Result

The new platform enables Merlin to link financial and non-financial usage data across multiple partners. Now, Merlin can gain deeper insights into trends, for use both internally and for its Members. Merlin can now see they are close to answering questions such as 'What is the best day to release a new single in a particular market for an artist in country or pop?' or 'How does the popularity of artists and genres compare across different music platforms?' The consolidated system has also significantly reduced the operational burden on Merlin’s Analytics Team as all data is available within the same data warehouse.

Because Merlin works with labels, distributors, and rightsholders around the world, it has a unique variety of data that is highly versatile and granular. The new platform provides Merlin with the opportunity to generate unique, previously unobtainable insights for its Members.

Receive data insights, use cases and behind-the-scenes peeks once a month?

Sign up for our email list and stay 'up to data':