A scalable data platform in Azure

TM Forum

Customer case
Data Engineering
Data projects

Ian Gardiner

Business Manager

4 min

14 Mar 2023

TM Forum, an alliance of over 850 global companies, engaged our company as a data partner to identify and solve data-related challenges.

The organisation was facing issues with their data ingestion process. The tool they were using was a low-code solution with limited available resources, and most Data Engineers were not familiar with it. Additionally, the tool was not used in other departments within TM Forum, making it difficult to transfer knowledge.

To address these issues, we worked with TM Forum to set up a scalable and future-proof data platform in Microsoft Azure. This cloud provider was already in use in several departments within TM Forum, and Azure has good integrations with Databricks for big data processing via Spark. We set up the data ingestion and data processing using Python code, which is transferable and in line with the skills of the platform's users.

Our approach

Our approach included a Proof of Concept, where we rebuilt one of the data pipelines in Microsoft Azure to test the feasibility of our solution. Within a few weeks, we were able to recreate all functionality of this data pipeline and set up a future-proof structure.

In this process, we used infrastructure as code via Terraform to create reusable modules. This approach provided better insight into platform status, clear documentation, and version management. Also, it allowed for easy rollbacks in case of pipeline failure. Find more details on the (dis)advantages of Infrastructure as Code in our blog.

We built the data pipelines in four steps:

Loading data: via a standard integration (Salesforce) or a custom API connection (Python code in Databricks), we loaded the data
Transforming data: we transformed the data, enriched it, aligned definitions, and renamed columns.
Checking data quality: we implemented data quality checks based on client definitions, ensuring that only data meeting the quality standards was included in the platform.
Pushing data to the data warehouse in Snowflake: Analysts at TM Forum were able to work with the data and bring valuable insights

infrastructure TM Forum — Data Platform Architecture

Result

All data pipelines are now secured in Azure, managed, and monitored via Data Factory. The tool checks daily if all data is properly retrieved, transformed, and verified. This ensures that analysts have access to high-quality data in Snowflake and allows them to extract relevant insights.

Future

In future, we will work with TM Forum to identify which features in the cloud platform can be leveraged for analysis. In addition, business logic will be implemented on top of the ingested raw data, before it’s served to the business, for example via dashboards. We will also make the management of the platform easily transferable to the internal organization so that they can maintain and expand it themselves.

Want to know more?

Ian will be happy to talk to you about what we can do for you and your organisation as a data partner.

Ian Gardiner

Business Manager+31(0) 20 308 43 90 +44(0) 77 8654 1313 ian.gardiner@digital-power.com

Receive data insights, use cases and behind-the-scenes peeks once a month?

You might also like:

Measure ecommerce events in GA4 and Universal Analytics with only the updated datalayer pushes

With our variable in the Google Tag Manager Community Template Gallery it is easy to start using all the new ecommerce analytics capabilities that Google Analytics 4 offers while fully supporting the 'old' enhanced ecommerce of Universal Analytics. Find out how the variable works.

Switching from Universal Analytics to Google Analytics 4 (GA4)

On 14 October 2020, Google launched the new version of Analytics: Google Analytics 4 (GA4). Soon after the launch, it became clear that a number of important functionalities from Universal Analytics (GA3) were missing, and therefore the time to switch seemed far away. Fortunately, we see that the development team on the side of Google has not been idle. Some nice features have since been introduced within GA4 that have narrowed the gap between GA3 and GA4. This article answers the questions that are increasingly being asked about GA4.

How do I set up Google Tag Manager?

A tag management system such as Google Tag Manager (GTM) enables you to measure visitor behaviour on your website. You can also implement marketing pixels (such as Google Ads and Facebook) and cookie banners via this platform. This article gives you tips to keep in mind when setting up GTM. This allows you to collect reliable and usable data, and you will be less dependent on your web developers.

The impact of ITP on analytics and the user experience

Intelligent Tracking Prevention (ITP) was launched by Apple in 2017 in an effort to restore "the balance the balance between privacy and the need for on-device data storage". With Intelligent Tracking Prevention, Apple aims to reduce cross-site tracking (following users across websites) by limiting the use of cookies. Find out what this means for you.

The quality of web analytics implementations

How good is your web analytics implementation? How much confidence is there within the company regarding those figures? In this article we first explain why a web analytics tool will never have 100% accurate data and why that is not a bad thing. Then we look at the practice: how good are most implementations really?

What is Tagbird, what do you use it for, and what can you do with it?

Tagbird is a Chrome extension developed by Digital Power. You can download it from the Chrome Web Store and add it to your browser. It is a debug/visualisation tool that provides a simple and clear insight into, among other things, the data layer, tag management events and analytics requests of a website. So you can quickly and easily test your entire analytics implementation with Tagbird.

Which data traineeship is right for you?

You are almost done with your studies and looking for an employer that offers you the opportunity to learn everything about the field of data. Or you are no longer challenged in your current position and would like to become more technical. In both cases, you do not want to follow unpaid courses, but you would like to get started as soon as possible for real customers, with a serious salary. Does this sound familiar? Then these data traineeships are really something for you.

What are cookies?

Cookies. This word comes up a lot in the world of marketing and online analytics. But what exactly are those cookies? And are there different types of cookies?

What to do about broken Looker Studio dashboards?

Did the following scenario happen to you last week? While enjoying your morning coffee, you take a look at your pride and joy: a meticulously created Looker Studio dashboard built with the GA4 connector. You rub the sleep from your eyes and see your charts are all broken. Frantic phone calls from colleagues are pouring in. What is happening? Find out in this post what the consequences of Google’s newly imposed quota on the Google Analytics Data API are.

Integration web and app data contributes to a 360-degree customer view

Univé is a Dutch insurance company that offers insurance, financial products, and services to both consumers and businesses. The company is focused on providing high-quality service and helping customers make responsible financial decisions. Since 2014, we have been working closely with Univé.