A day in the life of a Data Engineer

Oskar's experiences

Article
Data Engineering

Oskar van den Berg

Data Engineer

4 min

28 Feb 2023

For developing modern data applications, a Data Engineer is essential. But what does it actually mean to be a Data Engineer and what exactly do you do? Our colleague Oskar, Data Engineer at Digital Power, explains.

Different types of Data Engineering assignments

As a Data Engineer at Digital Power you can perform different types of assignments. What does that look like in practice?

"In a more Scrum environment, your day is the same as any other Scrum development team. You start with a cup of coffee and do the daily stand-up. Then you work with a tool on the tasks that you can handle and you discuss the progress with your team. In a typical assignment, you will work a lot with Python and one of the major cloud providers: AWS, Azure or GCP. Stakeholder management mainly involves the interaction with your Product Owner and the demos you give.

If you are working on a smaller assignment on a project basis, the share of stakeholder management will be bigger. You design and develop the solution yourself, and you also maintain contact with the customer. You often do this in collaboration with a colleague from Digital Power. This way you come to the best solution together, and you learn from each other.

The field of Data Engineering is very broad. For example, a colleague can review your setup for a CI/CD street, while you give your opinion on their application of Kubernetes."

Learning on the job

The work of the Data Engineer is relatively new and still evolving. This means that as a Data Engineer you learn a lot by simply doing it.

"This is also why I love working as a Data Engineering consultant. I have seen and been able to solve problems for several large clients. Intakes and the influence I have on the customer demand are very important here. This is where you determine the scope of the problem to be solved and find out what knowledge and experience is needed to."

Eventually, I can learn many new techniques or solve new problems during an assignment. In this way, I continue to develop myself and take on new challenges. In the past year, I started working on the realisation of a datalake house architecture for the first time, whereas previously I had mainly worked with (no)SQL databases. And all that from one employer."

Knowledge sharing at the core

Besides learning by doing, you can of course also learn a lot from your colleagues. Different assignments also present different challenges. And how cool is it if you can then share those learning moments?

"Internally at Digital Power I also continue to develop, for example by transferring knowledge, certainly also gaining new knowledge through internal meet-ups and training courses, and by mentoring colleagues within the Data Engineering domain. This allows me to combine the opportunities and pleasure of a varied job with the benefits of a permanent employer."

A job as a Data Engineer

Has reading Oskar's experience made you excited? You will find several opportunities to become a Data Engineer with us. If you already have experience with the profession, you can immediately apply for our Data Engineer vacancy.

Do you already have work experience as a data professional, but no specific knowledge of Data Engineering? Get retrained! We offer a special programme for this. Read more about it here.

This is an article by Oskar van den Berg, Data Engineering Consultant at Digital Power

Oskar started developing his own websites at the age of 10. During his career, he increasingly focused on Data Engineering. Through Digital Power he works for major clients such as De Nederlandsche Bank and ASML.

Oskar van den Berg

Data Engineeroskar.vandenberg@digital-power.com

Receive data insights, use cases and behind-the-scenes peeks once a month?

You might find this interesting too

Converting billions of streams into actionable insights with a new data & analytics platform

Merlin is the largest digital music licensing partner for independent labels, distributors, and other rightsholders. Merlin’s members represent 15% of the global recorded music market. The company has deals in place with Apple, Facebook, Spotify, YouTube, and 40 other innovative digital platforms around the world for its’ member’s recordings. The Merlin team tracks payments and usage reports from digital partners while ensuring that their members are paid and reported to accurately, efficiently, and consistently.

Valuable insights from Microsoft Dynamics 365

Agrico is a cooperative of potato growers. They cultivate potatoes for various purposes such as consumption and planting future crops. These potatoes are exported worldwide through various subsidiaries. All logistical and operational data is stored in their ERP system, Microsoft Dynamics 365. Due to the complexity of this system with its many features, the data is not suitable for direct use in reporting. Agrico asked us to help make their ERP data understandable and develop clear reports.

Insight into the complete sales funnel thanks to a data warehouse with dbt

Our consultants log the assignments they take on for our clients in our ERP system AFAS. In our CRM system HubSpot, we can see all the information relevant before signing a collaboration agreement. When we close a deal, all the information from HubSpot automatically transfers to AFAS. So, HubSpot is mainly used for the process before entering a collaboration, while AFAS is used for the subsequent phase. To tighten our people's planning and improve our financial forecasts, we decided to set up a data warehouse to integrate data from both data sources.

A standardised way of processing data using dbt

One of the largest online shops in the Netherlands wanted to develop a standardised way of data processing within one of its data teams. All data was stored in the scalable cloud data warehouse Google BigQuery. Large amounts of data were available within this platform regarding orders, products, marketing, returns, customer cases and partners.

Reliable reporting using robust Python code

The National Road Traffic Data Portal (NDW) is a valuable resource for municipalities, provinces, and the national government to gain insight into traffic flows and improve infrastructure efficiency.

Setting up a future-proof data infrastructure

Valk Exclusief is a chain of 4-star+ hotels with 43 hotels in the Netherlands. The hotel chain wants to offer guests a personal experience, both in the hotel and online.

A scalable data platform in Azure

TM Forum, an alliance of over 850 global companies, engaged our company as a data partner to identify and solve data-related challenges.

A fully automated data import pipeline

Stichting Donateursbelangen aims to strengthen trust between donors and charities. They believe that that trust is based on collecting money honestly, openly, transparently and respectfully. At the same time effectively using the raised donation funds to make an impact. To further this goal, Stichting Donateursbelangen wants to share information about charities with donors through their own search engine.

Central data storage with a new data infrastructure

Dedimo is a collaboration of five mental healthcare initiatives. In order to continuously enhance the quality of their care, they organize internal processes more efficiently. Therefore, they use perceptions from the data that is internally available. Previously, they acquired the data themselves from different source systems with ad hoc scripts. They requested our help to make this process more robust, efficient and to further professionalise it. They asked us to facilitate the central storage of their data, located in a cloud data warehouse. The goal was to set up the data infrastructure within this environment, since they were already used to working with Google Cloud Platform (GCP).

A scalable machine-learning platform for predicting billboard impressions

The Neuron provides a programmatic bidding platform to plan, buy and manage digital Out-Of-Home ads in real-time. They asked us to predict the number of expected impressions for digital advertising on billboards in a scalable and efficient way.