If you are looking for a job as a data professional, you will increasingly see the Data Engineer position come up. But what does it actually mean to be a Data Engineer and what exactly do you do? Our colleague Oskar, Data Engineer at Digital Power, explains.
What is a Data Engineer?
To start with the question, what exactly is a Data Engineer? The expectations and general views on this are quite different. Ultimately, a Data Engineer is someone who designs and builds solutions for data storage and data analysis. For example, think of building data pipelines, moving data from A to B and adjusting (transforming) it if necessary before placing it at B. Or setting up a database, working with distributed systems such as Spark and Hadoop, setting up an infrastructure and actually much more.
Once working as a Data Engineer, you will gain specific knowledge in this domain, and you will specialise more. Every Data Engineer has preferences for what she or he likes to work on. And if you look at it even more abstractly, as a Data Engineer you are exactly in between business, IT and analytics. You must have some knowledge of everything and be able to deal with all stakeholders and explain clearly what you are doing.
"A Data Engineer is someone who designs and builds solutions for data storage and data analysis"
The functions of a Data Engineer and a Data Scientist are not even far apart, they just work for a different purpose. Oskar explains how this works:
"The tooling we use doesn't differ that much, but the approach is different. A Data Scientist tends more towards an analyst for whom knowledge of mathematics and statistics play an important role. A Data Engineer will function more as a software programmer or IT person.
Both profiles work in a position where business, IT and analysis intersect. Thus, depending on your position in the organisation, you will have to serve stakeholders to the same extent. Think of it as a painter touching up your window frame. You wouldn't ask that him to fix the crack in your glass either. You have a glazier for that! But they do work together on the same frame."
Different types of Data Engineering assignments
As a Data Engineer at Digital Power you can perform different types of assignments. What does that look like in practice?
"In a more Agile assignment, your day is the same as any other agile development team. You start with a cup of coffee and do the daily stand-up. Then you work with a tool on the tasks that you can handle and you discuss the progress with your team. Stakeholder management mainly involves the interaction with your Product Owner and the demos you give.
For more small, in-house assignments, the share of stakeholder management will be greater. You design and develop the solution yourself, and you also maintain contact with the customer. You often do this in collaboration with a colleague from Digital Power. This way you come to the best solution together, and you learn from each other."
Learning on the job
The work of the Data Engineer is relatively new and still evolving. This means that as a Data Engineer you learn a lot by simply doing it.
"This is also why I like working as a Data Engineering consultant so much. I have seen and been able to solve problems for various large customers. Intakes and the influence I have on customer demand are of great importance here. Here you determine the scope of the problem that needs to be solved and you discover what knowledge and experience are needed to do this.
Eventually, I can learn many new techniques or solve new problems during an assignment. In this way, I continue to develop myself and take on new challenges. And all that from one employer."
Knowledge sharing at the core
Besides learning by doing, you can of course also learn a lot from your colleagues. Different assignments also present different challenges. And how cool is it if you can then share those learning moments?
"Internally at Digital Power I also continue to develop, for example by transferring knowledge, certainly also gaining new knowledge through internal meet-ups and training courses, and by mentoring colleagues within the Data Engineering domain. This allows me to combine the opportunities and pleasure of a varied job with the benefits of a permanent employer."
A job as a Data Engineer
Has reading Oskar's experience made you excited? You will find several opportunities to become a Data Engineer with us. If you already have experience with the profession, you can immediately apply for our Data Engineer vacancy.
Do you already have work experience as a data professional, but no specific knowledge of Data Engineering? Get retrained! We offer a special programme for this. Read more about it here.
This is an article by Oskar van den Berg, Data Engineering Consultant at Digital Power
Oskar started developing his own websites at the age of 10. During his career, he increasingly focused on Data Engineering. Through Digital Power he works for major clients such as De Nederlandsche Bank and ASML.
Receive data insights, use cases and behind-the-scenes peeks once a month?
Sign up for our email list and stay 'up to data':