Migration to the cloud: How does this work in practice?

Why are an increasing number of organisations migrating their data infrastructure to the cloud?

There are several reasons for this:

1. The development of the cloud has progressed so rapidly that various advanced functionalities are challenging to achieve on-premise. This is because on-premise environments require a significant amount of knowledge and expertise. Upgrades to new software, frameworks, and technologies are also necessary. Modern tools and techniques are primarily offered on a cloud-first basis. Additionally, expanding hardware may be necessary when adding new functionalities, requiring a considerably higher number of Full-Time Equivalents (FTEs) for environment management.
2. The cloud is relatively accessible and cost-effective. The time-to-production is shorter, system setups are simpler, and there is more dynamic and flexible access to resources. For instance, if you need extra GPU capacity for a short period, you can rent it from the cloud provider for an hour. On-premise, you would need to purchase the actual hardware (GPU) for this purpose.
3. Cloud providers benefit from significant economies of scale. As a result, they can offer complete services, including security, user management, logging, and monitoring, much more affordably than if you were to do it yourself. Additionally, due to their scale, they can provide many more functionalities that you can easily deploy yourself, reducing dependence on engineers.
4. Cloud solutions generally guarantee a minimum uptime of 99.9%. To achieve the same result on-premise, you would need a large team with extensive expertise.

What options are available for storing your data in the cloud?

If you want to process your data for analytical purposes in a data platform, you need to store the data. The structure in which you store and bring together the data is called the data model.

A data lake or data warehouse serves as a storage location for data, typically not (directly) used by operational systems.

A data warehouse has a structured method of storage, whereas a data lake provides much more freedom. However, this doesn't make a data warehouse 'better' than a data lake, just different.

Often, a data lake is better suited for storing raw, relatively unprocessed data. Nowadays, a data lake house architecture is also frequently chosen. This is a setup where a separate processing engine runs alongside the data lake, which writes back the results of the transformations.

Read more about data storage here.

What does a cloud migration look like in practice?

The way you migrate your data infrastructure from an on-premise environment to the cloud depends on two factors: the scale of the solution and the complexity of the environment. The more significant the migration, the greater the need for a thorough analysis beforehand, as described below in the discovery phase.

We recommend breaking down your cloud migration project into three phases:

1. Discovery Phase

• Analyse your existing processes. Which processes are analytical, transactional, or operational? To what extent is each process business-critical? The higher the degree, the greater the impact and risk.
• Explore cloud solutions that can replace your current on-premise infrastructure. You can choose between two main options: Compute services, such as virtual machines that you largely set up and manage yourself, or managed services that take over maintenance and management tasks. With managed cloud services, you need to invest more in reprogramming your processes, but they work quickly and intuitively, allowing you to go into production swiftly. Opting for self-managed compute services will keep you closer to your on-premise setup, with a lower initial investment, though you may not fully leverage the cloud solution.
• Align your migration project with the guidelines of the internal IT & Operations department. If there are none, establish them. Security and governance play a crucial role. Who has access to which data? Who is responsible? Who has decision-making authority?

Check out our webinar on data governance.

Based on the discovery phase, create a step-by-step plan with a broad scope of the project. Other deliverables from this phase include a technical design and an overview of all technical and functional requirements. When creating the technical design, you'll encounter critical choices: will you build and manage everything yourself, or will you use third-party tools like Data Factory, Fivetran, and DBT? Opting for such tools requires less technically advanced knowledge, making your migration faster and simpler. However, you should consider ongoing subscription costs or pay-per-use. Moreover, if you keep everything in-house, your project becomes much more complex.

2. Proof of Concept (POC) or Minimum Viable Product (MVP) Phase

In this optional phase, you work on a small scale to address the most uncertain factor of your project. You test its functionality and try to prove your assumptions. If it works and aligns, you can build on it. If not, you can make adjustments early on.

While a PoC is used solely to demonstrate the effectiveness of the technology and the technical design, an MVP is already a fully functional product but on a small scale. For instance, a PoC may not need to contain production data, while an MVP usually does.

3. Implementation Phase

Once there is a final agreement, your technical team can set up the infrastructure. Continuous collaboration with end-users and ongoing discussions with stakeholders are crucial during this phase. Your engineering team must not only be technically proficient but also adept at explaining how certain things work. After all, you want to avoid a scenario where your new platform doesn't align with the end-users' needs.

What are the key conditions for a successful migration?

Clear communication with end-users and stakeholders is essential. Therefore, you need a competent Product Owner or Project Manager who understands both the technical aspects and the end-users' perspective. Additionally, working with professionals who have executed similar projects and faced comparable challenges is crucial. An experienced specialist prevents reinventing the wheel and can foresee potential issues in advance. Knowing not only how things should be done but also how they should not be done is vital. Identifying errors early on saves a significant amount of time and money.

What are the risks and drawbacks of a cloud migration?

During a migration to the cloud, you are relocating a part of your company's infrastructure. Challenges may arise during the migration, as in any other IT project. It's important to remain flexible in finding solutions and keep stakeholders engaged, meaning working in an agile manner. This flexibility is largely dependent on the project's size and the initially estimated complexity. Integrating a medium-sized database or data warehouse typically won't present many unforeseen complexities.

When a new data model needs to be implemented alongside the migration, it must be well-documented. The time for design and implementation varies based on the model's complexity, making it challenging to scope.

Consider that after the migration, your on-premise specialists may need to be retrained. If not feasible, you might need to hire new personnel or outsource the service. Note that when using managed services in the cloud, there is relatively less maintenance work, often focusing more on maintaining the data model, where data expertise becomes more critical than IT.

Who are the stakeholders, and who performs the migration?

The primary stakeholders include your company's IT organisation, data end-users, and senior management. The CEO and CTO are mainly involved in the decision to migrate to the cloud but don't play a role in the execution. IT support is crucial for aspects like providing system access and advising on the most efficient connection to local data sources.

The project team for a cloud migration ideally consists of:

• A project manager overseeing the entire project, skilled in stakeholder management, expectation management, and change management.
• A Solution Architect or Cloud Architect determining the necessary infrastructure.
• One or more specialists with technical experience executing the implementation. The overarching term 'Data Engineer' is often used in the market for this role. In practice, you need someone specialized in areas such as DevOps, platform engineering, or a Python programmer.

For structuring a data warehouse, an Analytics Engineer is necessary, while a Data Analyst can assist in setting up the right dashboards for your business inquiries.

Read a practical case study here.

What does a migration to the cloud cost?

The license for a cloud solution is more expensive than the on-premise variant. However, this creates a distorted image because, in the long run, you will spend less money on a cloud platform for the following two reasons:

• Cloud solutions provide standard services, including security, user management, and login monitoring. Maintenance and management tasks are also largely automated, resulting in savings on Full-Time Equivalents (FTEs).
• All costs associated with managing your on-premise infrastructure disappear. For example, you will no longer incur costs for hardware.

The costs to the cloud provider depend on various factors, such as the amount of data, the refresh rate of source data, the complexity of the data model, and the frequency and complexity of executed queries.

The migration project itself incurs costs as well. The size of your investment depends heavily on the requirements, the scale of the solution, and the complexity of the environment. If you plan to revise your data model or connect new sources, the investment may increase. Cloud migration is often linked to a business question arising from a need for more insight into data or better technical performance.

From a technical perspective, you can execute a cloud migration for a few thousand euros. However, achieving the desired added value and fully meeting the guidelines of your organisation, such as security or data quality, may require additional investments.

Need help with your cloud migration?

If you want insights into possible costs and would like advice on your migration project, we are happy to have a conversation

Receive data insights, use cases and behind-the-scenes peeks once a month?

Sign up for our email list and stay 'up to data':