The challenge
A client from the FMCG sector was developing a central data platform based on cloud services, designed to handle large volumes of event data and integrations with external systems (APIs).
The key requirements
As the number of data sources, pipelines, and dependencies between ingestion, transformation, and analytics layers grew, challenges emerged related to maintaining data quality, platform stability, and compliance with internal architectural and security standards.
Additional challenges included ensuring:
- reliable event-driven pipelines,
- a consistent approach to data governance and access control,
- automation of deployments and monitoring,
- rapid response to incidents in the production environment.
The client needed experienced engineers who could take end-to-end responsibility for key areas of the data platform.
The solution
Our consultants joined the client’s team to support the development and maintenance of the data platform built on Microsoft Azure services. The scope included:
- building and operational maintenance of data ingestion and transformation pipelines in an event-driven architecture (Event Hubs, Event Grid, Functions, Logic Apps),
- working with the data lake and processing layer (ADLS Gen2, Databricks, ADF, Synapse),
- implementing and maintaining CI/CD, monitoring, and alerting using Azure DevOps,
- collaborating with the Data Architect on data quality, architecture, compliance, and security,
- integrating the data layer with analytics tools (e.g., Power BI),
- resolving production issues from diagnosis to long-term improvements.
The conclusions
- A stable and scalable data platform supporting event processing and API integrations.
- Improved data quality and security through consistent governance and access control policies.
- Faster response to incidents due to automation and improved monitoring.
- Reduced workload for the client’s teams and greater predictability in the development of the data platform.
