Join our innovative, fast-growing data team at the forefront of cloud data architecture on Microsoft Azure. We're building scalable, secure, and modern data platforms using cutting-edge Azure services and Databricks unified analytics platform. If you're passionate about creating high-performance data infrastructure and solving complex big data challenges in a cloud-native environment, this is the perfect opportunity for you.
As a Senior Data Engineer specializing in Azure and Databricks, you will architect and implement enterprise-grade cloud-native data solutions on the Microsoft Azure ecosystem. This is a hands-on engineering role with significant architectural influence, where you'll work extensively with Azure Data Factory, Databricks, Delta Lake, and other modern data tools to create efficient, maintainable, and scalable data pipelines using medallion architecture and lakehouse patterns.
What you do
4–7 years of hands-on experience in data engineering with strong focus on cloud data platforms. Proven experience with big data technologies and distributed computing
Technical Expertise
Azure Mastery: Deep expertise in Azure data services including Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, and Azure SQL Database
Databricks Proficiency: Advanced proficiency in Databricks including cluster management, notebook development, and Delta Lake
Big Data Processing: Strong experience with Apache Spark (PySpark, Scala), distributed computing concepts, and performance optimization
Programming: Proficiency in Python, Scala, and SQL for data processing and transformation
Data Architecture
Experience with lakehouse architecture, medallion architecture, and modern data warehouse design patterns
Streaming Technologies: Knowledge of real-time data processing using Azure Event Hubs, Kafka, and Spark Streaming
Cloud Platforms: Deep understanding of Microsoft Azure ecosystem and native data services
Data Management & DevOps
Data Governance: Understanding of data governance frameworks, Unity Catalog, and data quality practices
DevOps: Experience with CI/CD practices, Git-based workflows, and Infrastructure as Code
Orchestration: Knowledge of data orchestration tools like Azure Data Factory, Airflow, or similar platforms
Security: Understanding of Azure security best practices, RBAC, and data encryption
Advanced Technologies
Experience with Azure Machine Learning and MLOps practices
Knowledge of containerization (Docker, Kubernetes) and Azure Container Instances
Experience with Power BI for data visualization and reporting
Understanding of data mesh architecture and domain-driven design principles
Experience with Azure DevOps for CI/CD pipeline management
Familiarity with monitoring tools like Azure Monitor, Application Insights, and Databricks monitoring
Specialized Skills
Experience with real-time analytics and complex event processing
Knowledge of graph databases and Azure Cosmos DB
Understanding of data lake optimization techniques and performance tuning
Experience with multi-cloud or hybrid cloud architectures
Technology Stack
Core Azure Services
Data Storage: Azure Data Lake Storage Gen2, Azure Blob Storage, Azure SQL Database
Data Processing: Databricks, Azure Synapse Analytics, Azure Data Factory
Streaming: Azure Event Hubs, Azure Stream Analytics, Kafka
Analytics: Databricks SQL, Azure Analysis Services, Power BI
Machine Learning: Azure Machine Learning, MLflow, Databricks ML
Big Data & Processing
Compute: Apache Spark (PySpark, Scala), Databricks Runtime
Data Formats: Delta Lake, Parquet, JSON, Avro
Architecture: Medallion Architecture, Lakehouse, Data Mesh
Orchestration: Azure Data Factory, Airflow, Azure Logic Apps
Development & DevOps
Programming: Python, Scala, SQL, PowerShell
Version Control: Git, Azure DevOps, GitHub
Infrastructure: ARM Templates, Terraform, Azure Resource Manager
Monitoring: Azure Monitor, Application Insights, Databricks monitoring
Security & Governance
Security: Azure Active Directory, Key Vault, RBAC
Governance: Unity Catalog, Azure Purview, Data Lineage
Compliance: GDPR, data encryption, access controls
Certifications
Microsoft Azure Data Engineer Associate (DP-203)
Databricks Certified Associate Developer for Apache Spark
Databricks Certified Professional Data Engineer
Azure Solutions Architect or Azure Data Fundamentals certifications
What we ask
Data Architecture & Engineering
Design and implement enterprise-scale, robust data solutions using Azure and Databricks
Architect scalable ELT/ETL pipelines using Azure Data Factory, Azure Synapse Analytics, and Databricks
Build automated data ingestion processes from various sources including streaming and batch data
Develop and maintain data transformation workflows using PySpark, Scala, and SQL on Databricks
Data Platform & Optimization
Implement lakehouse architecture using Delta Lake and Databricks Delta tables
Design and optimize Databricks clusters for performance, cost management, and resource utilization
Apply medallion architecture (Bronze, Silver, Gold) for data processing and transformation
Optimize Spark jobs and queries for maximum performance and cost efficiency
Big Data & Analytics
Build streaming data pipelines using Azure Event Hubs, Kafka, and Databricks Structured Streaming
Implement machine learning pipelines using MLflow and Databricks ML capabilities
Design and maintain data models for analytical workloads and real-time processing
Work with large-scale datasets and implement partitioning strategies for optimal performance
Quality & Governance
Develop comprehensive data quality tests and monitoring using Great Expectations and custom frameworks
Implement data governance, lineage, and security policies within Azure and Databricks
Create and maintain comprehensive data documentation and lineage tracking using Unity Catalog
Build data validation and testing frameworks for proactive data monitoring
DevOps & Automation
Build and maintain CI/CD pipelines for Databricks notebooks and data workflows
Develop custom Python/Scala scripts for data processing, manipulation, and automation tasks
Implement Infrastructure as Code (IaC) using ARM templates or Terraform for Azure resources
Work with orchestration tools for pipeline scheduling and dependency management
Collaboration & Analytics
Collaborate with data scientists, analysts, and ML engineers to support advanced analytics use cases
Enable self-service analytics capabilities using Databricks SQL and Azure Analytics services
Work closely with stakeholders to understand data requirements and deliver scalable solutions
What we offer
You’ll join an international network of data professionals within our organisation. We support continuous development through our dedicated Academy. If you're looking to push the boundaries of innovation and creativity in a culture that values freedom and responsibility, we encourage you to apply.
At Valtech, we’re here to engineer experiences that work and reach every single person. To do this, we are proactive about creating workplaces that work for every person at Valtech. Our goal is to create an equitable workplace which gives people from all backgrounds the support they need to thrive, grow and meet their goals (whatever they may be). You can find out more about what we’re doing to create a Valtech for everyone here.
Please do not worry if you do not meet all of the criteria or if you have some gaps in your CV. We’d love to hear from you and see if you’re our next member of the Valtech team!