Azure Data Engineer

Job Summary

We are seeking an experienced Azure Data Engineering Tech Lead to design, develop, optimize, and maintain scalable enterprise data engineering and Lakehouse solutions. The ideal candidate will have strong hands-on expertise in Azure Databricks, PySpark, Azure Data Factory, ADLS Gen2, Azure SQL, Python, and Azure Cloud, along with proven technical leadership capabilities.

Key Responsibilities

• Design, develop, and maintain scalable Lakehouse solutions using Azure Databricks and PySpark.
• Build and optimize data pipelines supporting batch and near-real-time processing across ERP systems, APIs, relational databases, NoSQL platforms, and on-premises sources.
• Optimize Spark jobs using DataFrame APIs, Spark SQL, partitioning strategies, and file format optimization.
• Implement and manage data governance, access controls, and lineage using Unity Catalog.
• Develop and orchestrate workflows using Azure Data Factory and Databricks Workflows.
• Design secure and scalable ADLS Gen2 storage solutions using Bronze, Silver, and Gold architecture.
• Develop and optimize T-SQL queries, stored procedures, and metadata layers using Azure SQL.
• Implement Azure security, monitoring, networking, and cost optimization best practices.
• Develop reusable Python modules and packages, implement unit testing, and integrate solutions into CI/CD pipelines.
• Leverage Git, Azure DevOps, and infrastructure-as-code tools for automated deployments.
• Lead technical design discussions, prepare HLD/LLD documents and architecture diagrams, mentor engineers, and conduct code reviews.
• Collaborate with cross-functional teams and stakeholders to translate business requirements into scalable technical solutions.

Required Qualifications & Technical Skills

• 12+ years of overall professional experience.
• 4+ years of experience with Azure Databricks and PySpark.
• 2+ years of experience with Databricks Workflows and Unity Catalog.
• 3+ years of experience with Azure Data Factory (ADF).
• 3+ years of experience with Azure Data Lake Storage Gen2 (ADLS Gen2).
• 3+ years of experience with Azure SQL.
• 5+ years of experience with the Azure Cloud platform.
• 2+ years of experience in Python programming and package development.
• Strong knowledge of Delta Lake, Lakehouse architecture, distributed data processing, and performance optimization.
• Experience with Git, Azure DevOps, CI/CD, technical documentation, and code quality practices.
• Bachelor's degree in any discipline.
• Excellent communication, analytical, problem-solving, and technical leadership skills.

Good-to-Have Skills

• Azure Entra ID / Azure Active Directory and GitHub Actions.
• Apache Airflow, Dagster, or Azure Logic Apps.
• Kafka, Azure Event Hubs, and event-driven architectures.
• Change Data Capture (CDC), preferably using Debezium.
• Azure Synapse and Databricks Lakehouse migration experience.
• Terraform, Bicep, or ARM templates.
• Google Cloud Pub/Sub and Google Cloud Storage.

What We’re Looking For

• A technically strong data engineering professional who can lead solution design, build reliable and secure data pipelines, optimize enterprise-scale workloads, and drive engineering best practices.
• The ideal candidate will combine hands-on development expertise with the ability to mentor engineers and collaborate effectively with business and technical stakeholders.