Data Engineer

Azure Databricks Engineer Location: Minneapolis, MN or Dallas, TX (Onsite) Job Summary We are seeking an experienced Azure Databricks Engineer to design, develop, and optimize scalable data solutions on the Azure cloud platform. The ideal candidate should have strong expertise in Azure Databricks, PySpark, ETL development, data warehousing, and cloud-based data engineering.

Key Responsibilities

• Design, develop, and maintain data pipelines using Azure Databricks and PySpark .
• Build scalable ETL/ELT solutions for processing large volumes of structured and unstructured data.
• Integrate data from multiple sources including databases, APIs, data lakes, and streaming platforms.
• Develop and optimize Spark jobs for performance, scalability, and reliability.
• Work with Azure Data Factory (ADF) for data orchestration and scheduling.
• Implement data quality, validation, monitoring, and governance processes.
• Create and maintain data models within Azure Data Lake and cloud data warehouses.
• Collaborate with business stakeholders, data analysts, and architects to understand requirements.
• Troubleshoot production issues and provide performance tuning recommendations.
• Follow best practices for CI/CD, security, and cloud architecture.
• Required Skills Strong experience with Azure Databricks .
• Expertise in PySpark , Spark SQL, and Python.
• Experience with Azure Data Factory (ADF) .
• Hands-on experience with Azure Data Lake Storage (ADLS Gen2) .
• Strong SQL development and query optimization skills.
• Experience in ETL/ELT pipeline development.
• Understanding of data warehousing concepts and dimensional modeling.
• Experience with Git, Azure DevOps, and CI/CD processes.
• Knowledge of Delta Lake, medallion architecture, and data governance practices