Senior Databricks Data Engineer

Job Title: Senior Databricks Data Engineer

Location: New York City, NY

Job Summary

We are seeking an experienced

Senior Databricks Data Engineer

to design, develop, and optimize scalable data engineering solutions using Databricks and modern cloud technologies. The ideal candidate will have strong hands-on experience with Apache Spark, PySpark, Python, SQL, Delta Lake, and Unity Catalog, along with expertise in building reliable ETL/ELT pipelines and enterprise data platforms.

The successful candidate will be responsible for developing batch and streaming data solutions, implementing Lakehouse architectures, and ensuring data quality, security, and performance across cloud environments. This role requires strong technical expertise, problem-solving skills, and the ability to collaborate with cross-functional teams to deliver efficient, production-ready data solutions.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, PySpark, Python, and SQL.
• Build and optimize ETL/ELT workflows for data ingestion, transformation, validation, and integration.
• Implement and maintain Lakehouse architectures using Delta Lake and the Bronze/Silver/Gold medallion framework.
• Manage data governance, access controls, metadata, and data security using Unity Catalog.
• Develop and optimize batch processing and real-time streaming solutions to support business and analytical requirements.
• Design and implement data models that support reporting, analytics, and downstream data consumption.
• Optimize Spark jobs, SQL queries, and data processing workflows to improve performance and cost efficiency.
• Integrate Databricks solutions with cloud platforms such as Microsoft Azure, AWS, or Google Cloud Platform (GCP).
• Implement data quality checks, error handling, and validation processes to ensure data accuracy and reliability.
• Use Git and CI/CD pipelines to manage source code, automate deployments, and maintain development standards.
• Troubleshoot data pipeline failures, resolve production issues, and monitor data processing performance.
• Collaborate with data architects, analysts, software engineers, and business stakeholders to translate requirements into technical solutions.
• Maintain technical documentation and follow best practices for data engineering, testing, security, and deployment.

Required Skills

• Strong hands-on experience with Databricks and enterprise data engineering solutions.
• Proficiency in Apache Spark, PySpark, Python, and SQL.
• Experience with Delta Lake and Unity Catalog.
• Strong understanding of ETL/ELT development and end-to-end data pipeline implementation.
• Experience with batch processing and streaming data architectures.
• Knowledge of data modeling, data warehousing, and Lakehouse architecture.
• Experience working with cloud platforms such as Azure, AWS, or GCP.
• Familiarity with Git, version control, CI/CD pipelines, and automated deployment practices.
• Understanding of data quality, governance, security, and performance optimization.
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent communication and collaboration skills.