Senior Databricks Data Engineer

Job Title: Senior Databricks Data Engineer

Location: New York City, NY

Experience: Senior Level

Job Summary:

We are seeking an experienced Senior Databricks Data Engineer to design, develop, and optimize scalable data pipelines and cloud-based Lakehouse solutions. The ideal candidate will have strong expertise in Databricks, Apache Spark, PySpark, Python, SQL, Delta Lake, and Unity Catalog, with experience building reliable batch and streaming data processing solutions.

Key Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, Apache Spark, and PySpark.
• Build and optimize batch and real-time streaming data processing workflows.
• Implement Lakehouse architecture using Delta Lake and Unity Catalog.
• Develop data transformation and integration solutions using Python and SQL.
• Design and maintain data models to support analytics, reporting, and business intelligence.
• Implement data quality checks, validation, monitoring, and performance optimization.
• Manage data governance, access controls, and data lineage using Unity Catalog.
• Deploy and maintain data engineering solutions across Azure, AWS, or GCP environments.
• Implement version control and automated deployment processes using Git and CI/CD pipelines.
• Troubleshoot pipeline failures and improve data reliability, scalability, and processing efficiency.
• Collaborate with data architects, analysts, and cross-functional teams to deliver business data solutions.

Required Qualifications:

• Strong hands-on experience with Databricks and enterprise data engineering.
• Proficiency in Apache Spark, PySpark, Python, and SQL.
• Experience with Delta Lake, Unity Catalog, and Lakehouse architecture.
• Strong knowledge of ETL/ELT processes and data pipeline development.
• Experience with batch processing and streaming data architectures.
• Understanding of data modeling, data warehousing, and data integration.
• Hands-on experience with at least one cloud platform: Azure, AWS, or GCP.
• Experience with Git, CI/CD pipelines, and software development best practices.
• Strong analytical, problem-solving, and communication skills.

Preferred Qualifications:

• Experience optimizing large-scale distributed data processing workloads.
• Knowledge of data governance, security, and data quality frameworks.
• Familiarity with workflow orchestration, automated testing, and production monitoring.
• Experience supporting enterprise analytics and cloud data platform modernization.