Platform Engineer

Databricks Platform Engineer

Hybrid | United States

Available Work Locations: New York, NY; Houston, TX; Lisle, IL; Atlanta, GA; Austin, TX; Bellevue, WA; Bridgewater, NJ; Somerset, NJ; Charlotte, NC; Franklin, TN; Milwaukee, WI; Rogers, AR; San Ramon, CA; Wayne, PA.

CornerStone Technology Talent Services is seeking an experienced Databricks Platform Engineer to help design, administer, secure, automate, and optimize an enterprise-scale Databricks environment.

This opportunity is ideal for someone who operates beyond traditional data engineering and has hands-on experience owning the Databricks platform itself. The successful candidate will establish platform standards, automate deployments, strengthen governance and security, optimize performance and cloud consumption, and provide engineering teams with a reliable foundation for enterprise analytics and data workloads.

We are looking for a hands-on engineer who understands how Databricks functions as an enterprise platform—from workspace architecture and cluster management through Unity Catalog, Infrastructure as Code, CI/CD, security, governance, observability, and cost optimization.

What You'll Be Doing

• Design, configure, administer, and continuously improve enterprise Databricks environments.
• Manage Databricks workspaces, clusters, compute policies, jobs, permissions, configurations, and platform services.
• Establish reusable platform standards, architecture patterns, guardrails, and engineering best practices.
• Own and support Unity Catalog, including catalogs, schemas, permissions, RBAC, data access policies, governance, and lineage.
• Build and maintain automated infrastructure and deployment processes using Terraform and Infrastructure as Code.
• Develop and support CI/CD pipelines for Databricks configurations, notebooks, jobs, workflows, and platform components.
• Partner with data engineers, analytics engineers, architects, security teams, and application teams to enable scalable development.
• Integrate Databricks with cloud storage, data lakes, enterprise applications, BI platforms, orchestration tools, and other data services.
• Support and optimize Apache Spark workloads across large-scale distributed environments.
• Monitor platform health, cluster utilization, workload performance, reliability, and cloud consumption.
• Troubleshoot performance, configuration, deployment, access, and integration issues.
• Automate repetitive administration and operational processes.
• Implement platform controls that support enterprise security, compliance, governance, and operational standards.
• Drive ongoing improvements around performance, scalability, reliability, automation, and cost optimization.

Required Experience

• 5+ years of experience in Data Engineering, Data Platform Engineering, Cloud Platform Engineering, or Data Platform Administration.
• 3+ years of hands-on Databricks experience focused on administration, platform engineering, architecture, or enterprise platform ownership.
• Strong hands-on experience with Azure Databricks.
• Deep understanding of Apache Spark, distributed computing, workload tuning, and performance optimization.
• Hands-on experience implementing and managing Unity Catalog.

Strong knowledge of

Role-Based Access Control (RBAC)

Identity and access management

Data governance

Platform security

Permissions and access policies

Experience building infrastructure using Terraform or comparable Infrastructure-as-Code technologies.

Experience designing or supporting CI/CD pipelines and automated Databricks deployments.

Strong understanding of cloud data architecture and services within Azure; experience with AWS or GCP is also valuable.

Experience operating enterprise data platforms where reliability, security, scalability, and governance are critical.

Bachelor's degree in Computer Science, Engineering, Data Science, Information Technology, or a related technical discipline.

Highly Desired

Experience designing or supporting a Lakehouse Architecture.

Strong experience with Delta Lake.

Experience with Databricks Workflows, Jobs, cluster policies, SQL warehouses, or related platform capabilities.

Experience integrating Databricks with enterprise data lakes and cloud storage.

Knowledge of Structured Streaming, Kafka, Azure Event Hubs, or other streaming technologies.

Experience implementing observability, logging, monitoring, alerting, and automated operational controls.

Experience with Databricks platform performance and FinOps / cloud cost optimization.

Exposure to secrets management, service principals, private networking, or enterprise cloud security patterns.

Certifications That Stand Out

Relevant certifications are valuable but not required, including:

Databricks Certified Data Engineer Professional

Databricks Certified Data Engineer Associate

Databricks platform-related certifications

Microsoft Certified: Azure Data Engineer

Microsoft Certified: Azure Solutions Architect

Comparable AWS or GCP cloud certifications

The Candidate We Want to Meet

The ideal candidate isn't simply someone who has used Databricks to build notebooks or Spark pipelines.

We're looking for someone who has helped own the Databricks platform.

You should be able to speak confidently about questions such as:

How should an enterprise Databricks environment be structured and governed?

How have you implemented Unity Catalog and RBAC?

How do you manage Databricks infrastructure through Terraform?

How do you automate deployments through CI/CD?

How do you establish cluster policies and platform guardrails?

How do you troubleshoot poorly performing Spark workloads?

How do you monitor and optimize Databricks compute costs?

How do you support multiple development teams without sacrificing governance or security?

Candidates with direct experience solving these problems in a large-scale enterprise environment will stand out.

Why Work With CornerStone Technology Talent Services?

CornerStone Technology Talent Services connects experienced technology professionals with organizations investing in meaningful enterprise transformation initiatives.

Our recruiting team understands the difference between general data engineering and specialized Databricks platform engineering, allowing us to match professionals with opportunities that align with their actual technical expertise.

If you're an experienced Databricks Platform Engineer looking for an opportunity where you can influence architecture, automation, governance, performance, and platform standards, we'd like to connect.