Data Engineer III

Position Title:

Data Engineer III

Pay Rate :

$83,000 - $121,000

Multiple factors, including Individual experience, skills and abilities will determine where an employee is ultimately placed in the pay range.

Category/Shift

Salaried Full-Time

Physical Location :

75 5th Street NW

Atlanta, GA 30308

United States

The Job You Will Perform:

The Data Engineer is an important member of the Cost Innovation Team, which is a strategic, innovative, and cross-functional team that is focused on creating value for the company through the advanced analytics of manufacturing data. This position offers a flexible working arrangement provided that the home office is within a typical commuting distance to the A2C office.

Core Responsibilities

Analytics & Data Science Enablement

• Partner with analysts, data scientists, engineers, and business stakeholders to translate functional requirements into scalable technical data solutions.
• Deliver self-service, analytics-ready datasets that empower downstream teams to consume data without deep knowledge of underlying source systems.
• Enable industrial analytics use cases by providing trusted, contextualized time-series, asset, event, and operational data for analytics platforms and applications.
• Eliminate redundant data-preparation and contextualization efforts across disparate analytics and application teams.
• Help establish standard data definitions, reusable data products, and consistent business context across enterprise and industrial analytics environments.
• Support downstream analytics workflows, including advanced analytics, data science, visualization, and industrial process analytics.

Data Ingestion & Integration

• Design, build, and maintain robust pipelines to ingest data from relational databases, operational systems, industrial historians, APIs, cloud platforms, and flat files.
• Establish reliable and secure connections between source systems, cloud data platforms, and industrial data platforms.
• Write optimized SQL queries and extraction logic for high-performance data retrieval.
• Implement flexible ingestion patterns including batch, incremental, streaming, and replication-based approaches based on business latency requirements.
• Integrate high-volume time-series data and industrial metadata while maintaining relationships between assets, measurements, events, and other operational entities.
• Support ingestion and integration patterns used by platforms such as Cognite Data Fusion (CDF), Seeq, and Microsoft Fabric.

Data Storage & Lakehouse Engineering

• Design and manage scalable storage structures across raw (bronze), intermediate (silver), and curated (gold) layers.
• Leverage Spark-native formats and engines such as Delta Lake to support scalable analytics workloads.
• Design curated data structures that support both enterprise analytics and industrial data consumption.
• Enforce data retention, archiving, and purging policies to manage data lifecycles.
• Monitor and optimize storage, compute costs, and cloud resource utilization.

Data Transformation

• Develop Python and SQL-based transformations to clean, standardize, enrich, and unify data.
• Develop, tune, and proactively optimize scalable PySpark applications to efficiently process high-volume datasets.
• Build modular, reusable transformation components rather than siloed, application-specific logic.
• Resolve complex data-quality anomalies including schema drift, missing values, duplicates, timestamp alignment, and inconsistent identifiers.
• Engineer efficient incremental processing patterns to handle high-volume, frequently changing datasets.
• Transform raw operational and historian data into analytics-ready time series, calculations, conditions, events, and contextualized datasets for downstream consumption.
• Enable reusable data transformations that can support platforms such as Seeq, Cognite, Fabric, and custom analytics applications.

Semantic Modeling & Industrial Data Contextualization

• Translate complex source schemas into intuitive, business-aligned semantic entities, asset hierarchies, and industrial data models.
• Build and maintain relationships between assets, equipment, time series, events, files, and other operational data.
• Standardize naming conventions, identifiers, metadata, and relationship mappings across source systems.
• Perform data contextualization to connect operational data with relevant equipment, process, and business context.
• Support asset-centric and knowledge-graph-style data models commonly used in industrial data platforms such as Cognite.

Pipeline Orchestration & Operations

• Architect end-to-end workflows that seamlessly coordinate ingestion, transformation, contextualization, validation, and publication.
• Orchestrate data movement between source systems, lakehouse environments, industrial data platforms, and analytics applications.
• Implement resilient error handling including retries, automated recovery, checkpointing, and alerting.
• Monitor pipeline health, SLA compliance, data freshness, and data drift using centralized observability tools.
• Design pipelines capable of supporting both traditional enterprise data and high-frequency industrial time-series workloads.

Data Governance, Security & DevOps

• Enforce data security and privacy standards, including row- and column-level security, RBAC (Role-Based Access Control), and masking of sensitive data.
• Apply consistent access-control and governance patterns across enterprise data platforms and industrial analytics environments.
• Embed automated validation checks as needed to enforce data quality.
• Maintain Git-based development workflows using branching, code reviews, and pull requests.
• Implement automated CI/CD pipelines for code testing, data validation, configuration management, and repeatable environment deployments.
• Support deployment and lifecycle management of data pipelines, transformation logic, semantic models, and analytics configurations across development, test, and production environments.
• Maintain comprehensive documentation of data architectures, pipeline schemas, source-to-target mappings, contextual relationships, and lineage maps.

Preferred Experience

• Bachelor’s Degree in Computer Science, Software Engineering, Data Science, Engineering, or related field + 2 years of relevant experience; equivalent Master’s degree accepted; non-relevant bachelor’s degree requires 5+ years of equivalent hands-on experience.
• Experience working with:

+ SQL and Python for data engineering and transformation.

+ PySpark and Microsoft Fabric for lakehouse transformations.