Data Engineer – Python, PySpark, SQL & ETL

Job Summary

Synechron is seeking a Data Engineer with 5+ years of experience in building data marts, ETL pipelines, and large-scale data-processing solutions.

The role will focus on designing, developing, optimizing, and supporting scalable data pipelines using Python, PySpark, Spark, SQL, and data-warehousing technologies. The position will cover the end-to-end software development lifecycle, including development, testing, UAT support, production deployment, issue resolution, and post-production support.

The role will contribute to business objectives by delivering reliable and efficient data solutions, improving data quality and accessibility, supporting informed decision-making, and enabling consistent data operations across enterprise environments.

Software Requirements

Required Software Skills

• Python: Strong hands-on experience developing data-processing applications, ETL workflows, automation scripts, and production services.
• PySpark and Spark: Strong experience building distributed data-processing and ETL solutions.
• Pandas: Experience with data preparation, analysis, transformation, validation, and profiling.
• SQL: Strong experience with data extraction, transformation, validation, analysis, and query optimization.
• Oracle SQL: Strong knowledge of Oracle SQL, complex queries, joins, aggregations, subqueries, stored procedures, and performance optimization.
• ETL development: Experience designing, developing, testing, deploying, and supporting ETL and data-pipeline solutions.
• Data warehousing: Strong understanding of data-warehouse concepts, dimensional modelling, data marts, transformations, fact tables, dimensions, and data quality.
• Hadoop, Hive, and MapReduce: Experience working with distributed data-processing and big-data technologies.
• SQL and NoSQL databases: Experience working with structured, semi-structured, and unstructured data.
• Jupyter Notebook: Experience using notebooks for data analysis, development, validation, troubleshooting, and technical exploration.
• CI/CD pipelines: Familiarity with automated build, testing, deployment, and release processes.
• Testing and data validation: Experience validating data transformations, pipeline results, data quality, and application behavior.
• Software engineering practices: Understanding of coding standards, version control, documentation, code reviews, testing, and maintainable development practices.

Preferred Software Skills

• Experience with banking or financial-services data and reporting requirements.
• Exposure to cloud-based data platforms and data-processing services.
• Familiarity with workflow scheduling, orchestration, monitoring, and alerting tools.
• Experience with automated data-quality checks, reconciliation, lineage, and metadata management.
• Exposure to infrastructure automation, containerization, and modern DevOps practices.
• Familiarity with tools supporting data pipeline observability and performance analysis.

Version and Experience Guidance

• Specific software versions are not mandated in the requirements.
• Candidates should have experience with current, production-supported versions of Python, PySpark, Spark, Pandas, Oracle SQL, Hadoop, Hive, MapReduce, and related tools.
• Candidates should be able to assess compatibility, manage dependencies, support upgrades, and follow version-control practices.
• Experience should include production data engineering, pipeline support, and end-to-end SDLC participation rather than only academic or prototype development.

Overall Responsibilities

• Design, develop, maintain, and optimize scalable ETL and data-pipeline solutions using Python and PySpark.
• Build and enhance data marts and data-transformation processes.
• Develop distributed data-processing solutions using Spark, Hadoop, Hive, and MapReduce.
• Work with structured, semi-structured, and unstructured data from multiple sources.
• Develop and optimize Oracle SQL queries for data extraction, transformation, analysis, and validation.
• Analyze and debug data issues using Oracle SQL, PySpark, Pandas, and structured data-analysis techniques.
• Perform end-to-end SDLC activities, including development, testing, UAT support, bug fixing, production deployment, and post-production support.
• Collaborate with cross-functional teams to define data requirements, validate solutions, resolve defects, and improve data quality.
• Implement data validation, reconciliation, completeness, accuracy, and consistency checks.
• Support CI/CD implementation for data pipelines, applications, and deployment workflows.
• Monitor production environments and troubleshoot pipeline failures, data-quality issues, performance problems, and processing delays.
• Create and maintain technical documentation, data mappings, transformation logic, support procedures, and operational runbooks.
• Apply software engineering best practices, coding standards, testing practices, and maintainable design principles.
• Contribute to performance improvements through query optimization, efficient transformations, appropriate partitioning, resource management, and pipeline tuning.
• Identify opportunities to improve automation, reliability, data quality, maintainability, and sustainable use of processing resources.
• Contribute to measurable outcomes, including reliable data delivery, reduced processing time, improved data accuracy, fewer production incidents, and successful release execution.

Technical Skills (By Category)

Programming Languages

Essential

• Strong Python development experience.
• Strong PySpark and Spark programming experience.
• Experience using Pandas for data analysis, transformation, profiling, and validation.
• Ability to write modular, maintainable, testable, and production-ready code.
• Ability to develop reusable data-processing components, ETL logic, validation utilities, and automation scripts.

Preferred

• Experience with additional programming or scripting languages used in data engineering and distributed processing.
• Familiarity with asynchronous or event-driven data-processing patterns.
• Experience developing reusable libraries and shared data-engineering components.

Databases/Data Management

Essential

• Strong Oracle SQL experience.
• Experience with SQL and NoSQL databases.
• Understanding of data-warehouse concepts and data-mart design.
• Experience working with structured, semi-structured, and unstructured data.
• Understanding of data modelling, dimensional modelling, schemas, fact tables, dimensions, keys, joins, aggregations, and transformations.
• Experience with data quality, validation, reconciliation, completeness, accuracy, and consistency checks.
• Ability to optimize queries and investigate data-related application or pipeline issues.
• Understanding of data lineage, source-to-target mapping, metadata, data retention, and controlled data access.

Preferred

• Experience with banking or financial-services data structures and reporting requirements.
• Familiarity with data lakes, analytical data stores, and distributed data architectures.
• Experience with partitioning, indexing, compression, schema evolution, and high-volume data processing.
• Exposure to automated data-quality frameworks, data reconciliation, and metadata management.

Cloud Technologies

Essential

• Understanding of scalable data-processing and data-pipeline deployment environments.
• Ability to support development, testing, UAT, staging, and production data environments.
• Understanding of resource management, performance, availability, monitoring, and operational support for data platforms.

Preferred

• Experience with cloud-based data platforms and distributed processing services.
• Familiarity with cloud storage, managed data-processing services, data warehouses, and orchestration tools.
• Exposure to cloud-native CI/CD, infrastructure automation, and containerized data workloads.
• Understanding of processing optimization, resource utilization, and sustainab