Sr Data Engineer

RESPONSIBILITIES

Work independently to provide solutions and leverage technical experience to align partners to optimal path for delivery Partner with team to craft, develop, and implement data pipelines and ETL systems for the data lake Perform data validation to ensure data quality and reliability Work with BI architect and Data Engineering Lead to build technical requirements and specifications for Business Intelligence products Identify and fix data pipeline Production issues Build clear documentation of procedures and protocols Implement data governance, data security and privacy policies while collaborating multi-functionally with technical and non-technical team members QUALIFICATIONS: Bachelor’s degree in computer science, Information Systems, Engineering, Statistics, or related field 7-9 years’ experience as a Data Engineer designing, developing, and implementing batch and real-time data pipelines and ETL systems.

4-5 years' proven track record with AWS Data technologies experience with data lake technologies (S3, Glue, Redshift, RDS, DynamoDB, Data Pipeline, EMR) and data lake architectures Sophisticated knowledge in data warehousing solutions (Panoply, Redshift, etc.) and hands on experience using Cloud based database systems (Azure SQL, Bigtable, etc.)

Direct experience using Python, Perl, Scala, Java, SAS, C++, as well as hands on development using real-time platform like Kafka, Kinesis etc. Experience working with Infrastructure as Code.

Terraform experience is preferred, ability to write new, sophisticated queries with no supervision Firm grasp of Agile development methodologies, CI/CD automation, Test Driven Development and knowledge of standard methodologies in database engineering and data security Familiarity with machine learning methodologies and high level cloud migration & tools knowledge