Data engineer senior

The Senior Data Engineer will play a vital role collaborating as part of a Data team to create and enhance data ingestion pipelines and address data challenges. The Senior Data Engineer will work closely with the Architect, systems engineers, software engineers, and data scientists on the following key tasks.

Primary Tasks and responsibilities:

  • Create and maintain optimal data pipeline and data lake
  • Assemble large, complex data sets that meet functional / non-functional business requirements
  • Perform tests, validate data flows and prepare all ETL processes, incorporating business requirements into all design specifications
  • Design and develop all data mapping techniques for all data models in systems
  • Document technical and system specifications for all ETL processes
  • Analyze and interpret complex data on all target systems, coordinating with internal and external stakeholders to validate requirements
  • Perform root cause analysis, resolve and validate all production data issues.

Technical profile requirements

  • Preferably +5 years of experience as Data Engineer, minimum +2 years
  • Experience developing flexible data ingest and enrichment pipelines, to easily accommodate new and existing data sources
  • Experience with continuous integration and deployment (CI/CD) pipelines and their enabling tools
  • Experience developing Microservices to fit data cleansing, transformation and enrichment needs
  • Very good knowledge & experience of cloud technologies : Python is mandatory, others are a nice to have (Spark, Hive or other data processing technologies)
  • Experience with AWS cloud services is mandatory
  • Working knowledge of database technologies (SQL) is mandatory
  • Experience with an ETL tool is mandatory
  • Experience with Microservices architecture components
  • Docker and Kubernetes are a nice to have
  • Experience with Jira, Confluence and extensive experience with Agile methodologies are a nice to have