Job Description

Posted 3 weeks ago

Data Engineer
Inside: IR35
Clearance: SC and NPPV3

Project Overview
A new cloud based application is being created to replace a legacy on-premises application. Data from the legacy application must be migrated to the new application and undergo comprehensive data mapping and processing to conform with the new application’s data store technology and structure. It is expected that data will be bulk migrated, followed by continuous data capture over a period of months while a user transition from one application to another takes place.
The on-premises source database is Oracle with roughly 50-100GB of data to be migrated. The target data is an AWS PostgreSQL database. This will not be a simple migration, and several steps to convert, map and assess the source data will need to be implemented before the data can be added to the target database.
A data migration pipeline has been designed and now must be developed, refined, tested, and automated to perform the migration of the secure data. The tools used to perform the migration and processing are all AWS native applications so deep technical experience in these areas is expected. Where appropriate, build documentation and guides will need to be produced to enable other teams and dependencies to work alongside the data migration team.
 
Key Technical Skills experience / technical expertise required AWS Specific
Glue
Lambda
Simple Storage Service (S3)
DB experience (postgres for queries / quality check etc.)
Database Migration Service (DMS)
Schema Conversion Tool (SCT)
Elastic Compute Cloud (EC2)
Automation and events  Skills
Document creation / review (build docs etc.)
Logging
Identity and Access Management (IAM)
Python experience (for Glue and Lambda if not using the GUI).
Data Mapping Project Experience Data migrations – preferably on-premises to AWS and Oracle to PostgreSQL.
 
Key Soft Skills Pro-activeness.

Communication – proficiency talking to both technical and non-technical people