Data Engineering

The Home Office's Data Services & Analytics (DSA) Unit intends to put in place a £12m contract over a 2 year term with an option to extend (+1) with a single Service Provider for the provision of Data Engineering Services

Additional information: Suppliers should note that while the Home Office intends to conduct a formal Early Market Engagement (EME) event. The procurement exercise will be conducted under RM6100 Technical Services 3 Framework Lot 2. The Successful Supplier must have detailed, deep or full stack engineering demonstrable experience and hands-on knowledge and accredited to UK Government OFFICIAL and OFFICIAL-SENSITIVE:• The Data Engineering function in DSA is morphing from one with a traditional strong ETL focus to a broader set of skills within a multi-disciplinary team space including, dev-ops, QA, data analysis (mapping and modelling), ETL pipeline build, Python and Java code development and working closely with product teams to create bespoke components in the cloud.• Deliver a capability to create, testable, performant and reusable code utilising the industry stand capabilities of Gitlab, Drone, Docker etc to facilitate automation.• Provide DE expertise support to the Data Integration and Application Support function• Provide expert level knowledge across all aspects of ETL data processing and be able to build end to end pipelines on DSAs' bespoke Java based platform which is hosted in AWS.• Provide expert level AWS Cloud engineering capabilities to facilitate where appropriate the use of AWS commodity services to deliver DSAs growing cloud-based capability• Structured Query language (SQL). Expert level understanding and development of complex SQL code is a major requirement• Change data capture approaches to simplifying data movements within data pipelines• Load and manage large volumes of data including both structured/unstructured sources in many formats including (but not exclusively) JSON, XML, SQL Server, Oracle, Access and MySQL.• Enable data governance criteria for Data access, deletion/removal and general compliance.• Actively contribute to the delivery of capability to support enhanced data quality and data lineage.• Have an expert level of understanding and demonstrable experience of delivery of streamed data using Kafka based technologies. The event will provide suppliers with an overview of the requirement, bidding process and aim to address any initial questions and will be recorded should suppliers not be able to attend the event.• The EME event will be held at 10am on 14th December 2021 Suppliers interested in attending should express their interest via email to TechUK using the following link:https://www.techuk.org/what-we-deliver/events/home-office-data-services-and-analytics-market-engagement-briefing.html.