Data Engineer

16 يناير، 2023

0 1 2 دقائق

Data Engineer

الوصف الوظيفي

Data Engineers are tasked with transforming data into a format that can be easily analyzed and utilized.The Data Engineer works closely other stakeholders to ensure the quality of data and the systems implemented is upheld to the highest standards possibleSupport in the development of use cases and build the data models and data flows required to support analytical model developmentBuild and monitor the integrations of the company including ETL / ELTBuild and deploy inference pipelines to productionize the models built and refactor code for new future algorithms as a reusable librarySet up of CI/CD (continuous integration and continuous deployment) pipelines required for the delivery of the company's data services (analytics, data mart, etc.) on their own data platformSet up model monitoring capabilities and model retraining workflow to ensure continuous improvement of data modelsWork with the data architect and external vendors in the implementation of any technology infrastructure or toolsWork closely with Data Architect to produce conceptual and logical data architecture design across the platformBuild, test and maintain data connectors and data pipelines required for data acquisition in the platformSet up data management tools to ensure data is handled appropriately on the platformPerform the relevant tests required to ensure data quality and perform holistic analysis of the datasets defined for the companyMaintain the data quality of dashboards and reports consistently with the relevant internal and external stakeholders (e.g., Data Director, etc.)Work with data owners, stewards and data governance and compliance team to understand the business rules and to deploy them into the ETL / ELTs as well as integrate them with the existing dashboardsTest the outcome of ETL integration, and initiate reconciliation between ETL outputs and ingestions inputsBuild the test automation rules relating to data ingestion, if applicableProvide advice on data topics to ecosystem stakeholders whenever required
 
 RequirementsMinimum of
5 years of experience in a
Data Engineer rolePreferred
degree in Computer Science, Statistics, Informatics, Information Systems or another quantitative fieldExperience in Big Data platforms such as
AWS, Azure, Spark, CouchDB, Hive, Pig, etc.Experience in integrating data into analytical platforms, ingestion technologies, data profiling, source-target mappings, ETL development, SQL optimization, testing and implementationExperience with building processes supporting data transformation, data structures, metadata, dependency, and workload managementExperience supporting and working with cross-functional teams in a dynamic environmentStrong analytic skills related to working with unstructured datasets

المهارات

Minimum of 5 years of experience in a Data Engineer role Preferred degree in Computer Science, Statistics, Informatics, Information Systems or another quantitative field Experience in Big Data platforms such as AWS, Azure, Spark, CouchDB, Hive, Pig, etc. Experience in integrating data into analytical platforms, ingestion technologies, data profiling, source-target mappings, ETL development, SQL optimization, testing and implementation Experience with building processes supporting data transformation, data structures, metadata, dependency, and workload management Experience supporting and working with cross-functional teams in a dynamic environment Strong analytic skills related to working with unstructured datasets