Back to all jobs
Career Portal Details
Senior Data Engineer
Location
Corporate H/O - 12th floor
Closing Date
01/31/2023
Description
- Design, develop and maintain Data Catalog, Metadata Management repository/module and Data Lineage through Data Management / Quality tools.
- Ensure that Master data management capabilities meets quality requirement.
- Report on data issues, data quality improvement initiatives and measurement metrics.
- Perform Root Cause Analysis and effectively resolve critical issues within agreed timelines.
- Collaborate with stakeholders, Data Governance team, Data Engineers/ IT team, Data Owners, Data Stewards, Data source administrators, platform and software developers to ensure standards, policy, measurement metrics are being met
- Ensure that Data Management infrastructure, Policies & Procedures are kept up to date with Data Ecosystem;
- Proactively identify gaps as part of Agile Data Quality Improvement initiatives
- Collaborate with IT Data Engineering team on Commissioning new pipelines (New source systems, new applications, new products, change requests) and on ETL issues involving Data Quality
- Translate business requirements to technical design, data models and manage data quality engineering tasks.
- Work with stakeholders, data architects, data owners and stewards to implement data management process, business rules, business decisions, policy, and analyze gap as part of the organization’s data governance requirements.
- Document, communicate progress and share improvement plan on Data Quality issues/reported incidents to stakeholders (Heads of Divisions, Data Scientists, ML Engineers, Analysts and Other Data Consumers)
- Monitor, across data-lifecycle stages that privacy (data protection), business decision, access, rules, and regulatory requirements are adhered to consistently from sources, storages (hot and cold), metadata updates, job failure logs/data logs, and downstream reporting servers.
- Facilitate data audits by auditors
Entry Requirements
- Bachelor’s Degree in Computer Science, Engineering, IT, Business or relevant major with good programming and technical skills. Master’s degree in a relevant field will be preferred.
- 2-5 years of experience within the field of Data Engineering, Data Management, implementing real-time ingestion/streaming, maintaining Big Data Lake/Warehouse and experience in ETL tasks on premise and cloud.
- Good understanding of Data lineage, Data Catalog, master data management, metadata management and Data Management tools.
- Experience using ETL tools and developing complex pipelines through either Scala/Java/Python or opensource tools.
- Expertise in SQL, Apache Spark, Nifi, Airflow, Kafka; Experience in designing and maintaining relational databases and non-SQL data storage. Data Engineering certifications will be an added advantage.
- Experience in software development and optimizing code for performance and scalability will be an added advantage