Data Engineering
Develop and implement efficient and stable processes to collect, store, extract, transform, load and integrate data at various stages in the data pipeline. This also involves processing varying amounts of data from a variety of sources and preparing data in a structure that is easily access and analysed according to business requirements.
Proficiency Level
Level 1 (Follow)
N/A
Level 2 (Assist)
- Assist in developing and implementing data pipelines and data stores.
- Perform administrative tasks to provide accessibility, retrievability, security and protection of data.
Level 3 (Apply)
- Design and implement data pipelines and data stores to acquire and prepare data.
- Apply data engineering standards and tools to create and maintain data pipelines and extract, transform and load data.
- Carry out routine data quality checks and remediation.
- Implement data management processes and systems to map data sources.
- Implement processes and relationships, and transform and process multiple streams of data.
Level 4 (Ensure)
- Design, implement, and maintain complex data engineering solutions to acquire and prepare data.
- Create and maintain data pipelines to connect data within and between data stores, applications and organisations.
- Carry out complex data quality checking and remediation.
- Translate business requirements into data structures and processes to standardise data, verify data reliability and validity, as well as store, extract, transform, load and integrate data.
Level 5 (Strategise)
- Plan and drive the development of data engineering solutions ensuring that solutions balance functional and non-functional requirements.
- Monitor application of data standards and architectures including security and compliance.
- Contribute to organisational policies, standards, and guidelines for data engineering.
- Lead the selection and development of data engineering methods, tools and techniques.
- Develop organisational policies, standards, and guidelines for the development and secure operation of data services and products.
- Ensure adherence to technical strategies and architectures.
- Plan and lead data engineering activities for strategic, large and complex programmes.