Job Summary:
A data engineer is responsible for collecting, managing, and converting raw data into information that can be interpreted by data scientists and business analysts. Data accessibility is the incumbent’s ultimate goal, which is to enable organizations to utilize data for performance evaluation and optimisation.
Key Accountabilities:
- Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
- Collaborate within and across the tech team to collect business requirements, define successful analytics outcomes, and design data models.
- Design, develop, and maintain cloud data warehouse / data lake (databricks) to support business needs.
- Design, develop, and maintain data pipelines to ingest and transform internal & 3rd party data sources.
- Identify, implement, and communicate strategies and best practices for data quality, building trust in data & reports produced by the data & tech team.
Qualifications and Competencies:
- 2 to 5 years of working experience in any related field.
- Bachelor’s degree in Computer Science, Applied Mathematics, or any equivalents.
- Able to create a data warehouse (databricks) and is familiar with Kimball data warehousing principles or similar.
- Able to create and maintain reliable batch ETL pipelines (ingest data from production database and bring it into the data warehouse).
- Familiar with the implementation of the above in any cloud environment - can be AWS, GCP, Azure, Digital Ocean, Alicloud, or other.
- Clear understanding on how to tune performance of SQL queries.
- Able to design and conduct tests on data pipeline to ensure data quality is maintained.