Principal Data Engineer

SUMMARY OF RESPONSIBILITIES

We are seeking a highly skilled Data Engineer to join our team. In this role, you will be responsible for designing and implementing data pipelines to support the company's various data initiatives. The ideal candidate will have extensive experience with data sources such as MS SQL, MySQL, Oracle, and PostgreSQL, and be comfortable with leveraging tools such as AWS Glue for ETL, Apache Kafka/AWS MSK for data streaming, Data Lake and Data Warehouses. The candidate must also be able to work with a variety of ETL functions to transform the data ready to support consumption to build multiple use cases, which may include ML model training and prediction using appropriate models to solve problems.

KEY AREAS OF RESPONSIBILITIES

• Assess existing data dictionaries and build up the data catalogue for various consumption
• Design and implement data pipelines to support the company's data initiatives, while ensuring data security and compliance
• Work with a variety of data sources such as MS SQL, MySQL, Oracle, and PostgreSQL
• Leverage AWS Glue for ETL and Data Lake, Data Warehouse for data storage and subsequent consumption
• Use and manage Apache Kafka/AWS MSK for data streaming and stream processing
• Work with ETL functions to transform the data for consumption to support multiple use cases
• Support ML model training and prediction using appropriate models to solve problems
• Collaborate with cross-functional teams to ensure data quality and accuracy
• Monitor data pipeline performance and make optimizations as needed
• Ensure data security and compliance by implementing appropriate measures, such as access controls, encryption or anonymization
• Stay up-to-date with the latest technologies and industry trends related to data engineering and data security and compliance

QUALIFICATIONS & EXPERIENCE

Minimum Qualifications

• Bachelor’s degree in computer science, Information Systems Technology, or Software Engineering 
• At least 5-7 years of relevant experience in Information Technology, especially in the field of Data Engineering and/or Data Science
• Relevant work experience in the Financial Services and/or Technology sectors would be an added advantage
• Excellent in both English and Bahasa Malaysia

FUNCTIONAL COMPETENCIES

• Extensive experience with MS SQL, MySQL, Oracle, and PostgreSQL
• Comfortable with AWS Glue for ETL, Apache Kafka/AWS MSK for data streaming, Data Lake and Data Warehouse such as AWS Redshift
• Ability to work with ETL functions to transform the data that will be consumed to support multiple use cases
• Strong programming skills, including Python and/or Scala
• Familiarity with machine learning frameworks such as TensorFlow, PyTorch, scikit-learn
• Experience leveraging large scale distributed data processing platforms, such as AWS EMR with analytics framework such as Apache Spark, Presto, stream processing such as Apache Flink
• Knowledge and experience in iteratively selecting, building, training, tuning and deploying ML models to solve problems
• Experience leveraging AWS SageMaker and its corresponding suite of tools and framework
• Strong problem-solving and critical thinking skills
• Excellent communication and collaboration skills
• Experience with handling data security and compliance, including implementing access controls, encryption, data anonymization