P

Data Engineer

Pty
9 days ago
Full-time
Remote
South Africa
Automation

Introduction

Are you ready to join an innovative and collaborative tech company with a global team that designs and develops its own in-house software systems?

At Matrix Design Africa, we build powerful solutions, including a fully-fledged ERP system, that drive efficiency and help organizations make the most of their resources

The Data Engineer will support the Senior Data Engineer to enable the data scientists and other team members in the team by collecting, structuring and storing data within a cloud and on-premises environments, assisting the business to take actionable steps to improve its performance and meet its strategic goals

Read more about MDA here: https://matrixteam.africa/index.html

Duties & Responsibilities

Duties include enabling and supporting the data strategies, implementing automated data pipelines, finding ways to improve a company’s efficiencies with regards to movement of data between environments and providing support to data analysts and data scientists in their efforts to analyze the data and extract insights.

Some of your responsibilities would include:

  • Document data processes, pipelines, and any changes made to the data systems
  • Conduct initial data validation and quality checks to identify and report inconsistencies.
  • Proactively monitor data systems and pipelines, troubleshoot issues, and implement solutions.
  • Write and optimize complex Python & SQL queries for data extraction, transformation, and loading
  • Integrate data from multiple sources and ensure consistency across different data systems.
  •  Manage and optimize sophisticated ETL processes and ensure seamless data integration from diverse sources
  • Design and implement data models and architecture to support business needs and analytics.
  • Provide mentorship to junior engineers.
  • Co-lead data engineering projects, manage timelines, and coordinate with cross-functional teams

Desired Experience & Qualification

Requirements:

  • A tertiary qualification demonstrating proficiency in programming and/or data design and manipulation
  • Strong Azure development skills: 2-5 years of experience
  • Strong MS SQL development skills: 5-8 years of experience
  • Intermediate Python development skills: 2-5 years of experience

Technical Skills:

  • Deployment experience in IoT devices and IoT platforms
  • Familiarity with IoT technology in a Mining context
  • Experience with IoT protocols (e.g. TCP/IP, OPCUA, MQTT)
  • Experience with programming languages such as Python, R, C++, etc
  • Understanding of connected industrial concepts (e.g. Digital twin)
  • Building ETL pipelines for data translation from various sources, Azure Data Factory
  • Data Modelling for effective storage and retrieval
  • Big Data processing for handling large-scale datasets
  • Data Governance related activities to build datasets comply with regulations
  • CI/CD pipelines., DevOps, GitHub, ARM Templates
  • Database and Datawarehouse design, and best practises for data security
  • Experience working with unstructured datasets and building supporting data transformation, data structures, metadata, dependencies and workload management.
  • Some exposure to big data tools like Spark, Hadoop, Kafka, Azure Data Factory etc.
  •  Languages: PySPARK, Python, T-SQL, Azure Python SDK, Apache Spark
  • Technologies for streaming data sensor data (HTTPs, MQTT), e.g. Azure Stream Analytics, Azure Event Hub, Azure IOT Hub, Azure Streaming Analytics
  • Some knowledge of object-oriented/object function scripting languages

Interested?