Our client is seeking an experienced Python/Spark/AI Developer to join a technically driven team focused on modernising data platforms and building scalable AI-enabled solutions. This role is ideal for a developer who enjoys working with distributed data, clean and testable Python, APIs, and modern lakehouse technologies.
Key Requirements
- Bachelor's degree in Computer Science, Engineering or equivalent experience.
- 4+ years' professional Python development experience.
- 2+ years' production experience developing Spark or comparable distributed data pipelines.
- Strong Python 3.12 skills, including type hints, Pydantic and modern packaging practices.
- Solid Apache Spark/PySpark knowledge, including Spark SQL, DataFrame APIs, partitioning and performance optimisation.
- Experience with Delta Lake or equivalent technologies such as Iceberg or Hudi.
- Strong SQL skills and the ability to translate legacy T-SQL logic into Spark SQL.
- Practical experience with pytest and automated testing.
- Experience developing REST APIs with FastAPI or equivalent technologies.
- Familiarity with Docker-based development environments.
- Strong Git and CI/CD practices.
Advantageous:
- Experience integrating LLMs and AI services.
- Knowledge of data privacy, PII handling, POPIA/GDPR concepts and data governance.
- Exposure to Hive Metastore, Trino or Apache Ranger.
- Azure data platform experience, including ADLS Gen2, Synapse or ADF.
- Experience with observability tools and structured logging.
- Previous data migration and pipeline parity experience.
- Databricks or equivalent Spark certification.
Should you meet the requirements for this position, please email your CV to it.jobs@mspstaffing.co.za . You can also contact the IT team on 031 350 3607 or visit our website at https://mspstaffingza.co/ NOTE: When replying to the advert, also include the reference number in the subject line. Correspondence will only be conducted with short listed candidates. Should you not hear from us within 3 days, please consider your application unsuccessful.
