Senior Data Engineer - Octopus by RTG

وصف الوظيفة

At Octopus by RTG, we are dedicated to bridging exceptional organizations worldwide with outstanding talent. Currently, we are in search of a Senior Data Engineer to join our innovative team.

Key Responsibilities:

  • Design, develop, and manage efficient data pipelines aimed at facilitating machine learning workflows and Generative AI applications.
  • Implement data ingestion, transformation, and storage solutions for both structured and unstructured datasets.
  • Maintain data quality, integrity, and consistency throughout the entire data pipeline.
  • Enhance the data infrastructure to ensure scalability, peak performance, and cost-effectiveness.
  • Implement workflows for real-time data processing.
  • Work collaboratively with machine learning engineers and data scientists to integrate data pipelines seamlessly with applications and models.

متطلبات الوظيفة

  • Proficiency in programming languages for data processing (e.g., Python, Scala, Java).
  • Strong experience with big data technologies (e.g., Hadoop, Spark) and ETL tools.
  • Familiarity with data storage systems (e.g., SQL databases, NoSQL databases, data lakes).
  • Strong Experience with vector databases and embedding stores
  • Experience with cloud platforms and data services (e.g., AWS Redshift, Google BigQuery, Azure Data Factory).
  • Knowledge of data modeling, warehousing, and real-time processing frameworks (e.g., Kafka, Flink).
  • Strong problem-solving skills and ability to work in cross-functional teams.