Data Engineer

About Company

Job description
The Data team comprises of software and system engineers and is responsible for safely receiving and storing trading-related data for the India teams, as well as operating and improving our shared data access and data processing systems. This is a critical role in the organisation as the data platform drives a huge range of trader analysis, simulation, reporting and insights. The ideal candidate should have work experience in systems engineering, preferably with prior exposure to financial markets and with proven working knowledge in the fields of Linux administration, orchestration and automation tools, systems hardware architecture as well as storage and data protection technologies.

Your Core Responsibilities:
• Manage and monitor all our distributed systems, storage infrastructure and data processing platforms (HDFS, Kubernetes, Dremio, in-house data pipelines)
• Heavy focus on systems automation and CI/CD so we can deploy hardware, software solutions rapidly
• Interact closely with other systems and network engineers, traders and developers across the office- to support and troubleshoot their queries
• Keep up with the latest technology trends in the industry; propose, evaluate and implement new solutions

Your Skills and Experience:
• Experience in a similar role managing large-scale multi petabyte data infrastructure with 5-7 years of experience
• Advanced knowledge of Linux system administration and Linux internals; with the ability to manage and troubleshoot issues in a Linux environment
• Expert knowledge in at least one of the following technologies: Kafka, Spark, Cassandra/Scylla, or HDFS.
• Strong working knowledge of Docker, Kubernetes, and Helm
• Experience with data access technologies such as Dremio and Presto
• Familiarity with workflow orchestration tools like Airflow and Prefect
• Familiarity with cloud platforms such as AWS, GCP, or Azure
• Experience with CI/CD pipelines and version control systems like Git
• Knowledge of best practices in data security and compliance.
• Demonstrated talent for solving problems proactively and creatively, result oriented
• Fast learner and excellent troubleshooter
• High degree of flexibility and adaptability Read more