Sr Azure Data Engineer

Job description

Key Responsibilities:
1.   Design, develop, and deploy scalable data pipelines on the Azure cloud platform using Azure
Data Factory (ADF).
2.   Architect end-to-end data solutions to support business requirements and analytical
insights, with a focus on Databricks (PySpark) for data processing.
3.   Implement and optimize data processing workflows using technologies such as Databricks,
Spark, and PySpark.
4.   Integrate diverse data sources including structured and unstructured data for analysis and
reporting purposes.
5.   Collaborate with cross-functional teams to understand data requirements and deliver
actionable insights.
6.   Ensure data quality, reliability, and security across all data processes and systems.
7.   Provide technical guidance and mentorship to junior team members, particularly in ADF and
Databricks (PySpark).
8.   Stay abreast of emerging technologies and best practices in the field of Big Data and cloud
computing, with a focus on ADF and Databricks.
Preferred Qualifications:
1.   Azure/AWS certification (e.g., Azure Data Engineer, AWS Certified Big Data - Specialty).
2.   Experience with DevOps practices and tools for CI/CD pipelines.
3.   Familiarity with containerization technologies such as Docker and Kubernetes.
4.   Knowledge of data visualization tools such as Power BI or Tableau.

Qualifications:
1.   Bachelor's or Master's degree in Computer Science, Engineering, or related field.
2.   Minimum of 3 years of experience in designing and implementing data solutions on the
Azure cloud platform, with a focus on ADF and Databricks (PySpark).
3.   Proficiency in Big Data technologies such as Hadoop, Spark, and other relevant tools.
4.   Strong programming skills in languages such as Python, Scala, or Java.
5.   Extensive experience with SQL and relational databases.
6.   Hands-on experience with cloud services such as Azure or AWS.
7.   Proven track record of delivering complex data projects on time and within budget.
8.   Excellent problem-solving skills and attention to detail.
9.   Strong communication and interpersonal skills, with the ability to collaborate effectively
with cross-functional teams.