Azure Data Engineer

Job description

Job Summary:

We are looking for an experienced Azure Data Engineer with 8+ years of hands-on experience in designing, developing, and implementing end-to-end data solutions on Microsoft Azure. The ideal candidate will have a strong background in data integration, data warehousing, data modeling, and building scalable data pipelines using Azure services.

Key Responsibilities:

1. Design and develop scalable and secure data pipelines using Azure Data Factory, Azure Synapse Analytics, and Databricks.
2. Implement ETL/ELT processes to ingest data from various structured and unstructured sources.
3. Build and maintain data lakes and data warehouses on Azure (e.g., Azure Data Lake Storage Gen2, Synapse).
4. Work closely with data analysts, scientists, and business stakeholders to understand data requirements and deliver high-quality data solutions.
5. Optimize and monitor data pipelines for performance, reliability, and cost-efficiency.
6. Apply data governance, security, and compliance standards, including role-based access controls and encryption.
7. Perform data modeling (dimensional and relational) and develop data dictionaries and documentation.
8. Integrate with external tools such as Power BI, Azure DevOps, or third-party APIs as needed.
9. Ensure CI/CD practices in data development and deployment workflows.
10. Must have experience in Event hub, Realtime data Streaming (using kafka).

Required Skills & Experience:

8 years of experience in data engineering or BI development, with a focus on Azure Data Services.

1. Strong experience with:
2. Azure Data Factory (ADF)
3. Azure Synapse Analytics
4. Azure SQL Database / SQL Server / SQL DW
5. Azure Data Lake Storage (ADLS Gen2)
6. Azure Databricks / Apache Spark
7. Proficiency in writing complex SQL queries and performance tuning.
8. Experience with Python, PySpark, or Scala for data transformation and orchestration.
9. Experience with CI/CD tools (Azure DevOps, GitHub Actions, etc.) and automated deployment strategies.
10. Knowledge of data security and compliance standards (GDPR, HIPAA, etc.).
11. Familiarity with Power BI or other visualization tools is a plus.

Preferred Qualifications:

1. Microsoft Azure certification(s) such as:
2. DP-203: Azure Data Engineer Associate
3. AZ-900 / DP-900 (Foundational)
4. Experience with Agile/Scrum methodology.
5. Knowledge of Data Governance, Master Data Management (MDM), and Data Quality frameworks.
6. Familiarity with real-time data processing (Event Hubs, Stream Analytics) is a plus.

Soft Skills:

1. Strong analytical and problem-solving skills.
2. Excellent communication and collaboration abilities.
3. Ability to work independently and in a cross-functional team environment.
4. Eagerness to learn and adapt to new technologies.