Data Engineer

Job description

Must have skills:

• Have 5+ years of experience with developing data applications using big data technologies such as Hadoop, Spark, Flink, Dataflow etc.


• Experience with workflow orchestration tools such as Airflow/Luigi/Azkaban etc.


• Experience with coding languages like Python/Java/Scala


• Experience with at least one cloud platform AWS/GCP/Azure


• Hands-on experience and highly advanced knowledge of SQL, Data Modeling, ETL Development, and Data Warehousing


• Experienced in scalable, configurable, parameterized, modular programming practices for data engineering.


• Knowledge and experience with Data Management and Data Storage best practices.


• Exposure to large databases, BI applications, data quality and performance tuning


• Good to have understanding of job management, resiliency


• Good to have prior experience with Graph, Time-series databases

Role and Responsibilities:

• Architect highly metadata-driven data pipelines with algorithms for data deduplication, data harmonization, fuzzy matching, identity resolution


• Design and architect relational, time series and graph databases to run OLAP queries


• Design and develop SDKs and APIs to enable configurable data consumption paradigms


• Build tools to monitor the health of the data pipelines and data infrastructure


• Develop and lead the Data Engineering team