Must have skills: Have 5+ years of experience with developing data applications using big data technologies such as Hadoop, Spark, Flink, Dataflow etc. Experience with workflow orchestration tools such as Airflow/Luigi/Azkaban etc. Experience with coding languages like Python/Java/Scala Experience with at least one cloud platform AWS/GCP/Azure Hands-on experience and highly advanced knowledge of SQL, Data Modeling, ETL Development, and Data Warehousing Experienced in scalable, configurable, parameterized, modular programming practices for data engineering. Knowledge and experience with Data Management and Data Storage best practices. Exposure to large databases, BI applications, data quality and performance tuning Good to have understanding of job management, resiliency Good to have prior experience with Graph, Time-series databases Role and Responsibilities: Architect highly metadata-driven data pipelines with algorithms for data deduplication, data harmonization, fuzzy matching, identity resolution Design and architect relational, time series and graph databases to run OLAP queries Design and develop SDKs and APIs to enable configurable data consumption paradigms Build tools to monitor the health of the data pipelines and data infrastructure Develop and lead the Data Engineering team
