Principal Data Engineer

The Enterprise Data Platform team is looking for a Principal Data Engineer to help revolutionize how we think about and deliver customer insights, playing an important role in the future success of the company.

The ideal candidate will be a platform evangelist, having a deep appreciation for the business as well as strong technical skills. You should be self-directed, a creative problem solver, and able to adjust to a fast-paced environment.  This position plays the important role of bridging the gap between our business stakeholders and our technical data teams.

What you'll do:

• Architecting large-scale data intelligence solutions, performing architectural assessments, crafting architectural options and analysis around Snowflake Data Warehouse and/or Google BigQuery
• Design and evolve an open lakehouse on AWS: S3 with Apache Iceberg/Delta table formats, a medallion architecture (bronze → silver → gold), and Trino as the federated query engine.
• Finalize preferred solution alternative working with IT and Business stakeholders to build data ingestion pipeline using tools like AWS EMR, Snowpipe, AWS DMS, etc.etc
• Need to have professional knowledge of cloud platforms like AWS, GCP, etc.
• Developing ETL pipelines in and out of data the warehouse using but not limited to Python, Scala,SnowSQL, etc.
• Own orchestration patterns in Dagster: asset-based pipelines, code-location design, cross-asset lineage, sensors and schedules, partitioning and backfills.
• Set dbt standards for the enterprise data model: layering (staging/intermediate/marts), testing, documentation, and CI for model changes.
• Hands on experience with building productionized data ingestion and processing pipelines using Java, Spark, Scala, Python
• In-depth understanding of Data Warehouse/ODS, ETL concept and modeling structure principles
• Tune Trino for performance and cost: query planning, catalog configuration (Glue/Hive/REST), resource groups, and scaling on Kubernetes.
• Set technical direction and write architecture decision records for review by peers and leadership.
• Mentor engineers across distributed teams in the US and India, and raise the bar through code and design reviews.

Who you are:

• 12+ years of experience working with data in the following ways: SQL development, Shell Scripting, Python, Snowflake, AWS, software engineering or database development
• Ideal candidate has led platform architecture for a multi-team data organization
• Experience developing end-to-end data pipelines to transform and process data for reporting layer
• Experience with large-scale data sets using both structured and unstructured data
• Production experience with at least two of: Trino/Presto, Iceberg/Delta/Hudi, Dagster/Airflow, dbt, Spark
• Full grasp of continuous integration and delivery, including branching strategies, automated deployment and functional automation within CI and CD Pipelines
• Strong communication skills across both business and technical communities
• Strong data modeling: dimensional/Kimball, conformed dimensions, slowly changing dimensions (SCDs), and modeling for multiple brands or tenants
• Experience with Kubernetes (EKS), infrastructure as code (Terraform/CDK), and containerized data services
• Bachelor or Master’s degree in Computer Science, Information Technology or Equivalent
• Performing in an agile, self-directed team environment
• Nice to have: experience with semantic layers (Cube, dbt Semantic Layer), data catalogs, ML/identity resolution, marketing or ad-tech data, and big data
• Experience with billing data and systems is a big plus

#LI-HK1 #LI-Remote