Lead Data Engineer

Title: Lead Data Engineer

Duties: The Lead Data Engineer at Earnest LLC in San Francisco, CA will lead data infrastructure and framework development to support and optimize predictive models. Build backend services in python to support ML model building and data transformations. Build and maintain ETL/ELT framework for data modeling and transformations, enable monitoring logging and CI/CD using AWS, Airflow, DBT, SQL, Snowflake. Build data integrations to extract data from various data resources such as Postgres DB, API, CSV files using Airflow DAGs, AWS DMS service. Create snowpipes to Load the data into Snowflake data warehouse. Build data models using DBT and SQL for data transformations to satisfy business requirements. Deploy and maintain Airflow for data orchestration and build DAGs to run data transformations on schedule. Manage Snowflake data warehouse that includes access management, usage monitoring, cost optimizations and data integrations. Build ML platform to support faster model development, training, experimentation, and production deployment by using technologies like Sagemaker, Kubeflow, AWS, Ansible, Terraform.  Set up model development environments by creating Sagemaker notebook instances and train models using various AWS services. Deploy Kubeflow, Sagemaker pipelines for automated model training, experimentation and serving. Use ansible, terraform, Jenkins and other CI/CD tools to automate tasks and cloud service deployments. Set up technical direction by designing system architecture, proposing technologies, planning and leading projects in roadmap to solve technical and business problems. Write and present ERDs for system architecture changes or additions and lead project implementations. Mentor and guide junior members of the team to build their skills and onboard them onto new tools and technologies. This position is 100% remote; can telecommute from anywhere in the U.S. Salary: $185,000 - $204,000 per year. 

Requirements:  Bachelor's degree in Computer Science, Software Engineering, or a closely related field, plus 5 years of software development experience.  These 5 years of experience must include experience with each of the following: (1) 3 years of SQL and data transformation experience; (2) 3 years of programming language experience, including python; (3) 2 years of Snowflake experience; (4) 2 years of cloud services experience, including AWS; and (5) 1 year of experience using data tools, including Airflow or DBT. 

This notice is subject to Earnest LLC's employee referral program.

Interested candidates must apply online at www.earnest.com/careers [earnest.com]. 

#LI-DNI

Tags