SHIVAM DOBARIYA
Senior Data Engineer
Senior Data Engineer with 5+ years building Databricks-based Lakehouse platforms and AWS-native data pipelines at enterprise scale. Currently lead the design and delivery of Databricks orchestration and PySpark ETL on Volkswagen AG's Cloud Analytics Platform, processing 80M+ records daily. Delivered a 59% PySpark runtime reduction on a flagship pipeline through query and join optimization. Hands-on with Databricks Workflows, Delta Lake, Apache Spark, and AWS (S3, IAM, VPC, Glue, Lambda); previously shipped FastAPI services over Snowflake / RDS / DynamoDB and Power BI dashboards for the OSCE / United Nations. Lead a distributed engineering team across India and Bulgaria.
Experience
Senior Data Engineer
e-Zest Solutions
Dec 2022 – Mar 2025
- Led a team of 4 engineers delivering data infrastructure for the Organization for Security and Co-operation in Europe, supporting diplomatic and humanitarian data programmes.
- Designed FastAPI REST services with OAuth / API-key authentication and IAM roles, integrating Snowflake, RDS, and DynamoDB into a hybrid storage layer — improving downstream data accessibility by 30%.
- Built automated PySpark / Glue ETL pipelines and orchestrated production workflows with Airflow (MWAA) and Step Functions, with retry logic, SLA monitoring, and alerting on critical paths.
- Reduced Snowflake compute costs by 18% through warehouse right-sizing, query profiling, and partitioning strategies; cut average query execution time by 45%.
- Built dashboards and reporting solutions in Power BI and Tableau over the warehouse layer, enabling self-serve analytics for business stakeholders.
- Containerized FastAPI services with Docker and deployed on Kubernetes; integrated with Jenkins and GitHub Actions CI/CD for repeatable, gated rollouts.
- Automated high-load data generation, validation, and reporting with scalable Glue jobs — saving ~70 engineer-hours per month.
Data Analyst
Augmented Systems LLP
Nov 2021 – Dec 2022
- Built data pipelines to extract, clean, and transform transactional data, improving fraud detection accuracy by 20%.
- Developed Power BI dashboards analyzing sales engineers' time allocation across customer accounts, increasing productivity by 15%.
- Implemented email notification mechanisms for stagnant model workflows, reducing processing delays by 30%.
Education
William Carey University
Bachelor of Business Administration (BBA), Statistics & Finance
Skills
Certifications
- AWS Kinesis Immersion Day