Lead AWS Data Engineer

A Leading Technology Solutions Provider is seeking an authoritative Lead AWS Data Engineer to direct enterprise cloud data platform initiatives across Delhi, Mumbai, Bengaluru, or Hyderabad. In this senior technical leadership position, you will serve as the core design authority responsible for migrating, modernizing, and scaling mission-critical data architectures on AWS. Furthermore, you will architect enterprise data lakes using Medallion patterns (Bronze/Silver/Gold) and manage Apache Iceberg tables. Consequently, this role is vital for supporting analytics, advisor payouts, and financial platforms.

The Lead AWS Data Engineer must combine extensive hands-on programming mastery in Python and PySpark with deep architectural expertise across AWS data services and Terraform. Leading engineering pods, you will re-architect legacy ODI workflows into scalable AWS Glue jobs and manage Airflow orchestration on MWAA. Therefore, the organization is looking for a technical leader who enforces IAM security, KMS encryption, and operational excellence standards seamlessly. If you want to direct enterprise-scale lakehouse engineering, this position offers an ideal platform.

Key Responsibilities

• Act as technical lead and design authority for enterprise data platform engineering on Amazon Web Services.
• Own end-to-end architectural decisions regarding platform scalability, high availability, security, and cost optimization.
• Design and build scalable ETL pipelines utilizing Amazon S3, AWS Glue (PySpark), MWAA (Airflow), and Step Functions.
• Lead on-premises to cloud migration tracks, re-architecting legacy ODI ETL processes into modern PySpark Glue jobs.
• Architect enterprise data lakes using Medallion design (Bronze, Silver, Gold) and manage Apache Iceberg tables.
• Establish orchestration workflows using MWAA, authoring Python Airflow DAGs integrated with Athena, Glue, and S3.
• Implement AWS security standards (IAM, KMS, secrets management), VPC Endpoints, and Terraform infrastructure provisioning.
• Serve as technical escalation point for complex production incidents, configuring CloudWatch metrics and alerts.

Requirements and Qualifications

• Bachelor’s degree in Computer Science, Software Engineering, Information Technology, or an allied discipline.
• 10+ Years of data engineering experience with demonstrated technical leadership on Amazon Web Services.
• Advanced programming proficiency in Python and PySpark for large-scale distributed data processing.
• Deep expertise in AWS data services (Glue, S3, MWAA/Airflow, Step Functions, Athena, CloudWatch) and Apache Iceberg.
• Hands-on experience with Infrastructure as Code (Terraform), on-prem to cloud data migrations, and Medallion architectures.