ACTIVE

Hi, I'm Ali asgar Malekpur

Data Engineer | Cloud & Pipeline Specialist

Data Engineer with 3 years of experience building production-grade ETL/ELT pipelines and cloud-scale data platforms on AWS. Delivered measurable results across high-volume workloads — including a 97% pipeline runtime reduction processing billions of records — using PySpark, Apache Airflow, Delta Lake, and Kubernetes-deployed Docker containers.

⬇ Download Resume Get In Touch →
3+ Years Experience
97% Runtime Reduction
99.9% Pipeline Uptime
# ETL Pipeline Optimization from pyspark.sql import SparkSession def process_data(df): # Billions of records, TB scale result = df.filter(... .transform(clean_data) .write("s3://bucket") return result # Runtime: 5 days → 4 hours (-97%) ⚡
// Career

Professional Experience

Data Engineer

Modak Analytics · Hyderabad, Telangana

3 Years
Python PySpark Apache Airflow AWS Docker Kubernetes Delta Lake ETL/ELT SQL GitHub Actions Grafana
  • Designed and delivered scalable PySpark ETL/ELT pipelines ingesting billions of records across terabytes of enterprise metadata, cutting runtime from 5 days to 4 hours (97% improvement) via partition tuning, caching, broadcast joins, and Spark UI-driven bottleneck detection
  • Built and managed multi-stage Apache Airflow DAGs with dynamic task mapping orchestrating the full Raw Data Lake → Hive Data Warehouse flow, cutting batch processing time by 50% and improving ETL throughput by 80% with strict SLA adherence
  • Wrote multithreaded Python ingestion pipelines to load terabyte-scale datasets into AWS S3, cutting execution time by 70% and lowering infrastructure costs across batch workloads
  • Designed dimensional data models with SCD Type 2 and partition pruning, and authored complex SQL for extraction and transformation supporting downstream BI and reporting
  • Integrated automated data validation, monitoring, and quality checks at every pipeline stage, enforcing governance and compliance standards alongside platform and analytics teams
  • Containerized ETL jobs with Docker and deployed on Kubernetes (Amazon EKS) with horizontal auto-scaling, maintaining 99.9% uptime; monitored pipeline health via Grafana and secured credentials with AWS Secrets Manager
  • Set up CI/CD workflows via Git and GitHub Actions for version-controlled deployments across staging and production, significantly reducing release errors
  • Engineered an AI-powered metadata pipeline on Airflow using an internal LLM gateway to auto-generate business descriptions for 5,000+ tables and 60,000+ columns, enriching the enterprise data catalog at scale
  • Connected to Palantir Foundry via REST APIs for data ingestion, reference normalization, and object transformation with validated lineage
// Technical Arsenal

Skills & Technologies

💻

Programming

Building efficient data workflows and automation scripts

Python SQL Java Pandas PySpark Multithreading
⚙️

Data Engineering

Large-scale ETL/ELT pipelines, lakehouse and stream processing

Apache Airflow Apache Kafka Spark Hive Hadoop Delta Lake Medallion Architecture
☁️

Cloud Platforms

Scalable infrastructure deployment and management

AWS Databricks Snowflake Palantir Foundry
🐳

DevOps & Containers

CI/CD, orchestration, and infrastructure as code

Docker Kubernetes Git GitHub Actions Grafana
🗄️

Databases

SQL and NoSQL database systems and optimization

MySQL PostgreSQL Oracle MongoDB MSSQL
🤖

AI & GenAI

LLM-powered applications, agents, and semantic search

LangChain LangGraph RAG AI Agents (ReAct) OpenAI / Gemini APIs FAISS ChromaDB FastAPI Streamlit
📊

Analytics & Monitoring

Data visualization, catalogs, and system monitoring

QuickSight Grafana Data Catalogs Data Lineage Data Quality
// Portfolio

Featured Projects

PROJECT 01

AI-Powered RAG Assistant

Built a Retrieval-Augmented Generation assistant enabling intelligent querying over custom enterprise documents with LLMs. Implemented chunking, embedding generation, and semantic vector search using FAISS, orchestrated dynamic prompt workflows with LangChain and LangGraph, integrated OpenAI and Gemini APIs, and shipped a real-time Streamlit interface backed by FastAPI for asynchronous, high-performance request handling.

LangChain LangGraph OpenAI API FAISS FastAPI Streamlit
View on GitHub ↗
🤖
🥉
PROJECT 02

End-to-End Medallion Data Pipeline

Designed and implemented a Bronze-Silver-Gold medallion architecture on Databricks for ingesting, transforming, and loading structured datasets into Delta Lake tables. Integrated ingestion from Cloudflare R2 and external cloud storage with automated landing/staging/archive movement, applied robust Silver-layer data quality validations, and built incremental upsert logic in Gold layer tables with Databricks Workflows orchestration and CI/CD integration.

Databricks PySpark Delta Lake AWS S3 SQL
View on GitHub ↗
PROJECT 03

YouTube Data Analytics Pipeline

Designed and orchestrated batch ETL workflows using Apache Airflow and PySpark to process large-scale YouTube analytics data. Implemented serverless ingestion with AWS Lambda, built partitioned data lakes on S3 queryable via Athena, and created interactive QuickSight dashboards enabling data-driven insights and trend analysis.

Apache Airflow PySpark AWS Lambda AWS S3 AWS Athena QuickSight
View on GitHub ↗
📺
// Credentials

Certifications

Astronomer

Certification for Apache Airflow 3 Fundamentals

Databricks

Databricks Fundamentals • Generative AI • AWS Platform Architect

Snowflake

Hands-On Essentials: Data Warehouse • Data Lake • Data Engineering

AWS

AWS Cloud Practitioner Essentials

Neo4j

Neo4j Certified Professional

// Background

Education

Bachelor of Technology, Computer Science & Engineering
Parul Institute of Engineering and Technology
April 2019 - March 2023 CGPA: 8.49
// Let's Connect

Ready to Build
Something Great?

I'm currently open to new opportunities and exciting data engineering projects. Whether you need help with pipeline optimization, cloud infrastructure, or building scalable data solutions, let's connect.

📧

Email

aliasgarmalekpur@gmail.com
📱

Phone

+971 526810313
💼

LinkedIn

Connect with me
💻

GitHub

View my code