rajin User

I’m a Senior Data Engineer with over 8 years of experience in data pipeline development and cloud environments, focused on building scalable, reliable distributed data infrastructure. I’m expert in PySpark and SQL, and I consistently optimize performance and delivery speed through automation and data-driven decision-making. My key achievements include processing over 40 TB of records through scalable PySpark ETL pipelines and improving query performance by 20% using Delta Lake optimization strategies like partitioning and incremental loading. I’ve also reduced ingestion latency from 1 hour to 5 minutes by implementing near real-time CDC pipelines with Debezium, Kafka Connect, and Azure Event Hubs. In recent roles, I designed and optimized PySpark ETL pipelines for enterprise supply chain analytics, migrated SAP HANA datasets to Azure Data Lake and Delta Lake, and monitored structured and unstructured datasets with Databricks and PySpark. Earlier, I built automated extraction and reporting pipelines using Python, SQL, Selenium, and REST APIs, reducing manual reporting effort by 8hrs, and I supported production reliability for multiple client-facing systems with AWS-hosted applications.