About Me in a Nutshell
Hello viewers! I'm Rahul, a senior Data Engineer deeply passionate about crafting robust and scalable data solutions. With over 7 years of experience in the tech industry, I've dedicated my time to designing, developing and deploying enterprise grade Data Warehouses, Data Mart, Batch and Streaming pipelines, high performance Geospatial analytics, and Data Governance.
My journey has given me hands-on expertise across distributed systems and data engineering, leveraging various cutting edge technologies. I've had the opportunity to work on diverse projects, from building high-performance applications to developing intricate data pipelines and advanced analytics products.
I created this blog as a platform to share my insights, technical deep dives, and best practices in software development, cloud computing, and data processing. My goal is to create a valuable resource for fellow developers, helping you navigate complex challenges, enhance your skills, and stay on top of the ever-evolving tech landscape. Join me as we explore the exciting world of technology together!
My Expertise
Work Experience
Technical Solutions Consultant, Google
Google, Aug 2022 - Present
🚩 Lead project delivery including, but not limited to, project scope, deliverables, estimates, quality, detailed project plans, and change requests.
🚩 Oversee projects through the complete life-cycle including scoping, planning, execution, implementation/rollout, and closing.
🚩 Help customers with current infrastructure assessment, design, and architect target infrastructure, develop a migration plan, and implement Google Cloud best practices.
🚩 Proactively partner with Global Business and technical teams to shape clients' need into opportunities and solutions.
🚩 Facilitate teamwork with decision-makers at higher levels of management to identify bottlenecks, identify appropriate solutions, and improve services. Provide direction for internal and external teams in the areas of new solutions/systems, user/partner/client ecosystem, and industry environment. Provide stakeholders with some direction on how to handle escalations.
Consultant - AI & Data Engineering, Deloitte US India
Deloitte India (Office of the US), Aug 2021 - Aug 2022
🚩 Developed Data Pipelines for 𝗕𝗮𝘁𝗰𝗵 𝗮𝗻𝗱 𝗦𝘁𝗿𝗲𝗮𝗺𝗶𝗻𝗴 𝗗𝗮𝘁𝗮 𝗜𝗻𝗴𝗲𝘀𝘁𝗶𝗼𝗻 using 𝗔𝗶𝗿𝗳𝗹𝗼𝘄 (Python) and 𝗗𝗮𝘁𝗮𝗳𝗹𝗼𝘄. Created custom Dataflow templates using Apache Beam library and Java.
🚩 Performed 𝗘𝗧𝗟 operations in 𝗕𝗶𝗴𝗤𝘂𝗲𝗿𝘆 and configured Scheduled Queries and External Transfers.
🚩 Optimized SQL queries and Airflow DAG with parallel execution which lowered execution time by 35 percent for Data Pipelines.
🚩 Communicate effectively with vendors and Data customers to fine-tune Data Platform. Participated in 𝗦𝗼𝗹𝘂𝘁𝗶𝗼𝗻𝘀 𝗱𝗲𝘀𝗶𝗴𝗻 𝗮𝗻𝗱 𝗗𝗮𝘁𝗮 𝗺𝗼𝗱𝗲𝗹𝗹𝗶𝗻𝗴 for new data sources and application enhancements.
🚩 Integrated with downstream applications for effective Campaigns, and solve data requirements for Mobile/Web applications.
🚩 Performed Customer segmentation and Master Data Management that enhanced campaigns and helped increase returning customers by 18 percent.
🚩 Developed 𝗖𝗜/𝗖𝗗 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀 using 𝗝𝗲𝗻𝗸𝗶𝗻𝘀 for automatic deployments and releases across environments.
🚩 Developed Cloud Functions code for 𝗦𝗲𝗿𝘃𝗲𝗿𝗹𝗲𝘀𝘀 𝗰𝗼𝗺𝗽𝘂𝘁𝗶𝗻𝗴 as per Business requirement which lower billing by 12 percent.
Data Warehouse Consultant and Team Lead, ICICI Bank
Resourced by Aptus Data Labs, Apr 2018 - Aug 2021
🚩 Lead analysis, architecture, design, and development of 𝐃𝐚𝐭𝐚 𝐖𝐚𝐫𝐞𝐡𝐨𝐮𝐬𝐞 𝐚𝐧𝐝 𝐁𝐮𝐬𝐢𝐧𝐞𝐬𝐬 𝐈𝐧𝐭𝐞𝐥𝐥𝐢𝐠𝐞𝐧𝐜𝐞 𝐬𝐨𝐥𝐮𝐭𝐢𝐨𝐧𝐬
🚩 Work closely with Project managers to develop work plan for Data Warehouse projects
🚩 Assist Business development teams, to 𝐞𝐬𝐭𝐢𝐦𝐚𝐭𝐞 𝐚𝐧𝐝 𝐩𝐥𝐚𝐧 𝐩𝐫𝐨𝐣𝐞𝐜𝐭𝐬
🚩 𝐃𝐞𝐯𝐞𝐥𝐨𝐩, 𝐫𝐞𝐟𝐢𝐧𝐞 𝐚𝐧𝐝 𝐬𝐜𝐚𝐥𝐚 𝐝𝐚𝐭𝐚 𝐦𝐚𝐧𝐚𝐠𝐞𝐦𝐞𝐧𝐭 𝐚𝐧𝐝 𝐚𝐧𝐚𝐥𝐲𝐭𝐢𝐜𝐬 𝐩𝐫𝐨𝐜𝐞𝐝𝐮𝐫𝐞𝐬, systems, workflows, and best practices
🚩 Work closely with Data scientists to assist on 𝐟𝐞𝐚𝐭𝐮𝐫𝐞 𝐞𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐦𝐨𝐝𝐞𝐥 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐟𝐫𝐚𝐦𝐞𝐰𝐨𝐫𝐤𝐬, 𝐚𝐧𝐝 𝐦𝐨𝐝𝐞𝐥 𝐝𝐞𝐩𝐥𝐨𝐲𝐦𝐞𝐧𝐭𝐬 𝐚𝐭 𝐬𝐜𝐚𝐥𝐞
🚩 Work with application developers and DBAs to diagonise and resolve query performance issues
🚩 Present and persuade the design architecture to various stakeholders (customer, server, network, security and other teams)
🚩 𝐀𝐝𝐨𝐩𝐭 𝐢𝐧𝐧𝐨𝐯𝐚𝐭𝐢𝐯𝐞 𝐚𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐚𝐥 𝐚𝐩𝐩𝐫𝐨𝐚𝐜𝐡𝐞𝐬 to leverage in-house data integration capabilities consistent with architectural goals of the enterprise
🚩 Roll up my sleeves and work with project teams to support project objectives through the application of sound architectural principles
🚩 Possess excellent interpersonal, verbal, and written skills and the ability to interact with all levels of stakeholders, support personnel, and clients
Data Engineer, Aptus Data Labs
Aptus Data Labs, Jan 2018 - Aug 2021
🚩 Responsible for developing and maintaining on-premise Data Warehouse with Peta-bytes of data for Financial clients.
🚩 Performed Security Audit, Data Governance, and vulnerability closure of Data Warehouse and Analytical Data Mart.
🚩 Developed Batch Data ingestion using Sqoop and Hadoop Distributed File System (HDFS). Extensively used ETL to transfer and extract data from various sources (RDBMS, APIs, Flat files) to target data warehouse.
🚩 Created a distributed computing environment using Jupyter Hub and Spark for Data Science workflows with multi-user Active Directory Authentication and Authorization.
🚩 Implemented hybrid multi-cloud architecture and migrated applications to the cloud using cloud-native services.
🚩 Worked on Python and Django to create an in-house teamwork management application to track and plan all tasks efficiently.
🚩 Worked on Cloud Development in GCP, AWS and Azure platform