Senior/Lead Data Engineer

Location: Gurgaon, Bangalore, Delhi NCR,None,None

About US:-
We turn customer challenges into growth opportunities.
Material is a global strategy partner to the world’s most recognizable brands and innovative companies. Our people around the globe thrive by helping organizations design and deliver rewarding customer experiences.
We use deep human insights, design innovation and data to create experiences powered by modern technology. Our approaches speed engagement and growth for the companies we work with and transform relationships between businesses and the people they serve.
Srijan, a Material company, is a renowned global digital engineering firm with a reputation for solving complex technology problems using their deep technology expertise and leveraging strategic partnerships with top-tier technology partner. Be a part of an Awesome Tribe

Responsibilities:
- Design and implement scalable and robust ETL pipelines to facilitate the extraction, transformation,     and loading of large datasets from various sources into a structured data warehouse.
- Utilize Databricks to process big data workloads, leveraging Spark for data transformation and 
 aggregation, and optimize performance of big data queries.
- Manage and orchestrate containerized applications using Docker and Kubernetes, ensuring seamless  deployment, scaling, and management of applications across clusters.
- Develop and maintain data architectures on Azure, including data lakes, data warehouses, and 
other relevant data storage solutions, ensuring security, scalability, and compliance with industry standards.
- Integrate and manage Elasticsearch clusters for efficient searching, logging, and analytics of structured and unstructured data.
- Implement and maintain asynchronous programming models to optimize data processing workflows and improve system responsiveness and scalability.
- Collaborate with AI teams to integrate large language models (LLMs) into data processing pipelines, enhancing data analysis and decision-making capabilities.
- Develop and enforce best practices and standards for data management, including data security, backup procedures, and disaster recovery methods.
- Monitor and troubleshoot performance issues across all platforms and applications, proposing and implementing improvements to system architectures.
- Stay up-to-date with industry trends and advancements in data engineering technologies, applying new techniques and tools as applicable to improve existing systems.

**Must-Have Skills:**
- Advanced proficiency in Python programming, with a strong grasp of software development best practices and efficient code production.
- Deep knowledge and hands-on experience with Azure data services, including Azure Delta Lake, Service Bus, Queues and Topics, and Event Hub.
- Expertise in software design patterns, particularly those relevant to data engineering such as producer-consumer, singleton, and others.
- Proven ability to monitor and scale data pipelines for performance and efficiency, identifying bottlenecks and implementing solutions to enhance data flow and processing.
- Extensive experience with big data technologies, including Databricks, Spark, and Hadoop.
- Proficient in managing and deploying applications using Docker and Kubernetes.
- Solid understanding and experience with Elasticsearch and its deployment and maintenance.
- Experience with asynchronous and concurrent programming models.
- Proficient in developing and managing ETL pipelines for large-scale data.

Why work for Material
In addition to fulfilling, high-impact work, company culture and benefits are integral to 
determining if a job is a right fit for you. Here’s a bit about who we are and highlights around 

What we offer.
Who We Are & What We Care About:-

• Material is a global company and we work with best-of-class brands worldwide. We also create and launch new brands and products, putting innovation and value creation at the center of our practice. Our clients are in the top of their class, across industry sectors from technology to retail, transportation, finance and healthcare.
• Material employees join a peer group of exceptionally talented colleagues across the company, the country, and even the world. We develop capabilities, craft and leadingedge market offerings across seven global practices including strategy and insights, design, data & analytics, technology and tracking. Our engagement management team makes it all hum for clients.
• We prize inclusion and interconnectedness. We amplify our impact through the people, perspectives, and expertise we engage in our work. Our commitment to deep human understanding combined with a science & systems approach uniquely equips us to bring a rich frame of reference to our work.
• A community focused on learning and making an impact. Material is an outcomes focused company. We create experiences that matter, create new value and make a difference in people's lives.]

What We Offer:-
• Professional Development and Mentorship.
• Hybrid work mode with remote friendly workplace. (6 times in a row Great Place To 
Work (Certified).
• Health and Family Insurance.
• 40+ Leaves per year along with maternity & paternity leaves.
• Wellness, meditation and Counselling sessions.