Senior Data Engineer (Must have Databricks & AWS Experience)

Job Summary

We are looking for a Senior Data Engineer to lead the design, development, and optimization of scalable data platforms and pipelines. This role is hands-on and requires strong experience with Databricks, Python (Spark), SQL, and cloud data services, primarily on AWS. The ideal candidate can translate business requirements into reliable data solutions, guide technical decisions, and ensure data systems are performant, secure, and production-ready.

Design, build, and maintain enterprise-grade ETL/ELT pipelines using Databricks, Python (PySpark), and SQL.

Lead the architecture and optimization of data lakes and data warehouse solutions using Delta Lake and cloud storage.

Integrate data from APIs, cloud storage, third-party systems, and structured and semi-structured sources.

Own performance tuning, cost optimization, and reliability of data pipelines and Databricks workloads.

Define and enforce data engineering standards, best practices, and reusable frameworks.

Implement data validation, monitoring, and logging to ensure data accuracy and completeness.

Collaborate closely with analytics, BI, and business teams to deliver clean, trusted datasets.

Support data modeling, metadata management, and data governance initiatives.

Manage and optimize cloud infrastructure across AWS, with exposure to Azure and Linux environments.

Review code, mentor junior data engineers, and provide technical guidance to the team.

Participate in Agile ceremonies and contribute to planning, estimation, and delivery.

Document data architectures, pipelines, and operational procedures for long-term maintainability.

Bachelor’s degree in Computer Engineering, Data Science, Statistics, Physics, or a related field.

5+ years of experience in data engineering or related roles.

Strong hands-on experience with Databricks for large-scale data processing.

Advanced proficiency in Python (Apache Spark/PySpark) and SQL.

Solid experience designing and maintaining ETL/ELT pipelines and cloud-based data architectures.

Hands-on experience with AWS services (S3, Glue, Athena, Redshift, EC2); Azure exposure is a plus.

Experience working with Delta Lake, data modeling, and performance optimization.

Familiarity with Git/GitHub and CI/CD practices in an Agile environment.

Experience supporting analytics and reporting tools such as Power BI and DAX.

Nice to Have

Experience with Airflow or other orchestration tools

Exposure to LS Central, Dynamics 365 (Business Central), or Microsoft SQL Server

Experience with Tableau, JavaScript, HTML, CSS, or AL

Prior experience mentoring or leading data engineering teams