Senior Data Engineer (Databricks / Spark Streaming)

Senior Data Engineer (Databricks / Spark Streaming) Location: Remote (Mexico) About the Role We are looking for a Senior Data Engineer to design, build, and scale our data infrastructure with a focus on real-time and batch processing pipelines. You will work extensively with Databricks and Apache Spark Structured Streaming to deliver reliable, high-throughput data platforms that power analytics, machine learning, and business-critical applications. This is a fully remote position open to candidates based in Mexico. What You'll Do Design, build, and maintain scalable ETL/ELT pipelines using Databricks and Apache Spark Develop and optimize real-time streaming pipelines using Spark Structured Streaming (Kafka, Kinesis, Event Hubs, or similar) Architect and manage Delta Lake tables, implementing best practices for schema evolution, partitioning, and performance tuning Build and maintain data pipelines across the medallion architecture (bronze/silver/gold layers) Collaborate with data scientists, analysts, and software engineers to understand data needs and deliver production-grade solutions Implement CI/CD workflows for data pipelines using tools such as Databricks Asset Bundles, GitHub Actions, or Azure DevOps Monitor, troubleshoot, and optimize pipeline performance, cost, and reliability Establish and enforce data quality, governance, and observability standards (e.g., using Unity Catalog, Great Expectations, or similar) Mentor junior engineers and contribute to engineering best practices and documentation Participate in architecture discussions and provide technical guidance on data platform strategy Required Qualifications 8+ years of experience in data engineering, with a strong focus on distributed data processing Hands-on production experience with Databricks (clusters, jobs, workflows, Unity Catalog) Strong expertise in Apache Spark , including Spark Structured Streaming for real-time data processing Proficiency in Python and/or Scala for data pipeline development Advanced SQL skills for data transformation and optimization Experience with Delta Lake and lakehouse architecture concepts Experience with AWS — Databricks-supported environments Familiarity with message streaming systems such as Kafka , Kinesis , or Event Hubs Experience with orchestration tools (e.g., Databricks Workflows, Airflow) Solid understanding of data modeling, partitioning strategies, and performance optimization Experience with version control (Git) and CI/CD practices for data engineering Nice to Have Databricks certifications (e.g., Databricks Certified Data Engineer Professional) Experience with infrastructure-as-code (Terraform) Experience with dbt for transformation workflows Familiarity with MLOps or feature store concepts Experience working in a fully remote, distributed team environment Prior experience in a regulated industry (finance, healthcare, etc.) What We're Looking For Strong communication skills and comfort working async with distributed, cross-functional teams A proactive, ownership-driven mindset — comfortable taking projects from design through production Ability to balance pragmatic delivery with long-term architectural quality Fluent English and Spanish (written and verbal)