Databricks SME

Key Responsibilities

Architecture & Platform Design

Design enterprise Databricks Lakehouse architectures aligned with the Databricks Well-Architected Framework

Define reference architectures for

batch, streaming, analytics, and ML workloads

Select and standardize

cluster, compute, and workspace architectures

Design multi-workspace strategies (dev/test/prod, shared vs. isolated)

Ensure architectures meet

scalability, availability, and performance requirements

Well-Architected Framework Alignment

Apply Databricks best practices across all pillars, including:

Security & Governance

(Unity Catalog, IAM, data access controls)

Reliability & Resilience

(job retries, checkpointing, failure isolation)

Performance Efficiency

(cluster sizing, autoscaling, caching)

Cost Optimization

(compute policies, workload separation, monitoring)

Operational Excellence

(monitoring, automation, CI/CD, runbooks)

Implementation & Engineering

Lead

Databricks workspace, cluster, and Unity Catalog implementations

Implement Delta Lake, Delta Live Tables (DLT), and Structured Streaming

Build and optimize

• ETL/ELT pipelines using Spark and SQL
• Integrate Databricks with cloud services (S3/ADLS/GCS, IAM, Key Vault, networking)
• Establish CI/CD pipelines for notebooks, jobs, and infrastructure

Security, Governance & Compliance

• Implement Unity Catalog for centralized governance
• Define data classification, lineage, and audit strategies
• Enforce least-privilege access and secure networking patterns
• Support compliance requirements (HIPAA, SOC 2, PCI, GDPR as applicable)

Operations & Optimization

• Monitor platform health, performance, and cost
• Troubleshoot production issues across jobs, clusters, and data pipelines
• Perform workload tuning and cost-performance optimization

Define

SLOs, alerts, and operational metrics

Collaboration & Leadership

• Partner with Data Engineering, Analytics, ML, Platform, and Security teams
• Translate business requirements into technical architectures
• Provide architectural guidance and technical mentorship
• Communicate risks, tradeoffs, and recommendations to leadership

Required Qualifications

Experience

7+ years

• in data engineering, analytics, or platform architecture
• 3–5+ years hands-on Databricks experience in production environments
• Proven experience applying the Databricks Well-Architected Framework

Experience designing

cloud-native lakehouse architectures

Experience supporting

mission-critical data platforms

Technical Skills

Databricks Lakehouse Platform

Apache Spark (PySpark / Scala / Spark SQL)

Delta Lake, Delta Live Tables, Structured Streaming

• Unity Catalog (governance, lineage, access controls)
• Cloud platforms: AWS, Azure, or GCP
• Infrastructure as Code (Terraform strongly preferred)

CI/CD tools (GitHub Actions, Azure DevOps, GitLab, etc.)

Data formats and protocols (Parquet, JSON, Avro)

Certifications Required

Databricks Certified Data Engineer Professional

Databricks Certified Professional Architect

(or equivalent advanced certification)

Preferred / Additional Certifications

AWS Certified Solutions Architect (Associate or Professional)

Azure Solutions Architect Expert

Google Professional Data Engineer

Databricks Machine Learning Professional

Snowflake or other cloud data platform certifications

Soft Skills

• Strong architectural decision-making and documentation skills
• Excellent communication with technical and non-technical stakeholders
• Ability to lead design reviews and architecture governance forums
• Strong troubleshooting and performance-tuning mindset

Nice-to-Have Experience

MLflow and MLOps architectures

• Real-time analytics and streaming pipelines
• Multi-region or cross-account data architectures
• Consulting or MSP delivery experience