Databricks platform refactoring for a critical SLA

Problem

The platform relied on a highly complex framework that made maintenance hard and prevented data delivery within the business-critical 15-minute SLA.

Architecture

flowchart TB
  subgraph Before
    A1[Complex framework] --> A2[Hard to maintain]
    A2 --> A3[SLA missed > 15min]
  end
  subgraph After
    B1[Redesigned framework] --> B2[Optimized PySpark + SQL]
    B2 --> B3[Parallel orchestration]
    B3 --> B4[15min SLA met]
  end
  A3 -.refactor.-> B1
  classDef bad fill:#1a1018,stroke:#ef4444,color:#e2e8f0
  classDef good fill:#0d1525,stroke:#10b981,color:#e2e8f0
  class A1,A2,A3 bad
  class B1,B2,B3,B4 good

Solution

• Completely redesigned the framework, simplifying architecture and maintenance.
• Optimized PySpark and SQL transformations (partitioning, joins, caching).
• Restructured orchestration to parallelize steps and shorten the critical path.
• Introduced observability to track the SLA in production.

Results

• 15-minute SLA consistently achieved.
• More scalable, stable and maintainable platform.
• Significant reduction in team maintenance effort.

Stack