Matt Faltyn

Engineering initiative / rocky

Making Analytics Execution Fail Closed

Across 27 merged Rocky contributions, I found a recurring systems risk: control-plane claims could diverge from what the warehouse would execute or had already committed. This case study follows six changes that made those boundaries fail closed.

My role
Upstream contributor
System
Rust analytics engine
Contribution
27 merged PRs
Evidence window
July 18 – September 1, 2026

01 / Pattern

The recurring failure class

Rocky is a SQL transformation engine that plans, compiles, executes, and recovers analytical pipelines across multiple warehouses. Correctness depends on agreement between user intent, generated plans, warehouse state, and the engine’s durable ledger.

The failures appeared at different layers: a preview could describe work apply would skip, a failed metadata probe could expose a live target to replacement, committed rows could outpace their watermarks, and resume could silently start without authoritative state. Compiler gaps could also approve contracts the executed SQL could violate. The common risk was control-plane confidence without execution-plane truth.

Explore rocky

02 / Contract

The execution-safety contract

I used four connected invariants to reason from command intent through compilation, mutation, and recovery.

  1. Plan must match apply

    The previewed scope, persisted plan, and executed work must identify the same models and layers.

  2. Uncertainty must not authorize mutation

    Missing metadata or recovery authority must fail before destructive replacement or an unintended fresh run.

  3. State must follow durable writes

    A checkpoint may advance for committed work, never for failed work, and recovery must not repeat an already durable delta.

  4. Unsafe contracts must fail early

    When semantics cannot be preserved or a schema promise is false, compilation should stop before warehouse execution.

03 / Initiative

How I drove the initiative

The work tightened one execution chain rather than optimizing isolated commands.

Align intent and execution

Made filtered plans truthful and refused imported incremental logic when the engine could not preserve its semantics.

Protect durable data

Moved bootstrap, watermark, and resume decisions onto evidence that reflected committed warehouse and authoritative checkpoint state.

Reject unsafe contracts early

Strengthened compiler and type boundaries so invalid assumptions become diagnostics instead of production data failures.

04 / Judgment

Decisions and tradeoffs

  1. Refuse semantics the engine cannot preserve

    Raw dbt models using is_incremental() are rejected instead of being converted into plausible but behaviorally different SQL.

  2. Make uncertain bootstrap non-destructive

    Bootstrap uses non-replacing creation unless metadata positively confirms that full replacement is intentional.

  3. Checkpoint tables independently

    After partial failure, successful table watermarks persist because their warehouse writes are already durable; failed tables enqueue no watermark.

  4. Require authority before recovery

    An explicit resume must resolve an existing replication checkpoint from authoritative state or stop before discovery and materialization.

  5. Correct contracts without widening public interfaces

    Outer-join nullability is propagated inside compilation while public IR, CLI JSON, dependencies, and generated SQL remain unchanged.

05 / Validation

Public validation and influence

Project review repeatedly tested the safety claims against executable behavior and used the findings to strengthen adjacent paths.

  • Destructive path tested

    Review of #1207 fault-injected target discovery and verified that uncertain bootstrap fails without replacing live data.

  • Durability checked

    Review of #1411 traced every watermark enqueue to a successful durable write and checked recovery without duplication.

  • Follow-on fix merged

    Review of #1544 exposed the remaining DAG flag path; a project reviewer closed it in merged follow-up #1546.

These six pull requests are representative evidence within a broader contribution history. They establish the stated plan, mutation, state, recovery, and compiler invariants; they do not claim organization-wide adoption or user-impact metrics.

06 / Evidence

Selected evidence

Six representative pull requests connect the initiative from command intent to warehouse execution. The broader public record contains 27 merged Rocky contributions.

Align intent and execution

Protect durable data

Reject unsafe contracts early

07 / Next

Continue exploring the work

Related case studyPrevious initiative: Apache Polaris reliabilityDiscuss data platform engineering
All my Rocky contributions Back to Open Source