Contract Tests vs Schema Diffs

This comparison belongs to Breaking Change Detection in the API Versioning & Deprecation reference. Teams frequently adopt one technique and assume it covers the other’s ground. It does not: a diff has complete coverage of a narrow thing, contract tests have narrow coverage of a broad thing, and the gap between them is where production incidents live.

The decision trigger

Incident Which technique would have caught it
A response field was renamed schema diff
An endpoint nobody remembered was deleted schema diff
The default sort order flipped contract test
A new validation rule rejected old payloads contract test / replay
The error type URI was reworded contract test
An optional field started returning null always contract test
A type was narrowed from number to integer schema diff
The server stopped returning a documented field response validation

Coverage models compared

Dimension Schema diff Consumer contract tests
What it compares two documents recorded expectations vs a running service
Coverage every operation in the document only paths a consumer exercises
Sees behaviour no yes
Sees undocumented behaviour no yes
Needs a running service no yes
Needs consumer participation no yes
Runtime seconds minutes
Fails on prose edits possibly (exclude them) no
Catches implementation drift no partially
Setup cost one CI job broker, publishing, verification per consumer
Three layers of compatibility checking The document layer is checked by a schema diff. The implementation layer is checked by validating live responses against the schema. The consumer layer is checked by contract tests recording what each consumer actually depends on. document what the specification promises schema diff implementation what the server actually returns response validation consumers what each client actually depends on contract tests drift here is invisible to a diff expectations here are invisible to both

The middle layer nobody runs

Most teams have a diff or contract tests, and almost none validate live responses against their own schema. That middle layer catches implementation drift — the server quietly diverging from its own document — which neither other technique sees. The document says the field exists, so the diff is happy; no consumer test asserts it, so the suite passes; and a client generated from the spec expects a field that never arrives.

// response-validation.test.ts — assert the server matches its own document
import { describe, it, expect } from "vitest";
import Ajv2020 from "ajv/dist/2020";
import addFormats from "ajv-formats";
import spec from "../dist/openapi.json" assert { type: "json" };

const ajv = addFormats(new Ajv2020({ strict: false, allErrors: true }));
for (const [name, schema] of Object.entries(spec.components.schemas)) {
  ajv.addSchema(schema as object, `#/components/schemas/${name}`);
}

const CASES = [
  { path: "/invoices",             schema: "InvoiceListEnvelope" },
  { path: "/invoices/inv_seed_1",  schema: "Invoice" },
  { path: "/customers",            schema: "CustomerListEnvelope" },
];

describe("live responses match the published schema", () => {
  for (const { path, schema } of CASES) {
    it(`${path} conforms to ${schema}`, async () => {
      const res = await fetch(`${BASE}${path}`, { headers: AUTH });
      const body = await res.json();
      const validate = ajv.getSchema(`#/components/schemas/${schema}`)!;
      if (!validate(body)) {
        throw new Error(
          `response does not match ${schema}:\n` +
          (validate.errors ?? []).map(e => `  ${e.instancePath} ${e.message}`).join("\n"),
        );
      }
      expect(res.status).toBe(200);
    });
  }
});

This is cheap to add — the schemas already exist — and it is the only check that keeps the document honest. Everything else in this reference assumes the spec describes reality; this is the test that verifies the assumption.

What consumer contract tests add

A consumer-driven contract records what a specific client actually depends on, which is usually a small subset of what the document permits. That specificity is the value: the provider learns which of its documented promises are load-bearing.

// consumer side — recorded expectation, published to a broker
import { PactV3, MatchersV3 } from "@pact-foundation/pact";

const { like, eachLike, integer } = MatchersV3;

const provider = new PactV3({
  consumer: "billing-dashboard",
  provider: "invoices-api",
});

describe("invoice list", () => {
  it("returns items with the fields the dashboard renders", async () => {
    await provider
      .given("three invoices exist")
      .uponReceiving("a request for the first page")
      .withRequest({ method: "GET", path: "/invoices", query: { limit: "50" } })
      .willRespondWith({
        status: 200,
        headers: { "Content-Type": "application/json" },
        body: {
          data: eachLike({
            id: like("inv_1"),
            total_minor: integer(4200),
            currency: like("GBP"),
            status: like("paid"),          // dashboard branches on this
          }),
          meta: { has_more: like(true) },
        },
      })
      .executeTest(async mock => {
        const page = await listInvoices(mock.url, { limit: 50 });
        expect(page.data[0].status).toBeDefined();
      });
  });
});

The provider then verifies every published expectation before release. What this catches that a diff cannot: the dashboard depends on status being present and on the specific values it branches on, so a server that starts returning a new status value — perfectly additive at the document level — fails verification.

Consumer-driven contract verification loop The consumer's test suite publishes recorded expectations to a broker. The provider's pipeline fetches them, replays each interaction against a running build, and blocks release if any expectation fails. consumer tests record expectations publish broker versioned contracts fetch provider verify replay eachinteraction release or block covers only what consumers exercise — untested endpoints are invisible here

Cost is the other axis, and it explains why so many teams have one technique and not the other:

Setup and running cost of each technique A schema diff is one CI job with no running service and complete structural coverage. Contract tests need a broker, consumer participation and a running provider, but see behaviour. schema diff consumer contract tests one CI job, no service needed runs in seconds covers every operation zero consumer participation broker, publishing, verification runs in minutes covers exercised paths only every consumer must take part the cheap technique has complete coverage of the narrow thing — start there

Choosing what to run

Not every API needs all three layers. The useful rule is to scale with consumer visibility:

Situation Diff Response validation Contract tests
Internal API, one consumer, same team yes yes probably not
Internal platform API, many teams yes yes yes
Public API, unknown consumers yes yes limited value
API with a published SDK yes yes replace with SDK tests

Contract tests need consumer participation, which makes them powerful inside an organisation and impractical for a public API where you cannot enumerate — let alone instrument — your callers. For public APIs, spend the same effort on a broad behavioural test suite of your own plus traffic replay, and rely on the diff for structural coverage. That is the same reasoning behind the release gates in Client SDK Generation Pipelines.

Standard compliance

Concern Standard Note
Document comparison OpenAPI 3.0 / 3.1 The input to a diff
Response body validation JSON Schema 2020-12 The mechanism for the middle layer
Error contract stability RFC 9457 type URIs asserted by contract tests
Compatibility classification convention No RFC defines “breaking” for HTTP APIs

Running both without duplicating effort

The two techniques share one artefact: the schema. Use it as the single source for both, so a change to the document propagates automatically:

// shared-fixtures.ts — one schema, three consumers of it
import spec from "../dist/openapi.json" assert { type: "json" };

// 1. The diff compares this file against the released baseline (CI job).
// 2. Response validation compiles these schemas and checks live bodies.
export const schemaFor = (name: string) => spec.components.schemas[name];

// 3. Contract-test fixtures are generated FROM the schema, so a field added
//    to the document appears in the fixtures without a manual edit.
export function exampleFor(name: string): Record<string, unknown> {
  const schema = schemaFor(name) as any;
  const out: Record<string, unknown> = {};
  for (const [prop, def] of Object.entries<any>(schema.properties ?? {})) {
    if (!schema.required?.includes(prop)) continue;
    out[prop] = def.examples?.[0] ?? def.example ?? defaultForType(def.type);
  }
  return out;
}

Hand-maintained fixtures are the usual reason a contract-test suite decays: they drift from the schema, tests start failing for uninteresting reasons, and the suite gets muted. Generating the required-field skeleton from the document keeps them honest.

Common mistakes

Mistake Correct approach
Assuming a diff covers behaviour Add contract tests or a behavioural suite
Assuming contract tests cover every endpoint Keep the diff for full structural coverage
Never validating live responses against the schema Add the middle layer; it is nearly free
Hand-maintaining contract fixtures Generate required-field skeletons from the schema
Contract tests for a public API with unknown consumers Use your own behavioural suite plus traffic replay
Muting a flaky contract suite Fix the fixture drift; a muted gate is no gate

FAQ

Do I need consumer contract tests if I already diff my OpenAPI document?

If behaviour matters to your consumers — and it does — then yes. A diff proves one narrow thing very thoroughly: that the document did not change in a structurally incompatible way, across every operation, in seconds, with no running service required. It cannot see a changed default, a reversed sort order, a new validation rule enforced in code, or an error identifier that was reworded, because none of those are expressed in the document. Those are exactly the changes that pass review, pass CI, and break a consumer at deploy time. The two techniques overlap so little that running both is closer to addition than to duplication.

Can consumer-driven contract tests replace a schema diff?

No, because their coverage follows usage rather than the document. If no consumer test touches /v1/legacy-report, it can be deleted, renamed or have every field removed without a single failure — while a diff reports it immediately. Contract tests are also asymmetric in a way people forget: they verify what consumers currently depend on, so a consumer that has not yet shipped its dependency on a field gets no protection. Keep the diff as the floor of structural coverage and let contract tests add behavioural depth on the paths that matter most.

Where does response validation against the schema fit?

It is the layer between the other two, and the one most often missing. A diff compares documents; contract tests compare consumer expectations against a running service; response validation compares the running service against its own document. It catches implementation drift — a field the code stopped populating, a status value the schema does not list, an envelope that lost its meta member — none of which any diff can see and none of which a consumer test will catch unless that consumer happens to depend on the drifted field. The schemas already exist, so adding it is a few dozen lines in your integration suite.

Back to Breaking Change Detection