Sparse Fieldsets & Projection

Sparse fieldsets let API clients declare exactly which attributes they need, transforming monolithic responses into lean, purpose-built payloads. Without an explicit projection contract, every consumer over-fetches by default — serializing columns that are never read, bloating TLS payloads, and fragmenting HTTP caches unpredictably. Part of the Query Patterns & Data Shaping Strategies reference.

The Projection Contract Failure

The specific failure this page addresses: a GET /users/123 endpoint returns 42 JSON fields including computed aggregates, PII, and internal flags. A mobile client reads three of them. The result is a 12 KB payload where 400 bytes are used, cache hit rates drop because ?fields= variants multiply independently, and any internal field rename becomes a latency-silent breaking change for the client.

Projection solves this by making the response shape an explicit, validated contract element — not an afterthought. It pairs directly with advanced filtering operators (which control which resources return) and with offset vs cursor pagination (which controls how many return). Together, the three form the full query contract surface for list and detail endpoints.

Request/Response Projection Flow

The diagram below shows how a ?fields= request moves through a layered API stack — from client through gateway, service, ORM, and back — and where each validation step fires.

Spec Definition

Define the fields parameter in OpenAPI 3.1 with an explicit pattern constraint. The style: form + explode: false combination produces a single comma-separated string — the most cache-friendly serialisation.

# openapi.yaml (OpenAPI 3.1.0)
paths:
  /users/{id}:
    get:
      operationId: getUser
      parameters:
        - name: fields
          in: query
          required: false
          style: form
          explode: false
          description: >
            Comma-separated list of resource attributes to include.
            Omitting this parameter returns the full resource.
          schema:
            type: string
            pattern: "^[a-zA-Z0-9_.][a-zA-Z0-9_.,]*$"
            example: "id,username,email,profile.avatar_url"
      responses:
        "200":
          description: Projected user resource
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/User"
        "400":
          description: One or more requested fields are invalid or not permitted
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/FieldProjectionError"

components:
  schemas:
    FieldProjectionError:
      type: object
      required: [type, status, invalid_fields]
      properties:
        type:
          type: string
          example: "https://api-contract.com/errors/invalid-field"
        status:
          type: integer
          example: 400
        invalid_fields:
          type: array
          items:
            type: string
          example: ["secret_api_key", "internal_score"]
        allowed_fields:
          type: array
          items:
            type: string

RFC and Standard Alignment

Implementation Walkthrough — Server Side

Parse, validate, and push projection to the ORM in three sequential steps: no step may be skipped.

// Node.js / Express — TypeScript
import { RequestHandler } from "express";

// Static allowlist per resource type — never derived from DB schema at runtime
const USER_PROJECTABLE_FIELDS = new Set([
  "id", "username", "email", "createdAt",
  "profile.avatarUrl", "profile.bio",
]);

// Always-included fields regardless of what the client requests
const USER_REQUIRED_FIELDS = new Set(["id"]);

export function parseProjection(
  rawFields: string | undefined,
  allowed: Set<string>,
  required: Set<string>
): Set<string> {
  if (!rawFields) return new Set([...allowed]); // no ?fields → full resource

  // Sort + deduplicate for cache-key stability before any work
  const requested = [...new Set(rawFields.split(",").map((f) => f.trim()))].sort();

  const invalid = requested.filter((f) => !allowed.has(f));
  if (invalid.length) {
    throw Object.assign(new Error("invalid fields"), {
      status: 400,
      invalid_fields: invalid,
      allowed_fields: [...allowed].sort(),
    });
  }

  // Merge required fields so id is always present
  return new Set([...required, ...requested]);
}

// Express route handler
export const getUser: RequestHandler = async (req, res, next) => {
  try {
    const projection = parseProjection(
      req.query.fields as string | undefined,
      USER_PROJECTABLE_FIELDS,
      USER_REQUIRED_FIELDS
    );

    // Push projection directly to the ORM — never SELECT * then filter in memory
    const user = await db.user.findUnique({
      where: { id: req.params.id },
      select: Object.fromEntries([...projection].map((f) => [f, true])),
    });

    if (!user) return res.status(404).json({ status: 404, detail: "User not found" });

    // Normalise cache key: sorted, deduplicated field list
    const cacheKey = [...projection].sort().join(",");
    res.setHeader("Vary", "Accept, fields");
    res.setHeader("X-Cache-Fields", cacheKey);

    res.json(user);
  } catch (err: any) {
    if (err.status === 400) {
      return res.status(400).json({
        type: "https://api-contract.com/errors/invalid-field",
        status: 400,
        invalid_fields: err.invalid_fields,
        allowed_fields: err.allowed_fields,
      });
    }
    next(err);
  }
};

# Python / FastAPI equivalent
from fastapi import FastAPI, Query, HTTPException
from typing import Optional
import re

app = FastAPI()

USER_PROJECTABLE_FIELDS = {
    "id", "username", "email", "created_at",
    "profile_avatar_url", "profile_bio",
}
USER_REQUIRED_FIELDS = {"id"}

_FIELD_PATTERN = re.compile(r"^[a-zA-Z0-9_.][a-zA-Z0-9_.,]*$")

def parse_projection(
    raw: Optional[str],
    allowed: set[str],
    required: set[str],
) -> set[str]:
    if not raw:
        return set(allowed)  # no ?fields → full resource

    if not _FIELD_PATTERN.match(raw):
        raise HTTPException(status_code=400, detail={"invalid_fields": ["<malformed>"]})

    # Sort + dedup for cache-key stability
    requested = sorted(set(raw.split(",")))
    invalid = [f for f in requested if f not in allowed]
    if invalid:
        raise HTTPException(
            status_code=400,
            detail={
                "type": "https://api-contract.com/errors/invalid-field",
                "status": 400,
                "invalid_fields": invalid,
                "allowed_fields": sorted(allowed),
            },
        )
    return required | set(requested)

@app.get("/users/{user_id}")
async def get_user(user_id: str, fields: Optional[str] = Query(default=None)):
    projection = parse_projection(fields, USER_PROJECTABLE_FIELDS, USER_REQUIRED_FIELDS)
    # Push to ORM — only fetch projected columns
    row = await db.fetch_user(user_id, columns=list(projection))
    if not row:
        raise HTTPException(status_code=404)
    return row

Implementation Walkthrough — Client Side and CI

Cache-Key Normalisation Middleware

HTTP intermediaries cache on the raw URI. Two requests for ?fields=email,id and ?fields=id,email produce different cache keys but identical responses. Normalise at the edge or in an ingress middleware before the request reaches your service.

// Express middleware: normalise ?fields before routing
import { Request, Response, NextFunction } from "express";

export function normaliseFieldsParam(
  req: Request,
  _res: Response,
  next: NextFunction
): void {
  const raw = req.query.fields as string | undefined;
  if (raw) {
    // sort + deduplicate + lowercase-normalise
    const normalised = [...new Set(raw.split(",").map((f) => f.trim().toLowerCase()))]
      .sort()
      .join(",");
    req.query.fields = normalised;
    // Rewrite the URL so downstream cache keys are stable
    req.url = req.url.replace(/fields=[^&]*/, `fields=${normalised}`);
  }
  next();
}

CI Gate: Spectral Rules

Block undocumented endpoints and missing fields parameter definitions before merge.

# .spectral.yaml
rules:
  projection-parameter-required:
    description: All GET endpoints on resource paths must declare a fields query parameter.
    given: "$.paths[?(@property.match(/\\/[a-z]+\\/\\{[^}]+\\}$/))].get"
    severity: warn
    then:
      field: parameters
      function: schema
      functionOptions:
        schema:
          type: array
          contains:
            type: object
            properties:
              name:
                const: fields
            required: [name]

  projection-error-response-defined:
    description: Endpoints with a fields parameter must define a 400 response.
    given: "$.paths.*.get[?(@.parameters && @.parameters[?(@.name=='fields')])]"
    severity: error
    then:
      field: responses.400
      function: defined

Edge-Case Handling

Nested field paths. A dot-notation field like profile.avatarUrl must be expanded into a nested ORM select before hitting the database. Flatten the projection tree at parse time, not during serialisation, to avoid N+1 relationship loads.

Bulk and list endpoints. On GET /users, every row in the result set must receive the same projection. Do not project row-by-row in the serialiser — pass the projected column list to the ORM’s bulk findMany call so the SELECT pushdown applies across the entire result.

Conditional request interplay. If a client sends both ?fields= and If-None-Match, the ETag must be computed over the projected payload, not the full resource. A narrower ?fields= request against the same resource produces a different ETag.

Empty fields parameter. ?fields= (present but empty string) is distinct from the parameter being absent. Treat it as a client error (400) rather than silently falling back to the full resource — the intent is ambiguous and silent fallback masks client bugs.

Deprecated fields. When a field is deprecated, include it in the allowlist with a Deprecation header in the response (Deprecation: true; sunset="2026-12-31"). Remove it from the allowlist only after the sunset date passes.

Validation and Testing Patterns

// Jest + Supertest — projection validation suite
import request from "supertest";
import app from "../src/app";

describe("GET /users/:id projection", () => {
  test("returns only requested fields plus id", async () => {
    const res = await request(app)
      .get("/api/v1/users/123")
      .query({ fields: "email,username" })
      .expect(200);

    expect(Object.keys(res.body)).toEqual(["id", "email", "username"].sort());
  });

  test("400 on unknown field", async () => {
    const res = await request(app)
      .get("/api/v1/users/123")
      .query({ fields: "id,secret_api_key" })
      .expect(400);

    expect(res.body.invalid_fields).toContain("secret_api_key");
    expect(res.body.allowed_fields).toBeDefined();
  });

  test("400 on empty fields parameter", async () => {
    await request(app)
      .get("/api/v1/users/123")
      .query({ fields: "" })
      .expect(400);
  });

  test("cache key is stable regardless of field order", async () => {
    const [a, b] = await Promise.all([
      request(app).get("/api/v1/users/123").query({ fields: "email,id" }),
      request(app).get("/api/v1/users/123").query({ fields: "id,email" }),
    ]);
    expect(a.headers["x-cache-fields"]).toBe(b.headers["x-cache-fields"]);
  });
});

Contract-level validation belongs alongside the adding sparse fieldset support to OpenAPI specs workflow, which covers OpenAPI diff gating and openapi-diff PR checks.

SDK Generation Impact

Projection collapses the static type of a resource to a subset, which requires generated SDK clients to use conditional or partial types rather than the full resource model.

TypeScript — Pick<T, K>

// Generated or hand-authored SDK type wrapper
type UserFields = "id" | "username" | "email" | "createdAt" | "profile.avatarUrl" | "profile.bio";

async function getUser<F extends UserFields>(
  id: string,
  fields: F[]
): Promise<Pick<User, F>> {
  const res = await fetch(`/api/v1/users/${id}?fields=${fields.sort().join(",")}`);
  if (!res.ok) throw await res.json();
  return res.json() as Pick<User, F>;
}

// Usage: TypeScript narrows the return type to only the requested fields
const user = await getUser("123", ["id", "email"]);
// user.username  // ← compile-time error: not in the Pick

Python — Pydantic model_dump(include=...)

from pydantic import BaseModel
from typing import Optional

class User(BaseModel):
    id: int
    username: str
    email: str
    created_at: str
    profile_avatar_url: Optional[str] = None
    profile_bio: Optional[str] = None

def fetch_user(user_id: str, fields: list[str]) -> dict:
    import httpx
    params = {"fields": ",".join(sorted(set(fields)))}
    resp = httpx.get(f"/api/v1/users/{user_id}", params=params)
    resp.raise_for_status()
    user = User(**resp.json())
    # Validate that only requested fields (plus id) are present
    return user.model_dump(include=set(fields) | {"id"})

Go — omitempty struct tags

type User struct {
    ID        string  `json:"id"`                       // always present
    Username  string  `json:"username,omitempty"`
    Email     string  `json:"email,omitempty"`
    CreatedAt string  `json:"created_at,omitempty"`
    // Never exported via projection — use json:"-"
    PasswordHash string `json:"-"`
    InternalScore int   `json:"-"`
}

For OpenAPI codegen workflows (openapi-generator-cli, oapi-codegen), use a vendor extension to instruct the generator to emit partial types:

# openapi.yaml — vendor extension for codegen
x-codegen:
  projection:
    strategy: partial          # emit Pick<T,K> in TS; Optional fields in Python
    requiredFields: [id]       # always include in generated types
    fallbackBehavior: full-resource   # when fields param absent

Anti-Patterns Quick Reference

FAQ

Related

• Query Patterns & Data Shaping Strategies — parent reference covering the full query contract surface
• Advanced Filtering Operators — controls which resources return; pairs with projection to form the complete request contract
• Offset vs Cursor Pagination — controls how many resources return; combine with projection to minimise list-endpoint payload size
• Sorting & Multi-Field Ordering — the third query parameter axis; ordering interacts with cache keys the same way fields does
• Adding Sparse Fieldset Support to OpenAPI Specs — step-by-step guide for retrofitting projection into an existing OpenAPI spec with CI gating