API Design: REST, gRPC, GraphQL, Idempotency and Pagination

An API is a contract you can’t easily take back. Learn HTTP method and status semantics, Problem Details errors, ETags against lost updates, idempotency keys tested on PostgreSQL, offset vs cursor pagination measured, and when REST, gRPC or GraphQL fits.

An API is the part of a system other people build on. Once a mobile app ships with your endpoint in it, you can’t change that endpoint’s meaning without breaking someone. So API design is mostly about the things that are hard to change later: what each operation promises, what happens when a request is sent twice, how a client walks through a long list, how errors look, and how the contract grows.

Parts 1 and 3 followed requests over the network. This part designs what those requests say. We ran the parts that can go wrong against a real PostgreSQL database, so the pagination and retry behaviour you’ll see was measured, not described.

Try this first

A feed shows the newest posts first, 10 per page. The app asks for page 2 with ?limit=10&offset=10.

1. A reader loads page 1 and sees posts 30 down to 21. Before they scroll, 5 new posts are published. What does page 2 show?
2. Instead of new posts, 3 of the posts on page 1 are deleted. What does page 2 show now?
3. A payment request times out, and the app sends it again. Two servers receive the two copies at the same moment. Both check the database for an existing charge, find none, and create one. How do you stop the customer paying twice?

Write your answers down. We ran all three, and the answers are in the sections on pagination and idempotency.

What an API contract promises

Every API, whatever its style, makes promises in five areas:

Area	The question it answers
Operations	What can a client do, and to what?
Semantics	Is it safe to repeat? Can a cache store it? What if two clients do it at once?
Errors	How does a client tell “try again” from “you sent something wrong”?
Collections	How does a client read a list that’s too big for one response?
Evolution	What can change without breaking existing clients?

HTTP answers many of these for you, if you use it the way it’s defined. So we start there, and come back to gRPC and GraphQL once the ideas are in place.

HTTP methods: safe and idempotent

RFC 9110 defines two properties that decide what software between the client and your server may do with a request.

Safe means read-only in intent: “the client does not request, and does not expect, any state change on the origin server”. GET, HEAD, OPTIONS and TRACE are safe. Safe doesn’t mean nothing happens. The RFC’s own example is an access log that grows with every request, and that still counts as safe. What matters is that the client didn’t ask for a change.

Idempotent means “the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request”. PUT, DELETE and every safe method are idempotent.

Method	Safe	Idempotent	Meaning
GET	yes	yes	Read the resource
HEAD	yes	yes	GET without the body
OPTIONS	yes	yes	What’s supported here?
PUT	no	yes	Create or replace the resource with this representation
DELETE	no	yes	Remove the association between the URL and the resource
POST	no	no	Process this, by the resource’s own rules
PATCH	no	no, by default	Apply these changes (RFC 5789)

Why this matters in practice: RFC 9110 says idempotent requests “can be repeated automatically if a communication failure occurs before the client is able to read the server’s response”. Clients, proxies and load balancers rely on that. They may retry a GET or a PUT on their own. For POST the rule is the opposite: “A proxy MUST NOT automatically retry non-idempotent requests.”

Three details that are often misread:

• Idempotent doesn’t mean the same response. Deleting an order twice might return 204 and then 404. The effect, “the order is gone”, is the same, so DELETE is still idempotent. The RFC says so directly: the response “might differ”.
• PATCH isn’t idempotent by default, but it can be. RFC 5789 says a PATCH “can be issued in such a way as to be idempotent”. Setting a field to a value is idempotent in effect. Appending to a list isn’t. A PATCH must also be applied atomically: all of it, or none of it.
• A body on GET has “no generally defined semantics”. Some servers and proxies drop it or reject the request. If a query is too big for a URL, the usual answer is POST, which loses GET’s caching and automatic retries. RFC 10008, published in June 2026, defines a new method for this, QUERY: it carries a body and is safe, idempotent and cacheable. It’s so new that servers, proxies and client libraries may not support it yet, so check before you rely on it.

If a GET changes data, crawlers, link previews and retries will change it for you. RFC 9110 goes further: if a resource does something unsafe when accessed with a safe method, “the resource owner MUST disable or disallow that action”.

Status codes that carry meaning

A status code is read by software you don’t control: browsers, caches, retry logic, monitoring. Use the one whose meaning fits.

Code	Meaning	Use it for
200	OK	A successful read, or an update that returns the new state
201	Created	A new resource; send Location with its URL
202	Accepted	Accepted but not done yet; point to where the client can check progress
204	No Content	Success with nothing to return; it can’t carry a body
304	Not Modified	A conditional GET found the resource unchanged: use your cached copy
400	Bad Request	The request is malformed
401	Unauthorized	Missing or invalid authentication; the response MUST include WWW-Authenticate
403	Forbidden	Understood, but refused: typically, not allowed
404	Not Found	No such resource, or you won’t say whether it exists
405	Method Not Allowed	Wrong method for this URL; MUST include Allow
409	Conflict	Conflicts with the resource’s current state; the user may be able to fix it and retry
412	Precondition Failed	A condition such as If-Match evaluated to false
415	Unsupported Media Type	The body’s format isn’t accepted
422	Unprocessable Content	Well-formed, but the instructions can’t be processed
428	Precondition Required	This server requires a conditional request (RFC 6585)
429	Too Many Requests	Rate limited; may include Retry-After (RFC 6585)
500	Internal Server Error	Something unexpected broke on the server
503	Service Unavailable	Temporarily overloaded or down; may include Retry-After

A few that trip people up:

• 401 is about authentication, despite its name. 403 is the authorization refusal. RFC 9110 also allows 404 instead of 403 when you don’t want to reveal that a resource exists.
• 422 is now core HTTP. It came from WebDAV as “Unprocessable Entity”, and RFC 9110 took it in as “Unprocessable Content”.
• Don’t invent a status code per error. RFC 9205, the IETF’s advice for building on HTTP, says applications “should not specify a one-to-one relationship between status codes and application errors”. Proxies and CDNs generate status codes too. Put the specific error in the body.

Errors as a contract: Problem Details

Every team invents an error format, and every client has to learn it. RFC 9457, “Problem Details for HTTP APIs”, is the standard one. It replaced RFC 7807 in 2023. The media type is application/problem+json, and the body has five standard members:

Member	What it holds
type	A URI that identifies the kind of problem. Clients key on this. Absent means about:blank
title	A short summary of the problem type. It shouldn’t change between occurrences
status	The HTTP status code, “only advisory”; it MUST match the real one
detail	An explanation of this occurrence, for humans. Clients shouldn’t parse it
instance	A URI for this specific occurrence

You can add your own members, such as a list of invalid fields. This is the body ASP.NET Core produced for a 412 in the program later in this post:

{"type":"https://tools.ietf.org/html/rfc9110#section-15.5.13","title":"Precondition Failed","status":412,"detail":"The article changed since you read it","traceId":"0HNOJ4F8P8631:00000003"}

traceId is ASP.NET Core’s own extension member. RFC 9457 also warns that problem details “are not a debugging tool”: don’t put stack traces in them.

Lost updates, and If-Match

Alice and Bob open the same article. Alice changes the title and saves. Bob changes the title and saves a moment later. Bob’s save replaces Alice’s, and neither of them is told. That’s a lost update, and it happens with any “read, change, write back” API.

HTTP’s fix is the conditional request:

1. Every response carries an ETag, a validator that changes whenever the resource changes. A version number works.
2. The client sends it back in If-Match when it writes.
3. If the resource has changed since, the ETag no longer matches. The server MUST NOT apply the write, and normally responds 412 Precondition Failed.

RFC 9110 names exactly this use: If-Match exists “to prevent the “lost update” problem”. Choose a scenario and step through it:

Details that matter when you build it:

• If-Match uses strong comparison. A weak ETag, written W/"...", never matches If-Match. Use a real version number or a content hash.
• Check what sits in front of your server. A proxy that changes a response can weaken its ETag. nginx has done this since version 1.7.3: its changelog says strong ETags “are changed to weak” when a response is modified, for example when nginx compresses it. Clients that copy that ETag into If-Match then get 412 on every write.
• A server can require conditions. Respond 428 Precondition Required to a write without If-Match. RFC 6585 created 428 for the lost-update problem.
• If-None-Match: * stops two clients creating the same resource with PUT. The write only succeeds if nothing exists yet.
• The check and the write must be one atomic step. Compare and update in one database statement, such as UPDATE ... WHERE id = $1 AND version = $2, and check that one row changed. Otherwise two writers can both pass the check.

PUT, PATCH, and the two patch formats

PUT replaces. The body is the whole new state. If a client sends a PUT without a field it didn’t know about, it has removed that field. That’s why PUT from older clients can quietly delete data added by newer ones.

PATCH changes part of a resource. There are two standard formats:

	JSON Merge Patch (RFC 7396)	JSON Patch (RFC 6902)
Media type	application/merge-patch+json	application/json-patch+json
Looks like	The fields to change: {"title": "New", "note": null}	A list of operations: [{"op": "replace", "path": "/title", "value": "New"}]
Removing a field	Set it to null	remove
Setting a field to null	Impossible: null means remove	replace with null
Arrays	Replaced whole	Change one element
Conditions	None	test fails the whole patch if a value differs

Merge patch is simple and fits most APIs. JSON Patch is precise, and its test operation gives you a condition inside the patch. Either way, send If-Match with it.

REST, precisely

REST is the architectural style Roy Fielding described in his 2000 dissertation. It’s a set of constraints:

1. Client-server, with the concerns separated.
2. Stateless: “each request from client to server must contain all of the information necessary to understand the request”. Session state stays on the client.
3. Cache: responses are labelled cacheable or not.
4. Uniform interface: identification of resources, manipulation through representations, self-descriptive messages, and “hypermedia as the engine of application state”.
5. Layered system: a client can’t tell whether it’s talking to the server or to something in between.
6. Code on demand, which Fielding calls “only an optional constraint”.

The fourth one is where most “REST APIs” stop. Hypermedia means the client follows links in responses, the way a browser follows links in HTML, rather than building URLs from documentation. Fielding wrote in 2008 that an API not driven by hypertext “cannot be a REST API. Period.”

A hypermedia response carries the actions that are possible right now, as links:

{
  "id": 42,
  "status": "paid",
  "links": {
    "self": { "href": "/orders/42" },
    "cancel": { "href": "/orders/42/cancellation", "method": "POST" },
    "invoice": { "href": "/orders/42/invoice" }
  }
}

Once the order ships, the server stops sending the cancel link, and a client that follows links stops offering it. The client never needs to know the rule.

So most JSON APIs called REST are, more precisely, HTTP APIs that use resources and HTTP semantics. That’s fine. The parts that pay off in practice are the ones this post covers: resources with stable URLs, methods used as defined, meaningful status codes, cacheable reads and conditional writes.

Modelling resources:

• Nouns in URLs, verbs in methods: GET /orders/42, not GET /getOrder?id=42.
• Collections and items: /orders and /orders/42. POST /orders creates one, and the server chooses the id. RFC 9110 says a service that picks the URL for the client “SHOULD be implemented using the POST method rather than PUT”.
• Actions that aren’t a field change become resources too. POST /orders/42/cancellation records an intent, keeps a history, and can return 409 if the order has already shipped.
• Long operations: return 202 Accepted with a status resource the client can poll, such as /exports/7.

Idempotency and safe retries

Why retries create duplicates

A client sends a request and the connection times out. Did the server process it? The client can’t know. Maybe the request never arrived. Maybe it ran, and the response was lost on the way back.

For a GET or a PUT, retrying is fine. For “charge this card”, retrying may charge twice. gRPC’s documentation describes the same trap: a DEADLINE_EXCEEDED error “may be returned even if the operation has completed successfully”.

So any operation that isn’t naturally idempotent needs a way to make retries safe. The standard answer is an idempotency key.

Idempotency keys

The client generates a unique key per operation, such as a UUID, and sends it with every attempt of that operation:

POST /payments HTTP/1.1
Content-Type: application/json
Idempotency-Key: "8e03978e-40d5-43e8-bc93-6894a57f9324"

{"orderId": "order-42", "amountCents": 4999}

The server remembers each key with the result. A repeat returns the saved result instead of doing the work again.

There’s an IETF draft for this header, draft-ietf-httpapi-idempotency-key-header. Its latest version, -07, dates from October 2025 and expired in April 2026 without becoming an RFC. So treat it as well-documented common practice, not a standard. Its guidance:

Situation	Suggested response
Key missing where required	400
Key reused with a different request body	422
Retry while the first request is still running	409
Retry after the first request finished	The saved result, success or error

Stripe works much the same way. Its docs say it saves “the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeds or fails”, including 500 errors, and that keys can be pruned once they’re at least 24 hours old. There are exceptions: it saves nothing when validation fails, or when another request with the same key is still running, because the work never started. Note that the draft sends the key as a quoted string, while Stripe sends it bare.

Which results to save. Save a result once the work has started, success or failure, so a retry can’t run it twice. Don’t save rejections that happen before any work: validation errors, 429 and the in-progress 409. A saved 5xx says “we don’t know how this ended”. Stripe’s advice is to “treat the result of a 500 request as indeterminate”. So the client shouldn’t simply try again with a new key. It should look up what happened, for example by fetching the order’s payment status.

The draft also warns about scope. If keys are guessable, one client could replay another’s saved response. So store the key together with the client’s identity, such as the account id, not on its own.

Measured: two retries arrive at the same moment

The hard part isn’t remembering keys. It’s two copies of the same request arriving at two servers at the same time. We ran that on PostgreSQL 18.6, with two real database sessions, A and B, stepped in a fixed order. Each line in the figure is a statement that ran, with what PostgreSQL returned.

The five handlers:

Handler	What it does	Charges
Check, then insert	Both sessions look for a charge, both find none, both insert one	2
The same, under REPEATABLE READ	Each snapshot sees no charge, and both inserts commit	2
The same, under SERIALIZABLE	Both insert, but B’s COMMIT fails with error 40001, “could not serialize access”	1
A unique constraint on the charge’s key	B’s INSERT waits for A, then fails with error 23505, duplicate key	1
Insert the key first	A claims the key with INSERT ... ON CONFLICT DO NOTHING. B’s claim waits for A, then returns 0 rows, so B reads A’s result and returns it	1
Insert the key first, and A dies	A rolls back, as it would if its connection died, so B’s waiting claim succeeds and B does the work	1

What the run shows:

• Checking first doesn’t work under PostgreSQL’s default isolation, or under REPEATABLE READ. Both sessions see “no charge” and both insert. Nothing failed, so nothing was logged. The customer paid twice. SELECT ... FOR UPDATE doesn’t help either, because there’s no row yet to lock.
• The database can stop it, in three ways. SERIALIZABLE detects the conflict and fails one transaction, which the application must then retry. A unique constraint makes the second insert wait and fail. Claiming the key first makes the second request wait and then find the answer.
• Claiming the key first gives the cleanest API. B doesn’t get an error to translate. It gets the saved result, and replies exactly as the first request did. If A dies, its claim disappears with its transaction, and B takes over.

The call you can’t roll back

The lab kept everything in one database transaction. A real payment calls a payment provider over the network, and you can’t hold a database transaction open across that call, or roll back a charge the provider has already made. So production handlers split the work:

1. Claim the key and commit, with status in_progress and a hash of the request body.
2. Call the provider, passing an idempotency key of your own. Generate it in step 1 and store it with the claim, so every attempt, including a recovery job’s, sends the same key and the provider can deduplicate them. The provider must remember keys for longer than your whole retry and recovery window.
3. Save the result against the key and commit, with status done.

Then decide what the other cases do:

• A retry finds in_progress: return 409, as the draft suggests, and let the client retry later.
• A retry’s body hash differs: return 422.
• A worker crashed between steps 1 and 3: the key stays in_progress forever unless something recovers it. Give each claim a lease time, and let a recovery job ask the provider what happened to that key. A slow worker can outlive its lease, so the recovery job’s call is only safe because it sends the same provider key.
• Keys expire: document for how long. After that, a repeat is a new request.

Explain it like I’m ten

You post a birthday card to your friend, and you’re not sure it arrived. If you post a second card, your friend might get two. So you write a number on the envelope: “card number 7”. Your friend keeps a list of numbers they’ve already opened. When a card with number 7 arrives again, they don’t open it twice. They just tell you, “I got number 7, thanks.”

Now imagine both cards arrive at the same time, and two people at your friend’s house check the list at once. Both see that 7 isn’t on it, and both open a card. The fix is for the first person to write 7 on the list before opening the card. The second person sees it’s already there, and waits to hear what was inside.

The precise version

• The number is the idempotency key. The list is a table with a unique key column, and “writing it on the list first” is INSERT ... ON CONFLICT DO NOTHING, run before the work.
• “Waiting to hear what was inside” is the second transaction blocking on the first one’s uncommitted row, as in the lab. Across a network call, where you can’t hold the transaction open, it’s a stored in_progress status and a 409.
• Where the analogy breaks: your friend’s list lasts forever. Real keys expire, so a client that retries days later makes a new request.

Pagination

A list endpoint that returns everything works until the list has a million rows. Then every response is huge, slow and expensive. So you return a page, and the client asks for the next one. There are two common ways to say “the next one”.

Offset pagination, and what happens when rows change

Offset pagination says “skip n rows, then give me 10”: ?limit=10&offset=10, which becomes LIMIT 10 OFFSET 10 in SQL. It’s simple, and it lets a client jump to page 7. But the offset counts positions, and positions move when rows are inserted or deleted.

Keyset pagination says “give me 10 rows after this one”. The query asks for rows past the sort key of the last row the client saw: WHERE (created_at, id) < ($1, $2). APIs usually hand that position to the client as an opaque cursor. (Some APIs call an encoded offset a cursor too. In this post, cursor means keyset.)

We ran both against PostgreSQL, with the scenarios from the start of this post. Choose one and step through it:

Scenario	Offset page 2	Cursor page 2
5 new posts arrive before page 2	Ids 25 to 16: ids 25, 24, 23, 22 and 21 again	Ids 20 to 11
3 posts on page 1 are deleted	Ids 17 to 8: ids 20, 19 and 18 never shown	Ids 20 to 11

So the answers to the first two questions: with 5 inserts, page 2 repeated 5 posts the reader had just seen. With 3 deletes, 3 posts were skipped, and nothing told the reader. Cursor pagination returned exactly the next 10 posts both times, because it asks for “older than id 21”, not “positions 11 to 20”.

Offset still works well in some cases:

• The data doesn’t change while people page through it, such as last year’s invoices.
• Inserts only land after the reader’s position. An oldest-first list with new rows at the end doesn’t shift the rows already read. Deletes still cause skips there.
• Users need to jump to a page number, and the list is small enough for that to be cheap.

Measured: what a deep page costs

PostgreSQL’s documentation says it plainly: “The rows skipped by an OFFSET clause still have to be computed inside the server; therefore a large OFFSET might be inefficient.” We measured how inefficient.

The table has 1,000,000 posts and an index on (created_at, id). At each depth we ran EXPLAIN (ANALYZE, BUFFERS) 45 times for the offset query and 45 times for the cursor query that returns the same 20 rows, alternating the two. Move the slider:

Rows to skip	OFFSET: index rows read	OFFSET: median time (10th–90th percentile)	Cursor: index rows read	Cursor: median time
0	20	0.038 ms (0.031–0.052)	20	0.037 ms
1,000	1,020	0.35 ms (0.28–0.48)	20	0.043 ms
10,000	10,020	3.3 ms (2.7–4.2)	20	0.047 ms
100,000	100,020	29 ms (24–33)	20	0.056 ms
500,000	500,020	128 ms (88–155)	20	0.063 ms
999,980	1,000,000	174 ms (150–262)	20	0.036 ms

The offset query reads every skipped row through the index, so its cost grows with the depth: 1,000,000 rows and 35,084 pages for the last 20 posts. The cursor query jumps straight to its position in the index and reads 20 rows at every depth.

The times come from one laptop, running PostgreSQL in Docker, so read them as a shape, not as figures to plan with. Two things favour OFFSET here. The table was written in time order, so its rows sit on disk in the same order as the index. And we gave PostgreSQL 1 GB of buffer cache, so every page was already in memory. A table in random physical order, or one bigger than memory, makes deep offsets slower still. The rows read don’t depend on the machine. And one slow page isn’t the whole cost: a crawler or an export job walking every page pays for all the skipped rows again on each page.

Getting cursor pagination right

• The sort must be unique. ORDER BY created_at alone gives ties, and a cursor between tied rows repeats or skips them. Add a unique tiebreaker: ORDER BY created_at DESC, id DESC. PostgreSQL’s docs warn that without a unique order, LIMIT gives “an unpredictable subset of the query’s rows”.
• The sort key must not change. Sorting by updated_at or a score moves rows across the cursor while the client pages, so they repeat or vanish. Sort by something fixed, such as the creation time and id.
• All sort columns go in the same direction. (created_at, id) < ($1, $2) matches ORDER BY created_at DESC, id DESC. A mixed order, such as created_at DESC, id ASC, needs the longer WHERE created_at < $1 OR (created_at = $1 AND id > $2), and an index to match.
• The sort columns must be NOT NULL. A row comparison such as (created_at, id) < ($1, $2) returns null, not true or false, when it meets a null, so those rows silently drop out.
• You need an index that matches the sort, here (created_at, id). Without one, the database sorts the whole table for every page.
• Make the cursor opaque. Google’s API guidelines say page tokens must not be parseable by users, “because if users are able to deconstruct these, they will do so“. They also warn that base64 alone isn’t enough. Sign or encrypt the token if clients mustn’t build their own.
• A cursor belongs to its query. If the client changes the filter or the sort, reject the old cursor with 400 rather than guessing.
• Paging backwards flips the comparison and the order, WHERE (created_at, id) > ($1, $2) ORDER BY created_at, id, then reverses the rows before returning them.
• Signal the end explicitly. Return no next cursor at the end. A short page isn’t a reliable signal: Google’s guidelines allow an API to return fewer results than requested before the end.
• Cap the page size, and pick a default.
• Totals are expensive. SELECT count(*) on a large filtered table can cost more than the page. Leave totals out, or make them approximate or optional.
• Plan pagination from the start. Adding it to an endpoint that used to return everything breaks existing clients, which only ever read the first page.

What cursors give up: jumping to page 7, and knowing how many pages there are. For a feed, a timeline or a sync job, that’s usually fine. For a table in an admin screen, offset over a filtered, bounded result is often the better fit.

Responses can carry cursors in the body or in a Link header (RFC 8288) with rel="next", as GitHub’s API does. Both work. A body field is easier for most clients to read.

gRPC

gRPC is a remote procedure call framework: you define services and messages in Protocol Buffers, and generate client and server code in many languages. It runs over HTTP/2.

syntax = "proto3";

package orders.v1;

service OrderService {
  rpc GetOrder(GetOrderRequest) returns (Order);
  rpc ListOrders(ListOrdersRequest) returns (ListOrdersResponse);
  rpc WatchOrder(WatchOrderRequest) returns (stream OrderEvent);
}

message GetOrderRequest {
  string order_id = 1;
}

message ListOrdersRequest {
  int32 page_size = 1;
  string page_token = 2;
}

message ListOrdersResponse {
  repeated Order orders = 1;
  string next_page_token = 2;
}

message WatchOrderRequest {
  string order_id = 1;
}

message Order {
  // Field 3 was coupon_code. Its number and name stay reserved forever.
  reserved 3;
  reserved "coupon_code";

  string order_id = 1;
  int64 total_cents = 2;
  optional string note = 4;
}

message OrderEvent {
  string order_id = 1;
  string status = 2;
}

This file compiles with buf build (buf 1.73.0). When we added a new field numbered 3, the compiler refused it: “use of reserved field number 3“.

What you get:

• Four kinds of call: unary (one request, one response), server streaming, client streaming, and bidirectional streaming. Messages within one call arrive in order.
• A compact binary encoding with generated, typed clients.
• HTTP/2 multiplexing, so many calls share one connection (Part 3).

What to design for:

• Set a deadline on every call. gRPC’s docs: “By default, gRPC does not set a deadline which means it is possible for a client to end up waiting for a response effectively forever.” Java and Go pass deadlines on to downstream calls by default. In .NET you turn that on with EnableCallContextPropagation().
• Choose a retry policy deliberately. “Retries are enabled by default, but there is no default retry policy.” Without one, gRPC only retries when it’s certain the server hasn’t processed the call. Retrying a non-idempotent call needs the same idempotency key design as HTTP.
• Use the right status code. gRPC has 17. UNAVAILABLE means retry with backoff. FAILED_PRECONDITION means don’t retry until something is fixed. ABORTED means restart the whole read-modify-write sequence. UNAUTHENTICATED and PERMISSION_DENIED are the 401 and 403 of gRPC. A status is a code and a message. For structured details, gRPC’s docs point to Google’s richer error model, which sends google.rpc.Status messages, such as a list of invalid fields, in the response’s trailing metadata.
• Browsers can’t call gRPC directly. gRPC sends its status in HTTP/2 trailers, on an HTTP 200 response, and browsers don’t expose trailers or HTTP/2 framing to JavaScript. gRPC-Web, translated by a proxy such as Envoy or in-process by ASP.NET Core’s Grpc.AspNetCore.Web middleware, supports unary and server-streaming calls only. ASP.NET Core can also expose a gRPC service as JSON over HTTP with gRPC JSON transcoding.
• Evolve messages by the rules. In the binary format, adding fields is safe, because old code ignores fields it doesn’t know, and removing a field is safe as long as its number is never reused. Protobuf’s docs list the results of reuse as anything from a parse error to “Leaked PII/SPII” and “Data corruption”. The JSON format is stricter: ProtoJSON parsers generally don’t accept unknown fields, so a reader using the old schema fails on a new field, and on a removed one. That matters as soon as anything uses JSON, including JSON transcoding. Reserve both the number and the name of a removed field, as the Order message does.
• Mark scalar fields optional when “not set” differs from zero or an empty string. Without it, proto3 can’t tell them apart. The protobuf docs recommend optional for proto3 scalar fields.

GraphQL

GraphQL lets the client say exactly which fields it wants, and returns that shape. A schema defines the types:

type Query {
  orders(first: Int!): [Order!]!
}

type Order {
  id: ID!
  totalCents: Int!
  customer: Customer!
  shipment: Shipment
}

type Customer {
  id: ID!
  name: String!
}

type Shipment {
  carrier: String!
  trackingNumber: String!
}

A query picks from it: { orders(first: 25) { id totalCents customer { name } } }. One request replaces several REST calls, and a mobile screen gets only the fields it shows.

The latest edition of the specification is from September 2025. It defines the query language and how results look, and says nothing about transport. The GraphQL over HTTP specification, which covers methods and status codes, is still a draft.

We ran the next two behaviours on graphql-js 16.14.2.

The N+1 problem. Each field has a resolver. Resolve customer by loading the customer, and the query above makes 1 database call for the orders plus 1 per order: 26 calls for 25 orders. The standard fix is DataLoader, which, in its README’s words, collects “all individual loads which occur within a single frame of execution” and calls a batch function once. With DataLoader the same query made 2 calls: one for the orders, and one for the five distinct customers.

Create a new DataLoader per request. Its cache would otherwise serve one user’s data to another.

Partial results. When one field fails, GraphQL still returns the rest. We made the shipment service fail for the second order:

{
  "errors": [
    {
      "message": "shipping service timed out",
      "locations": [{ "line": 1, "column": 43 }],
      "path": ["orders", 1, "shipment"]
    }
  ],
  "data": {
    "orders": [
      {
        "id": "o1",
        "customer": { "name": "Customer 1" },
        "shipment": { "carrier": "DHL", "trackingNumber": "JD014600" }
      },
      {
        "id": "o2",
        "customer": { "name": "Customer 2" },
        "shipment": null
      }
    ]
  }
}

That’s useful for a screen that can show everything but the shipment. It worked here because shipment is nullable. Had it been declared Shipment!, the null would have spread up to the whole order, and, since the list is [Order!]!, to all the orders. So keep fields that depend on other services nullable.

It also means an HTTP status alone can’t tell a client whether a GraphQL request fully worked. Clients must check errors. The spec reserves an extensions entry on each error for your own details, which is where a machine-readable error code usually goes.

What to design for:

• Cost and depth limits. A client can ask for orders, their customers, those customers’ orders, and so on. One small query can fan out into millions of rows. Limit depth, count the cost of a query before running it, or accept only queries you’ve registered in advance.
• HTTP caching mostly doesn’t apply. Queries usually go to one URL by POST, so CDNs and browsers can’t cache them. The GraphQL over HTTP draft allows queries over GET, and forbids mutations over GET. graphql-js ran our mutation without asking how it arrived, so that rule is your HTTP layer’s job.
• Authorization goes on fields and objects, not endpoints. There’s one endpoint, so “can this user call this URL?” decides nothing.
• Introspection shows your schema to anyone who can query it. That’s great for tools, and something to switch off or restrict on a public API if the schema itself is sensitive.

Choosing between REST, gRPC and GraphQL

	HTTP API (“REST”)	gRPC	GraphQL
Best fit	Public APIs, browsers, cacheable resources	Service-to-service calls, streaming, many languages	Client-driven reads for varied screens
Contract	OpenAPI (optional)	.proto files (required)	Schema (required)
Encoding	Usually JSON	Protobuf binary	Usually JSON
Browser support	Native	Through gRPC-Web and a proxy	Native
HTTP caching	Yes, for GET	No	Hard
Streaming	SSE or WebSockets (Part 3)	Built in, four kinds	Subscriptions, usually over WebSockets
Main risk	Chatty clients, over-fetching	Browser access, harder to inspect by hand	N+1 queries, expensive queries

Many systems use more than one: gRPC between internal services, and an HTTP or GraphQL API at the edge for browsers and mobile apps. The choice is about who calls the API and what they need, not about which is newest.

Versioning and evolution

Most change should need no new version. Additive changes don’t break well-written clients:

• a new endpoint, or a new optional field in a request;
• a new field in a response, if clients ignore fields they don’t know;
• a new enum value, if clients were written to expect unknown values.

That last condition matters. protobuf’s docs point out that adding an enum value “would be a compilation break for any code with an exhaustive switch on that enum”. Tell clients from the start to ignore unknown fields and handle unknown values.

Breaking changes include removing or renaming a field, changing a type or format, making an optional field required, changing a default, and tightening validation. Google’s API guidelines add one that’s easy to miss: renaming is the same as removing and adding.

When you must break something, there are three common ways to mark the new version:

Where	Example	Trade-off
Path	/v2/orders	Visible, easy to route and cache; a new URL for the same resource
Header	Api-Version: 2	Keeps URLs stable; easy to forget, harder to test in a browser
Media type	Accept: application/vnd.example.order.v2+json	The most HTTP-native; the least familiar to client developers

RFC 9205 advises IETF specifications against fixed paths such as /v1. It also calls a prefix like /app/v1 “common practice” for a single-deployment API, and that’s what most of us build. Pick one approach and use it everywhere.

Tell clients before you remove anything:

• Deprecation (RFC 9745, 2025) says a resource is or will be deprecated, as a structured date such as @1688169599. Deprecation alone “does not change any behavior of the resource.”
• Sunset (RFC 8594) gives the date it may stop responding. It uses the older HTTP-date format.
• Watch who still calls the old version before you switch it off, and contact them.

A small API in C

This ASP.NET Core minimal API has three of the patterns in this post: a conditional update with If-Match, an idempotent create, and cursor pagination. It starts itself, calls its own endpoints with HttpClient, and prints each exchange. It runs as a .NET 10 file-based app with dotnet run api.cs.

The payment store here is a dictionary behind a lock, so it only works in one process. Across servers, that lock becomes the database’s unique key, as in the lab.

// File-based apps default to native AOT, which turns off reflection-based JSON; this sketch keeps it simple.
#:sdk Microsoft.NET.Sdk.Web
#:property PublishAot=false
using System.Net.Http.Json;
using System.Text;
using Microsoft.AspNetCore.Hosting.Server;
using Microsoft.AspNetCore.Hosting.Server.Features;

var builder = WebApplication.CreateBuilder(args);
builder.Logging.ClearProviders();
builder.Services.AddProblemDetails();          // errors as RFC 9457 application/problem+json
builder.WebHost.UseUrls("http://127.0.0.1:0");
var app = builder.Build();

var articles = new Dictionary<int, Article> { [1] = new(1, "Draft", 1) };
var paymentsByKey = new Dictionary<string, (string Hash, Payment Payment)>();
var nextPaymentId = 1;

// --- Conditional update: If-Match stops lost updates ---------------------------------
app.MapGet("/articles/{id:int}", (int id, HttpContext http) =>
{
    lock (articles)
    {
        if (!articles.TryGetValue(id, out var a)) return Results.Problem(statusCode: 404, detail: "No such article");
        http.Response.Headers.ETag = $"\"v{a.Version}\"";
        return Results.Ok(a);
    }
});

app.MapPut("/articles/{id:int}", (int id, ArticleUpdate body, HttpContext http) =>
{
    var ifMatch = http.Request.Headers.IfMatch.ToString();
    if (ifMatch == "") return Results.Problem(statusCode: 428, detail: "Send If-Match with the ETag you last read");
    lock (articles)   // the version check and the write must be one step; in a database, one UPDATE ... WHERE version = ...
    {
        if (!articles.TryGetValue(id, out var a)) return Results.Problem(statusCode: 404, detail: "No such article");
        if (ifMatch != $"\"v{a.Version}\"") return Results.Problem(statusCode: 412, detail: "The article changed since you read it");
        articles[id] = a = a with { Title = body.Title, Version = a.Version + 1 };
        http.Response.Headers.ETag = $"\"v{a.Version}\"";
        return Results.Ok(a);
    }
});

// --- Idempotent create: the same key returns the same result -------------------------
app.MapPost("/payments", (PaymentRequest body, HttpContext http) =>
{
    var key = http.Request.Headers["Idempotency-Key"].ToString();
    if (key == "") return Results.Problem(statusCode: 400, detail: "Idempotency-Key header is required");
    var hash = $"{body.OrderId}|{body.AmountCents}";
    lock (paymentsByKey)   // one process here; across servers this is the database's unique key
    {
        if (paymentsByKey.TryGetValue(key, out var seen))
            return seen.Hash == hash
                ? Results.Created($"/payments/{seen.Payment.Id}", seen.Payment)
                : Results.Problem(statusCode: 422, detail: "Idempotency-Key reused with a different request");
        var payment = new Payment(nextPaymentId++, body.OrderId, body.AmountCents);
        paymentsByKey[key] = (hash, payment);
        return Results.Created($"/payments/{payment.Id}", payment);
    }
});

// --- Cursor pagination: a token the client passes back, not a row number -------------
var posts = Enumerable.Range(1, 7).Select(i => new Post(i, $"post {i}")).ToList();
app.MapGet("/posts", (int? limit, string? cursor) =>
{
    var size = Math.Clamp(limit ?? 3, 1, 100);
    var after = cursor is null ? int.MaxValue : int.Parse(Encoding.UTF8.GetString(Convert.FromBase64String(cursor)));
    var rows = posts.Where(p => p.Id < after).OrderByDescending(p => p.Id).Take(size + 1).ToList();
    var page = rows.Take(size).ToList();   // the extra row only tells us whether another page exists
    var next = rows.Count > size ? Convert.ToBase64String(Encoding.UTF8.GetBytes(page[^1].Id.ToString())) : null;
    return Results.Ok(new PostPage(page, next));
});

await app.StartAsync();
var address = app.Services.GetRequiredService<IServer>().Features.Get<IServerAddressesFeature>()!.Addresses.First();
using var client = new HttpClient { BaseAddress = new Uri(address) };

async Task Show(string label, HttpRequestMessage request)
{
    var response = await client.SendAsync(request);
    var etag = response.Headers.ETag is { } e ? $" ETag {e.Tag}" : "";
    var type = response.Content.Headers.ContentType?.MediaType;
    var body = await response.Content.ReadAsStringAsync();
    var shown = body;
    if (type == "application/problem+json")
    {
        var problem = System.Text.Json.JsonDocument.Parse(body).RootElement;
        shown = $"{problem.GetProperty("title")}: {problem.GetProperty("detail")}";
    }
    Console.WriteLine($"{label,-32} {(int)response.StatusCode}{etag}  {shown}");
}

HttpRequestMessage Put(string ifMatch, string title)
{
    var r = new HttpRequestMessage(HttpMethod.Put, "/articles/1") { Content = JsonContent.Create(new ArticleUpdate(title)) };
    if (ifMatch != "") r.Headers.TryAddWithoutValidation("If-Match", ifMatch);
    return r;
}

HttpRequestMessage Pay(string key, int amount)
{
    var r = new HttpRequestMessage(HttpMethod.Post, "/payments") { Content = JsonContent.Create(new PaymentRequest("order-42", amount)) };
    r.Headers.Add("Idempotency-Key", key);
    return r;
}

await Show("GET article", new(HttpMethod.Get, "/articles/1"));
await Show("PUT with If-Match \"v1\"", Put("\"v1\"", "Alice's title"));
await Show("PUT with stale If-Match \"v1\"", Put("\"v1\"", "Bob's title"));
await Show("PUT without If-Match", Put("", "Bob's title"));
await Show("POST payment, key k-7f3a", Pay("k-7f3a", 4999));
await Show("POST retry, same key and body", Pay("k-7f3a", 4999));
await Show("POST same key, different amount", Pay("k-7f3a", 5999));
Console.WriteLine($"payments created: {nextPaymentId - 1}");

var pageOne = await client.GetFromJsonAsync<PostPage>("/posts?limit=3");
var pageTwo = await client.GetFromJsonAsync<PostPage>($"/posts?limit=3&cursor={Uri.EscapeDataString(pageOne!.Next!)}");
Console.WriteLine($"page 1: {string.Join(", ", pageOne.Items.Select(p => p.Id))}  next={pageOne.Next}");
var pageThree = await client.GetFromJsonAsync<PostPage>($"/posts?limit=3&cursor={Uri.EscapeDataString(pageTwo!.Next!)}");
Console.WriteLine($"page 2: {string.Join(", ", pageTwo.Items.Select(p => p.Id))}  next={pageTwo.Next}");
Console.WriteLine($"page 3: {string.Join(", ", pageThree!.Items.Select(p => p.Id))}  next={pageThree.Next ?? "(none: the end)"}");
await app.StopAsync();

record Article(int Id, string Title, int Version);
record ArticleUpdate(string Title);
record PaymentRequest(string OrderId, int AmountCents);
record Payment(int Id, string OrderId, int AmountCents);
record Post(int Id, string Title);
record PostPage(List<Post> Items, string? Next);

It prints:

GET article                      200 ETag "v1"  {"id":1,"title":"Draft","version":1}
PUT with If-Match "v1"           200 ETag "v2"  {"id":1,"title":"Alice's title","version":2}
PUT with stale If-Match "v1"     412  Precondition Failed: The article changed since you read it
PUT without If-Match             428  Precondition Required: Send If-Match with the ETag you last read
POST payment, key k-7f3a         201  {"id":1,"orderId":"order-42","amountCents":4999}
POST retry, same key and body    201  {"id":1,"orderId":"order-42","amountCents":4999}
POST same key, different amount  422  Unprocessable Entity: Idempotency-Key reused with a different request
payments created: 1
page 1: 7, 6, 5  next=NQ==
page 2: 4, 3, 2  next=Mg==
page 3: 1  next=(none: the end)

Some things the run shows:

• A stale ETag got 412, and a missing one got 428. Neither write was applied.
• The retry got the same 201 and the same payment id. Only one payment was created.
• Reusing the key with a different amount got 422. ASP.NET Core 10 still gave 422 its old WebDAV title, “Unprocessable Entity”, and set type to RFC 4918, where RFC 9110 now calls it “Unprocessable Content”.
• The last page returned no cursor. Asking for one row more than the page size tells the server whether another page exists.

This sketch cuts corners a real API shouldn’t:

• If-Match is compared as one exact string, while the header may carry a list of ETags or *.
• The idempotency keys aren’t scoped to a client, and the key is sent bare, where the draft’s example quotes it.
• A malformed cursor causes a 500 instead of a 400.
• The cursor is base64 of an id, which clients can read and forge.

One thing it does get right: each version check and its write happen under one lock. We tested it: 16 PUTs with the same If-Match sent at once, 200 times, in three runs. Without the lock, more than one PUT got 200 in 17, 22 and 24 of the 200 rounds. With the lock, none did.

The same building blocks elsewhere:

	Problem Details	Routing	gRPC	GraphQL
C#/.NET	AddProblemDetails, TypedResults.Problem	Minimal APIs	Grpc.AspNetCore, gRPC-Web, JSON transcoding	Community libraries
Java	Spring’s ProblemDetail (Spring Framework 6+); for Spring MVC’s own exceptions, Spring Boot needs spring.mvc.problemdetails.enabled=true	Spring MVC	grpc-java	Spring for GraphQL
Go	Write the JSON yourself; it’s five fields	net/http patterns such as GET /posts/{id} (Go 1.22+)	grpc-go	Community libraries
Rust	Write the JSON yourself	axum 0.8	tonic 0.14	async-graphql 7

Two defaults to check in .NET:

• ASP.NET Core’s rate limiting middleware rejects with 503 unless you change it. The default is in its source: RejectionStatusCode { get; set; } = StatusCodes.Status503ServiceUnavailable. Set it to 429, so clients know they were limited rather than that the service is down.
• OpenAPI: .NET 10’s built-in document generation writes OpenAPI 3.1 by default. The current OpenAPI Specification is 3.2.1, and Microsoft’s docs say .NET 11 will default to 3.2.

Trade-offs

Offset versus cursor. Offset allows page jumps and totals, and repeats or skips rows when data changes, with a cost that grows with depth. Cursors are stable and cheap at any depth, and only go forward and back.

Strict versus lenient contracts. Rejecting unknown fields catches client bugs early, and makes every addition a breaking change. Ignoring them lets the contract grow, and hides typos.

Idempotency keys cost storage and design. Every key is a row with an expiry, and every handler needs the in-progress and crash cases handled. For operations that are naturally idempotent, such as “set the status to shipped”, a PUT with a condition is simpler.

One flexible endpoint versus many specific ones. GraphQL moves query design to the client and the cost control to the server. Specific endpoints are easier to cache, limit and reason about, and need more of them.

Binary versus text. gRPC’s protobuf is compact and typed, and you can’t read it with curl. JSON is readable everywhere, and bigger and looser.

Common mistakes

Retrying POST without an idempotency key. Timeouts are ambiguous, and every retry layer makes duplicates more likely. Add keys to anything that creates or charges.

Checking for a duplicate, then inserting. The lab showed two charges under PostgreSQL’s default isolation, with no error. Let a unique key decide.

Offset pagination on a changing feed. Readers see repeats and miss items, and deep pages get slower with every row skipped.

A cursor sorted on a non-unique column. Rows that share a timestamp get repeated or skipped at page boundaries. Add the id as a tiebreaker.

Returning 200 with {"error": ...}. Caches, retries and monitoring all see success. Use the status code, and put details in a Problem Details body.

Read, change, write back, without If-Match. Concurrent edits silently overwrite each other.

No deadline on gRPC calls. The default is to wait forever.

Sharing a DataLoader across requests. One user’s cached data can appear in another user’s response.

Reusing a protobuf field number. Old data decodes into the wrong field. Reserve removed numbers and names.

Breaking changes without a signal. Announce with Deprecation and Sunset, and check traffic before removing anything.

Interview questions

Try to answer each one before opening the model answer.

1. What’s the difference between safe and idempotent methods? Is POST ever idempotent?

Show a strong answer

• Safe (RFC 9110): the client doesn’t request a state change. GET, HEAD, OPTIONS, TRACE. Logging still happens, and that’s allowed.
• Idempotent: repeating the request has the same intended effect as sending it once. PUT, DELETE and the safe methods. The response may differ, such as 204 then 404 for DELETE.
• Why it matters: clients and proxies may retry idempotent requests automatically after a connection failure. A proxy must not automatically retry POST.
• POST isn’t idempotent by method, but an operation can be made idempotent with an idempotency key, where the server stores the result per key and replays it.
• PATCH is not idempotent by default, and can be written to be, for example by setting fields to values.

Likely follow-up: “Is PUT /counter with body {"increment": 1} idempotent?” Not in effect, so it’s misusing PUT. PUT’s body should be the new state, such as {"value": 5}. Use POST for an increment, or PATCH with a condition.

2. Design the retry behaviour for a “create payment” endpoint.

Show a strong answer

• Client: generate an idempotency key per payment attempt (a UUID), and send the same key on every retry. Retry with backoff on timeouts, 429, 503, and the 409 your API documents for “still in progress”, with its own problem type so clients can tell it from other conflicts.
• Server: store keys scoped to the client, with a unique constraint, a hash of the request, a status and the saved response. 1. Claim the key with INSERT ... ON CONFLICT DO NOTHING and commit it as in_progress. 2. Call the payment provider with your own idempotency key, stored with the claim, so every retry and recovery sends the same one. 3. Save the result and mark the key done.
• Repeats:
• finished: return the saved response, including errors from work that started; after a saved 5xx, the client checks the payment’s status rather than paying again with a new key;
• still in progress: 409;
• same key with a different body: 422.
• Crashes: a lease on in_progress, and a recovery job that asks the provider what happened.
• Expiry: document how long keys last, such as 24 hours.
• Why not “check, then insert”: two concurrent retries both see no record. We saw two charges in PostgreSQL under READ COMMITTED.

Likely follow-up: “What if the key store and the payments table are in different databases?” Then you can’t make the claim and the charge atomic. Use the provider’s idempotency, a transactional outbox (Part 28), or a reconciliation job that compares your records with the provider’s.

3. Offset or cursor pagination for an infinite-scrolling feed? Why?

Show a strong answer

• Cursor. New posts arrive at the top all the time. With offset, every insert shifts positions, so the next page repeats items the reader has seen. Deletes make it skip items. We measured 5 repeats after 5 inserts, and 3 skipped posts after 3 deletes.
• Cost: offset reads and discards every skipped row. In our test, the page at depth 999,980 read 1,000,000 index rows and had a median of 174 ms, while the cursor query read 20 rows in well under a millisecond.
• How: sort by a unique key, such as (created_at, id), with NOT NULL columns and a matching index. Query with WHERE (created_at, id) < (last_created_at, last_id). Return an opaque cursor, and no cursor at the end.
• What you give up: jumping to page N and total counts, which a feed doesn’t need.

Likely follow-up: “The feed is ranked, not sorted by time. Now what?” A cursor needs a stable order. Either snapshot the ranked list for the session and page through the snapshot by position, or rank within time windows and page by the window plus a tiebreaker.

4. How do you stop two users overwriting each other’s edits?

Show a strong answer

• Optimistic concurrency with conditional requests. Return an ETag (a version number or content hash) on GET. Require If-Match on PUT and PATCH. If the ETag doesn’t match the current version, respond 412 and don’t apply the write.
• Require the condition: respond 428 to writes without If-Match.
• Make check and write atomic: UPDATE ... SET ..., version = version + 1 WHERE id = $1 AND version = $2, and check the affected row count.
• Creates: If-None-Match: * so a PUT fails if the resource already exists.
• What the client does on 412: re-read, show or merge the changes, and try again.
• Alternative: pessimistic locking (check out, edit, check in) when conflicts are frequent and merges are costly, at the price of stale locks.

Likely follow-up: “Why not a last-modified timestamp?” If-Unmodified-Since has one-second resolution, so two changes in the same second look identical. If-Match uses strong comparison on an ETag, which changes with every change.

5. When would you choose gRPC over a JSON HTTP API, and what problems come with it?

Show a strong answer

• Choose gRPC for internal service-to-service calls with many languages, a strict contract with generated clients, streaming, and high call volumes where protobuf’s size and HTTP/2 multiplexing help.
• Problems:
• Browsers can’t call it directly (status in HTTP/2 trailers), so you need gRPC-Web through a proxy, which has no client or bidirectional streaming, or JSON transcoding.
• It’s harder to debug by hand than JSON.
• HTTP caches don’t help.
• HTTP/2 connections are long-lived, so a layer 4 load balancer pins them to one backend. You need layer 7 or client-side balancing (Part 30).
• Design rules:
• Always set deadlines, because there’s no default.
• Configure retry policies deliberately.
• Use status codes correctly: UNAVAILABLE is retryable, FAILED_PRECONDITION is not.
• Never reuse field numbers, and reserve removed ones with their names.

Likely follow-up: “How do you evolve a proto without breaking clients?” Add fields with new numbers. Never renumber, retype or reuse. Reserve deleted numbers and names. Remember that removal breaks the JSON encoding even though binary tolerates it. And don’t assume adding an enum value is harmless to clients with exhaustive switches.

6. What is the N+1 problem in GraphQL, and how do you fix it?

Show a strong answer

• Cause: resolvers run per field, per object. A list of N orders whose customer resolver loads one customer each makes 1 + N queries.
• Fix: batching with DataLoader. Loads requested in the same tick are collected into one batch call, and repeated keys are de-duplicated. On graphql-js, 25 orders went from 26 database calls to 2.
• Scope: one DataLoader per request, because its cache must not cross users.
• Related protections: query depth and cost limits, pagination on list fields, and timeouts on resolvers.

Likely follow-up: “Can the database do the join instead?” Yes. Resolvers can look ahead at the requested fields and build one joined query. It’s faster for predictable shapes, and more complex to keep correct as the schema grows. Batching is the general-purpose fix.

7. How do you version an API, and what counts as a breaking change?

Show a strong answer

• Avoid versions for additive changes: new endpoints, new optional request fields, new response fields. That requires clients that ignore unknown fields and tolerate unknown enum values, which you ask of them from day one.
• Breaking changes: removing or renaming fields, changing types or formats, making something required, changing defaults or meaning, tightening validation, changing error codes clients depend on.
• Marking a new version: in the path (/v2), a header, or a media type. Pick one approach for the whole API. Paths are the most common for single-deployment APIs.
• Retiring: announce with the Deprecation (RFC 9745) and Sunset (RFC 8594) headers and documentation, measure who still calls the old version, and contact them before switching it off.
• gRPC: a new package such as orders.v2 for breaking changes, and field rules for everything else.

Likely follow-up: “How long do you support the old version?” Long enough for the slowest important client to move. For mobile apps that’s the life of old app versions still in use, which you measure from traffic, and then state as a date in Sunset.

8. What should an error response contain?

Show a strong answer

• The right HTTP status, so generic software behaves correctly: retries on 503, no retry on 400, re-authentication on 401.
• A machine-readable body, ideally RFC 9457 Problem Details: a type URI clients can switch on, a stable title, status, a human detail, and extension members such as invalid fields or a trace id.
• Retry guidance where relevant: Retry-After on 429 and 503.
• Required headers: WWW-Authenticate on 401 and Allow on 405.
• Nothing internal: no stack traces or SQL. RFC 9457 says problem details “are not a debugging tool”. Log the details, and return a trace id to correlate.
• Consistency: the same shape from every endpoint, including errors produced by middleware and gateways.

Likely follow-up: “Should validation errors be 400 or 422?” Both appear in practice. 422 means the syntax was fine and the content couldn’t be processed, which fits validation failures. The important thing is to pick one and document it, and to list every invalid field in the body.

Sources

• Lab: system-design/checks/part04_lab.py (PostgreSQL 18.6 in Docker, pinned by digest: pagination, deep offsets, idempotency), system-design/checks/part04_graphql/ (graphql-js 16.14.2, DataLoader 2.2.3), and the C# program above (.NET 10.0.302)
• RFC 9110: HTTP Semantics: safe and idempotent methods (9.2), methods (9.3), validators (8.8), conditional requests (13), status codes (15)
• RFC 6585: Additional HTTP Status Codes (428, 429)
• RFC 5789: PATCH, RFC 7396: JSON Merge Patch, RFC 6902: JSON Patch
• RFC 9457: Problem Details for HTTP APIs
• RFC 9205: Building Protocols with HTTP
• RFC 8288: Web Linking, RFC 9745: Deprecation, RFC 8594: Sunset
• draft-ietf-httpapi-idempotency-key-header-07 (an expired Internet-Draft, not an RFC)
• Stripe: Idempotent requests (vendor documentation)
• PostgreSQL 18: LIMIT and OFFSET and Row Constructor Comparison
• Google AIP-158: Pagination and AIP-180: Backwards compatibility (vendor guidance)
• Roy Fielding, Architectural Styles and the Design of Network-based Software Architectures, chapter 5 (2000), and REST APIs must be hypertext-driven (2008)
• gRPC: core concepts, deadlines, retry, status codes, gRPC over HTTP/2
• Protocol Buffers: proto3 language guide, field presence, JSON mapping
• GraphQL specification, September 2025, GraphQL over HTTP (draft), DataLoader
• Microsoft Learn: problem details in ASP.NET Core, gRPC-Web, gRPC deadlines, OpenAPI document generation; ASP.NET Core source, RateLimiterOptions.cs
• Spring Framework: error responses, Go 1.22 routing enhancements, OpenAPI Specification

What to remember

• Safe and idempotent are promises other software acts on: retries, caches and proxies. GET must not change data, and POST must not be retried blindly.
• Use status codes for their meaning, and put the specifics in an RFC 9457 Problem Details body.
• ETags with If-Match turn lost updates into 412 responses. Make the version check and the write one atomic statement.
• Make non-idempotent operations safe to retry with idempotency keys, and let a unique key in the database decide who does the work. Checking first isn’t enough.
• Offset pagination repeats and skips rows when data changes, and gets slower with depth. Cursor pagination on a unique, indexed, NOT NULL sort key does neither.
• HTTP APIs suit browsers and caching, gRPC suits internal calls and streaming, and GraphQL suits client-shaped reads, as long as you control N+1 and query cost.
• Most change should be additive. When something must break, version it once, and announce the removal before you do it.

An API is a promise about what happens when things go wrong: a retry, a race, a changed list or a new client version. Design those cases first.