Blog

Testing and Shipping a Go REST API

Written by

Test a Go REST API with httptest, a fake store, log assertions, the race detector and fuzzing. Then ship it with server timeouts, graceful shutdown and one static binary.

An API that passes a few manual curl requests isn’t finished. You need tests that check every status code and every error path, and a server that copes with slow clients, stops without dropping requests, and ships as something you can copy to a machine and run.

This post does both for the task-list API built in the part on JSON REST APIs. It adds real test files to that module, then covers timeouts, graceful shutdown and building one binary. Every program below was run on Go 1.26, and its output is pasted from the run.

The API under test

The task API is a Go module with a task package for storage, an api package for HTTP, and a tasksd command that runs the server. Its five routes answer with status codes from 200 to 500, and every error is JSON in one shape. This post adds four test files to the module:

19-tasks-api/
├── go.mod
├── cmd/
│   └── tasksd/
│       ├── main.go
│       └── main_test.go
└── internal/
    ├── task/
    │   ├── task.go
    │   └── memstore.go
    └── api/
        ├── api.go
        ├── api_test.go
        ├── handlers.go
        ├── json.go
        ├── json_test.go
        ├── validate.go
        ├── middleware.go
        └── middleware_test.go

Each test file uses the same package name as the code next to it, so the tests can reach unexported names such as decodeJSON. The part on packages and testing covered go test, table-driven tests and t.Helper, so this post uses them without explaining them again.

httptest.NewRecorder or httptest.NewServer

The net/http/httptest package gives you two ways to test a handler, and they test different things. httptest.NewRecorder returns a ResponseWriter that stores the status, headers and body. You call ServeHTTP yourself, with no network. httptest.NewServer starts a real server on a free loopback port, and you talk to it with a real client.

Most of the API’s tests use the recorder, through two small helpers:

// newTestAPI returns the API over a store that already holds one task,
// "Buy milk", with ID 1. Its logs are thrown away.
func newTestAPI(t *testing.T) http.Handler {
	t.Helper()
	store := task.NewMemStore()
	if _, err := store.Create(t.Context(), task.Task{Title: "Buy milk"}); err != nil {
		t.Fatalf("seeding the store: %v", err)
	}
	return New(store, slog.New(slog.DiscardHandler))
}
// do sends one request straight to h, with no network, and returns
// the recorded response. A non-empty body is sent as JSON.
func do(h http.Handler, method, path, body string) *httptest.ResponseRecorder {
	req := httptest.NewRequest(method, path, strings.NewReader(body))
	if body != "" {
		req.Header.Set("Content-Type", "application/json")
	}
	rec := httptest.NewRecorder()
	h.ServeHTTP(rec, req)
	return rec
}

newTestAPI builds the whole API, middleware included, over a store that already holds one task. slog.DiscardHandler throws the logs away. t.Context() returns a context that’s cancelled just before the test’s cleanup functions run, so a store call in a test gets it, the way a handler’s store call gets r.Context(). Both arrived in Go 1.24.

Use the recorder to test what a handler decides: status, headers, body. It’s fast, and a failure points straight at your code. Use a real server when the network is part of the test: a client, connection handling, timeouts or shutdown.

A table over every endpoint

A table-driven test is a good fit for an API, because each row is one request and the response you expect. This one covers every route, including the errors:

func TestEndpoints(t *testing.T) {
	tests := []struct {
		name       string
		method     string
		path       string
		body       string
		wantStatus int
		wantHeader map[string]string
		wantBody   string
	}{
		{"list", "GET", "/tasks", "", 200, nil,
			`[{"id":1,"title":"Buy milk","done":false}]`},
		{"get", "GET", "/tasks/1", "", 200, nil,
			`{"id":1,"title":"Buy milk","done":false}`},
		{"get missing", "GET", "/tasks/99", "", 404, nil,
			`{"error":"task not found"}`},
		{"get bad id", "GET", "/tasks/abc", "", 404, nil,
			`{"error":"task not found"}`},
		{"create", "POST", "/tasks", `{"title":"Walk the dog"}`, 201,
			map[string]string{"Location": "/tasks/2"},
			`{"id":2,"title":"Walk the dog","done":false}`},
		{"create blank title", "POST", "/tasks", `{"title":"  "}`, 422, nil,
			`{"error":"validation failed","fields":{"title":"must not be empty"}}`},
		{"create unknown field", "POST", "/tasks", `{"title":"a","id":7}`, 400, nil,
			`{"error":"unknown field \"id\""}`},
		{"create too big", "POST", "/tasks", `{"title":"` + strings.Repeat("a", maxBodyBytes) + `"}`, 413, nil,
			`{"error":"request body must not be larger than 1048576 bytes"}`},
		{"update", "PUT", "/tasks/1", `{"title":"Buy milk","done":true}`, 200, nil,
			`{"id":1,"title":"Buy milk","done":true}`},
		{"update missing", "PUT", "/tasks/99", `{"title":"x"}`, 404, nil,
			`{"error":"task not found"}`},
		{"delete", "DELETE", "/tasks/1", "", 204, nil, ``},
		{"delete missing", "DELETE", "/tasks/99", "", 404, nil,
			`{"error":"task not found"}`},
		{"wrong method", "PATCH", "/tasks/1", "", 405,
			map[string]string{"Allow": "DELETE, GET, HEAD, PUT"},
			`{"error":"method not allowed"}`},
		{"unknown path", "GET", "/users", "", 404, nil,
			`{"error":"not found"}`},
	}
	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			rec := do(newTestAPI(t), tt.method, tt.path, tt.body)

			if rec.Code != tt.wantStatus {
				t.Errorf("status = %d, want %d", rec.Code, tt.wantStatus)
			}
			for name, want := range tt.wantHeader {
				if got := rec.Header().Get(name); got != want {
					t.Errorf("%s = %q, want %q", name, got, want)
				}
			}
			if got := strings.TrimSuffix(rec.Body.String(), "\n"); got != tt.wantBody {
				t.Errorf("body = %s, want %s", got, tt.wantBody)
			}
			if rec.Code != http.StatusNoContent {
				if ct := rec.Header().Get("Content-Type"); ct != "application/json" {
					t.Errorf("Content-Type = %q, want application/json", ct)
				}
			}
		})
	}
}

Each row gets a fresh API, so the delete row can’t break the get row. Comparing the whole body checks the error shape too: fields appears only on the 422, and the 405 is JSON with an Allow header.

With -v, each row shows up as a named subtest. I’ve removed the durations from the --- PASS lines, because they change on every run:

$ go test -v -run TestEndpoints ./internal/api
=== RUN   TestEndpoints
=== RUN   TestEndpoints/list
=== RUN   TestEndpoints/get
=== RUN   TestEndpoints/get_missing
=== RUN   TestEndpoints/get_bad_id
=== RUN   TestEndpoints/create
=== RUN   TestEndpoints/create_blank_title
=== RUN   TestEndpoints/create_unknown_field
=== RUN   TestEndpoints/create_too_big
=== RUN   TestEndpoints/update
=== RUN   TestEndpoints/update_missing
=== RUN   TestEndpoints/delete
=== RUN   TestEndpoints/delete_missing
=== RUN   TestEndpoints/wrong_method
=== RUN   TestEndpoints/unknown_path
--- PASS: TestEndpoints
    --- PASS: TestEndpoints/list
    --- PASS: TestEndpoints/get
    --- PASS: TestEndpoints/get_missing
    --- PASS: TestEndpoints/get_bad_id
    --- PASS: TestEndpoints/create
    --- PASS: TestEndpoints/create_blank_title
    --- PASS: TestEndpoints/create_unknown_field
    --- PASS: TestEndpoints/create_too_big
    --- PASS: TestEndpoints/update
    --- PASS: TestEndpoints/update_missing
    --- PASS: TestEndpoints/delete
    --- PASS: TestEndpoints/delete_missing
    --- PASS: TestEndpoints/wrong_method
    --- PASS: TestEndpoints/unknown_path
PASS

A final ok line follows, with a duration that varies. The 415 case is in a short test of its own, because it needs a different Content-Type.

A real server, with t.Cleanup and t.Context

One test in the API package runs over a real connection, to check that a client can follow the Location header from a create. Two helpers set it up:

// startServer runs h on a real loopback port until the test ends.
func startServer(t *testing.T, h http.Handler) *httptest.Server {
	t.Helper()
	srv := httptest.NewServer(h)
	t.Cleanup(srv.Close)
	return srv
}
// send makes a real HTTP request and returns the response with its body read.
func send(t *testing.T, method, url, body string) (*http.Response, string) {
	t.Helper()
	req, err := http.NewRequestWithContext(t.Context(), method, url, strings.NewReader(body))
	if err != nil {
		t.Fatalf("building request: %v", err)
	}
	if body != "" {
		req.Header.Set("Content-Type", "application/json")
	}
	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		t.Fatalf("%s %s: %v", method, url, err)
	}
	defer resp.Body.Close()
	b, err := io.ReadAll(resp.Body)
	if err != nil {
		t.Fatalf("reading body: %v", err)
	}
	return resp, strings.TrimSuffix(string(b), "\n")
}

startServer can’t use defer srv.Close(), because that would close the server as soon as the helper returns. t.Cleanup(srv.Close) runs it when the test finishes instead. send builds each request with t.Context(), so a request still waiting when the test ends is cancelled.

func TestCreateThenFollowLocation(t *testing.T) {
	srv := startServer(t, New(task.NewMemStore(), slog.New(slog.DiscardHandler)))

	if _, body := send(t, "GET", srv.URL+"/tasks", ""); body != "[]" {
		t.Fatalf("empty list = %s, want []", body)
	}

	resp, _ := send(t, "POST", srv.URL+"/tasks", `{"title":"Buy milk"}`)
	loc := resp.Header.Get("Location")
	if resp.StatusCode != http.StatusCreated || loc == "" {
		t.Fatalf("create: status %d, Location %q", resp.StatusCode, loc)
	}

	resp, body := send(t, "GET", srv.URL+loc, "")
	if resp.StatusCode != http.StatusOK || body != `{"id":1,"title":"Buy milk","done":false}` {
		t.Errorf("GET %s = %d %s", loc, resp.StatusCode, body)
	}
}

The first request checks that an empty list is [], not null. The rest reads the new task’s URL from the response, the way a client would.

A fake store for the 500 path

The 500 path is the hardest one to test, because MemStore never fails. Store is an interface, though, so a test can pass New anything with the five methods. This fake fails in whatever way the test chooses:

// fakeStore is a Store whose every method calls fail. The test decides
// what fail does: return an error, or panic.
type fakeStore struct {
	fail func() error
}

func (f fakeStore) List(context.Context) ([]task.Task, error) { return nil, f.fail() }

func (f fakeStore) Get(context.Context, int64) (task.Task, error) { return task.Task{}, f.fail() }

func (f fakeStore) Create(context.Context, task.Task) (task.Task, error) {
	return task.Task{}, f.fail()
}

func (f fakeStore) Update(context.Context, task.Task) (task.Task, error) {
	return task.Task{}, f.fail()
}

func (f fakeStore) Delete(context.Context, int64) error { return f.fail() }

One test makes fail return an error, and another makes it panic. The handlers can’t tell this from a real store, which is the point of putting storage behind an interface.

A 500 has two halves that matter. The client must learn nothing about the cause, and the log must record all of it. To check the log, the test gives the API a logger that writes into a bytes.Buffer:

// newTestLogger returns a logger that writes JSON lines into buf. It drops
// the attributes that change on every run: time, duration and stack.
func newTestLogger(buf *bytes.Buffer) *slog.Logger {
	opts := &slog.HandlerOptions{
		ReplaceAttr: func(groups []string, a slog.Attr) slog.Attr {
			switch a.Key {
			case slog.TimeKey, "duration", "stack":
				return slog.Attr{}
			}
			return a
		},
	}
	return slog.New(slog.NewJSONHandler(buf, opts))
}

The time, the request duration and a panic’s stack trace change on every run, so ReplaceAttr drops them, and what’s left can be compared exactly. Here’s the test, with its helper:

// assertLog fails the test unless buf holds exactly the want lines.
func assertLog(t *testing.T, buf *bytes.Buffer, want ...string) {
	t.Helper()
	got := strings.Split(strings.TrimSuffix(buf.String(), "\n"), "\n")
	if len(got) != len(want) {
		t.Fatalf("got %d log lines, want %d:\n%s", len(got), len(want), buf)
	}
	for i := range want {
		if got[i] != want[i] {
			t.Errorf("log line %d:\n got %s\nwant %s", i+1, got[i], want[i])
		}
	}
}
func TestStoreErrorIs500(t *testing.T) {
	var logs bytes.Buffer
	store := fakeStore{fail: func() error {
		return errors.New("dial tcp db.internal:5432: connection refused")
	}}
	h := New(store, newTestLogger(&logs))

	rec := do(h, "GET", "/tasks/1", "")

	if rec.Code != http.StatusInternalServerError {
		t.Errorf("status = %d, want 500", rec.Code)
	}
	if body := rec.Body.String(); body != `{"error":"internal server error"}`+"\n" {
		t.Errorf("body = %q leaks more than it should", body)
	}
	assertLog(t, &logs,
		`{"level":"ERROR","msg":"internal error","err":"dial tcp db.internal:5432: connection refused","request_id":"req-1"}`,
		`{"level":"INFO","msg":"request","method":"GET","path":"/tasks/1","status":500,"request_id":"req-1"}`,
	)
}

The error names a database host and port, which an attacker would like to see. The body is only {"error":"internal server error"}. The log has the real error, then the request line with status 500, and both carry req-1, which leads from a user’s bug report to the cause.

Testing panic recovery and request IDs

Panic recovery gets tested with fakeStore too, this time with a fail that panics:

func TestPanicIs500(t *testing.T) {
	var logs bytes.Buffer
	store := fakeStore{fail: func() error { panic("store exploded") }}
	h := New(store, newTestLogger(&logs))

	rec := do(h, "DELETE", "/tasks/1", "")

	if rec.Code != http.StatusInternalServerError {
		t.Errorf("status = %d, want 500", rec.Code)
	}
	if body := rec.Body.String(); strings.Contains(body, "exploded") {
		t.Errorf("body = %q leaks the panic value", body)
	}
	if id := rec.Header().Get("X-Request-Id"); id != "req-1" {
		t.Errorf("X-Request-Id = %q, want req-1", id)
	}
	assertLog(t, &logs,
		`{"level":"ERROR","msg":"panic","value":"store exploded","request_id":"req-1"}`,
		`{"level":"INFO","msg":"request","method":"DELETE","path":"/tasks/1","status":500,"request_id":"req-1"}`,
	)
}

The panic starts inside the store and passes through the real middleware chain from New. If someone reorders New so that logRequests sits inside recoverPanics, the request line disappears from the log and this test fails.

A panic after the status has been sent is a different story:

func TestPanicAfterWriteHeader(t *testing.T) {
	var logs bytes.Buffer
	s := &server{logger: newTestLogger(&logs)}
	h := s.recoverPanics(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
		fmt.Fprint(w, "[")
		panic("encoder broke halfway")
	}))

	rec := do(h, "GET", "/tasks", "")

	// The 200 is already on its way, so the recovery can't turn it into a 500.
	if rec.Code != http.StatusOK {
		t.Errorf("status = %d, want 200", rec.Code)
	}
	if want := `[{"error":"internal server error"}` + "\n"; rec.Body.String() != want {
		t.Errorf("body = %q, want %q", rec.Body.String(), want)
	}
}

This test calls recoverPanics directly, with a handler that sends a 200 and one byte, then panics. The recovery still calls writeError, but WriteHeader(500) comes too late, so the status stays 200. The error JSON is added to what was already sent, and the client gets [{"error":"internal server error"}, which isn’t valid JSON. That’s why writeJSON marshals the whole body before it writes a status. The test pins the behaviour down, so nobody believes the recovery covers it.

TestRequestIDs in the same file sends three requests and checks for req-1, req-2 and req-3, then checks that a second New starts again at req-1.

The race detector on concurrent requests

The race detector watches memory while tests run and reports two goroutines touching the same variable without a lock. It only sees code that actually runs at the same time, so the test has to make that happen:

func TestConcurrentCreates(t *testing.T) {
	h := New(task.NewMemStore(), slog.New(slog.DiscardHandler))

	const n = 50
	var wg sync.WaitGroup
	for range n {
		wg.Go(func() {
			rec := do(h, "POST", "/tasks", `{"title":"Buy milk"}`)
			if rec.Code != http.StatusCreated {
				t.Errorf("status = %d, want 201", rec.Code)
			}
		})
	}
	wg.Wait()

	var tasks []task.Task
	rec := do(h, "GET", "/tasks", "")
	if err := json.NewDecoder(rec.Body).Decode(&tasks); err != nil {
		t.Fatalf("decoding list: %v", err)
	}
	if len(tasks) != n {
		t.Fatalf("got %d tasks, want %d", len(tasks), n)
	}
	for i, tk := range tasks {
		if tk.ID != int64(i+1) {
			t.Errorf("tasks[%d].ID = %d, want %d", i, tk.ID, i+1)
		}
	}
}

Fifty goroutines send POST /tasks to one handler at once, using sync.WaitGroup.Go from Go 1.25. Then the test checks for exactly fifty tasks with IDs 1 to 50. t.Errorf is safe to call from those goroutines, but t.Fatalf must run on the test’s own goroutine.

To see the test do its job, I deleted the two lines that lock the mutex in MemStore.Create and ran it with -race. Here are the lines of the report that stay the same between runs, with the duration removed from the --- FAIL line:

$ go test -race -run TestConcurrentCreates ./internal/api
==================
WARNING: DATA RACE
...
  example.com/tasks/internal/task.(*MemStore).Create()
...
--- FAIL: TestConcurrentCreates
    testing.go:1712: race detected during execution of test
FAIL

The race detector flagged it in 20 runs out of 20. Without -race, 19 runs crashed with fatal error: concurrent map writes and one passed. That one pass is the reason to run go test -race ./... before every release: a race that doesn’t crash today is still a bug.

Fuzzing the decoder

A fuzz test hands a function thousands of generated inputs and checks a property that must hold for all of them. decodeJSON reads bytes from strangers, so it’s the natural target:

// FuzzDecodeJSON feeds decodeJSON random bodies. Whatever arrives, it must
// not panic, and every error must be a *requestError with a 4xx status.
func FuzzDecodeJSON(f *testing.F) {
	f.Add(`{"title":"Buy milk","done":true}`)
	f.Add(`{"title":`)
	f.Add(`["Buy milk"]`)
	f.Add(`{"title":"a"}{"title":"b"}`)

	f.Fuzz(func(t *testing.T, body string) {
		req := httptest.NewRequest("POST", "/tasks", strings.NewReader(body))
		req.Header.Set("Content-Type", "application/json")

		var in taskInput
		err := decodeJSON(httptest.NewRecorder(), req, &in)
		if err == nil {
			return
		}
		var re *requestError
		if !errors.As(err, &re) {
			t.Fatalf("decodeJSON(%q) returned %T, want *requestError", body, err)
		}
		if re.status != http.StatusBadRequest && re.status != http.StatusRequestEntityTooLarge {
			t.Fatalf("decodeJSON(%q) status = %d, want 400 or 413", body, re.status)
		}
	})
}

Whatever the body, decodeJSON must not panic, and any error must be a *requestError with a 400 or a 413. The f.Add calls are seeds for the fuzzer to mutate. Plain go test runs only the seeds. To generate new inputs, pass -fuzz with a time limit:

$ go test -fuzz=FuzzDecodeJSON -fuzztime=10s ./internal/api
fuzz: elapsed: 0s, gathering baseline coverage: 0/4 completed
...
PASS

The progress lines in between depend on your machine. A failing input would be saved under testdata/fuzz/FuzzDecodeJSON/, and plain go test would run it as a seed from then on. -fuzz takes one package at a time, so ./... is refused.

What each server timeout protects against

An http.Server with no timeouts waits for a slow client forever, and each waiting client holds a connection and a goroutine. The server in cmd/tasksd sets four:

	srv := &http.Server{
		Handler:           handler,
		ReadHeaderTimeout: 5 * time.Second,
		ReadTimeout:       10 * time.Second,
		WriteTimeout:      10 * time.Second,
		IdleTimeout:       60 * time.Second,
		ErrorLog:          slog.NewLogLogger(logger.Handler(), slog.LevelError),
	}

In plain words, each one limits a different kind of waiting:

  • ReadHeaderTimeout stops a client that sends its request headers very slowly, or never finishes them. That’s the Slowloris attack: open thousands of connections, send a few bytes on each now and then, and the server runs out of connections without ever seeing a full request.
  • ReadTimeout limits reading the whole request, body included, so a client can’t trickle a body in forever either.
  • WriteTimeout limits how long the response may take, counted from the end of the request headers. It stops a client that reads the response too slowly, and puts an upper bound on the whole request.
  • IdleTimeout closes a kept-alive connection that has sat unused between requests.

This program sends half a request header and then waits, once against a server with no ReadHeaderTimeout and once against a 50ms one. The client waits 2 seconds, a wide margin:

package main

import (
	"errors"
	"fmt"
	"net"
	"net/http"
	"net/http/httptest"
	"os"
	"time"
)

// slowClient connects, sends half a request header, and then waits up to
// 2 seconds for the server to do anything.
func slowClient(timeout time.Duration) {
	h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintln(w, "hello")
	})
	srv := httptest.NewUnstartedServer(h)
	srv.Config.ReadHeaderTimeout = timeout
	srv.Start()
	defer srv.Close()

	conn, err := net.Dial("tcp", srv.Listener.Addr().String())
	if err != nil {
		fmt.Println(err)
		return
	}
	defer conn.Close()
	fmt.Fprint(conn, "GET / HTTP/1.1\r\nHost: example.com\r\n") // no blank line: the header never ends

	conn.SetReadDeadline(time.Now().Add(2 * time.Second))
	_, err = conn.Read(make([]byte, 1))
	switch {
	case errors.Is(err, os.ErrDeadlineExceeded):
		fmt.Printf("ReadHeaderTimeout %v: after 2s the server is still holding the connection open\n", timeout)
	default:
		fmt.Printf("ReadHeaderTimeout %v: the server hung up: %v\n", timeout, err)
	}
}

func main() {
	slowClient(0)
	slowClient(50 * time.Millisecond)
}

It prints:

ReadHeaderTimeout 0s: after 2s the server is still holding the connection open
ReadHeaderTimeout 50ms: the server hung up: EOF

With the timeout, the server closed the connection without sending anything, not even a 408, and got its connection back.

WriteTimeout surprised me. I expected it to stop a slow handler, and it doesn’t:

package main

import (
	"errors"
	"fmt"
	"io"
	"net/http"
	"net/http/httptest"
	"time"
)

func main() {
	handlerDone := make(chan string, 1)
	h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		time.Sleep(500 * time.Millisecond) // slow work, 10 times the WriteTimeout
		_, err := fmt.Fprintln(w, "report ready")
		handlerDone <- fmt.Sprintf("handler: finished, write error = %v", err)
	})
	srv := httptest.NewUnstartedServer(h)
	srv.Config.WriteTimeout = 50 * time.Millisecond
	srv.Start()
	defer srv.Close()

	_, err := http.Get(srv.URL)
	fmt.Println("client got EOF:", errors.Is(err, io.EOF))
	fmt.Println(<-handlerDone)
}

It prints:

client got EOF: true
handler: finished, write error = <nil>

The handler ran its full 500ms, and its write reported no error, because the bytes only went into the server’s buffer. The timeout struck when the server tried to send them, and the client got a closed connection instead of a response. WriteTimeout protects the connection, not your handler’s time. To give up on slow work, pass r.Context() into it, or wrap the handler in http.TimeoutHandler, which answers 503 when time runs out.

Graceful shutdown

Stopping a server by killing the process cuts off every request in the middle. A graceful shutdown stops accepting new connections, lets the requests already running finish, and only then exits. Here’s the second half of serve in cmd/tasksd:

	errc := make(chan error, 1)
	go func() {
		logger.Info("listening", "addr", ln.Addr().String())
		errc <- srv.Serve(ln)
	}()

	select {
	case err := <-errc:
		return err
	case <-ctx.Done():
	}

	logger.Info("shutting down")
	shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer cancel()
	if err := srv.Shutdown(shutdownCtx); err != nil {
		return err
	}
	if err := <-errc; !errors.Is(err, http.ErrServerClosed) {
		return err
	}
	return nil
}

The sequence has five steps:

  1. srv.Serve(ln) blocks until the server stops, so it runs in a goroutine. errc has room for one value, so that goroutine can always send its result and exit.
  2. The select waits for whichever comes first: Serve failing, or ctx being cancelled, which signal.NotifyContext in run does on Ctrl+C or SIGTERM.
  3. srv.Shutdown closes the listener, so new connections are refused, closes idle connections, and waits for active ones to finish their requests.
  4. The wait gets a new 10-second context from context.Background(), because ctx is already cancelled and would end the wait at once. If requests are still running at the deadline, Shutdown returns context.DeadlineExceeded.
  5. After Shutdown, Serve returns http.ErrServerClosed. That’s the normal way to stop, so serve returns nil for it.

Keep the 10 seconds shorter than your platform waits between SIGTERM and killing the process. Kubernetes waits 30 seconds by default, but docker stop waits only 10.

Explain it like I’m ten

Think of a shop at closing time. The shopkeeper doesn’t push everyone out onto the street. First she locks the front door, so nobody new can come in. People who are just looking around without buying are asked to leave. The customers already standing at the till get to finish paying.

When the last customer has paid, she turns off the lights and goes home. If someone is still counting coins after ten minutes, she closes anyway.

The precise version

Shutdown closes every listener registered with the server, so the operating system refuses new connections to that port. It then closes connections that are idle, meaning between requests, or new with no request started for a few seconds. Then it polls the remaining connections, and returns when all of them have finished their current request and gone idle, or when its context ends. It doesn’t cancel the requests’ contexts, and it doesn’t interrupt handlers. A handler that never returns keeps Shutdown waiting until the deadline.

Where the analogy breaks: a shopkeeper can tell a customer “we’re closing, please hurry”. Shutdown tells running handlers nothing. If a handler does long work, it has to learn about the shutdown some other way, for example through a function registered with srv.RegisterOnShutdown. Also, closing the door isn’t polite on a real network: a client that tries to connect gets “connection refused”, so a load balancer should stop sending traffic before the shutdown begins.

ctx cancelled: Shutdown starts last request done: Shutdown returns listener accepting closed refused idle conn keep-alive closed at once request A handler running response sent serve() srv.Serve(ln) Shutdown waits returns nil

Graceful shutdown over time. When ctx is cancelled, the listener closes and new connections are refused, idle connections close at once, and Shutdown waits until the request already running has sent its response.

Proving shutdown works in a test

The shutdown sequence can’t be checked by running main and pressing Ctrl+C in a test, but serve takes a context, a listener and a handler, so a test can drive every step. It listens on 127.0.0.1:0, which asks the operating system for any free port, and passes serve a handler that blocks until the test releases it:

func TestServeShutsDownGracefully(t *testing.T) {
	ln, err := net.Listen("tcp", "127.0.0.1:0")
	if err != nil {
		t.Fatal(err)
	}
	addr := ln.Addr().String()

	started := make(chan struct{})
	release := make(chan struct{})
	slow := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		close(started)
		<-release
		io.WriteString(w, "finished")
	})

	ctx, cancel := context.WithCancel(t.Context())
	served := make(chan error, 1)
	go func() {
		served <- serve(ctx, ln, slow, slog.New(slog.DiscardHandler))
	}()

	// 1. Start a request, and wait until the handler is running.
	type result struct {
		body string
		err  error
	}
	inflight := make(chan result, 1)
	go func() {
		resp, err := http.Get("http://" + addr + "/slow")
		if err != nil {
			inflight <- result{err: err}
			return
		}
		defer resp.Body.Close()
		b, err := io.ReadAll(resp.Body)
		inflight <- result{string(b), err}
	}()
	<-started

	// 2. Ask the server to stop, as SIGTERM would.
	cancel()

	// 3. Wait until new connections are refused. Shutdown closes the
	// listener first, but nothing tells us when, so poll with a deadline.
	deadline := time.Now().Add(5 * time.Second)
	for {
		conn, err := net.Dial("tcp", addr)
		if err != nil {
			break
		}
		conn.Close()
		if time.Now().After(deadline) {
			t.Fatal("server still accepts connections 5s after cancel")
		}
		time.Sleep(10 * time.Millisecond)
	}

	// 4. The request in flight has not been cut off. Let it finish.
	select {
	case r := <-inflight:
		t.Fatalf("request ended before it was released: %+v", r)
	default:
	}
	close(release)

	if r := <-inflight; r.err != nil || r.body != "finished" {
		t.Errorf("in-flight request: body %q, err %v; want finished", r.body, r.err)
	}
	if err := <-served; err != nil {
		t.Errorf("serve returned %v, want nil", err)
	}
}

Channels fix the order, not sleeps. The handler closes started once it runs, so the test cancels only when the request is really in flight, and the handler can’t finish until the test closes release.

One step has to poll, because Shutdown doesn’t signal when the listener is closed. The test dials every 10 milliseconds until a dial fails, with a 5-second deadline, and closes each connection that gets through so Shutdown doesn’t wait on it.

A test that can’t fail proves nothing, so I changed srv.Shutdown(shutdownCtx) to srv.Close(), which closes every connection at once. The test failed: the in-flight request got EOF instead of finished.

The test passed 20 times in a row under go test -race -count=20 ./.... It doesn’t use testing/synctest, stable since Go 1.25: that package fakes the clock, but its documentation says to avoid real networking inside it.

Shipping one binary

A Go program builds into one executable that includes the runtime and every package it uses, so shipping the API means copying one file. First, give it a version: main.go declares a variable the build can overwrite.

// version is set at build time with -ldflags "-X main.version=v1.2.3".
var version = "dev"

func main() {
	addr := flag.String("addr", "localhost:8080", "address to listen on")
	showVersion := flag.Bool("version", false, "print the version and exit")
	flag.Parse()

	if *showVersion {
		fmt.Println("tasksd", version)
		return
	}

	logger := slog.New(slog.NewJSONHandler(os.Stderr, nil))
	if err := run(*addr, logger); err != nil {
		logger.Error("server stopped", "err", err)
		os.Exit(1)
	}
}

-X main.version=v1.0.0 sets that string at link time. Here’s the release build:

$ CGO_ENABLED=0 go build -trimpath -ldflags="-s -w -X main.version=v1.0.0" -o tasksd ./cmd/tasksd
$ ./tasksd -version
tasksd v1.0.0

What each flag does:

  • CGO_ENABLED=0 turns off cgo. This surprised me: a plain go build of the server was dynamically linked against the C library, though the module has no C code. The net package can use the C library for DNS lookups, so it links it whenever cgo is available. With cgo off, file reported a statically linked binary, which runs on any Linux of that architecture, even in an empty container.
  • -trimpath removes your machine’s directory paths from the binary, so stack traces show example.com/tasks/internal/api/handlers.go instead of a path in your home folder. It also helps two machines build the same bytes from the same code.
  • -ldflags="-s -w" drops the symbol table and DWARF debug information. The release build came out about a third smaller than a plain go build. Panics still print file names and line numbers, because the runtime keeps its own tables for that, but a debugger has less to work with.

go version -m reads the build settings back out of any Go binary:

$ go version -m tasksd
...
	path	example.com/tasks/cmd/tasksd
	mod	example.com/tasks	(devel)	
	build	-buildmode=exe
	build	-compiler=gc
	build	-trimpath=true
	build	CGO_ENABLED=0
	build	GOARCH=amd64
	build	GOOS=linux
	build	GOAMD64=v1

I’ve left out the first line, which names the exact Go release. -ldflags is missing from the list: Go doesn’t record it with -trimpath, because linker flags can contain paths. A program can read the same information with debug.ReadBuildInfo, but for your own version number, -X is simpler.

To build for another system, set GOOS and GOARCH. No extra toolchain is needed:

$ CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -trimpath -ldflags="-s -w" -o tasksd-linux-arm64 ./cmd/tasksd
$ GOOS=windows GOARCH=amd64 go build -trimpath -ldflags="-s -w" -o tasksd.exe ./cmd/tasksd

go tool dist list prints every supported pair. When I sent the Linux binary SIGTERM with kill, it logged shutting down and exited with status 0.

A static binary also makes a small container image. This Dockerfile is illustrative. I haven’t built or run it as part of this post:

FROM golang:1.26 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /tasksd ./cmd/tasksd

FROM scratch
COPY --from=build /tasksd /tasksd
USER 65532:65532
ENTRYPOINT ["/tasksd", "-addr", ":8080"]

The final image holds nothing but the binary: no shell, no CA certificates, no time zone data. This server needs none of them, but a service that calls HTTPS APIs would need certificates, or a distroless base image. -addr :8080 matters, because localhost:8080 only accepts connections from inside the container. The exec form of ENTRYPOINT makes tasksd process 1, so docker stop sends SIGTERM straight to it.

Where to go from here

The task API is a small, complete service, and each next step has a place in the design already:

  • A database. Write a Store on database/sql, like the SQLStore sketch in the part on JSON APIs, and run the same endpoint table against it with a test database.
  • Authentication. A middleware that checks a token and puts the user in the request context, the same pattern as withRequestID. For browser clients, look at http.CrossOriginProtection, added in Go 1.25, which rejects unsafe cross-origin requests.
  • An OpenAPI description of the routes, bodies and status codes, so clients can generate code from it.
  • Profiling. net/http/pprof serves CPU and memory profiles. Put it on a separate, private port, never on the public mux.

What to remember

  • Test handlers with httptest.NewRecorder for status, headers and body. Use httptest.NewServer when the network, a client or the server itself is part of the test.
  • Put storage behind an interface, and a small fake can make every error path happen, including a panic. Check the log too, with a slog handler writing to a buffer and ReplaceAttr dropping what changes.
  • Use t.Cleanup in helpers instead of defer, and t.Context() for anything that takes a context. Run go test -race ./... with tests that really send requests at the same time.
  • Set ReadHeaderTimeout against slow clients. WriteTimeout closes the connection but doesn’t stop your handler.
  • Shut down with srv.Shutdown and a deadline, and treat http.ErrServerClosed as success. Test it with a handler that blocks on a channel.
  • Ship with CGO_ENABLED=0 go build -trimpath -ldflags="-s -w", set the version with -X, and cross-compile with GOOS and GOARCH.

A server isn’t finished until you’ve tested how it fails and how it stops.

How useful was this post?

Click on a heart to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.