JSON/REST from Scratch: A Guide to Go's net/http Package

Go/Golang

What does it take to stand up a minimal REST service in Go, using only facilities provided by the standard library? This is a useful fingering exercise, and also provides an opportunity to dive a little deeper into the net/http package. The net/http package can be a bit confusing at first, but once the appropriate idioms have been identified, the resulting code is actually quite compact and convenient.

The intended service should provide an REST API endpoint that accepts and returns JSON datagrams. We want to build both a server and a client — both minimal: the emphasis is on the communication infrastructure, not business logic.

This project also gives us an opportunity to take a look at the net/httptest package, which provides some facilities and convenience functions for writing unit tests, testing both HTTP client and server code.

Server

The net/http package can appear a somewhat mysterious, because, by default, it does a few things implicitly. This is convenient, but also a bit opaque. In particular, the package sets up a Server instance, with a default request dispatcher. To the programmer, these default instances are not directly visible, which gives the confusing impression that some actions are taking place in a vacuum. For example when registering request handlers it is not clear with what they are being registered. Answer: with the default request dispatcher, which lives at the package level.

Aside: Package-Level State

This is actually an interesting and somewhat under-appreciated design feature of Go: state can be maintained at the package level, as opposed to the application level. In the present case, we don’t need to instantiate an application-level Server instance; instead, we can use the one that is instantiated and maintained by the package itself.

The same principle can be applied to many other long-lived services within an application: servers, clients, loggers, and so on. This is an alternative to the usual application start-up sequence, in which all required sub-systems must be explicitly brought up, and where the overall “application” typically maintains handles to them (which it then uses to invoke the subsystems' functionality).

The advantage is simplified start-up code, and the fact that we don’t need to maintain (and possibly pass around) handles to the various subsystems.

The disadvantage is mostly conceptual: the fact that state is being maintained, without an explicit, visible handle to it. (In my experience, few programmers are truly comfortable with this idiom!)

The Server Behavior

Each Server instance listens for requests on a single host:port combination, and forwards them to the Handler component that is registered with the server. The server starts a new goroutine for each new connection (not for each request!). Handlers must be prepared to handle this level of concurrency.

(Although natural, the concurrency behavior of the Server implementation is a bit obscure. The only place it is mentioned in the package reference is in the documentation for the function http.Server.Serve().)

The net/http package defines the Handler interface; anything implementing this interface can be registered with the Server instance. The interface itself defines only a single function:

type Handler interface {
	ServeHTTP(ResponseWriter, *Request)
}

In principle, an application can start from here, providing a Handler implementation that handles every request.

Request Dispatching with ServeMux

As a convenience, net/http provides a facility to match requests against URL patterns and to forward the requests to handlers accordingly: the ServeMux dispatcher (or “multiplexer”).

This component can appear a bit mysterious, but there is really nothing to it: it’s a collection of URL patterns, with their associated handlers. It implements the Handler interface, and can therefore be used in a Server instance. Its implementation of ServeHTTP() selects a matching handler, based on the request URL, and forwards the request accordingly. (Unfortunately, the ServeMux implementation does not provide a way to list all registered URL patterns!)

The default server that is created when the package is initialized by default installs a ServeMux instance that is ready to be used.

Handlers

Anything that provides a function with the signature

func ServeHTTP(ResponseWriter, *Request)

implements the Handler interface. If the handler needs to maintain state between invocations, it makes sense to define a struct. If the handler’s internal state may change, it must be protected by a mutex. Such a handler can be registered with the default instance of ServeMux using the Handle( pattern string, handler Handler ) function.

If the handler does not need to maintain state, it can be replaced by a function. The net/http package defines a named type HandlerFunc. Its underlying type is a function with the signature func(ResponseWriter, *Request). Because the underlying type is the same, any function with such a signature can be converted to the type HandlerFunc, using the conversion function HandlerFunc() that is automatically created for each named type.

The type HandlerFunc defines a method, called ServeHTTP() with the appropriate signature: it therefore implements the Handler interface. The implementation simply calls the underlying (wrapped) function. Converting (wrapping) any function with the appropriate signature to a HandlerFunc therefore creates a Handler instance.

Any function with the appropriate signature can be registered with the default ServeMux instance using HandleFunc( pattern string, f func( ResponseWriter, *Request ) ). This function will implicitly wrap the supplied function f in a HandlerFunc type, thereby converting it to an object that implements Handler, and then register it as handler for the specified pattern.

Names and Meanings

The names of some of the elements of net/http are a bit unfortunate. (What does ServeMux do — just based on the name? Is it a function or a type?)

There is some particular opportunity for confusion around the handlers, their types, and the functions used to register them.

  • The Handle() function does not handle anything, but merely registers a handler for a URL pattern.

  • The HandleFunc() function registers a function as request handler with the default dispatcher.

  • The HandlerFunc() call is a conversion function that wraps a function and returns an object implementing the Handler interface.

Request Forwarding

It is a common idiom to wrap handlers into one another, so that one handler forwards the request to another one. This works both with struct-based, stateful handlers, as well as stateless handlers implemented as HandlerFunc.

For example, to add logging to a Handler, the original handler can be wrapped:

func Logger( h Handler ) Handler {
    return HandlerFunc( w ResponseWriter, r *Request) {
	    // Perform logging ...
		h.ServeHTTP(w, r)
    }
}

Be aware that a function with the signature func(ResponseWriter, *Request) is itself not a Handler and can therefore not be used as argument to the Logger() function above. Instead, it first must be converted, using the http.HandlerFunc() conversion function: its return is a suitable argument to Logger(). (See line 27 in the listing below.)

The net/http package provides a few such wrappers. Examples include TimeoutHandler and StripPrefix.

Listing

 1import (
 2	"encoding/json"
 3	"fmt"
 4	"io"
 5	"log"
 6	"net/http"
 7	"sync"
 8)
 9
10// Datagrams
11type Input struct{ Query string }
12type Output struct{ Answer string }
13
14func main() {
15	// Inline function: simple GET request to check whether service is up!
16	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
17		io.WriteString(w, "Hello, World") // fmt.Fprint( w, "Hello, World")
18	})
19
20	// Handler function
21	http.HandleFunc("/action", handleJson)
22
23	// Handler instance
24	http.Handle("/counter", &statefulHandler{prefix: "Request Count"})
25
26	// Add logging to JSON handling
27	http.HandleFunc("/logaction", loggingHandler(http.HandlerFunc(handleJson)))
28
29	// Listen and Serve
30	log.Fatalln(http.ListenAndServe(":8888", nil))
31}
32
33func handleJson(w http.ResponseWriter, r *http.Request) {
34	input := Input{}
35	decoder := json.NewDecoder(r.Body)
36	decoder.Decode(&input)
37
38	output := Output{"Query was: " + input.Query}
39	encoder := json.NewEncoder(w)
40	encoder.Encode(output)
41}
42
43type statefulHandler struct {
44	mutex   sync.Mutex // mutex to protect counter
45	counter int
46	prefix  string
47}
48
49func (h *statefulHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
50	h.mutex.Lock()
51	defer h.mutex.Unlock()
52	h.counter += 1
53
54	output := Output{fmt.Sprintf("%s: %d", h.prefix, h.counter)}
55	json.NewEncoder(w).Encode(output)
56}
57
58func loggingHandler(h http.Handler) http.HandlerFunc {
59	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
60		log.Println("Log Message")
61		h.ServeHTTP(w, r)
62	})
63}

Reading and Writing JSON

When making REST calls, the JSON payload is usually transported in the body of the request. Note that many elements of the Request type are public and expected to be accessed directly (that is, not through an “accessor function”). This includes the Request.Body, which is an io.ReadCloser, and must be accessed using the APIs that can accept that type.

A response is written to the network using a ResponseWriter instance, which implements the io.Writer interface.

The encoding/json package provides functions to create a decoder or encoder from a io.Reader or io.Writer instance.

Client

The client is simpler. It does not need to match up handlers with URL patterns, and does not need to maintain state between calls.

This client implementation simply calls all the endpoints provided by the server and prints the responses.

Listing

 1import (
 2	bpkg "bytes"
 3	"encoding/json"
 4	"fmt"
 5	"io"
 6	"log"
 7	"net/http"
 8)
 9
10// Datagrams
11type Input struct{ Query string }
12type Output struct{ Answer string }
13
14func main() {
15	HOST := "http://localhost:8888"
16	MIME := "application/json"
17
18	// Simple GET: Test if server is alive
19	res, err := http.Get(HOST)
20	if err != nil {
21		log.Fatalln("GET failed", err)
22	}
23	bytes, err := io.ReadAll(res.Body)
24	res.Body.Close()
25	fmt.Println(string(bytes))
26
27	// Query/Response to stateless server
28	input := Input{"This is the query"}
29	bytes, err = json.Marshal(input)
30
31	res, err = http.Post(HOST+"/action", MIME, bpkg.NewReader(bytes))
32	if err != nil {
33		log.Fatalln("POST failed", err)
34	}
35
36	output := Output{}
37	json.NewDecoder(res.Body).Decode(&output)
38	res.Body.Close()
39	fmt.Printf("%+v\n", output)
40
41	// Query/Response to stateful server
42	input = Input{"This is another query"}
43	bytes, err = json.Marshal(input)
44
45	res, err = http.Post(HOST+"/counter", MIME, bpkg.NewReader(bytes))
46	if err != nil {
47		log.Fatalln("POST failed", err)
48	}
49
50	output = Output{}
51	json.NewDecoder(res.Body).Decode(&output)
52	res.Body.Close()
53	fmt.Printf("%+v\n", output)
54
55	// Query to logging handler - ignore response
56	input = Input{"This is the third query"}
57	bytes, err = json.Marshal(input)
58
59	res, err = http.Post(HOST+"/logaction", MIME, bpkg.NewReader(bytes))
60	if err != nil {
61		log.Fatalln("POST failed", err)
62	}
63}

Unit Tests

The net/httptest package provides helpers to write unit tests testing HTTP server and client implementations.

Server

The function NewRequest() can be used to obtain a fully formed http.Request that can be passed to a handler, and an instance of the type ResponseRecorder can be used in place of an http.ResponseWriter. The actual http.Response can later be retrieved from the ResponseRecorder using the Result() method. It is thus possible to exercise the entire server interface locally, without actually having to bring up a server instance.

import (
	"testing"

	"bytes"
	"encoding/json"
	"fmt"
	"net/http/httptest"
)

func Test_json(t *testing.T) {
	tests := []struct{ sent, want string }{
		{"Hello", "Query was: Hello"},
		{"Bye", "Query was: Bye"},
		{"", "Query was: "},
	}

	for _, test := range tests {
		bs, _ := json.Marshal(Input{test.sent})

		recorder := httptest.NewRecorder()
		request := httptest.NewRequest("POST", "/action", bytes.NewReader(bs))

		handleJson(recorder, request)

		out := Output{}
		response := recorder.Result()
		json.NewDecoder(response.Body).Decode(&out)

		if out.Answer != test.want {
			t.Error("Want:", test.want, "Got:", out.Answer)
		}
	}
}

func Test_counter(t *testing.T) {
	h := statefulHandler{prefix: "Prefix"}

	for i := 0; i < 5; i++ {
		recorder := httptest.NewRecorder()
		request := httptest.NewRequest("POST", "/counter", bytes.NewReader(nil))

		h.ServeHTTP(recorder, request)

		out := Output{}
		response := recorder.Result()
		json.NewDecoder(response.Body).Decode(&out)

		want := fmt.Sprintf("Prefix: %d", i+1)
		if out.Answer != want {
			t.Error("Want:", want, "Got:", out.Answer)
		}
	}
}

Client

On the client side, the package provides a Server implementation that is essentially a stub. It requires an implementation of the desired Handler, but otherwise listens on a local loopback device. Any client logic can be run against this stub and its results verified. (The server’s address can be found in the URL member variable of the server instance.)

It is still necessary to register appropriate handlers with the stub server. These can either be “real” handlers, coming from the server implementation, or special test handlers, with prescribed behavior. (The listing below chooses to provide a test handler, which mimics the real handler in the server code.)

import (
	"testing"

	bpkg "bytes"
	"encoding/json"
	"fmt"
	"io"
	"net/http"
	"net/http/httptest"
)

func Test_simple(t *testing.T) {
	// This is the test server stub
	srv := httptest.NewServer(
		http.HandlerFunc(
			func(w http.ResponseWriter, r *http.Request) {
				fmt.Fprint(w, "Hi!")
			}))

	// This is the client code to test!
	response, _ := http.Get(srv.URL)

	msg, _ := io.ReadAll(response.Body)
	if string(msg) != "Hi!" {
		t.Error("Want: Hi! Got: ", string(msg))
	}
}

func Test_json(t *testing.T) {
	// Test server using a "real" handler
	srv := httptest.NewServer(http.HandlerFunc(handleJson))

	// Client code preparing request for JSON handler
	input := Input{"This is a test query"}
	bytes, err := json.Marshal(input)

	res, err := http.Post(srv.URL+"/action", "", bpkg.NewReader(bytes))
	if err != nil {
		t.Error("HTTP Error:", err)
	}

	output := Output{}
	json.NewDecoder(res.Body).Decode(&output)
	res.Body.Close()
	if output.Answer != "Query was: This is a test query" {
		t.Error("Unexpected response:", output.Answer)
	}
}

// Test handler: same as server implementation
func handleJson(w http.ResponseWriter, r *http.Request) {
	input := Input{}
	decoder := json.NewDecoder(r.Body)
	decoder.Decode(&input)

	output := Output{"Query was: " + input.Query}
	encoder := json.NewEncoder(w)
	encoder.Encode(output)
}

Conclusion

This concludes our quick tour of the net/http and net/httptest packages.

Although some of the idioms may look a little unfamiliar at first, the resulting code, in particular for the server component, is actually quite compact and readable.

The client-side is a bit messier; a fact acknowledged by the library’s own maintainer in considerable depth. I would recommend using a third-party package for client-side development; personally, I had good experiences with the excellent Go Resty. Another alternative, with extensive discussion of its design rationale, is the Go requests library.

Resources

This guide only talks about the mechanics of the net/http package; it does not address questions of project layout, application architecture, or application lifecycle management. An interesting and detailed write-up on these topics that mentions some advanced idioms (collect all URL patterns and their handlers in a separate source file, conventionally called routes.go; use closures that return handlers to manage the handlers' state; ensure clean application shutdown; etc) can be found here.

Two somewhat older posts discuss configuration options (specifically timeouts) when exposing Go services to the internet: Post 1, Post 2.

Although the client-side facilities of the net/http library are reasonably convenient, several libraries and frameworks exist that improve on it. Personally, I have experience with Labstack Echo, but there are many more. Some of the most popular include gin, go-micro, go-zero, beego, kratos, and the somewhat older go-kit.

Listings