Shulou(Shulou.com)06/02 Report--

How to build a testable Go Web application, aiming at this problem, this article introduces the corresponding analysis and answer in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

Almost every programmer agrees that testing is important, but testing discourages test writers in a variety of ways. They may run slowly, they may use duplicate code, or they may test too much at a time, making it difficult to locate the root cause of test failure.

In this article, we will discuss how to design unit tests for Sourcegraph to make them easy to write, easy to maintain, fast to run, and available to others. We hope that some of the patterns mentioned here will help other Go web app writers and welcome suggestions for our testing methods. Before we start testing, let's take a look at our framework overview.

Frame

Like other web app, our website has three layers:

The web front end is used to serve HTML

HTTP API is used to return JSON

Data storage, running SQL queries against the database, returning Go structures or slices.

When a user requests a Sourcegraph page, the front end receives the HTTP page request and initiates a series of HTTP requests to the API server. Then the API server starts to query the data storage, which returns the data to the API server, then encodes it into JSON format and returns it to the web front-end server, which uses Go html/template packages to display and format the data into HTML.

The frame diagram is as follows: (for more details, check out recap of our Google Imax O talk about building a large-scale code search engine in Go.)

Test v0

When we started building Sourcegraph for the first time, we wrote the test in the easiest way to run. Each test goes into the database and initiates a HTTP GET request to the test API endpoint. The test parses the HTTP return and compares it with the expected data. A typical v0 test is as follows:

Func TestListRepositories (t * testing.T) {tests: = [] struct {url string; insert [] interface {} Want [] * Repo} {{"/ repos", [] * Repo {{Name: "foo"}}, [] * Repo {{Name: "foo"}}, {"/ repos?lang=Go", [] * Repo {{Lang: "Python"}, nil}, {"/ repos?lang=Go", [] * Repo {{Lang: "Go"}}, [] * Repo {{Lang: "Go"} } db.Connect () s: = http.NewServeMux () s.Handle ("/", router) for _, test: = range tests {func () {req, _: = http.NewRequest ("GET", test.url, nil) tx, _: = db.DB.DbMap.Begin () defer tx.Rollback () tx.Insert (test.data...) Rw: = httptest.NewRecorder () rw.Body = new (bytes.Buffer) s.ServeHTTP (rw, req) var got [] * Repo json.NewDecoder (rw.Body). Decode (& got) if! reflect.DeepEqual (got, want) {t.Errorf ("% s: got% v, want% v", test.url, got, test.want)}} ()}

Testing is easy to write at first, but it becomes painful as app evolves. Over time, we added new features. More features lead to more tests, longer running times, and longer dev cycles. More features also need to be changed and added to new URL paths (there are now about 75), most of which are quite complex. The interior of each layer of Sourcegraph also becomes more complex, so we want to test independently of other layers.

We encountered some problems in the test:

1. Testing is slow because they interact with the actual database-inserting test cases, initiating queries, and rolling back each test transaction. Each test runs for about 100 milliseconds, as we add more tests to add up.

two。 Testing is difficult to ReFactor. The test writes dead HTTP paths and query parameters with strings, which means that if we want to change a URL path or query parameter set, we have to manually update the URL in the test. This pain will increase as the complexity and number of our URL routing increases.

3. There are a lot of scattered and fragile sample code. Installing each test requires ensuring that the database is running properly and has the correct data. Such code is reused in multiple cases, but it is different enough to introduce bug into the installation code. We find ourselves spending a lot of time debugging our tests instead of the actual app code.

4. Test failure is difficult to diagnose. As app becomes more complex, because each test accesses three application layers, the root cause of test failure is difficult to diagnose. Our tests are more like integration tests than unit tests.

*, we put forward the need to develop a publicly available API client. We want to make API easy to imitate so that our API users can also write testable code.

Advanced test objectives:

As our app evolves, we realize the need for tests that meet these high requirements:

The goal is clear: we need to test each layer of app separately.

Comprehensive: all three layers of our app will be tested.

Fast: tests need to run very fast, which means that there is no longer database interaction.

DRY: although each layer of our app is different, they share many common data structures. Testing needs to take advantage of this to eliminate duplicate sample code.

Easy to imitate: API external users should also be able to use our internal test mode. Projects built on our API should be able to easily write good tests. After all, our web front end is not unique-it's just another API user.

How do we rebuild the test?

Well-written, maintainable tests and good, maintainable application code are inseparable. Refactoring the application code allows us to greatly improve our test code, which is a step for us to improve the test.

1. Build a Go HTTP API client

The step to simplify testing is to use Go to write a high-quality client for our API. Previously, our website was AngularJS app, but because we mainly served static content, we decided to move the front-end HTML generation to the server. After doing so, our new front end can use Go's API client to communicate with the API server. Our client go-sourcegraph is open source, and the go-github library has a huge impact on it. The client code, especially the endpoint code that gets warehouse data (repository data), is as follows:

Func NewClient () * Client {c: = & Client {BaseURL:DefaultBaseURL} c.Repositories = & repoService {c} return c} type repoService struct {c * Client} func (c * repoService) Get (name string) (* Repo, error) {resp, err: = http.Get (fmt.Sprintf ("% s/api/repos/%s", c.BaseURL, name) if err! = nil {return nil Err} defer resp.Body.Close () var repo Repo return & repo, json.NewDecoder (resp.Body) .Decode (& repo)}

In the past, our v0 API test wrote a large number of URL paths and built HTTP requests as ad-hoc, but now they can use this API client to build and initiate requests.

two。 Unified interface between HTTP API client and data warehouse

Next, we unify the interface between HTTP API and the data warehouse. In the past, our API http.Handlers directly initiated SQL queries. Now our API http.Handlers only needs to parse http.Request and then call our data warehouse, data warehouse and HTTP API client to implement the same interface.

Using the method of HTTP API client (* repoService) .Get above, we now have (* repoStore) .Get:

Func NewDatastore (dbh modl.SqlExecutor) * Datastore {s: = & Datastore {dbh: dbh} s.Repositories = & repoStore {s} return s} type repoStore struct {* Datastore} func (s * repoStore) Get (name string) (* Repo, error) {var repo * Repo return repo, s.db.Select (& repo, "SELECT * FROM repo WHERE name=$1", name)}

Unifying these interfaces puts our web app behavior description in one place, making it easier to understand and reason. And we can reuse the same data types and parameter structures in both the API client and the data warehouse.

3. Centralized URL path definition

Previously, we had to redefine URL paths at multiple layers of the application. In the API client, our code looks like this

Resp, err: = http.Get (fmt.Sprintf ("% s/api/repos/%s", c.BaseURL, name))

This approach can easily lead to errors because we have more than 75 path definitions, many of which are complex. The centralized URL path definition means refactoring the path independently from the API server in a new package. The definition of the path is declared in the path package.

Const RepoGetRoute = "repo" func NewAPIRouter () * mux.Router {m: = mux.NewRouter () / define the routes m.Path ("/ api/repos/ {Name:.*}") .Name (RepoGetRoute) return m} while the http.Handlers were actually mounted in the API server package: func init () {m: = NewAPIRouter () / / mount handlers m.Get (RepoGetRoute). HandlerFunc ( HandleRepoGet) http.Handle ("/ api/" M)}

Http.Handlers is actually mounted in the API server package:

Func init () {m: = NewAPIRouter () / / mount handlers m.Get (RepoGetRoute) .HandlerFunc (handleRepoGet) http.Handle ("/ api/", m)}

Now we can use path packages to generate URL in the API client instead of writing them to death. (* repoService) .Get method is now as follows:

Var apiRouter = NewAPIRouter () func (s * repoService) Get (name string) (* Repo, error) {url, _: = apiRouter.Get (RepoGetRoute) .URL ("name", name) resp, err: = http.Get (s.baseURL + url.String () if err! = nil {return nil, err} defer resp.Body.Close () var repo [] Repo return repo Json.NewDecoder (resp.Body) .Decode (& repo)}

4. Create a replica with an ununified interface

Our v0 test tested both paths, HTTP processing, SQL generation, and DB queries. Failures are difficult to diagnose and tests are slow.

Now, we have independent tests for each layer and we mimic the functions of the adjacent layers. Because each layer of the application implements the same interface, we can use the same imitation interface in all three layers.

The implementation of the imitation is a simple simulation function structure, which can be specified in each test:

Type MockRepoService struct {Get_ func (name string) (* Repo, error)} var _ RepoInterface = MockRepoService {} func (s MockRepoService) Get (name string) (* Repo, error) {if .Get_ = = nil {return nil, nil} return s.Get_ (name)} func NewMockClient () * Client {return & Client {& MockRepoService {}}

The following is the use in the test. We mimicked the RepoService of the data warehouse and tested API http.Handler using the HTTP API client. (this code uses all of the above methods.)

Func TestRepoGet (t * testing.T) {setup () defer teardown () var fetchedRepo bool mockDatastore.Repo. (* MockRepoService) .Get _ = func (name string) (* Repo, error) {if name! = "foo" {t.Errorf ("want Get% Q, got% Q", "foo", repo.URI)} fetchedRepo = true return & Repo {name} Nil} repo, err: = mockAPIClient.Repositories.Get ("foo") if err! = nil {t.Fatal (err)} if! fetchedRepo {t.Errorf ("! fetchedRepo")}}

Review of Advanced Test objectives

Using the above pattern, we have achieved the test goal. Our code is:

The goal is clear: test one layer at a time.

Comprehensive: all three application layers are tested.

Fast: the test runs very fast.

DRY: we merged the common interfaces of three application layers and reused them in application code and testing.

Easy to imitate: a copycat implementation can be used in all three application layers, as can external API users who want to test libraries built on Sourcegraph.

That's the end of the story about how to rebuild and improve Sourcegraph's tests. These patterns and examples work well in our environment, and we hope that these patterns and examples will also help the rest of the Go community, it is clear that they are not correct in every scenario, and we are sure there is room for improvement.

This is the answer to the question about how to build a testable Go Web application. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.