Graphql-engine integration tests written in Haskell using the hspec testing framework.

For motivation, rationale, and more, see the test suite rfc.

Table of Contents

• Running the test suite
• Test suite structure
  • Harness
  • Test
• Adding a new test
  • Specifying backends
    • Setup action
    • Teardown action
  • Writing tests
  • Style guide

1. To run the Haskell integration test suite, we'll first need to start the backends:

   docker-compose up

   This will start up Postgres, SQL Server, Citus and MariaDB.

   Note: on ARM64 architecture we'll need additional steps in order to test mssql properly.

   Preparation

   1. Switch the docker image in docker-compose/sqlserver/Dockerfile to azure-sql-edge:

   - FROM mcr.microsoft.com/mssql/server:2019-latest@sha256:a098c9ff6fbb8e1c9608ad7511fa42dba8d22e0d50b48302761717840ccc26af
   + FROM mcr.microsoft.com/azure-sql-edge

   2. Install sqlcmd locally. On MacOS, this can be done with brew: brew install mssql-tools.

   Start the backends

   1. Run docker-compose up

   2. Initialize the SQL Server database

      (cd docker-compose/sqlserver && bash run-init.sh 65003)

2. Once the containers are up, you can run the test suite via

   cabal run tests-hspec

   You can also further refine which tests to run using the -m flag:

   cabal run tests-hspec -- -m "SQLServer"

   For additional information, consult the help section:

   cabal run tests-hspec -- --help

3. The local databases presist even after shutting down docker-compose. If this is undesirable, delete the databases using the following command:

   docker-compose down --volumes

Modules under the Harness/ namespace provide the infrastructure and supporting code for writing and running the tests. It includes quasiquoters, interacting with backends, interfacing with HTTP, constants, and so on.

Supporting code should be added under the Harness.* namespace instead of added ad-hoc in test specs, to improve readability and reuse.

Modules under the Test/ namespace define integration test specifications for various features and backends.

The module Test.HelloWorldSpec contains a starting point which can be built upon.

To create a new test:

1. Create a new module based on Test.HelloWorldSpec
2. Specify each relevant context on which the tests will run in spec
3. Specify the tests and expectations in tests

When creating a new test, make sure that:

1. The module name is suffixed by the word Spec
2. The module exports the entry point spec :: SpecWith State
3. The module is listed in the cabal file under other-modules

(1) and (2) are required for hspec-discover to find and run the test.

We often want to run the same tests several times with slightly different configuration. Most commonly, we want to assert that a given behaviour works consistently across different backends.

Harness.Test.Feature defines two functions for running test trees in terms of a list of Context as.

Each Context a requires:

• a unique name, of type String
• a setup action, of type State -> IO a
• a teardown action, of type (State, a) -> IO ()
• an options parameter, which will be threaded through the tests themselves to modify behavior for a particular Context

Of these two functions, whether one wishes to use Harness.Test.Feature.run or Harness.Test.Feature.runWithLocalState will depend on if their test can be written in terms of information provided by the global State type or if it depends on some additional "local" state.

More often than not, test authors should use Harness.Test.Feature.run, which is written in terms of Context (). This uses () for the local test which does not carry any "useful" state information, and is therefore omitted from the body of the tests themselves.

In the rare cases where some local state is necessary (either for the test itself, or as an argument to the teardown action for some Context), test authors should use Harness.Test.Feature.runWithLocalState. This function takes a type parameter for its local state, which will be provided to both the teardown action specified in Context as well as the body of tests themselves.

A setup action is a function of type State -> IO a which is responsible with creating the environment for the test. It needs to:

1. Clear and reconfigure the metadata
2. Setup tables and insert values
3. Track tables, add relationships, permissions

etc.

These actions can be created by running POST requests against graphql-engine using Harness.GraphqlEngine.post_, or by running SQL requests against the backend using Backend.<backend>.run_.

Its return value, IO a, matches the a of Context a: it is the additional local state that is required throughout the tests, in addition to the global State. Most tests do not need additional state, and use a State -> IO () function.

The teardown action is another function of type (State, a) -> IO () which is responsible for removing the environment created by the test or setup, so that other tests can have a "clean slate" with no artifacts. The (State, a) parameter is constructed from the a parameter of the Context a: it is the local state that is passed throughout the tests.

This action is responsible for freeing acquired resources, and reverting all local modifications: dropping newly created tables, deleting custom functions, removing the changes made to the metadata, and so on.

These actions can be created by running POST requests against graphql-engine using Harness.GraphqlEngine.post_, or by running SQL requests against the backend using Backend.<backend>.run_.

Test should be written (or reachable from) tests :: SpecWith State, or tests :: SpecWith (State, Foo) for tests that use an additional local state.

A typical test will look similar to this:

  it "Where id=1" \state ->
    shouldReturnYaml
      ( GraphqlEngine.postGraphql
          state
          [graphql|
query {
  hasura_author(where: {id: {_eq: 1}}) {
    name
    id
  }
}
|]
      )
      [yaml|
data:
  hasura_author:
  - name: Author 1
    id: 1
|]

• it specifies the name of the test
• shouldReturnYaml creates an Expectation which does the following:
  • Runs a POST request against graphql-engine which can be specified using the graphql quasi-quoter.
  • Compares the response to an expected result which can be specified using the yaml quasi-quoter.

Note: these quasi-quoter can also perform string interpolation. See the relevant modules under the Harness.Quoter namespace.

• Test suite should be very easy to read for a junior or intermediate Haskell developer. Be mindful of advanced feature use and abstractions!
• Prefer self-contained specs, even if there's some query duplication.
• Avoid functions or types in tests, other than calls to the Harness.* API. Any supporting code should be in the Harness.* hierarchy and apply broadly to the test suites overall.
• Consider reorganising tests if modules get much longer than 1/2 pagescrolls (~200-300 LOC?).