Test specs

Table of Contents

How to write spec tests

This guide describes how to write and run golden tests for product specs. Golden testing allows spec developers to validate internal logic, such as derived fields, calculations, and ratings, by comparing actual behavior against expected results.

In this context, a golden test runs a .tart script that loads saved policy facts and verifies that the computed outputs match expected values. If the logic changes or a calculation is incorrect, the test will fail and show exactly what diverged. This helps ensure regressions are caught early and that changes to product logic are intentional and reviewed.

How to write spec tests

Fill out a policy with the data you want to test against.
Sign in to the environment where that policy exists and dump the facts as a JSON file. Replace product 1 and policy 1 with your product and policy names:
```
./scripts/console.sh -r admin -e localhost # Select environment where the target policy exists

> load-product 1
1> load-policy 1
1:1:1> dump-facts test_facts.json
```
If successful, you will see a message like Wrote facts to "/Your/path/to/brossa/brossa/brossa/test_facts.json"

Create a test directory at the path of your spec, and move the facts file to this directory. For example:

mkdir ./brossa/brossa/catalog/testing_re/test
mv ./brossa/brossa/test_facts.json brossa/brossa/catalog/testing_re/test/test_facts.json

Now create the test .tart file in the testing directory. The .tart file allows us to assert the value of datapoints to verify the logic of the product:
```
touch ./brossa/brossa/catalog/testing_re/test/test.tart
```

Write the test .tart file. The following example shows a simple test verifying both logic from the dumped facts, and overriding datapoints to assert the logic:

Test Scenario 1
============================================

An example .tart file to test the facts and verify derived data inside the product.

---

// Load the facts that we dumped earlier
load-facts "test_facts.json";

// Verify a derived fact
assert (policy.period == 12);

// We can also override datapoints
policy.start_date = 2024-01-01;
policy.end_date = 2024-04-01;

// We can then verify the derived fact with the overriden datapoints
assert (policy.period == 3);

// We can also add a friendly message to a test with the @it annotation
@it("Test that the policy period calculated correctly")
assert (policy.period == 3);

// We can also skip tests with a message using the @xit annotation
@xit("Skip this test for now as it will fail")
assert (policy.period == -1);

From the root of the brossa directory, you can now run the tests with the golden.sh script:

./scripts/golden.sh testing_re

You should see the following output from the tests:

catalog/testing_re/test/test.tart: Test Scenario 1
  An example .tart file to test the facts and verify derived data inside the product.
    (policy.period == 12) [✔]
    (policy.period == 3) [✔]
    Test that the policy period calculated correctly [✔]
    Skip this test for now as it will fail [‐]
      # PENDING: No reason given

Finished in 0.0004 seconds
4 examples, 0 failures, 1 pending

Upon failure you will get an explain trace that shows the derivation and what went wrong. For example, if we try to assert the policy.period to an incorrect value, we would see the following:

catalog/testing_re/test/test.tart: Test Scenario 1
  An example .tart file to test the facts and verify derived data inside the product.
    (policy.period == 12) [✔]
    (policy.period == 3) [✔]
    Test that the policy period calculated correctly [✔]
    (policy.period == -1) [✘]

Failures:

  src/glossterm:Brossa[]/Language/Golden.hs:202:14:
  1) catalog/testing_re/test/test.tart: Test Scenario 1, An example .tart file to test the facts and verify derived data inside the product., (policy.period == -1)
      Assertion did not hold
        because (3 == -1) is false

  To rerun use: --match "/catalog/testing_re/test/test.tart: Test Scenario 1/An example .tart file to test the facts and verify derived data inside the product./(policy.period == -1)/" --seed 1767339374

Randomized with seed 1767339374

Finished in 0.0004 seconds
4 examples, 1 failure

You can have multiple .tart files and multiple facts .json files for a single spec. This allows you to test for multiple scenarios and policy configurations when working with more complex products.