Build Fixture-Based Regression Tests for MetricKit Diagnostics

MetricKit diagnostic payloads do not arrive on the same cadence as code changes. A one-line parser change today may not reveal a missing field until a crash or hang report arrives several days later. A more reliable approach is to sanitize, normalize, and preserve previously collected payloads as fixed fixtures, then replay the same inputs for every commit on a cloud Mac. This validates the diagnostic pipeline itself instead of waiting for the next intermittent event.

Define the Test Boundary First

Fixed fixtures are well suited to testing four layers of behavior: whether raw JSON can be read, whether system fields map correctly to the internal model, whether sensitive data is removed, and whether malformed input degrades gracefully. They cannot prove that the system will generate a payload, nor can they replace device-side callback validation.

Separate payload collection from application-level processing. The collection layer should only call jsonRepresentation(), write the result to a protected directory, and queue it for upload. The parsing layer should accept Data and return an internal representation that does not depend on MetricKit types. Unit tests should call only the latter, so the test target does not have to fabricate system objects.

The purpose of a fixture is not to imitate one successful parse. It is to freeze the input contract so that every parser change can answer two questions: which fields changed, and which information was discarded?

Create Normalized Fixtures That Can Be Committed

Raw payloads may contain bundle identifiers, device information, timestamps, call stack symbols, and local paths. Do not commit them directly. Keep the originals only in a restricted environment, then generate normalized copies suitable for the repository: replace timestamps with fixed values, identifiers with test values, and paths with $APP or $HOME. Preserve the format of call stack addresses without retaining real addresses.

You can add a schema field to the internal format, but do not rewrite the original MetricKit version. Organize the directory by diagnostic type:

Tests/Fixtures/MetricKit/
├── crash/basic.json
├── crash/missing-stack.json
├── hang/main-thread.json
├── disk-write/threshold.json
└── malformed/truncated.json

Each fixture should represent exactly one condition. If one file contains a crash, a hang, and a disk anomaly, failures become difficult to isolate. Fixture names should describe the input, not the expected result. Keep expected values in the test code, where reviewers can more easily notice assertions that were changed merely to match new output.

Apply a Schema Gate Before Running Tests

Running inexpensive jq checks before compiling the tests can quickly reject invalid JSON, missing versions, and paths that were not sanitized. In the example below, diagnostics is the team's normalized array; it does not assume that the raw system payload has the same structure.

set -euo pipefail

root="Tests/Fixtures/MetricKit"

find "$root" -name '*.json' -print0 |
while IFS= read -r -d '' file; do
  jq -e '
    type == "object" and
    .schema == 1 and
    (.diagnostics | type == "array") and
    all(.diagnostics[];
      (.kind | type == "string") and
      (.timestamp | type == "string") and
      (.stackID | type == "string")
    )
  ' "$file" >/dev/null

  if grep -E '/Users/|/private/var/|[A-F0-9]{16,}' "$file"; then
    echo "fixture contains unnormalized data: $file" >&2
    exit 1
  fi
done

The schema gate should not require every system field, because newly introduced optional fields would then cause meaningless failures. Check only the keys that internal processing actually depends on, and configure the decoder to ignore unknown fields.

Cover the Parsing Contract with Positive and Negative Fixtures

Prepare at least one set of valid input and three sets of failure cases. The important question is not merely whether parsing completes without throwing an error, but whether the output remains usable for aggregation, alerting, and investigation.

Assert the Internal Model, Not the Entire JSON Document

Full-document snapshots are easily disrupted by field ordering and irrelevant metadata. Prefer assertions on the number of diagnostics, their types, stable identifiers, and sanitization results. Add formatted JSON snapshots only when normalized output is exchanged across systems. Errors should also use comparable enum cases such as invalidJSON, missingRequiredField, and unsupportedDiagnostic; do not compare only unstable natural-language messages.

Integrate the Fixtures into Cloud Mac CI

On VMRunner cloud Macs, run the fixture checks before the unit tests and ensure that non-interactive jobs use a fixed working directory. One practical sequence is to check out the code, run the schema gate, execute the parser unit tests, generate test results, and finally verify that no uncommitted fixture changes have appeared in the workspace.

Fixture changes must be reviewed separately. When the system introduces a new field, first determine whether the parser needs to consume it. If it does, upgrade the internal schema and commit migration tests at the same time. If it does not, keep decoding permissive. Never use a CI script to overwrite baseline files automatically, because a genuine field loss could then be “approved” by newly generated but incorrect output.

Pre-Merge Checklist

• Raw payloads were sanitized outside the repository.
• Each fixture covers only one diagnostic condition.
• Valid, empty, missing-field, truncated, and unknown-type cases are all tested.
• Unknown optional fields do not cause the entire batch to fail.
• Local paths, long identifiers, and user content cannot pass the schema gate.
• Parsing failures return stable error categories.
• Fixture changes and parser changes are approved in the same review.

Once these constraints are in place, MetricKit diagnostic processing changes from “try it after data arrives” into ordinary, repeatable engineering tests. System payload delivery still requires device-side validation, but parsing, sanitization, and compatibility no longer depend on reports arriving by chance.

Frequently asked questions