Compliance Manifest

When keel runs a pipeline, it can emit a compliance manifest — a structured, machine-readable record that captures what ran, on what data, when, and who triggered it. The manifest is the link between a pipeline output and the exact source code and input files that produced it.

The manifest is designed for regulated industries where auditors need to answer:

• What code produced this output? (source hash)
• What input data did it run on? (SHA-256 of each input file)
• What did it write? (SHA-256 of each output file)
• How many rows were in each DataFrame? (row counts per dataset)
• Who ran it, and when? (user identity, UTC timestamp)
• Can I verify the record hasn't been tampered with? (optional HMAC signature)

Keel owns the computation layer. Storage guarantees — WORM storage, access control, audit trails — are the responsibility of the surrounding infrastructure.

Enabling the Manifest

The manifest is disabled by default. Enable it in keel.toml:

[compliance]
enabled = true
user = "ci-service"

With this configuration, every keel run invocation will emit a manifest.

Capturing Manifests with --audit-log

To record manifests across runs, use --audit-log on the keel run command:

keel run src/pipeline.kl --audit-log /var/log/keel-audit.jsonl

Each successful run appends one JSON line to the file. Multiple runs accumulate in the same file. The file is created if it does not exist. An error during execution does not append a manifest — partial runs leave no record.

The [compliance] Section

Full configuration reference:

[compliance]
enabled = true           # required to emit a manifest
user = "analyst"         # recorded in the manifest; defaults to KEEL_COMPLIANCE_USER
fields = [               # optional: restrict which fields appear in the manifest
  "timestamp",           # if omitted, all fields are included
  "user",
  "source_hash",
  "input_files",
  "output_files",
  "dataframes",
]
hmac_key = "..."         # optional: hex-encoded HMAC-SHA256 key for signing

All fields except enabled are optional. The hmac_key field is sensitive — prefer setting it via the environment variable rather than in keel.toml.

Environment Variable Overrides

Every [compliance] field can be overridden at runtime with an environment variable. This is useful in CI environments where credentials are injected by the pipeline runner:

Environment variables take precedence over keel.toml values. The env vars are read fresh on every execution — they are never cached between runs.

Manifest Fields

A manifest with all fields looks like this:

{
  "keel_version": "0.0.14",
  "timestamp_utc": "2026-06-22T14:32:00Z",
  "source_hash": "a3f7c2e1d4b9...",
  "user": "ci-service",
  "input_files": [
    {
      "path": "/data/positions.parquet",
      "sha256": "b2c8f1a3d7e9..."
    }
  ],
  "output_files": [
    {
      "path": "/out/report.csv",
      "sha256": "e4d9b1c2f8a7..."
    }
  ],
  "dataframes": [
    {
      "name": "positions",
      "row_count": 4218,
      "source_paths": ["/data/positions.parquet"],
      "operations": ["read_parquet", "filter", "group_by"]
    }
  ],
  "signature": "7f2a1c4d8e3b..."
}

Field reference

HMAC Signing

When KEEL_COMPLIANCE_HMAC_KEY is set (a hex-encoded key), the manifest body is signed with HMAC-SHA256. The signature field contains the hex-encoded MAC. To verify:

1. Remove the signature field from the manifest JSON.
2. Re-serialise the remaining fields in the same order.
3. Compute HMAC-SHA256 over the canonical JSON using your key.
4. Compare the result to the recorded signature.

The key must be shared with anyone who needs to verify. For public verifiability without key sharing, consider an asymmetric scheme at the infrastructure layer (keel does not currently implement asymmetric signing).

What Keel Does Not Own

The compliance manifest documents what happened during computation. It does not:

• Authenticate users. The user field is a string from configuration or an environment variable. Keel does not verify identity — that is the responsibility of the CI system or orchestration layer that sets KEEL_COMPLIANCE_USER.
• Guarantee storage integrity. Writing a manifest to a file is not the same as tamper-proof storage. Use WORM storage (e.g. S3 Object Lock) or a content-addressed store for regulatory guarantees.
• Audit database reads. If a pipeline reads from a database rather than a file, keel cannot hash the source data. Database audit trails are the responsibility of the database layer.
• Provide field-level transformation traces. The operations list in each DataFrameRecord names the operations applied; it does not record intermediate row counts at each step. Per-step traceability is a planned future extension.

Example: SOX Audit Pipeline

Add [compliance] to your project manifest:

[project]
name = "pnl-pipeline"
version = "1.0.0"
main = "src/main.kl"

[permissions.dataframe]
enabled = true
sandbox = "./data"

[compliance]
enabled = true

Set the signing key and user identity in CI:

export KEEL_COMPLIANCE_HMAC_KEY=<your-hex-key>
export KEEL_COMPLIANCE_USER=ci-service
keel run --audit-log /var/log/keel-audit.jsonl

After each run, keel-audit.jsonl gains one line. Ship the file to WORM storage to create an immutable record of every pipeline execution.

Example: Selective Fields for Pharma Submission

For a 21 CFR Part 11 submission package, you may want only the fields required by the submission:

[compliance]
enabled = true
fields = ["timestamp", "user", "source_hash", "input_files", "dataframes"]

This omits output_files and signature from the manifest. Unknown field names in the fields list are silently ignored — no error.

See Also

• Projects — keel.toml structure and permissions
• Data Lineage — Column-level lineage tracking that feeds the dataframes field in the manifest