Profiles & Semantic Validation

Introduction

A structurally valid Evidence Pack tells you nothing about whether it contains the right evidence. A pack could pass all integrity checks but be missing a required SOC 2 report, or include a pentest summary without the full findings. Profiles solve this by declaring what a pack must contain.

Relationship to Core Spec

This spec builds on the reserved fields defined in Evidence Pack v1.0:

• profile - References the profile this pack claims to conform to
• overlays - Additional constraints applied on top of the profile
• profile_lock - Pinned digests for reproducible validation

Artifact Semantic Type

For semantic validation to work, artifacts must declare their semantic type. This companion spec defines a new optional artifact field:

An artifact matches a profile requirement when artifact.semantic_type == requirement.type. The field is named semantic_type (not type) to distinguish it from the core spec's type field ("embedded"/"reference").

Core Concepts

How They Compose

// A pack declares its profile, overlays, and locks digests for reproducibility
{
  "spec_version": "1.0",
  "profile": "evidencepack/soc2-basic@v1",
  "overlays": ["evidencepack/hipaa-overlay@v1"],
  "profile_lock": [
    { "id": "evidencepack/soc2-basic@v1", "digest": "sha256:abc123..." },
    { "id": "evidencepack/hipaa-overlay@v1", "digest": "sha256:def456..." },
    { "id": "evidencepack/soc2-report@v1", "digest": "sha256:789def..." }
  ],
  "artifacts": [
    {
      "type": "embedded",
      "path": "artifacts/soc2-2025.pdf",
      "digest": "sha256:...",
      "semantic_type": "evidencepack/soc2-report@v1",
      "collected_at": "2025-03-15T10:00:00Z",
      "metadata": { "auditor": "Big4 LLP" }
    }
  ]
}

Validation proceeds in layers:

1. Resolve the profile and overlays (fetch or use cached, verify digests if locked)
2. Merge overlays onto the base profile to produce an effective profile
3. Validate each artifact against its declared type schema
4. Check that all required types are present with valid artifacts
5. Report findings (pass, fail, or warning for each requirement)

Types

A type is a reusable definition for a category of evidence artifact. It specifies what the artifact should contain, how to validate it, and what metadata to expect.

Type Definition

{
  "id": "evidencepack/soc2-report@v1",
  "name": "SOC 2 Type II Report",
  "description": "Annual SOC 2 Type II attestation report from an accredited auditor",
  "content_types": ["application/pdf"],
  "metadata_schema": {
    "type": "object",
    "properties": {
      "report_period_start": { "type": "string", "format": "date" },
      "report_period_end": { "type": "string", "format": "date" },
      "auditor": { "type": "string" },
      "trust_services_criteria": {
        "type": "array",
        "items": { "enum": ["security", "availability", "processing_integrity", "confidentiality", "privacy"] }
      }
    },
    "required": ["report_period_start", "report_period_end", "auditor"]
  },
  "controls": ["SOC2-CC1.1", "SOC2-CC1.2"]
}

Type Fields

Type Examples

Types MAY include an examples array for collector conformance testing. Each example provides a sample input (mocked API response) and expected output (valid artifact), enabling automated verification that collector implementations produce correct output.

{
  "id": "evidencepack/vcs-config@v1",
  "name": "Version Control Configuration",
  "examples": [
    {
      "name": "github-org-mixed",
      "description": "GitHub organization with mixed branch protection",
      "input_url": "https://registry.evidencepack.org/examples/vcs-config/github-org/input.json",
      "output_url": "https://registry.evidencepack.org/examples/vcs-config/github-org/output.json"
    }
  ]
}

Conformance testing: Collectors are conformant for a type when they produce output matching the expected output for all examples. The test procedure is:

1. Fetch the example input from input_url
2. Run the collector with the input
3. Validate the collector's output against the type's metadata_schema
4. Compare the collector's output to the expected output from output_url

Resolution & Registry

Types, profiles, and overlays are resolved from configured registries or local cache. A registry is an HTTPS endpoint with these URL patterns:

# Registry URL structure
GET https://registry.example.com/types/{namespace}/{name}@v{N}
GET https://registry.example.com/profiles/{namespace}/{name}@v{N}
GET https://registry.example.com/overlays/{namespace}/{name}@v{N}

# Example: fetch a type definition
GET https://registry.evidencepack.org/types/evidencepack/soc2-report@v1

# Response (same structure for types, profiles, overlays)
{
  "id": "evidencepack/soc2-report@v1",
  "digest": "sha256:abc123...",
  "definition": { ... }
}

The digest is the SHA-256 of the JCS-canonicalized definition bytes. Validators should compute the digest locally and verify it matches the registry-provided value.

Profiles

A profile is a named contract that declares what a pack must contain. Profiles reference types and specify requirements (required vs optional, cardinality, freshness).

Profile Structure

{
  "id": "evidencepack/soc2-basic@v1",
  "name": "SOC 2 Basic",
  "description": "Minimum evidence for a SOC 2 security review",
  "requirements": [
    {
      "type": "evidencepack/soc2-report@v1",
      "required": true,
      "cardinality": { "min": 1, "max": 1 },
      "freshness": { "max_age_days": 365 }
    },
    {
      "type": "evidencepack/pentest-report@v1",
      "required": true,
      "cardinality": { "min": 1 },
      "freshness": { "max_age_days": 365 }
    },
    {
      "type": "evidencepack/security-policy@v1",
      "required": false
    }
  ]
}

Requirement Fields

Profile Resolution

When validating a pack, the validator must resolve its declared profile:

1. Parse the profile field from the manifest
2. If profile_lock contains a matching id, verify the fetched profile's digest matches
3. Fetch the profile from a configured registry or local cache
4. If digest verification fails, reject validation (do not substitute)

Overlays

Overlays are additive constraints that modify a base profile. They're useful for expressing cross-cutting requirements (like HIPAA) that apply on top of existing profiles.

Overlay Structure

{
  "id": "evidencepack/hipaa-overlay@v1",
  "name": "HIPAA Overlay",
  "description": "Additional requirements for HIPAA compliance",
  "add_requirements": [
    {
      "type": "evidencepack/baa-agreement@v1",
      "required": true
    },
    {
      "type": "evidencepack/hipaa-risk-assessment@v1",
      "required": true,
      "freshness": { "max_age_days": 365 }
    }
  ],
  "modify_requirements": [
    {
      "type": "evidencepack/pentest-report@v1",
      "freshness": { "max_age_days": 180 }
    }
  ]
}

Merge Semantics

Overlays are applied in declared order. For each overlay:

• add_requirements -Append new requirements to the effective profile. If the type already exists, emit DUPLICATE_REQUIREMENT_TYPE error.
• modify_requirements -Update existing requirements (matched by type). Targets the effective profile at application time, so overlay B can modify a requirement added by overlay A.
• Modifications are shallow-merged: specified fields override, unspecified fields are preserved.
• If an overlay modifies a type not in the effective profile, emit OVERLAY_TARGET_NOT_FOUND error.

Semantic Validation

Semantic validation checks whether a pack satisfies its declared profile. This runs after structural validation (core spec Level 1) succeeds.

Validation Procedure

Semantic validation proceeds in eight steps:

1. Build lock index -Index profile_lock entries by id. Emit DUPLICATE_LOCK_ENTRY on duplicates.
2. Resolve profile -Resolve and verify digest against lock. Emit PROFILE_NOT_FOUND or DIGEST_MISMATCH.
3. Resolve and merge overlays -For each overlay in order, resolve, verify lock, and merge. Emit OVERLAY_TARGET_NOT_FOUND if modifying missing type.
4. Resolve types -For each type in effective profile, resolve and verify lock.
5. Check unpinned dependencies -Emit UNPINNED_DEPENDENCY warning for dependencies not in profile_lock.
6. Index artifacts -Group artifacts by semantic_type. Emit UNTYPED_ARTIFACT warning for missing type.
7. Check requirements -For each requirement: check cardinality, freshness, content_type, and metadata.
8. Check unknown artifacts -Emit UNKNOWN_ARTIFACT warning for typed artifacts not matching any requirement.

// Simplified pseudocode (see spec for full details)
function validateSemantic(pack):
  findings = []

  // 1. Build lock index from profile_lock
  lockedDigests = indexByIdChecking­Duplicates(pack.profile_lock)

  // 2-4. Resolve profile, overlays, types (verify against locks)
  effectiveProfile = resolveLocked(pack.profile, lockedDigests)
  for overlayId in pack.overlays:
    effectiveProfile = merge(effectiveProfile, resolveLocked(overlayId))

  // 6. Index artifacts by semantic_type
  artifactsByType = {}
  for artifact in pack.artifacts:
    if artifact.semantic_type:
      artifactsByType[artifact.semantic_type].append(artifact)
    else:
      findings.add(warning, UNTYPED_ARTIFACT, artifact.path)

  // 7. Check each requirement
  for req in effectiveProfile.requirements:
    artifacts = artifactsByType[req.type] or []
    minCount = req.cardinality?.min ?? (1 if req.required else 0)

    // Cardinality
    if minCount > 0 and len(artifacts) == 0:
      findings.add(error, MISSING_REQUIRED, req.type)

    // Freshness (uses collected_at, must be UTC Z format)
    if req.freshness?.max_age_days:
      for artifact in artifacts:
        if not artifact.collected_at:
          findings.add(error, MISSING_COLLECTED_AT)
        else if not validTimestamp(artifact.collected_at):
          findings.add(error, INVALID_COLLECTED_AT)
        else if daysSince(artifact.collected_at) > req.freshness.max_age_days:
          findings.add(error, STALE_ARTIFACT)

  return findings

Findings Format

Validation produces a list of findings. Each finding has a severity, message, and location:

{
  "findings": [
    {
      "severity": "error",
      "code": "MISSING_REQUIRED",
      "message": "Missing required artifact of type evidencepack/soc2-report@v1",
      "requirement": "evidencepack/soc2-report@v1"
    },
    {
      "severity": "error",
      "code": "STALE_ARTIFACT",
      "message": "Artifact is 400 days old, max allowed is 365",
      "path": "artifacts/pentest-2024.pdf",
      "requirement": "evidencepack/pentest-report@v1"
    },
    {
      "severity": "warning",
      "code": "UNKNOWN_ARTIFACT",
      "message": "Artifact not matched to any profile requirement",
      "path": "artifacts/misc/notes.txt"
    }
  ],
  "summary": {
    "errors": 2,
    "warnings": 1,
    "passed": false
  }
}

Finding Codes

Reference Profiles

Reference types, profiles, and overlays are published under the evidencepack/ namespace. Use these as starting points or reference implementations.

Test Vectors

Use these test vectors to verify your semantic validator implementation. Test vector files will be published in test-vectors/profiles/.