Skip to main content
Compliance evaluators run entirely within your environment. No data leaves your infrastructure.

​
PIIEvaluator

Scans LLM outputs for personally identifiable information using local regex patterns. Passes when no PII is detected. Fails with a per-type breakdown. 
from multivon_eval import PIIEvaluator

suite.add_evaluators(PIIEvaluator())
suite.add_evaluators(PIIEvaluator(jurisdiction="gdpr"))
suite.add_evaluators(PIIEvaluator(jurisdiction="hipaa"))
suite.add_evaluators(PIIEvaluator(redact=True))
suite.add_evaluators(PIIEvaluator(patterns={"employee_id": r"EMP-\d{6}"}))
Parameters:
ParameterDefaultDescription
jurisdiction"all""gdpr", "ccpa", "pipeda", "hipaa", or "all"
patternsNoneAdditional {name: regex} patterns
redactFalseReplace PII with [REDACTED-TYPE] in reports
threshold1.0Pass threshold (default: any PII = fail)
HIPAA jurisdiction adds patterns for 13 of the 18 HIPAA Safe Harbor PHI identifiers: medical record numbers, health plan beneficiary numbers, VINs, fax numbers, admission/discharge dates, device identifiers, account numbers, NPI/DEA/license numbers, certificate numbers, and URLs. The 5 identifiers that cannot be reliably detected via regex — names, geographic subdivisions below state, photographs, biometrics, and arbitrary unique identifiers — require upstream de-identification before evaluation. Sample output: 
PII detected (2 type(s)):
  email: "user@company.com"
  phone_us: "555-123-4567"

​
SchemaEvaluator

Validates that LLM outputs conform to a Pydantic model or JSON Schema dict. Passes when output is valid JSON matching the schema. Fails with per-field error messages. 
from pydantic import BaseModel
from multivon_eval import SchemaEvaluator

class Summary(BaseModel):
    title: str
    score: float
    tags: list[str]

suite.add_evaluators(SchemaEvaluator(Summary))

# JSON Schema alternative
suite.add_evaluators(SchemaEvaluator({
    "type": "object",
    "required": ["title", "score"],
    "properties": {
        "title": {"type": "string"},
        "score": {"type": "number", "minimum": 0, "maximum": 1},
    }
}))
Parameters:
ParameterDefaultDescription
schemarequiredPydantic model class or JSON Schema dict
strictFalseFail on extra fields not in schema
threshold1.0Pass threshold (default: any field error = fail)
Strips markdown code fences automatically before parsing. Sample output: 
Schema validation failed:
  score: Input should be a valid number
  tags: Field required

​
ComplianceReporter

Not an evaluator — a report writer. Produces tamper-evident NDJSON audit trails. See Compliance & Privacy guide for full documentation.