Capture forensic evidence on a detection

When a Detection / critical SecurityEvent lands on a pod-shaped subject, you usually want three things in the next 60 seconds:

1. the pod stops talking to anything (no more egress, no more lateral movement)
2. its filesystem is captured to write-once storage so an analyst can inspect it later
3. every step taken is signed and logged so the chain of custody is defensible

The forensics operator does all three, gated by an EventResponse chain that the attestor seals into in-toto bundles.

Prerequisites

• ugallu umbrella chart installed.
• A working WORM bucket (SeaweedFS, MinIO, or AWS S3 with Object Lock in COMPLIANCE mode). WORMConfig populated.
• The attestor configured with at least one signing mode (fulcio-keyless or openbao-transit).
• forensics-snapshot privileged namespace exists (ugallu-system-privileged).

Sanity check:

kubectl -n ugallu-system get deploy ugallu-forensics
kubectl -n ugallu-system-privileged get deploy ugallu-forensics-snapshot
kubectl get forensicsconfig default -o jsonpath='{.status.freezeBackend}'
# Cilium  (or CoreV1, depending on your CNI)

Configure the trigger predicate

The default predicate listens for Detection and Anomaly SE classes at high|critical severity and requires Status.Phase=Attested. You opt in to specific types via whitelistedTypes - an empty whitelist matches nothing.

apiVersion: security.ugallu.io/v1alpha1
kind: ForensicsConfig
metadata: { name: default }
spec:
  trigger:
    classes: [Detection, Anomaly]
    minSeverities: [high, critical]
    requireAttested: true
    namespaceAllowlist: []
    whitelistedTypes:
      - PrivilegedPodChange
      - HostPathMount
      - ExecIntoPod
      - HoneypotTriggered
      - CrossTenantSecretAccess
      - CrossTenantExec
  cleanup:
    autoUnfreezeAfter: 4h
  evidence:
    bucket: ugallu-forensics
    objectLock: COMPLIANCE
    retainDays: 365

kubectl apply -f forensics-config.yaml

Synthetic detection (test environment only)

Pick a target Pod that is safe to freeze - a throwaway Pod in a sandbox namespace, never anything serving real traffic:

kubectl create namespace forensics-sandbox
kubectl -n forensics-sandbox run forensics-victim --image=nginx --restart=Never
kubectl -n forensics-sandbox wait pod/forensics-victim --for=condition=Ready --timeout=60s

Now emit a synthetic SE:

apiVersion: security.ugallu.io/v1alpha1
kind: SecurityEvent
metadata:
  generateName: synth-
  namespace: ugallu-system
spec:
  type: HoneypotTriggered
  class: Detection
  severity: critical
  source:
    operator: ugallu-forensics-recipe
    cluster: sandbox
  subject:
    kind: Pod
    namespace: forensics-sandbox
    name: forensics-victim
  description: synthetic detection for the recipe

kubectl create -f synth.yaml

Once the attestor seals the bundle and flips SE.status.phase=Attested, the trigger predicate matches and forensics enters the pipeline.

Watch the pipeline

# every step is an EventResponse with a chain back-link
kubectl get eventresponses -A \
  --sort-by=.metadata.creationTimestamp \
  --selector=ugallu.io/incident-uid

# step labels
kubectl get eventresponse <name> -o yaml \
  | yq '.metadata.labels'
# ugallu.io/incident-uid: 1f4abc...
# ugallu.io/parent-er: ...
# ugallu.io/step: podfreeze | filesystem-snapshot | evidence-upload | podunfreeze

The pipeline runs three steps sequentially:

1. PodFreezeStep - applies a deny-all CiliumNetworkPolicy (or NetworkPolicy v1 on vanilla CNI) with egress carve-outs for DNS, the WORM endpoint, and the forensics namespace. Adds the label ugallu.io/frozen=<incident-uid> to the pod.

2. FilesystemSnapshotStep - injects an ephemeral container into the suspect pod. The container has only CAP_DAC_READ_SEARCH, walks /proc/<pid>/root, and tees a tar | gzip | sha256 stream to S3.

3. EvidenceUploadStep - builds a content-addressed manifest with the SHA-256 over the canonical JSON, writes it to s3://<bucket>/forensics/<incident>/manifest-<sha>.json under Object Lock, and references it from the IncidentCaptureCompleted SE.

Find the evidence

kubectl get securityevent -A \
  --field-selector spec.type=IncidentCaptureCompleted \
  -o yaml | yq '.items[].spec.evidence'

The evidence.url field contains the S3 path to the manifest. With S3 / SeaweedFS credentials:

aws --endpoint-url=https://worm.example.internal s3 cp \
  s3://ugallu-forensics/forensics/1f4abc/manifest-abc123.json \
  ./manifest.json
jq . manifest.json

The manifest references the snapshot tarball, the freeze evidence, and back-links each step’s EventResponse UID. Object Lock makes it impossible to overwrite or delete the manifest before the retention window expires.

Acknowledge the incident

When you’re done analysing, an authorised ServiceAccount stamps the IncidentCaptureCompleted SE:

kubectl -n ugallu-system annotate securityevent <name> \
  ugallu.io/incident-acknowledged=true

Forensics observes the annotation, runs PodUnfreezeStep, and the pod returns to its normal network policy. If you forget to ack, the cleanup.autoUnfreezeAfter timer (4h above) unfreezes the pod automatically. The deadline lives on the SE, so a controller restart honours the grace.

Cleanup

kubectl delete namespace forensics-sandbox
kubectl delete securityevent <synth-se-name>

The EventResponse chain stays in the cluster for the TTL window, then the ttl operator GCs them and writes an archive snapshot to WORM if archiveSnapshotEnabled: true.