honeypot

ugallu-honeypot is a tripwire. The operator deploys decoy Secret and ServiceAccount objects (real K8s resources, never referenced by any real workload) and listens to the audit-detection event bus for any operation that touches them. The instant something does, an SE fires.

Two detectors

• HoneypotTriggered - an audit event names a known decoy. This is a real-time access detection (get, update, token request, exec into the SA’s pods, etc.).
• HoneypotMisplaced - the deployer reconciler notices that a decoy lives somewhere it shouldn’t (e.g. a copy got created in a namespace that wasn’t in the config). Hints at lateral movement by an attacker who tried to hide a trail.

Decoy shape

HoneypotConfig.spec.decoys declares the decoys to materialise:

apiVersion: security.ugallu.io/v1alpha1
kind: HoneypotConfig
metadata: { name: default, namespace: ugallu-system }
spec:
  decoys:
    - kind: Secret
      namespace: prod
      name: aws-prod-creds
      labels: { ugallu.io/honeypot: "true" }
    - kind: ServiceAccount
      namespace: prod
      name: payments-deployer
      labels: { ugallu.io/honeypot: "true" }

The operator owns the decoy lifecycle - it creates them on first reconcile, updates labels/annotations on drift, and deletes them when the HoneypotConfig removes the entry. The decoys carry an ugallu.io/honeypot=true label so even a casual kubectl get -l won’t trip them up by accident; an attacker with get on the namespace will see them.

Topology

Two replicas, leader election on. The deployer index is authoritative on the leader; followers maintain a hot cache so a leader flip doesn’t drop in-flight detector matches.

Audit-bus subscription

The operator opens a gRPC stream against the audit-detection event bus (--audit-bus-endpoint, bearer token from --audit-bus-token). Every audit event with an objectRef.resource of secrets / serviceaccounts is hashed against the decoy index - O(1) match.

Internals

State machine

HoneypotConfig has no phase. The reconciler treats every CR write as a diff request between spec.decoys and the in-memory deployer index, and reflects deployment outcomes on status.deployedDecoys[] plus per-namespace counters.

Reconcile loop (Config + decoys)

on each HoneypotConfig event:
  cfg := Get(req)
  desired := cfg.Spec.Decoys
  current := index.Snapshot()
  for d in (desired - current): create Secret/SA + index.Add(d)
  for d in (current - desired): delete Secret/SA + index.Remove(d)
  for d in (desired & current with drift): patch Secret/SA labels
  patch Status.DeployedDecoys, Status.LastReconcileAt

Reconcile loop (audit-bus dispatcher)

goroutine, started as manager Runnable:
  open gRPC stream against --audit-bus-endpoint
  for event in stream:
    if event.objectRef.resource not in [secrets, serviceaccounts]: continue
    if entry := index.Lookup(event.objectRef.namespace, name); entry != nil:
      emitSE(HoneypotTriggered, critical, subject=entry)
    if entry := index.LookupByName(event.objectRef.name); entry != nil:
      if entry.Namespace != event.objectRef.namespace:
        emitSE(HoneypotMisplaced, high, subject=entry)

Error recovery

Operator restart: the index is rebuilt from scratch by re-Listing the existing Secrets / ServiceAccounts that carry the ugallu.io/honeypot=true label. Decoys themselves are durable in etcd, so no decoy is ever lost across restarts. The audit-bus gRPC stream reconnects with exponential backoff.

Crash recovery scenario

Pod killed after creating decoy Secrets but before flushing the status update: new pod restarts, re-Lists labelled Secrets, finds them already in cluster, rebuilds the in-memory index, patches the status on the next reconcile. No duplicate decoys are created.

Edge cases

• Two replicas + leader election. Only the leader writes decoys / patches status; followers maintain a hot index for the audit-bus dispatcher so a leader flip doesn’t drop in-flight matches.
• Index ordering. The dispatcher checks HoneypotTriggered first, then HoneypotMisplaced - so a triggered decoy always fires the higher-impact event.
• Allowlisted actors. ServiceAccount names listed in spec.allowlistedActors skip both detectors (cluster-admin drills don’t generate paging events).

Full RBAC

# ClusterRole
rules:
  - apiGroups: [security.ugallu.io]
    resources: [honeypotconfigs, honeypotconfigs/status]
    verbs: [get, list, watch, update, patch]
  - apiGroups: [security.ugallu.io]
    resources: [securityevents]
    verbs: [create]
  - apiGroups: [""]
    resources: [secrets, serviceaccounts]
    verbs: [get, list, watch, create, update, patch, delete]
  - apiGroups: [""]
    resources: [namespaces, pods, configmaps]
    verbs: [get, list, watch]   # detector enrichment
  - apiGroups: [""]
    resources: [events]
    verbs: [create, patch]
# Namespaced Role
  - apiGroups: [coordination.k8s.io]
    resources: [leases]
    verbs: [get, list, watch, create, update, patch, delete]

CRDs owned

• HoneypotConfig - singleton or per-namespace; status carries deploy / drift counters.

Key flags

--cluster-id, --cluster-name, --audit-bus-endpoint, --audit-bus-token.

Deployment

Deployment (2 replicas) in ugallu-system, leader election on, priorityClassName=system-cluster-critical. RBAC: CRUD on Secret / ServiceAccount cluster-wide (decoy lifecycle).

Telemetry

ugallu_honeypot_decoys_active{kind}, ugallu_honeypot_triggers_total{kind,namespace}, ugallu_honeypot_misplaced_total{kind}, ugallu_honeypot_audit_bus_events_total.