This Kubernetes detection lab shows how to validate runtime analytics without attempting a real container escape. You will build an isolated test workload, run harmless privilege and process checks, simulate a mounted-host discovery path, test service-account permissions, generate a controlled outbound request, and compare the resulting Kubernetes audit events, runtime logs, Falco findings, cloud alerts, and SIEM records. The same checklist can be repeated after cluster upgrades, admission-policy changes, runtime migrations, and scheduled purple-team exercises.
Overview
A useful container security test does not need to exploit a kernel, access a node filesystem, or retrieve production credentials. The objective is to produce observable behavior that resembles an early stage of suspicious activity while keeping every action bounded and reversible. This makes the exercise suitable for a payload emulation lab, SOC validation lab, or purple team lab.
The lab assumes a non-production Kubernetes cluster with a dedicated namespace, a test image supplied by your organization, and access to the relevant telemetry platforms. Do not run these checks in a namespace containing customer workloads or sensitive service accounts. Confirm that your monitoring team knows the test window, expected source namespace, test pod name, and cleanup plan.
Isolation checklist
- Create a namespace such as
detection-laband apply a resource quota. - Use a disposable service account with no unnecessary permissions.
- Apply a restrictive NetworkPolicy, then permit only the controlled destination required for the outbound test.
- Use an approved, minimal container image. Avoid images containing shells, package managers, cloud credentials, or offensive tooling unless the image has been reviewed for the lab.
- Record the cluster version, Kubernetes distribution, container runtime, Falco or equivalent agent version, and SIEM connector configuration.
- Define a stop condition: terminate the test if a command targets a node, production address, secret, or unapproved external system.
A simple starting point is kubectl create namespace detection-lab. Label the namespace so alerts can identify the exercise, for example with kubectl label namespace detection-lab purpose=security-validation. Treat the label as context, not as an exception that suppresses detections.
What to track
Track each test as a separate event rather than judging the lab only by whether an alert appeared. A detection may be technically correct but arrive without the fields an analyst needs. Capture the activity, expected evidence, actual evidence, alert latency, severity, and disposition.
| Lab activity | Safe implementation | Evidence to seek | Useful SIEM fields |
|---|---|---|---|
| Privilege checks | Run id, whoami, and a read-only check of the process security context. | Process execution, pod identity, UID, namespace, and container name. | cluster, namespace, pod, container, user, process, parent process, timestamp. |
| Unusual process launch | Start a benign child process supplied by the base image, such as sleep, from the test entrypoint. | Unexpected parent-child relationship or shell-to-process chain. | command line, executable, parent executable, image, pod labels. |
| Mounted-host discovery simulation | Mount an emptyDir volume at /simulated-host and list that directory. Do not mount the real node root. | Volume mount, directory enumeration, and workload metadata. | mount source, mount target, volume type, pod spec, user, syscall or file event. |
| Service-account access test | Use kubectl auth can-i with the disposable account, or request only an approved harmless resource. | Kubernetes API request, authorization decision, verb, resource, and subject. | user, groups, verb, resource, namespace, response status, user agent. |
| Controlled outbound request | Send one request to an approved internal sink or test endpoint and record the destination. | DNS, connection, egress policy, and workload identity. | destination, port, protocol, namespace, pod, policy decision, bytes, timestamp. |
Kubernetes audit logs are most valuable for API activity, such as service-account requests and workload changes. Container runtime logs and Falco events usually provide process, syscall, file, network, and container context. Managed-cluster security services may add workload, identity, image, or control-plane findings, but their field names and defaults vary. Confirm the actual schema in your environment instead of copying a field name from another platform.
Step-by-step workflow
- Deploy the baseline. Start a short-lived pod with the approved image, the disposable service account, read-only root filesystem where practical, and no host namespaces. Verify that the pod reaches a ready state and note its UID.
- Generate harmless identity telemetry. Execute
idandwhoamiinside the pod. These commands establish whether process telemetry records the effective user and whether the runtime associates events with the correct workload. - Generate a process-chain event. Launch a benign process through the container's normal entrypoint. The purpose is to test parent-child visibility, not to imitate a malicious binary. Compare the event with the expected image entrypoint and command line.
- Simulate mounted-host discovery safely. Define an
emptyDirvolume namedsimulated-host, mount it at/simulated-host, create a harmless marker file, and list the directory. This tests whether analytics notice a workload inspecting a host-like path without exposing the node filesystem. - Test service-account boundaries. Run
kubectl auth can-i get pods --as=system:serviceaccount:detection-lab:lab-runner -n detection-labfrom an authorized operator context. Then test a deliberately denied action, such as reading a resource outside the namespace. Record both decisions and ensure the denied request is not treated as proof of compromise by itself. - Generate controlled egress. Permit one destination through the lab NetworkPolicy and send a single request to the approved sink. Confirm that the destination, pod, namespace, and policy decision are visible. Do not query cloud metadata endpoints or scan an address range.
- Collect before cleanup. Wait for the normal ingestion delay used by your environment, export alert IDs and raw events, and take screenshots only when they add context. Then delete the workload and verify that no test resources remain.
Cadence and checkpoints
Run the complete workflow quarterly as a baseline purple-team exercise. Run a focused subset monthly if your team changes detection content frequently. Repeat the relevant checks after a Kubernetes version upgrade, container runtime migration, admission-controller change, CNI or NetworkPolicy change, Falco rule update, SIEM parser change, or managed-cluster security-agent update.
Use the following checkpoints:
- Before the test: confirm namespace isolation, approved destinations, alert routing, audit-policy coverage, and the expected test pod identity.
- During the test: record command start times and correlate them with pod creation, API calls, runtime events, and network records.
- After each activity: confirm whether the event arrived, which detector matched, and whether the alert included enough context for triage.
- After cleanup: verify deletion, check for leftover roles, bindings, pods, volumes, network-policy exceptions, and test labels.
A coverage matrix is useful here. Link each activity to its intended analytic, expected data source, alert owner, severity, and last successful test date. For a broader workflow, pair this lab with a detection coverage matrix and a rule regression process so that changes are reviewed rather than silently accepted.
How to interpret changes
A missing alert does not always mean a missing control. First determine whether the source event was generated and ingested. If the raw Kubernetes audit event exists but the SIEM alert does not, investigate parsing, field mapping, time normalization, suppression, and rule scheduling. If no raw event exists, review audit-policy coverage, runtime instrumentation, agent health, and the container's permissions.
Classify results using four practical outcomes:
- Detected and actionable: the expected analytic fired with namespace, pod, image, identity, process, and destination context.
- Detected but noisy: the analytic fired, but normal workloads create the same pattern. Tune with stable context such as workload identity, namespace purpose, or approved image—not a broad exclusion for an entire behavior.
- Observed but not alerted: telemetry exists, but the rule or routing logic needs review.
- Not observed: the data source, runtime sensor, audit policy, or connector may not cover the activity.
For false-positive reduction, compare the lab event with at least one known-good workload performing a similar operation. Ask whether the analytic depends on a brittle command-line string, an inconsistent image name, or a field that changed after an agent update. Prefer detections that combine behavior with workload identity, privilege level, image provenance, and network context. Document the reason for every tuning change and rerun the original test afterward.
When to revisit
Revisit this Kubernetes detection lab on a monthly or quarterly cadence and whenever recurring data points change. The most important changes are cluster upgrades, runtime migrations, new admission policies, CNI changes, service-account redesigns, Falco or EDR rule updates, SIEM schema changes, and new managed-cluster defaults. These changes can alter event names, process visibility, audit stages, network records, or alert enrichment without changing the workload behavior being tested.
Before each rerun, refresh the audit-policy example, runtime field mappings, Falco rule syntax, managed-cluster assumptions, and MITRE ATT&CK technique references used by your coverage matrix. Do not assume that a rule written for one runtime or distribution will behave identically in another.
Next action: create the isolated namespace, assign a disposable service account, document the five test activities, and run them once while an analyst watches the telemetry. Save the raw events and alert IDs as the baseline. On the next checkpoint, compare detection coverage, alert latency, field completeness, and cleanup results against that baseline. If the result changes, open a detection-engineering task, identify the broken stage, make the smallest justified correction, and rerun the same safe payload emulation workflow.