Linux Auditd Detection Lab: Validate Process Execution and Persistence Alerts Safely
Linux SecurityauditdTelemetryDetection EngineeringMITRE ATT&CKBlue TeamSIEM

Linux Auditd Detection Lab: Validate Process Execution and Persistence Alerts Safely

PPayloads Lab Editorial Team
2026-11-05
6 min read

Build a safe Linux auditd lab to validate process execution, persistence telemetry, and detection rules through repeatable checkpoints.

This Linux auditd detection lab gives blue teams a repeatable way to validate process-execution and persistence alerts without deploying real malware. You will build an isolated test host, collect audit records, run benign emulation actions, compare expected telemetry with your SIEM or EDR data, and record the results for future regression testing.

Overview

Linux process execution logs are valuable only when they arrive consistently, contain enough context, and produce an actionable detection. A successful test therefore checks more than whether a command ran. It verifies the complete path from endpoint configuration to auditd records, log forwarding, field normalization, analytics, alert routing, and analyst visibility.

This payload emulation lab is intentionally limited to harmless actions. It does not download tools, modify authentication controls, access credentials, scan other systems, or establish covert persistence. Use a disposable virtual machine or a dedicated laboratory host, take a snapshot before testing, and keep the system isolated from production networks. The exact audit fields and file paths can differ by distribution, auditd version, collection agent, and SIEM parser, so treat the examples as a starting point to validate in your environment.

The primary coverage areas are Linux process execution, shell and interpreter use, changes to common startup locations, and creation or modification of a user-level systemd unit. These behaviors can support MITRE ATT&CK technique simulations such as T1059.004, Unix Shell, and T1543.002, Systemd Service, when the surrounding context matches the technique. A benign test is not proof of malicious intent; it is a controlled way to confirm that telemetry and analytics respond to a recognizable behavior.

What to track

1. Lab and collector state

Record the operating system release, kernel version, auditd version, rules file location, log shipper, SIEM index or table, and the timestamp of each test. Confirm that the host clock is synchronized enough for event correlation. Note whether the endpoint uses /var/log/audit/audit.log, journald, or a collector-specific destination. Also record whether rules are loaded at boot or applied dynamically.

2. Process execution telemetry

Start with a harmless command that creates a distinctive process event:

mkdir -p /tmp/audit-lab
printf 'audit-lab-process\n' > /tmp/audit-lab/marker.txt
/bin/sh -c 'printf "child-process\n" > /tmp/audit-lab/child.txt'

For an auditd rule focused on execution, a common baseline is:

-a always,exit -F arch=b64 -S execve -k audit_lab_exec

Some systems also require a 32-bit architecture rule for legacy applications, while others use distribution-specific rule management. After loading the rule through the host's supported mechanism, confirm it is active with the local audit rule listing command. Search records for fields such as type=EXECVE, exe, comm, cmd, uid, auid, pid, and ppid. Do not assume every field will be present in every normalized event.

Test file monitoring with a dedicated, harmless path rather than modifying a real service or a user's existing startup files. For example:

mkdir -p /tmp/audit-lab/startup-test
printf '# benign lab marker\n' > /tmp/audit-lab/startup-test/marker.conf
printf '# updated benign marker\n' >> /tmp/audit-lab/startup-test/marker.conf

For a more representative user-level systemd exercise, use a disposable account and a unit that only writes a marker file, then remove it immediately after collection. Keep the unit under a laboratory home directory, do not enable it system-wide, and do not configure it to run with elevated privileges. Track the expected file creation, unit-loading, and process events separately. A file-integrity rule alone may show the change but not explain whether the file was loaded or executed.

4. Detection outputs

For every test, capture the raw event, normalized fields, rule name, alert timestamp, severity, destination, and analyst-facing description. A useful coverage record answers: did the event arrive, did the parser preserve the important fields, did the analytic match, and could an analyst distinguish the lab action from routine administration?

A Sigma-style process analytic might be expressed as follows, with field names adapted to the target backend:

title: Linux Auditd Suspicious Shell Execution in Lab Scope
logsource:
  product: linux
  service: auditd
detection:
  selection:
    type: EXECVE
    exe|endswith:
      - '/sh'
      - '/bash'
      - '/dash'
  condition: selection
level: low

This broad example is suitable for telemetry validation, not direct production deployment. Reduce noise by adding parent-process, user, path, command-line, and asset context. A persistence-oriented analytic can focus on changes to monitored startup or service paths:

title: Linux Auditd Write to Monitored Service or Startup Path
logsource:
  product: linux
  service: auditd
detection:
  path:
    name|startswith:
      - '/etc/systemd/system/'
      - '/etc/systemd/user/'
      - '/etc/cron.'
  change:
    type:
      - PATH
      - SYSCALL
  condition: path and change
level: medium

Because auditd records are commonly split across related event types, implement correlation in the SIEM where possible. A single PATH record may need to be joined with its SYSCALL, EXECVE, user identity, and process identifiers.

Cadence and checkpoints

Run the full test suite during initial deployment, after major Linux or kernel updates, and after changes to audit rules, endpoint agents, parsers, indexes, or detection content. A monthly or quarterly lightweight check is useful for environments where telemetry pipelines change frequently.

Use a checkpoint table with one row per emulation action:

  • Action: the exact benign command or file operation.
  • Expected record: event type, executable, user, path, and process identifiers.
  • Observed record: raw and normalized values.
  • Detection result: matched, missed, delayed, or generated a false positive.
  • Owner and date: person responsible for remediation and the next review.

Allow enough time between steps to distinguish events, but retain the original timestamps. If forwarding is asynchronous, measure the delay from endpoint generation to SIEM arrival. Repeat tests with a normal administrative account and, where appropriate, a dedicated service account so that identity-based conditions are exercised without using privileged credentials unnecessarily.

How to interpret changes

A missing alert does not immediately mean the rule is wrong. First determine where the chain failed. If no local audit event exists, inspect rule loading, architecture coverage, permissions, and audit backlog. If the local event exists but the SIEM has none, inspect the shipper, file permissions, rotation behavior, and filters. If the SIEM event arrives with missing fields, review parsing and field mapping before changing the analytic.

When an analytic fires too often, preserve the behavior signal and narrow the context rather than excluding an entire executable. Useful tuning dimensions include the initiating user, parent process, destination path, asset role, maintenance window, and whether a corresponding change ticket exists. Avoid broad exclusions for shells or systemd because those programs are common in both administration and abuse. Use suppression only when the benign pattern is stable, documented, and still observable for audit purposes.

Compare results with a detection coverage matrix so that one test can be mapped to endpoint telemetry, Sigma content, SIEM queries, and response procedures. The Detection Coverage Matrix Template can help organize this evidence. For format decisions, see Sigma vs YARA Rules; process and log analytics generally belong in Sigma or native SIEM logic, while YARA is more appropriate for file-content inspection.

When to revisit

Schedule a recurring review at least monthly or quarterly according to the rate of change in your Linux estate. Re-run the lab sooner when a distribution release, kernel, auditd package, log shipper, EDR agent, SIEM parser, rule version, or systemd configuration changes. Also revisit after an incident, a newly observed persistence pattern, a change in administrator workflows, or a move from one log backend to another.

At each review, refresh the test host snapshot, verify the current rule syntax and log paths, run the same safe actions, and compare results with the previous coverage record. Mark any change in event volume, field availability, alert latency, or false-positive rate. If behavior differs, preserve a raw sample and open a focused investigation rather than silently updating the rule.

Finish with three practical actions: confirm that process execution is visible, confirm that monitored persistence locations generate correlated records, and confirm that an analyst can explain the alert from the available fields. Keep the emulation commands, expected outcomes, and cleanup steps in version control. This turns a one-time Linux MITRE ATT&CK lab into a repeatable SOC validation lab that can expose telemetry regressions before they become detection gaps.

Related Topics

#Linux Security#auditd#Telemetry#Detection Engineering#MITRE ATT&CK#Blue Team#SIEM
P

Payloads Lab Editorial Team

Security Content Editors

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.