Skip to content

Module 10 — Anti-Analysis Techniques

Type 6 · Reconstruct — recover and defeat common Linux anti-analysis checks (ptrace self-debug, timing, /proc/cpuinfo hypervisor scan) via strace, an LD_PRELOAD hook, and CPU-flag manipulation, deliverable a documented bypass log plus a YARA rule on the evasion indicators proven against a benign control. (Secondary: Misconception Reveal — "nothing happened" actually means "it detected me.") Go to the hands-on lab →

Last reviewed: 2026-06

Malware Analysis — understand the tricks that slow you down, then bypass them so the analysis continues.

Difficulty: Advanced  ·  Estimated time: ~4–6 hrs (study + lab)  ·  Prerequisites: Foundations

In 60 seconds

Modern malware almost always carries at least one anti-analysis check, and the dangerous failure is misreading silence: "the sample did nothing" and "the sample detected me and went quiet" look identical but mean opposite things — the first clears it, the second is a missed threat. The checks fall into three families: environment, timing, and code-flow. You meet the canonical Linux ones — ptrace(PTRACE_TRACEME), a timing delta, a /proc/cpuinfo hypervisor scan — bypass each (strace, an LD_PRELOAD hook, a modified /proc), and write a rule that flags the evasion itself.

Why this matters

Modern malware almost always carries at least one anti-analysis check. Ransomware families check for analyst usernames and sandbox artefacts before encrypting; RATs sleep for ten minutes to outlast sandbox timeouts; rootkits detect ptrace and terminate. If you do not know what these checks look like, you cannot recognise them in decompiled code, and you cannot distinguish "the sample did nothing" from "the sample detected me and did nothing." The difference is significant: the first outcome clears the sample; the second means you missed the threat entirely. TrickBot is a documented real instance of the timing-evasion class: MITRE notes it uses "printf and file I/O loops to delay process execution as part of API hammering" — a time-based check (T1497.003) designed to burn through a sandbox's analysis window so the malicious behaviour never fires inside the timeout. An analyst who doesn't recognise the stall reports "benign — did nothing" on a live banking trojan. (TrickBot — MITRE ATT&CK S0266.)

Objective

Compile and run a C program that implements common Linux anti-analysis checks — a ptrace self-debug probe, a timing check, and a /proc/cpuinfo hypervisor flag scan — observe the checks firing, then bypass each one using strace, an LD_PRELOAD hook, and CPU flag manipulation, and document which bypass worked and why — then author a YARA rule on the anti-analysis indicators you recovered (the ptrace/PTRACE_TRACEME/timing/hypervisor strings or call sites) and prove it matches the demo binary but stays quiet on a benign control. Bypassing the checks and authoring a corpus-verified detection that flags the evasion itself are equal halves.

The core idea

The mental model

Anti-analysis falls into three families. Environment checks look for analyst artefacts — VM CPU flags in /proc/cpuinfo, sandbox usernames, short uptime, low screen resolution. Timing checks measure execution time between two points and bail if it's too long (a sign someone is single-stepping); TrickBot's printf/file-I/O delay loops are a sandbox-aimed variant — it stalls to outlast the analysis window rather than measure an interval. Code-flow tricks use exception handlers, self-modifying code, and TLS callbacks to create paths that only appear when no debugger is present.

The Linux ptrace(PTRACE_TRACEME) call is the canonical debugger check: a process can only be traced by one tracer at a time, so if ptrace(PTRACE_TRACEME, ...) returns -1, the process is already being debugged. Windows uses IsDebuggerPresent() which reads the PEB flag at a known offset, and NtQueryInformationProcess with class 7 (ProcessDebugPort). Both are trivially patchable — the defensive move is not to hide the debugger but to never let the check determine control flow unobserved.

The gotcha

The bypasses are not universal, and choosing the wrong one wastes the run. strace defeats some ptrace checks but the sample may detect strace itself; LD_PRELOAD hooks libc wrappers cleanly — unless the sample statically links libc (common in Go and Rust malware), where it does nothing and you need kernel-level interception or manual patching. And hypervisor detection via CPUID / /proc/cpuinfo reflects real hardware: you bypass it by changing the environment (a modified /proc/cpuinfo, hypervisor.cpuid.v0=FALSE, or bare metal), not by patching the binary.

Go deeper: the bypass hierarchy and freezing the clock

Work up the hierarchy: strace first (intercepts syscalls without a full debugger, so ptrace checks may not fire), then LD_PRELOAD (a preloaded .so returning success from ptrace() defeats most checks). Timing checks are defeated by normalising the clock: an LD_PRELOAD hook on clock_gettime/gettimeofday returning a fixed value makes the elapsed delta zero (GetTickCount/QueryPerformanceCounter on Windows). The insight is that the check compares two timestamps — manipulate either side and the comparison breaks in your favour.

Learn (~3 hrs)

Anti-debug fundamentals - MITRE ATT&CK T1497 — Virtualization/Sandbox Evasion — real-world usage examples with references to specific malware families; read the sub-techniques and "Mitigations" (~30 min). - MITRE ATT&CK T1027 — Obfuscated Files or Information — covers anti-analysis through code obfuscation and environment detection; understand the distinction between evasion and obfuscation (~20 min).

Linux-specific anti-analysis - Checkpoint Research — "Anti-Debug Tricks on Linux" — interactive catalogue of Linux-specific techniques with code examples and detection notes; work through ptrace, /proc checks, and timing sections (~40 min).

A real family that stalls the sandbox (~10 min) - TrickBot — MITRE ATT&CK S0266 — read the evasion techniques to see a documented timing check in the wild: printf/file-I/O delay loops (API hammering, T1497.003) that outlast a sandbox timeout. Grounds the timing-check family in a real banking trojan.

Bypasses - strace man page — syscall interception reference — focus on -e trace= filtering and -f for following forks (~20 min).

Key concepts

  • Anti-analysis checks fall into: environment checks, timing checks, code-flow tricks.
  • ptrace(PTRACE_TRACEME) returns -1 if already traced — the simplest debugger check.
  • LD_PRELOAD hooks libc wrappers before the syscall; works only against dynamically linked binaries.
  • Timing checks compare two timestamps — intercept clock_gettime to freeze the delta.
  • Hypervisor flags in /proc/cpuinfo are real hardware data; bypass requires modifying the environment, not the binary.
  • MITRE ATT&CK T1497 is the tagging for all sandbox/VM evasion techniques.
  • Real worked family: TrickBot (banking trojan) — uses printf/file-I/O delay loops (API hammering, T1497.003) to outlast a sandbox window; the timing-check class you bypass here is the same one it relies on
  • Author then verify: write the YARA rule on the anti-analysis indicators (with the ATT&CK tags) and prove it matches the sample, not the benign corpus — the build half

AI acceleration

Paste the anti-analysis C function into a model and prompt: "Identify every anti-debug or anti-VM check in this function and suggest the simplest bypass for each." The model is effective here because the check patterns are well-documented. Your job: confirm each bypass actually works by running the demo binary with and without it, not just by reading the suggestion.

AI caveat

A model spots well-documented check patterns reliably, but a suggested bypass is not a working one — static linking, a detected tracer, or real hardware flags can defeat it. Confirm each by running the demo binary with and without the bypass, not by reading the suggestion.

Check yourself

  • Why is "the sample did nothing" the most dangerous possible result to report uncritically?
  • Name the three families of anti-analysis check and one concrete example of each.
  • Why does LD_PRELOAD defeat a ptrace check on a typical binary but fail on Go/Rust malware?

Comments

Sign in with GitHub to comment. Choose the type: Feedback (errors or suggestions on this page) · Hints (help for fellow learners — no spoilers) · General (anything else).