created by AnotherOne from @Pwn3rzs Telegram channel.
After finding potential vulnerabilities through fuzzing (Week 2) or patch diffing (Week 3), the next critical step is analyzing crashes to determine if they're exploitable. This week focuses on crash triage, debugger mastery, and techniques for identifying how to reach vulnerable code paths from attacker-controlled input.
Once you've confirmed a crash is exploitable and built a PoC, you'll be ready for Basic Exploitation in Week 5.
[!IMPORTANT]
Before any crash analysis, ensure you can reproduce the crash reliably.
A crash that only happens "sometimes" or "on the fuzzer's machine" is nearly impossible to analyze or exploit.
This section establishes the mandatory checklist for achieving reproduction fidelity.
Reproduction Fidelity Checklist
Before analyzing any crash, verify these match between discovery and analysis environments:
# Full ASAN options for crash analysisexport ASAN_OPTIONS="\abort_on_error=1:\symbolize=1:\detect_leaks=1:\disable_coredump=0:\halt_on_error=1:\print_stats=1:\check_initialization_order=1:\detect_stack_use_after_return=1:\quarantine_size_mb=256"# UBSAN optionsexport UBSAN_OPTIONS="\print_stacktrace=1:\halt_on_error=1:\suppressions=ubsan_suppressions.txt"# Symbolizer path (required for readable stack traces)export ASAN_SYMBOLIZER_PATH=$(command -v llvm-symbolizer)
glibc Allocator Tuning (Linux):
bash
# Enable glibc heap consistency checks (catch corruption early)export MALLOC_CHECK_=3# Modern glibc tunable interface (glibc 2.26+)export GLIBC_TUNABLES="\glibc.malloc.check=3:\glibc.malloc.perturb=165"# What these do:# MALLOC_CHECK_=3: Abort on heap corruption detection# glibc.malloc.perturb=165: Fill freed memory with 0xA5 (helps detect UAF)
Core Dump Configuration (Linux):
bash
# Enable unlimited core dumpsulimit -c unlimited# Verify core pattern (where dumps go)cat /proc/sys/kernel/core_pattern# For local dumps in CWD (temporary, affects system):# echo 'core.%e.%p' | sudo tee /proc/sys/kernel/core_pattern
ASLR Control (Linux - for deterministic analysis):
bash
# Check current ASLR statecat /proc/sys/kernel/randomize_va_space# 0 = disabled, 1 = conservative, 2 = full# Disable ASLR for current shell (temporary, per-process)setarch $(uname -m) -R ./target < crash_input# Or system-wide (DANGEROUS - only for isolated VMs):# echo 0 | sudo tee /proc/sys/kernel/randomize_va_space
Input Path Matching
The crash may behave differently depending on HOW input reaches the target:
bash
# If fuzzer used stdin:./target < crash_input# If fuzzer used file argument:./target crash_input# If fuzzer used network:cat crash_input | nc localhost 8080# WRONG: Mixing input paths can change behavior!# Fuzzer: ./target @@ (file)# You: ./target < crash (stdin) # May not reproduce!
Example: stdin vs file difference:
c
// Some programs behave differently:// - stdin may be line-buffered// - File may be memory-mapped// - Network may have different read chunk sizes// This can affect:// - Buffer contents at crash time// - Heap layout (different allocation patterns)// - Race conditions (timing changes)
Quick Reproduction Test Script
bash
#!/bin/bash# repro_test.sh - Verify crash reproductionCRASH_INPUT="$1"TARGET="$2"EXPECTED_SIGNAL="${3:-11}" # Default: SIGSEGV (11)echo "[*] Testing reproduction of $(basename $CRASH_INPUT)"echo "[*] Target: $TARGET"echo "[*] Expected signal: $EXPECTED_SIGNAL"# Set up environmentulimit -c unlimitedexport ASAN_OPTIONS="abort_on_error=1:symbolize=1"# Run 10 timesCRASHES=0for i in {1..10}; do timeout 5s $TARGET < "$CRASH_INPUT" 2>/dev/null EXIT_CODE=$? # Check for crash signal (128 + signal number) if [ $EXIT_CODE -gt 128 ]; then SIGNAL=$((EXIT_CODE - 128)) if [ $SIGNAL -eq $EXPECTED_SIGNAL ] || [ $SIGNAL -eq 6 ]; then ((CRASHES++)) fi fidoneecho "[*] Crash rate: $CRASHES/10"if [ $CRASHES -ge 9 ]; then echo "[+] Reproduction: RELIABLE"elif [ $CRASHES -ge 5 ]; then echo "[!] Reproduction: FLAKY - investigate environment"else echo "[-] Reproduction: FAILED - check environment checklist"fi
Windows SDK Debugging Tools (includes cdb.exe for command-line/batch analysis):
bash
# Option 1: Install via winget (Windows SDK)winget install --source winget --exact --id Microsoft.WindowsSDK.10.0.26100# Option 2: Download from Microsoft# https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/# During installation, select "Debugging Tools for Windows"# After installation, cdb.exe is located at:# C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe# Add to PATH for convenience (run as Administrator):setx PATH "%PATH%;C:\Program Files (x86)\Windows Kits\10\Debuggers\x64" /M# Or use full path in scripts:"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe" -z dump.dmp -c "!analyze -v; q"
Configure Symbol Path:
bash
# In WinDbg Settings -> Default Symbol Path, or:# In WinDbg command window:.sympath SRV*C:\Symbols*https://msdl.microsoft.com/download/symbols# Or set environment variable permanently (recommended):setx _NT_SYMBOL_PATH "SRV*C:\Symbols*https://msdl.microsoft.com/download/symbols"# Create symbols cache directorymkdir C:\Symbols# Reload symbols (in debugger).reload /f
[!WARNING]
Pwndbg is installed per-user in ~/.gdbinit. If you run sudo gdb, it uses root's home directory and won't find your pwndbg config. Solutions:
For crash analysis of your own compiled test programs, you typically don't need sudo. Only use sudo when attaching to system processes or analyzing setuid binaries.
bash
# Option 1: Use gdb as regular user (recommended for most analysis)cd ~/crash_analysis_labgdb ./vuln_no_protect -c core.dump# Option 2: If you MUST use sudo (e.g., attaching to privileged process)sudo -E gdb ./program # -E preserves your environment including HOME# Option 3: Install pwndbg for root as wellsudo su -cd /rootgit clone https://github.com/pwndbg/pwndbgcd pwndbg && ./setup.shexit# Option 4: Explicitly source pwndbg in sudo gdb sessionsudo gdb -ex "source /home/<YOUR_USER>/tools/pwndbg/gdbinit.py" ./program
Configuring Core Dumps on Linux:
bash
# Check current core dump configurationcat /proc/sys/kernel/core_pattern# Enable core dumps for current shell (recommended for learning)ulimit -c unlimited
[!TIP]
For the exercises in this course, you typically only need:
bash
ulimit -c unlimited # In your current shell
On modern Ubuntu/Debian with systemd, cores are handled by systemd-coredump even if you set ulimit.
Use coredumpctl to list and debug them.
[!WARNING]
Optional: Local core files in CWD (modifies system-wide settings)
If you specifically need core files in your working directory instead of systemd-coredump:
bash
# This is SYSTEM-WIDE and may interfere with other toolingecho 'core.%e.%p' | sudo tee /proc/sys/kernel/core_pattern
Additional kernel settings that affect core dumps:
kernel.core_uses_pid: Append PID to core filename
fs.suid_dumpable: Controls dumps for setuid binaries (0=disabled, 1=enabled, 2=suidsafe)
Create these vulnerable C programs to generate real crashes:
bash
# Create a directory for crash analysis practicemkdir -p ~/crash_analysis_lab/{src,crashes,cores}cd ~/crash_analysis_lab/src
vulnerable_suite.c - Save this file for testing multiple vulnerability types:
c
// ~/crash_analysis_lab/src/vulnerable_suite.c - Compile with different flags for different exercises#include <stdio.h>#include <stdlib.h>#include <string.h>// 1. Stack Buffer Overflowvoid stack_overflow(char *input) { char buffer[64]; printf("[*] Copying input to 64-byte buffer...\n"); strcpy(buffer, input); // No bounds check! printf("[*] Buffer: %s\n", buffer);}// 2. Heap Buffer Overflowvoid heap_overflow(char *input) { char *buf = malloc(32); printf("[*] Allocated 32 bytes at %p\n", buf); strcpy(buf, input); // Overflow heap buffer printf("[*] Buffer: %s\n", buf); free(buf);}// 3. Use-After-Freevoid use_after_free() { char *ptr = malloc(64); strcpy(ptr, "Hello, World!"); printf("[*] Allocated at %p: %s\n", ptr, ptr); free(ptr); printf("[*] Freed, now accessing...\n"); printf("[*] UAF read: %s\n", ptr); // UAF read - may print stale data ptr[0] = 'X'; // UAF write - may corrupt allocator state}// 4. Double Freevoid double_free() { char *ptr = malloc(64); printf("[*] Allocated at %p\n", ptr); free(ptr); printf("[*] First free done\n"); free(ptr); // Double free!}// 5. NULL Pointer Dereferencevoid null_deref(int trigger) { char *ptr = trigger ? malloc(10) : NULL; printf("[*] ptr = %p\n", ptr); *ptr = 'A'; // NULL deref if trigger is 0}void print_usage(char *prog) { printf("Usage: %s <test_num> [input]\n", prog); printf("Tests:\n"); printf(" 1 <input> - Stack overflow (need ~100+ chars)\n"); printf(" 2 <input> - Heap overflow (need ~50+ chars)\n"); printf(" 3 - Use-after-free\n"); printf(" 4 - Double free\n"); printf(" 5 <0|1> - NULL deref (0=crash)\n"); printf("\nExample: %s 1 $(python3 -c \"print('A'*100)\")\n", prog);}int main(int argc, char **argv) { if (argc < 2) { print_usage(argv[0]); return 1; } int test = atoi(argv[1]); switch(test) { case 1: if (argc<3) return 1; stack_overflow(argv[2]); break; case 2: if (argc<3) return 1; heap_overflow(argv[2]); break; case 3: use_after_free(); break; case 4: double_free(); break; case 5: if (argc<3) return 1; null_deref(atoi(argv[2])); break; default: print_usage(argv[0]); return 1; } return 0;}
Build the test suite:
bash
cd ~/crash_analysis_lab/src# 1. Build WITHOUT mitigations (for basic crash analysis)gcc -g -fno-stack-protector -no-pie -z execstack \ vulnerable_suite.c -o ../vuln_no_protect# 2. Build WITH ASAN (for detailed memory error reports)gcc -g -O1 -fsanitize=address -fno-omit-frame-pointer \ vulnerable_suite.c -o ../vuln_asan# 3. Build with standard protections (see how mitigations affect crashes)gcc -g vulnerable_suite.c -o ../vuln_protected
Generate your first crashes:
bash
cd ~/crash_analysis_lab# Enable core dumpsulimit -c unlimited# Test 1: Stack overflow - generates a core dump./vuln_no_protect 1 $(python3 -c "print('A'*200)")# You should see: Segmentation fault (core dumped)# Check for core file: ls -la core* (if core_pattern writes to CWD) or use coredumpctl (systemd systems) or look at output of `cat /proc/sys/kernel/core_pattern`# Test 2: Stack overflow with ASAN - detailed report./vuln_asan 1 $(python3 -c "print('A'*200)") 2>&1 | tee crashes/stack_asan.txt# ASAN prints detailed overflow information# Test 3: Use-after-free with ASAN./vuln_asan 3 2>&1 | tee crashes/uaf_asan.txt# Test 4: NULL dereference - generates core dump./vuln_no_protect 5 0
Using coredumpctl (systemd systems):
bash
sudo apt install systemd-coredump# List recent core dumpscoredumpctl list# Show details of most recent crashcoredumpctl info# Debug most recent crash with GDBcoredumpctl debug# Debug specific crash by PIDcoredumpctl debug 12345# Extract core dump to file for offline analysiscoredumpctl dump -o crash.core# View where cores are storedcat /etc/systemd/coredump.conf# [Coredump]# Storage=external # 'external' = /var/lib/systemd/coredump/# Compress=yes# MaxUse=1G # Max disk space for cores
[Coredump]# Where to store cores: external (disk), journal, or noneStorage=external# Compress with zstd/lz4Compress=yes# Maximum size for stored coresProcessSizeMax=2G# Maximum total disk usageMaxUse=5G# Keep cores for this longKeepFree=1G
After editing, reload: sudo systemctl daemon-reload
[!NOTE]
ASAN often exits via SIGABRT, not SIGSEGV. This can be confusing when trying to capture core dumps.
bash
# ASAN default: aborts on error (SIGABRT = signal 6)# Core dumps may not be generated by default for SIGABRT# Method 1: Configure ASAN to allow core dumpsexport ASAN_OPTIONS="abort_on_error=1:disable_coredump=0"# Method 2: Check that coredumpctl captures SIGABRT# coredumpctl list# Should show crashes with signal=6 (SIGABRT)# Method 3: Use gdb to catch ASAN abortecho "1 $(python3 -c "print('A'*200)")" > crash_inputgdb ./vuln_asan(gdb) run < crash_input# ASAN prints report, then GDB catches SIGABRT(gdb) bt full # Get full backtrace# What "success" looks like with ASAN + core dump:# 1. ASAN prints detailed error report (allocation/free stacks)# 2. Program aborts with SIGABRT# 3. coredumpctl captures the core# 4. coredumpctl debug lets you examine state at abort
# install visual studio community# Open "x64 Native Tools Command Prompt for VS 2022"# Create lab directorymkdir C:\CrashAnalysisLab\srcmkdir C:\CrashAnalysisLab\dumpscd C:\CrashAnalysisLab\src# Save the source code above as vulnerable_suite_win.c, then:# 1. Build WITHOUT mitigations (for basic crash analysis)# /GS- disables stack cookies, /DYNAMICBASE:NO disables ASLRcl /Zi /Od /GS- vulnerable_suite_win.c /Fe:..\vuln_win.exe /link /DYNAMICBASE:NO /NXCOMPAT:NO# 2. Build WITH ASAN (Visual Studio 2019 16.9+ or VS 2022)cl /Zi /Od /fsanitize=address vulnerable_suite_win.c /Fe:..\vuln_asan.exe# 3. Build with standard protections (default mitigations)cl /Zi /Od vulnerable_suite_win.c /Fe:..\vuln_protected.exe
Generate your first Windows crashes:
bash
cd C:\CrashAnalysisLab# Test 1: Stack overflowvuln_win.exe 1 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA# Should crash with access violation# crash will be at C:\CrashDumps\# Test 2: Stack overflow with ASAN - detailed reportvuln_asan.exe 1 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA# ASAN prints detailed overflow information# Test 3: Use-after-free with ASANvuln_asan.exe 3# Test 4: NULL dereferencevuln_win.exe 5 0# Test 5: Double free (may not crash immediately without PageHeap)vuln_win.exe 4# Directory of C:\CrashDumps# 01/05/2026 03:11 PM <DIR> .# 01/05/2026 03:10 PM 9,879,181 vuln_win.exe.7452.dmp# 01/05/2026 03:11 PM 9,866,817 vuln_win.exe.7756.dmp# 01/05/2026 03:09 PM 10,543,599 vuln_win.exe.984.dmp
Using PowerShell to generate long strings:
bash
# PowerShell equivalent of Python one-linerscd C:\CrashAnalysisLab# Generate 200 'A' characters$payload = "A" * 200# Test stack overflow.\vuln_win.exe 1 $payload# Test with ASAN.\vuln_asan.exe 1 $payload 2>&1 | Tee-Object -FilePath C:\CrashDumps\stack_asan.txt# Test UAF with ASAN.\vuln_asan.exe 3 2>&1 | Tee-Object -FilePath C:\CrashDumps\uaf_asan.txt
Verify crashes are captured:
bash
# If WER LocalDumps is configured (see next section), check:dir C:\CrashDumps\# Or use Event Viewer:# Windows Logs -> Application -> Look for "Application Error" events
# Check WER service statusGet-Service WerSvc# Check LocalDumps configurationGet-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps"
Sysinternals ProcDump
ProcDump provides more control than WER and catches crashes in real-time:
Basic Crash Capture (using our test binary):
bash
winget install Microsoft.Sysinternals.Suite# First, ensure you've built the test suite (see "Building a Windows Vulnerable Test Suite" above)cd C:\CrashAnalysisLab# Options:# -ma : Full memory dump (recommended)# -e : Write dump on unhandled exception# -x : Launch and monitor (below)# Launch and monitor for crashesprocdump -ma -e -x dumps vuln_win.exe 1 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA# Monitor already-running processprocdump -ma -e -p <PID>
Advanced ProcDump Usage:
bash
cd C:\CrashAnalysisLab# Capture on first-chance exceptions (catches more bugs)procdump -ma -e 1 -x dumps vuln_win.exe 1 AAAA...# Capture on specific exception codesprocdump -ma -e 1 -f C0000005 -x dumps vuln_win.exe 5 0 # Access violation (NULL deref)# Capture multiple dumps (for intermittent crashes)procdump -ma -e -n 5 -x dumps vuln_win.exe 3 # UAF - capture up to 5 dumps# Monitor service (generic example)# procdump -ma -e -x C:\Dumps -w ServiceName.exe
ProcDump + Fuzzing Integration:
bash
# Monitor fuzzing target (generic example)# procdump -ma -e -x C:\FuzzDumps -accepteula target.exe @@# Batch process dumps from fuzzing run# for %d in (C:\CrashAnalysisLab\dumps\*.dmp) do cdb -z "%d" -c "!analyze -v; q" >> analysis.txt
Batch Dump Triage with CDB
Analyze multiple dumps automatically:
bash
# Set CDB path (adjust version number as needed)set CDB="C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe"# Single dump analysis (use actual dump from ProcDump)%CDB% -z C:\CrashAnalysisLab\dumps\vuln_win.exe_XXXXXX.dmp# Or if cdb is in PATH:cdb -z C:\CrashAnalysisLab\dumps\vuln_win.exe_XXXXXX.dmp -c "!analyze -v; q"
Batch triage script (batch_triage.cmd):
bash
@echo off# Set path to cdb.exe (adjust if needed)set CDB="C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe"for %%f in (C:\CrashAnalysisLab\dumps\*.dmp) do ( echo ======================================== >> triage_report.txt echo Analyzing: %%f >> triage_report.txt echo ======================================== >> triage_report.txt %CDB% -z "%%f" -c ".symfix; .reload; !analyze -v; q" >> triage_report.txt 2>&1)echo Done! Results in triage_report.txt
PowerShell Batch Analysis:
bash
# batch_analyze.ps1# Path to cdb.exe - adjust if your Windows SDK version differs$cdb = "C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe"# Verify cdb existsif (-not (Test-Path $cdb)) { Write-Error "cdb.exe not found at $cdb. Install Windows SDK Debugging Tools." exit 1}$dumps = Get-ChildItem "C:\CrashAnalysisLab\dumps\*.dmp"$results = @()foreach ($dump in $dumps) { Write-Host "Analyzing $($dump.Name)..." $output = & $cdb -z $dump.FullName -c "!analyze -v; !exploitable; q" 2>&1 | Out-String # Extract key info $exploitable = if ($output -match "Exploitability Classification: (\w+)") { $Matches[1] } else { "Unknown" } $bugcheck = if ($output -match "EXCEPTION_CODE: \(NTSTATUS\) (0x[0-9a-f]+)") { $Matches[1] } else { "Unknown" } $results += [PSCustomObject]@{ DumpFile = $dump.Name Exploitability = $exploitable ExceptionCode = $bugcheck }}$results | Export-Csv "triage_results.csv" -NoTypeInformation$results | Format-Table -AutoSize
#Symbols and Symbolization (Linux Quick Reference)
Meaningful backtraces (GDB, CASR, ASAN reports) require symbols.
2. Install debug symbols for system libraries (real-world targets):
bash
# Ubuntu/Debian: prefer -dbg packages when available (example: libc6-dbg).# Some packages ship -dbgsym via Ubuntu's ddebs repository.# Fedora/RHEL:# sudo dnf debuginfo-install glibc
3. Use debuginfod for "fetch symbols on demand" (when local symbols unavailable):
bash
# Set URL for your distribution (GDB/LLDB will auto-fetch symbols)# Ubuntu:export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com"# Fedora:# export DEBUGINFOD_URLS="https://debuginfod.fedoraproject.org"# Generic fallback:# export DEBUGINFOD_URLS="https://debuginfod.elfutils.org/"# Note: If you install -dbgsym packages locally (recommended),# GDB uses those directly without needing debuginfod.
4. Symbolize raw addresses when you only have PCs:
bash
sudo apt install -y elfutils binutilscd ~/crash_analysis_lab# IMPORTANT: Full source info requires debug symbols (-g flag at compile time)# Verify with: file ./target (look for "with debug_info, not stripped")# Find function addresses in your binarynm ./target | grep -E " T " | head -5# Example output:# 00000000000012b0 T double_free# 0000000000001624 T _fini# 00000000000011e0 T heap_overflow# 0000000000001000 T _init# 00000000000013c0 T main# Symbolize using an address from nm output or a crash backtrace# (PIE binaries show low addresses; add the runtime base for live processes)addr2line -e ./target -f -C 0x12b0# With debug info (-g at compile time):# double_free# /home/dev/crash_analysis_lab/src/vulnerable_suite.c:37## Without debug info, you only get the function name:# double_free# ??:0# NOTE: eu-addr2line (from elfutils) may show ??:0 even with debug info# due to DWARF5 compatibility issues. Prefer addr2line (from binutils).# eu-addr2line -e ./target -f -C 0x12b0 # May not resolve line numbers# Dynamic lookup example:addr2line -e ./target -f -C $(nm ./target | grep " T main" | awk '{print $1}')
Without them, you're staring at hex addresses instead of function names.
This section provides best practices for both Windows and Linux.
Linux Symbol Management
1. debuginfod (Automatic Symbol Fetching):
debuginfod can automatically fetch debug symbols on-demand from public servers when you don't have them installed locally.
bash
# Install debuginfod clientsudo apt install debuginfod# Configure debuginfod URL for your distribution# Ubuntu:export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com"# Fedora:# export DEBUGINFOD_URLS="https://debuginfod.fedoraproject.org"# Arch:# export DEBUGINFOD_URLS="https://debuginfod.archlinux.org"# For GDB, enable automatic fetchingecho "set debuginfod enabled on" >> ~/.gdbinit# For LLDBexport LLDB_DEBUGINFOD_URLS="https://debuginfod.elfutils.org/"
[!IMPORTANT]
debuginfod vs local debug packages: debuginfod queries remote servers for symbols you don't have locally.
If you install debug symbol packages (e.g., coreutils-dbgsym), the symbols are stored locally at /usr/lib/debug/ and GDB uses them directly without needing debuginfod.
Verification: Don't use debuginfod-find to verify your setup—it only queries remote servers. Instead, verify GDB can find symbols:
bash
# Install local debug symbols (recommended for common packages)sudo apt install coreutils-dbgsym# Verify GDB finds the symbolsgdb -q -ex "file /usr/bin/ls" -ex "info sources" -ex "quit" 2>&1 | head -5# Expected output (with pwndbg you'll see its banner first, then):# Reading symbols from /usr/bin/ls...# Reading symbols from /usr/lib/debug/.build-id/xx/xxxxx.debug...# ... followed by source file paths like ls.c, hash.c, etc.
When to use debuginfod: debuginfod is useful when you're analyzing crashes in binaries where you haven't installed the -dbgsym package.
GDB will automatically fetch symbols from the configured server.
# addr2line (from binutils) is preferred for symbolization# NOTE: eu-addr2line (from elfutils) may show ??:0 even with debug info# due to DWARF5 compatibility issues. Prefer addr2line.# IMPORTANT: ASAN reports are ALREADY SYMBOLIZED!# If your ASAN output shows:# #0 0x59cc1877a53e in use_after_free src/vulnerable_suite.c:33# The file:line info (src/vulnerable_suite.c:33) is already there!# You do NOT need to run addr2line on ASAN output.## addr2line is only needed for:# - Raw core dumps without ASAN# - Stripped binaries with separate debug info# - Non-ASAN crash logs that only show addresses## If ASAN output shows "??:0" instead of file:line, fix symbolization:# sudo apt install llvm# export ASAN_SYMBOLIZER_PATH=$(which llvm-symbolizer)# # Then re-run the crash# For non-ASAN crashes, use STATIC addresses from nm (not runtime addresses):# Runtime addresses like 0x59cc1877a53e include PIE base and won't work!nm ./vuln_asan | grep "T use_after_free"# Output: 00000000000014a3 T use_after_free# Use the static address with addr2line:addr2line -e ./vuln_asan -f -C 0x14a3# Output:# use_after_free# /home/dev/crash_analysis_lab/src/vulnerable_suite.c:27# addr2line options:# -f: Show function names# -C: Demangle C++ symbols# -i: Show inlined functions# Example: Look up a function by name and symbolize itaddr2line -e ./vuln_asan -f -C $(nm ./vuln_asan | grep "T print_usage" | awk '{print $1}')# To convert runtime address to static (for PIE binaries):# 1. Get the binary's load base from /proc/<pid>/maps or ASAN output# 2. Subtract base from runtime address# Example: If base is 0x59cc18779000 and crash addr is 0x59cc1877a53e:# Static offset = 0x59cc1877a53e - 0x59cc18779000 = 0x153e# addr2line -e ./vuln_asan -f -C 0x153e# With debuginfod (for system binaries without local debug packages):DEBUGINFOD_URLS="https://debuginfod.ubuntu.com" \ addr2line -e /usr/bin/crashed_binary -f -C 0x12345
4. Verifying Symbol Quality:
bash
# Check if binary has debug symbolsfile target# Look for: "with debug_info, not stripped"# Check symbol table sizenm target | wc -l# Check DWARF info presencereadelf --debug-dump=info target | head -50# Verify specific function is symbolizednm target | grep stack_overflow
Windows Symbol Management
1. Configuring _NT_SYMBOL_PATH:
bash
# Set symbol path permanently (user environment)setx _NT_SYMBOL_PATH "srv*C:\Symbols*https://msdl.microsoft.com/download/symbols"# Or in current sessionset _NT_SYMBOL_PATH=srv*C:\Symbols*https://msdl.microsoft.com/download/symbols# Multiple symbol sources (local + Microsoft + custom server)set _NT_SYMBOL_PATH=C:\MySymbols;srv*C:\Symbols*https://msdl.microsoft.com/download/symbols;srv*C:\ThirdParty*https://symbols.example.com/
2. WinDbg Symbol Commands:
bash
# open C:\CrashAnalysisLab\vuln_win.exe in windbg# Quick setup for Microsoft symbols.symfix C:\Symbols# Add additional symbol path.sympath+ C:\CrashAnalysisLab.reload# Show current symbol path.sympath# Force reload all symbols.reload /f# Reload specific module# .reload /f ntdll.dll# Enable verbose symbol loading (debugging symbol issues)!sym noisy.reload /f# Disable noisy mode when done!sym quiet# Check symbol status for modulelm m ntdll# Look for: "pdb symbols" vs "export symbols" vs "no symbols"# Verify specific symbol loadsx ntdll!Rtl* # List all Rtl* functions - only works with symbols
3. Troubleshooting Symbol Issues:
bash
# Symbol loading failed? Check these:!sym noisy.reload /f vuln_win.exe# Common issues:# 1. Symbol server timeout → Use local cache# 2. PDB mismatch → Check build matches binary# 3. Private symbols missing → Request from vendor# Verify PDB matches binary!lmi target# Check: "Checksum" matches between .exe and .pdb# Force load unverified symbols (use with caution).symopt+ 0x40 # SYMOPT_LOAD_ANYTHING.reload /f.symopt- 0x40 # Disable after
Cross-Platform Symbol Checklist
Linux
debuginfod URL configured
Debug packages installed for target libraries
Binary built with -g flag
eu-addr2line available for batch symbolization
Windows
_NT_SYMBOL_PATH environment variable set
Symbol cache directory exists and writable
Microsoft symbol server accessible
PDB files match target binaries (same build)
Both Platforms
Third-party library symbols obtained
Symbol server accessible (or offline cache populated)
Test symbolization: verify backtrace shows function names
Command Window: Type commands here
Registers Window: View CPU register state
Disassembly Window: View assembly code at current IP
Memory Window: Inspect memory contents
Call Stack Window: View function call hierarchy
Locals/Watch Window: Inspect variables
Crash Scenario: Stack buffer overflow in vulnerable application
Load Crash Dump:
bash
# Open crash dump fileFile → Open Dump file → select C:\CrashAnalysisLab\dumps\xxx.dmp (or one of the crashes from linux)# Or from command linecd C:\CrashAnalysisLab\dumpswindbg -z xxx.dmp# Verify dump loaded!analyze -v#FILE_IN_CAB: vuln_win.exe_260105_151715.dmp#COMMENT:#*** procdump -ma -e -x dumps vuln_win.exe 1 #AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA#AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA#AAAAAAAAAA#*** Unhandled exception: C0000005.ACCESS_VIOLATION#NTGLOBALFLAG: 70#APPLICATION_VERIFIER_FLAGS: 0#CONTEXT: (.ecxr)#rax=00000000000000d7 rbx=000000000052e6b0 rcx=0000000000000000#rdx=0000000000010000 rsi=0000000000000000 rdi=00000000005342d0#rip=000000014000744a rsp=000000000014fed8 rbp=0000000000000000# r8=7ffffffffffffffc r9=0000000000000000 r10=0000000000000000#r11=000000000014fcd0 r12=0000000000000000 r13=0000000000000000#r14=0000000000000000 r15=0000000000000000#iopl=0 nv up ei pl nz na po nc#cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010204#vuln_win!stack_overflow+0x3a:#00000001`4000744a c3 ret#Resetting default scope#EXCEPTION_RECORD: (.exr -1)#ExceptionAddress: 000000014000744a (vuln_win!stack_overflow+0x000000000000003a)# ExceptionCode: c0000005 (Access violation)# ExceptionFlags: 00000000#NumberParameters: 2# Parameter[0]: 0000000000000000# Parameter[1]: ffffffffffffffff#Attempt to read from address ffffffffffffffff#PROCESS_NAME: vuln_win.exe#READ_ADDRESS: ffffffffffffffff#ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.#EXCEPTION_CODE_STR: c0000005#EXCEPTION_PARAMETER1: 0000000000000000#EXCEPTION_PARAMETER2: ffffffffffffffff#IP_ON_HEAP: 4141414141414141#The fault address in not in any loaded module, please check your build's rebase#log at <releasedir>\bin\build_logs\timebuild\ntrebase.log for module which may#contain the address if it were loaded.
Initial Analysis Commands:
bash
# Show registers at crashr# Display call stackkkv # Verbose with frame pointerkP # With full source paths (if symbols loaded)kn # With frame numbers# Show current instructionu @ripu @rip L10 # Disassemble 10 instructions# Examine stackdps @rspdps @rsp L50 # Display 50 pointer-sized values
Using the vuln_win.exe test suite from the "Building a Windows Vulnerable Test Suite" section, generate heap-related crashes:
bash
# Generate heap overflow crash (Test 2)cd C:\CrashAnalysisLab"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\gflags.exe" /p /enable vuln_win.exe /fullvuln_win.exe 2 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA#AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA#AAAAAAAAAA# Crash dump saved to C:\CrashDumps\vuln_win.exe.<PID>.dmp# Generate use-after-free crash (Test 3)vuln_win.exe 3# May not crash without PageHeap - see PageHeap lab below# Generate double-free crash (Test 4)vuln_win.exe 4# May not crash without PageHeap - see PageHeap lab below
Load and Analyze Heap Overflow Dump:
bash
# open dump! via GUI: File → Open Crash Dump → select the .dmp file# Initial analysis0:000> !analyze -v# Look for: EXCEPTION_CODE: c0000005 (Access violation)# Look for: heap_overflow or HeapFree in the stack
# Crash often occurs in HeapFree or subsequent allocation0:000> k# ntdll!RtlUserThreadStart$filt$0+0x3f# ntdll!_C_specific_handler+0x93# ntdll!RtlpExecuteHandlerForException+0xf# ntdll!RtlDispatchException+0x437# ntdll!KiUserExceptionDispatch+0x2e# vuln_win!__entry_from_strcat_in_strcpy+0x1f# vuln_win!heap_overflow+0x45# vuln_win!main+0xdb# Check heap state0:000> !heap -s # Summary of all heaps0:000> !heap -a 0 # Analyze default process heap# Check what was the destination buffer0:000> dq @rdx L8# See how far past the buffer you wrote(WRITE_ADDRESS from !analyze -v)0:000> !address 0x01fac000# Examine the vulnerable function0:000> uf vuln_win!heap_overflow# Check source if symbols are good0:000> lsa vuln_win!heap_overflow
Identifying UAF with vuln_win.exe:
bash
# First, enable PageHeap for better UAF detection (run as Administrator)#cd C:\CrashAnalysisLab#"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\gflags.exe" /p /enable /full vuln_win.exe# Now run the UAF test (Test 3)vuln_win.exe 3# With PageHeap, this will crash immediately on UAF access# Load the crash dumpwindbg -z C:\CrashDumps\vuln_win.exe.<PID>.dmp0:000> !analyze -v# Typical UAF crash pattern0:000> k # ChildEBP RetAddr # 00 ntdll!RtlpLowFragHeapFree+0x42 # 01 vuln_win!use_after_free+0x15 # 02 vuln_win!main+0x89# Check if address was recently freed (requires PageHeap) -(READ_ADDRESS)0:000> !heap -p -a 0x01fabfc0 address 0000000001fabfc0 found in _DPH_HEAP_ROOT @ 1c01000 in free-ed allocation ( DPH_HEAP_BLOCK: VirtAddr VirtSize) 1c0c820: 1fab000 2000 00007ffd1074b2d3 ntdll!RtlDebugFreeHeap+0x0000000000000037 00007ffd106e370c ntdll!RtlpFreeHeap+0x000000000000178c 00007ffd10739300 ntdll!RtlFreeHeap+0x0000000000000620 000000014000753d vuln_win!use_after_free+0x000000000000005d# Don't forget to disable PageHeap after analysis#"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\gflags.exe" /p /disable vuln_win.exe
Classification: Use-After-Free - object accessed after being freed.
0:000> r raxrax=00000000000000000:000> u @ripmov qword ptr [rax], rcx # Writing to NULL# Usually not exploitable unless kernel-mode
2. Access Violation (Invalid Address):
bash
0:000> r raxrax=deadbeefdeadbeef # Invalid address# Could be:# - Uninitialized pointer# - Freed memory# - Corrupted pointer
3. Stack Cookie Violation:
bash
0:000> kntdll!RtlReportCriticalFailurentdll!RtlpReportHeapFailure<Application>!__security_check_cookie<Application>!function_with_stack_cookie# Stack overflow detected, but mitigated by /GS
u <address> # Unassemble at addressu <address> L<count> # Unassemble count instructionsub <address> # Unassemble backwarduf <function> # Unassemble entire function
Breakpoints:
bash
bp <address> # Set breakpointbp <module>!<function> # Set breakpoint on functionba r 1 <address> # Hardware breakpoint on readba w 4 <address> # Hardware breakpoint on write (4 bytes)bl # List breakpointsbc * # Clear all breakpoints