Malware Analysis Workflow

Understanding malicious code behavior - reverse engineering malware to understand its capabilities, indicators of compromise, and create defenses.

IMPORTANT: This is a READ-ONLY workflow. The goal is understanding, not enhancement. Never execute malware outside isolated environments. Never improve or weaponize malicious code.

Trigger

Incident response (what did it do?)
Threat intelligence (understanding adversary TTPs)
Creating detection signatures
Security research
CTF challenges

Goal

Understand malware capabilities
Extract indicators of compromise (IOCs)
Document behavior for threat intelligence
Create detection rules
NOT: execute, enhance, or weaponize

Prerequisites

Isolated analysis environment (VM, sandbox)
Sample acquisition (legally obtained)
Analysis tools
Understanding of target platform
Network isolation (air-gapped if possible)

Safety First

BEFORE ANY ANALYSIS:

1. ☐ Environment isolated from production
2. ☐ Network air-gapped or monitored sandbox
3. ☐ Snapshot taken for rollback
4. ☐ Host isolation verified
5. ☐ Sample legally obtained
6. ☐ Permission documented

Environment Setup

bash

# VM-based isolation (VirtualBox, VMware, QEMU)
# Snapshot BEFORE importing sample
virsh snapshot-create-as malware-vm clean-state

# Network isolation options:
# 1. Disconnected entirely
# 2. INetSim (fake services)
# 3. Remnux/FLARE-VM with controlled networking

# Analysis VMs:
# - REMnux (Linux-based analysis)
# - FLARE-VM (Windows-based analysis)
# - Cuckoo Sandbox (automated analysis)

Analysis Types

Type	When	Tools
Static	Safe first look	strings, file, disassembler
Dynamic	Understand runtime	sandbox, debugger, monitor
Code Review	Source available	normal RE techniques
Memory	Running sample	volatility, process dump

Core Strategy: Triage → Static → Dynamic → Document

┌─────────────────────────────────────────────────────────┐
│                      TRIAGE                              │
│  Quick classification: type, target, obfuscation       │
└─────────────────────┬───────────────────────────────────┘
                      ▼
┌─────────────────────────────────────────────────────────┐
│                      STATIC                              │
│  Analyze without execution: strings, disassembly       │
└─────────────────────┬───────────────────────────────────┘
                      ▼
┌─────────────────────────────────────────────────────────┐
│                     DYNAMIC                              │
│  Controlled execution: sandbox, debugger               │
└─────────────────────┬───────────────────────────────────┘
                      ▼
┌─────────────────────────────────────────────────────────┐
│                     DOCUMENT                             │
│  IOCs, capabilities, detection rules                   │
└─────────────────────────────────────────────────────────┘

Phase 1: Triage

File Identification

bash

# What is it?
file sample.bin
# sample.bin: PE32 executable (GUI) Intel 80386, for MS Windows

# Hashes for lookup
md5sum sample.bin
sha256sum sample.bin

# Check VirusTotal (passive - just lookup)
vt file sample.bin

# Entropy (high = packed/encrypted)
ent sample.bin
# Entropy > 7.5 suggests packing

# File size
ls -la sample.bin

Quick Classification

bash

# Strings (visible text)
strings -n 8 sample.bin | head -100

# Look for indicators
strings sample.bin | grep -iE "http|https|\.exe|\.dll|cmd|powershell"
strings sample.bin | grep -iE "password|login|credential"
strings sample.bin | grep -iE "bitcoin|wallet|ransom"

# Import table (what APIs does it use?)
objdump -p sample.bin | grep -A100 "Import"
# or
rabin2 -i sample.bin

Packing Detection

bash

# Common packers leave signatures
# DIE (Detect It Easy)
diec sample.bin

# PEiD signatures (for PE files)
# UPX, Themida, VMProtect, etc.

# Manual check: section entropy
rabin2 -S sample.bin
# .text with entropy > 7 = likely packed

# Try UPX unpack (most common)
upx -d sample.bin -o unpacked.bin

Phase 2: Static Analysis

Disassembly

bash

# Ghidra (free, powerful)
ghidraRun &
# Import sample, analyze, review

# radare2 (CLI)
r2 -A sample.bin
> afl          # List functions
> pdf @main    # Disassemble main
> iz           # List strings
> ii           # List imports

# Binary Ninja, IDA Pro (commercial)

PE Analysis (Windows)

bash

# PE structure
pefile sample.bin
# or
pescan sample.bin

# Imports (capabilities)
rabin2 -i sample.bin
# VirtualAlloc, WriteProcessMemory = injection
# RegSetValue = persistence
# InternetOpen = network communication

# Exports (if DLL)
rabin2 -E sample.bin

# Resources (embedded files)
wrestool -x -o resources/ sample.bin

ELF Analysis (Linux)

bash

# Headers
readelf -h sample.bin

# Sections
readelf -S sample.bin

# Symbols
readelf -s sample.bin
nm sample.bin

# Dependencies
ldd sample.bin  # WARNING: can execute! Use in isolated env

Script Analysis

bash

# PowerShell deobfuscation
# Often base64 + compression
echo "EncodedCommand" | base64 -d

# Python malware
# Decompile pyc
uncompyle6 malware.pyc > malware.py

# JavaScript
# Beautify first
js-beautify malware.js > pretty.js

String Analysis

bash

# FLOSS - FLARE Obfuscated String Solver
floss sample.bin
# Extracts obfuscated strings via emulation

# Regex for IOCs
strings sample.bin | grep -oE '\b[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\b'  # IPs
strings sample.bin | grep -oE 'https?://[^\s\"<>]+' # URLs
strings sample.bin | grep -oE '[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z]{2,}' # Emails

Phase 3: Dynamic Analysis

Sandbox Execution

bash

# Cuckoo Sandbox (automated)
cuckoo submit sample.bin
cuckoo analysis {id}

# ANY.RUN (online, interactive)
# Hybrid Analysis (online)
# Joe Sandbox (online)

# IMPORTANT: Only use trusted sandboxes
# Never upload sensitive samples to public services

Manual Dynamic Analysis

bash

# Windows (in FLARE-VM)
# Procmon - API calls, file/registry activity
# Process Hacker - process inspection
# Wireshark - network traffic
# FakeNet-NG - fake network services

# Linux
strace -f ./sample.bin 2>&1 | tee strace.log
ltrace -f ./sample.bin 2>&1 | tee ltrace.log

# Network monitoring
tcpdump -i any -w capture.pcap &

Debugging

bash

# x64dbg/OllyDbg (Windows)
# Set breakpoints on suspicious APIs:
# - VirtualAlloc
# - CreateRemoteThread
# - WriteProcessMemory
# - InternetConnect

# GDB (Linux)
gdb ./sample.bin
> break main
> run
> info registers
> x/20i $pc

Memory Analysis

bash

# Dump process memory
procdump -ma {pid}

# Volatility (memory forensics)
volatility -f memory.dump --profile=Win10x64 pslist
volatility -f memory.dump --profile=Win10x64 malfind
volatility -f memory.dump --profile=Win10x64 netscan

Phase 4: Document Findings

IOC Extraction

markdown

## Indicators of Compromise

### File Hashes
- MD5: d41d8cd98f00b204e9800998ecf8427e
- SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

### Network Indicators
- C2 Server: 192.168.1.100:443
- Domain: evil.example.com
- User-Agent: Mozilla/5.0 (compatible; Malware/1.0)

### Host Indicators
- File created: C:\Windows\Temp\payload.exe
- Registry key: HKCU\Software\Microsoft\Windows\CurrentVersion\Run\Updater
- Mutex: Global\MalwareMutex123
- Service: FakeUpdater

### YARA Rule
```yara
rule MalwareFamily_Variant {
    meta:
        description = "Detects MalwareFamily variant"
        author = "Analyst"
        date = "2024-01-01"
    strings:
        $s1 = "UniqueString1"
        $s2 = "UniqueString2"
        $api1 = "VirtualAlloc"
        $api2 = "WriteProcessMemory"
    condition:
        uint16(0) == 0x5A4D and
        all of ($s*) and
        any of ($api*)
}


### Capability Summary

```markdown
## Malware Capabilities

### Initial Access
- Delivered via: Phishing email attachment

### Execution
- Executes embedded PowerShell
- Unpacks payload in memory

### Persistence
- Registry Run key
- Scheduled task

### Defense Evasion
- UPX packed
- API hashing to hide imports
- Process injection into explorer.exe

### Collection
- Captures keystrokes
- Screenshots every 60 seconds

### Exfiltration
- HTTP POST to C2
- Data encoded in base64

MITRE ATT&CK Mapping

markdown

| Tactic | Technique | ID |
|--------|-----------|-----|
| Execution | PowerShell | T1059.001 |
| Persistence | Registry Run Keys | T1547.001 |
| Defense Evasion | Process Injection | T1055 |
| Collection | Keylogging | T1056.001 |
| Exfiltration | Exfiltration Over C2 | T1041 |

Common Malware Patterns

Unpacking Stub

Packed malware often:
1. Allocates memory (VirtualAlloc)
2. Decrypts/decompresses payload to new memory
3. Transfers execution to unpacked code
4. May use process hollowing

Breakpoint strategy:
- BP on VirtualAlloc, note returned address
- BP on VirtualProtect (when making executable)
- Dump memory at that point

API Hashing

python

# Malware hides API names by hashing
# Common: ROR13, CRC32, djb2

def ror13_hash(name):
    h = 0
    for c in name:
        h = ((h >> 13) | (h << 19)) & 0xFFFFFFFF
        h += ord(c)
    return h

# Build hash table of common APIs to identify
api_hashes = {ror13_hash(name): name for name in ['LoadLibraryA', 'GetProcAddress', ...]}

String Encryption

python

# XOR encryption (common, simple)
def decrypt_xor(data, key):
    return bytes(b ^ key[i % len(key)] for i, b in enumerate(data))

# Stack strings (character-by-character)
# MOV [esp], 'h'
# MOV [esp+1], 't'
# MOV [esp+2], 't'
# MOV [esp+3], 'p'

C2 Communication

Common patterns:
- HTTP(S) with hardcoded domains
- DNS tunneling (long subdomains)
- Custom binary protocol
- Beaconing at regular intervals

Detection:
- Unusual DNS queries
- HTTP to IP addresses (no domain)
- Regular interval connections
- Large POST data

Tools Reference

Tool	Purpose
Ghidra	Disassembler/decompiler (free)
IDA Pro	Disassembler/decompiler (commercial)
x64dbg	Windows debugger
radare2/rizin	CLI reverse engineering
Cuckoo	Automated sandbox
YARA	Pattern matching
Volatility	Memory forensics
PEStudio	PE analysis
FLOSS	String extraction
CyberChef	Data transformation

LLM-Specific Techniques

Code Understanding

This is decompiled malware code (for analysis only):

```c
void func_401000() {
    char buffer[256];
    HMODULE h = GetModuleHandleA(0);
    GetModuleFileNameA(h, buffer, 256);
    // ... more code
}

Explain:

What this function does
Its likely purpose in malware context
What APIs to look for related functionality


### IOC Extraction

Given this network capture summary:

DNS query: aG9zdC5ldmlsLmNvbQ== (base64)
HTTP POST to 192.168.1.100/gate.php
User-Agent: Mozilla/5.0 (Windows NT 10.0)
POST body: id=ABC123&data=

Extract and decode all IOCs.


### YARA Rule Generation

Based on these unique strings found in malware:

"XorDecrypt"
"C:\Users\Public\payload.exe"
User-Agent: "MalBot/1.0"

And these API patterns:

VirtualAlloc followed by WriteProcessMemory

Generate a YARA rule for detection.


## Failure Modes

| Failure | Detection | Recovery |
|---------|-----------|----------|
| VM escape | Host compromise | Air-gapped physical machine |
| Anti-analysis | Sample detects VM | Anti-anti-VM techniques |
| Heavy obfuscation | Cannot understand | Emulation, dynamic analysis |
| Time-based evasion | Sleeps forever | Patch sleep calls |

## Anti-patterns

- **Executing on production systems**: Always isolated VMs
- **Uploading sensitive samples publicly**: Use private sandboxes
- **Trusting strings alone**: May be decoys or red herrings
- **Incomplete analysis**: Document everything, even failures
- **Sharing samples recklessly**: Follow responsible disclosure

## Legal and Ethical

IMPORTANT:

Sample acquisition must be legal
- Research agreements
- Incident response authorization
- CTF competitions
- Public malware repositories (VirusTotal, MalwareBazaar)
Never execute outside controlled environments
Never enhance or weaponize
Follow responsible disclosure for findings
Document authorization for all analysis


## Open Questions

### AI-Generated Malware

How to analyze malware written/modified by AI?
- May lack human patterns (signatures, strings)
- More sophisticated obfuscation
- Rapidly mutating variants

### Anti-Analysis Evolution

Malware increasingly detects analysis:
- Timing-based (real system is faster)
- Hardware checks (VM artifacts)
- Behavior-based (no user activity)

### Attribution

When is attribution possible?
- Code reuse from known groups
- Infrastructure overlap
- Consistent TTPs
- But: false flags are common

## See Also

- [Reverse Engineering Binary](reverse-engineering-binary.md) - Binary format analysis
- [Cryptanalysis](cryptanalysis.md) - If malware uses crypto
- [Security Audit](security-audit.md) - Defensive perspective

Malware Analysis Workflow ​

Trigger ​

Goal ​

Prerequisites ​

Safety First ​

Environment Setup ​

Analysis Types ​

Core Strategy: Triage → Static → Dynamic → Document ​

Phase 1: Triage ​

File Identification ​

Quick Classification ​

Packing Detection ​

Phase 2: Static Analysis ​

Disassembly ​

PE Analysis (Windows) ​

ELF Analysis (Linux) ​

Script Analysis ​

String Analysis ​

Phase 3: Dynamic Analysis ​

Sandbox Execution ​

Manual Dynamic Analysis ​

Debugging ​

Memory Analysis ​

Phase 4: Document Findings ​

IOC Extraction ​

MITRE ATT&CK Mapping ​

Common Malware Patterns ​

Unpacking Stub ​

API Hashing ​

String Encryption ​

C2 Communication ​

Tools Reference ​

LLM-Specific Techniques ​

Code Understanding ​

Malware Analysis Workflow

Trigger

Goal

Prerequisites

Safety First

Environment Setup

Analysis Types

Core Strategy: Triage → Static → Dynamic → Document

Phase 1: Triage

File Identification

Quick Classification

Packing Detection

Phase 2: Static Analysis

Disassembly

PE Analysis (Windows)

ELF Analysis (Linux)

Script Analysis

String Analysis

Phase 3: Dynamic Analysis

Sandbox Execution

Manual Dynamic Analysis

Debugging

Memory Analysis

Phase 4: Document Findings

IOC Extraction

MITRE ATT&CK Mapping

Common Malware Patterns

Unpacking Stub

API Hashing

String Encryption

C2 Communication

Tools Reference

LLM-Specific Techniques

Code Understanding