Heap Leaks Detection Skill

1. Skill Overview

This skill investigates memory bloat, Vitest OOM crashes, RSS peak consumption, and heap snapshot deltas triggered during pnpm test execution. It implements a systematic forensic workflow to avoid drawing conclusions solely from raw RSS metrics.

Core Principle: Differences in snapshot object naming serve only as classification evidence, not definitive root cause conclusions — judgment is only confirmed after retainer chain or dominator tree analysis validates the hypothesis.

2. Core Capabilities

2.1 Test Memory Forensics Workflow

A complete standardized pipeline for reproducing and diagnosing test memory anomalies:

Reproduce Memory Pressure Scenarios

pnpm canvas:a2ui:bundle && \
OPENCLAW_TEST_MEMORY_TRACE=1 \
OPENCLAW_TEST_HEAPSNAPSHOT_INTERVAL_MS=60000 \
OPENCLAW_TEST_HEAPSNAPSHOT_DIR=.tmp/heapsnap \
OPENCLAW_TEST_WORKERS=2 \
OPENCLAW_TEST_MAX_OLD_SPACE_SIZE_MB=6144 \
pnpm test

Keep OPENCLAW_TEST_MEMORY_TRACE=1 enabled; the test wrapper prints per-file RSS summaries alongside heap snapshot generation
If the report targets a specific test shard or worker resource quota, retain the identical execution configuration for consistent reproduction

Resolve Actual Test Lane Identifiers

Extract real lane names from log lines starting with [test-parallel] start ... or via pnpm test --plan
Do not assume a single unit-fast lane; local test planning often splits workloads into multiple batches such as unit-fast-batch-*

Capture Multiple Interval Snapshots

Collect at minimum two spaced heap snapshots from the identical test lane
Compare snapshots generated under the same PID within the dedicated lane directory (e.g. .tmp/heapsnap/unit-fast-batch-2/)

2.2 Heap Snapshot Delta Comparison Tool

Dedicated analysis scripts for quantifying memory growth between capture points:

Direct Two-File Comparison

node .agents/skills/openclaw-test-heap-leaks/scripts/heapsnapshot-delta.mjs before.heapsnapshot after.heapsnapshot

Auto-select Earliest / Latest Snapshots Per PID

node .agents/skills/openclaw-test-heap-leaks/scripts/heapsnapshot-delta.mjs --lane-dir .tmp/heapsnap/unit-fast-batch-2

Common CLI arguments:

--top 40: Display the top 40 object types with the largest memory growth delta
--min-kb 32: Suppress delta entries smaller than 32KB to filter trivial noise
--pid 16133: Restrict analysis to a single target process ID

Analysis priority logic: Prioritize reviewing entries with the largest positive memory growth. Massive growth in module transpilation artifacts points to lane isolation flaws; massive growth in runtime business objects indicates genuine memory leaks. If object naming alone yields ambiguous signals, load both snapshots side-by-side in DevTools and audit retainer / dominator graphs before finalizing root cause statements.

2.3 Memory Growth Classification Framework

Standardized categorization to distinguish distinct memory pressure root causes:

Growth Type	Key Characteristics	Mitigation Approach
Retained Module Graph Bloat	Dominated by Vite/Vitest transpiled source strings, Module objects, system Contexts, bytecode, descriptor arrays, and property maps	Optimize test scheduling via timing and hotspot-aware workload peeling; resolve long-lived worker module retention
Genuine Runtime Leaks	Dominated by application business objects, in-memory caches, buffers, server handles, timers, mock state, SQLite connections, and runtime component state	Patch incomplete test teardown logic; fix missing `afterEach`/`afterAll` cleanup, unreset module state, orphaned global variables, unreleased database handles, and cross-test lingering event listeners / timers

If object type labels remain ambiguous, withhold definitive root cause claims and inspect retainer / dominator graphs in DevTools for validation.

2.4 Targeted Remediation Strategies

For Retained Module Graph Bloat

Prioritize scheduling adjustments driven by test timing and execution hotspots
Verify if problematic files are already indexed within test/fixtures/test-timings.unit.json
Only update test/fixtures/test-parallel.behavior.json when timing-based workload peeling fails to resolve bloat
Apply singletonIsolated tagging to files that run safely in isolation but inflate shared worker heaps
Explicitly flag files that should have been split out via timing rules but are missing entries in test/fixtures/test-timings.unit.json

For Genuine Runtime Leaks

Repair incomplete teardown logic in affected test suites or runtime modules
Audit for missing afterEach/afterAll hooks, unreset module state, orphaned global state, unreleased database connections, and event listeners / timers that persist across test files
Validate remediation with conclusive heap snapshot evidence

2.5 Runtime Closure Fix Validation Harness

The above workflow targets Vitest worker memory bloat diagnostics. A dedicated harness verifies whether runtime / closure cleanup logic fully releases captured state variables:

pnpm leak:embedded-run

This utility executes scripts/embedded-run-abort-leak.ts, iteratively running aborted embedded execution workflows N times within simulated function scopes, writing heap snapshots, and tracking live instance counts alongside RSS deltas via FinalizationRegistry to output PASS / FAIL verdicts.

Three supported operating modes:

closure-extracted (default): Post-fix production pattern (module-scoped helper isolation)
closure-inline: Pre-fix baseline pattern (closures nested inside runner logic, used for sensitivity benchmarking)
synthetic-leak: Intentional module-scoped object retention to validate the harness’s leak detection capability

Snapshots are stored under .tmp/embedded-run-abort-leak/ and analyzed using the same delta comparison script.

3. Primary Use Cases

Diagnose Vitest OOM crashes and excessive RSS memory spikes

Systematically trace root causes when pnpm test terminates due to out-of-memory errors or exhibits steadily rising RSS consumption
Differentiate genuine business logic leaks vs. retained module graph bloat

Eliminate false positives by distinguishing worker-lifetime transpilation artifacts from application-level object leaks via snapshot delta analysis and standardized classification
Validate closure and runtime cleanup patches

Use the dedicated pnpm leak:embedded-run harness to confirm closure teardown logic fully releases captured state references
Optimize parallel test scheduling

Identify memory-heavy test hotspots and update timing manifests to offload bloated workloads from shared worker lanes
Provide memory regression evidence for PR reviews

Attach heap snapshot delta analysis as supporting audit evidence for pull requests modifying memory-sensitive logic

4. Critical Guiding Principles

Never draw conclusions solely from RSS metrics: Always rely on heap snapshot data when available
Snapshot object naming patterns are classification evidence, not definitive proof; retainer / dominator graph analysis is mandatory to confirm root causes
Not all memory growth qualifies as a leak: Significant bloat within unit-fast or unit-fast-batch-* lanes typically stems from worker-lifetime module retention, not application code leaks
Prioritize snapshot evidence over subjective intuition: Consistent growth of identical retained object families across sequential snapshots on the same PID takes precedence; resolve ambiguous signals via retainer graph inspection
Resolve accurate test lane identifiers first — avoid assuming a single monolithic unit-fast lane
Capture a minimum of two spaced heap snapshots before reaching diagnostic conclusions
Use the dedicated embedded leak harness for runtime closure validation; do not repurpose the general test memory forensic workflow for this validation task

5. Mandatory Report Output Requirements

All analysis reports generated by this skill must include the following complete details:

Exact reproduction command used to trigger memory bloat
Target test lane and PID selected for snapshot comparison
Dominant retained object families identified in snapshot delta outputs
Final classification (genuine runtime leak / shared worker retained module bloat) plus confirmation via retainer / dominator analysis
Specific remediation patches or workload mitigation adjustments applied
Scope of validation performed, and any validation steps blocked by snapshot capture overhead

Summary

OpenClaw Test Heap Leaks is a specialized diagnostic skill for identifying and resolving memory leaks within OpenClaw’s test suites. It combines heap snapshot delta comparison, RSS trend tracking, and a dedicated runtime validation harness to clearly separate two distinct memory pressure root causes: genuine application object leaks and retained Vite/Vitest transpilation module graphs. It provides standardized, actionable remediation workflows and is intended for OpenClaw maintainers and contributors debugging Vitest OOM failures, preventing memory regressions, and validating closure runtime cleanup fixes.