OpenClaw Testing Skill

1. Skill Overview

This skill is used when deciding test scope, debugging failures, re-running CI pipelines, or validating code changes. It delivers a complete test routing model spanning lightweight local targeted tests through large-scale remote validation, enabling developers to maximize verification confidence with minimal resource overhead.

Core Principle: Route first by source trust level, then by validation scale.

2. Core Capabilities

2.1 Test Routing Model

The skill automatically selects the optimal validation path based on change trustworthiness and scope.

Validation Type	Applicable Scenarios	Execution Method
Local Targeted Testing	Trusted codebase, narrow focused test set, local dependencies ready	`node scripts/run-vitest.mjs <file>`
Remote Validation	Change gate checks, lint/type fanout, builds, Docker packaging, E2E, cross-platform testing	Blacksmith Testbox or AWS Crabbox

Heavyweight validation for trusted maintainers defaults to Blacksmith Testbox.

Untrusted contributor forks must run via keyless fork CI or sanitized direct AWS Crabbox environments.

2.2 Change Classification Commands

A suite of utilities to categorize edits and select appropriate validation workflows:

Command	Purpose
`pnpm changed:lanes --json`	Output which inspection lanes the current diff touches
`pnpm check:changed`	Lightweight local static checks or delegated heavy static pipelines (no Vitest)
`pnpm test:changed`	Intelligently scope Vitest targets to modified code
`pnpm verify`	Full static checks + complete Vitest suite
`pnpm test <file>`	Run isolated targeted tests for a single file

Command semantics breakdown:

pnpm check / pnpm check:changed: No Vitest execution; only type checking, linting, and gate validations
pnpm test / pnpm test:changed: Executes Vitest test cases
pnpm verify: Sequentially runs pnpm check followed by pnpm test

2.3 Test Target Resolver

Precise rules for selecting minimal required test coverage:

Direct test file edits: Execute that test file only
Source code edits: Prioritize explicit test mappings, sibling *.test.ts files, then dependency graph reachable test suites
Shared harness / config / root file edits: Skip all tests by default unless exact dedicated test mappings exist
Shared group message config & source reply prompt edits: Run core auto-reply regression tests + Discord/Slack delivery validation where mapped tests exist
Public SDK / contract edits: Avoid full plugin test sweeps; select minimal representative plugin & contract Vitest validations

2.4 Remote Validation Backends

Two remote execution backends are supported:

Backend	Description	Identifier Tag
Blacksmith Testbox	Default heavy validation backend for trusted maintainers	`provider=blacksmith-testbox`, `tbx_...` ID prefix
AWS Crabbox	Direct AWS cloud test runner	`provider=aws`, `cbx_...` ID prefix

Remote backend lifecycle rules:

Lazy initialization: Do not pre-provision backends for anticipated workloads
Acquire backend instance on first heavy command execution, cache its ID, reuse for subsequent heavy tasks; terminate before handoff
Single-shot heavy commands may use ephemeral one-off instances

Mandatory constraints for untrusted code on AWS Crabbox:

CRABBOX_ENV_ALLOW=CI must replace the repository’s internal OPENCLAW_* allowlist
--no-hydrate flag blocks injection of authenticated config files
All remote jobs launch with a brand-new temporary HOME directory
Spin up provisioned trusted Crabbox binaries from a clean trusted main checkout; pull PR code via --fresh-pr
Unset CRABBOX_AWS_INSTANCE_PROFILE and all CRABBOX_TAILSCALE* overrides
Upload trusted bootstrap script scripts/crabbox-untrusted-bootstrap.sh to provision Node 24 and repo-locked pnpm versions

2.5 CI Debug Workflow

Efficient CLI-based CI troubleshooting commands:

# List recent CI runs on main branch
gh run list --branch main --limit 10
# Fetch structured metadata for a specific run
gh run view <run> --json status,conclusion,headSha,url,jobs
# Pull full logs for a single failing job
gh run view <run> --job <job> --log

Debug best practices:

Inspect exact commit SHA; ignore unrelated updated commits on main
For canceled runs on the same branch, verify if a newer superseded execution exists
Only pull full logs for failed or relevant jobs
Prefer gh run view <run> --json jobs over PR summary views (summaries may be stale or noisy)

2.6 Docker Testing

Docker pipelines carry high compute cost; a two-phase check-first-then-run workflow is enforced.

Dry Run Planning Phase

# Preview full Docker test plan without execution
OPENCLAW_DOCKER_ALL_DRY_RUN=1 pnpm test:docker:all
# Export structured plan JSON for lane breakdown
OPENCLAW_DOCKER_ALL_LANES=install-e2e node scripts/test-docker-all.mjs --plan-json

Isolated Single Failed Lane Execution (Only on explicit request / GitHub outage)

OPENCLAW_DOCKER_ALL_LANES=<target-lane> \
OPENCLAW_DOCKER_ALL_BUILD=0 \
OPENCLAW_DOCKER_ALL_PREFLIGHT=0 \
OPENCLAW_SKIP_DOCKER_BUILD=1 \
pnpm test:docker:all

Docker Test Sharding for Release Pipelines

Release workflows split Docker workloads into segmented small job batches:

core
package-update-openai, package-update-anthropic, package-update-core
plugins-runtime-plugins, plugins-runtime-services
plugins-runtime-install-a through plugins-runtime-install-h
openwebui

2.7 Full Release Validation Workflow

Full Release Validation (.github/workflows/full-release-validation.yml) serves as the master manual pre-production verification pipeline. It runs a complete sub-matrix against the exact code commit SHA preceding release changelog generation:

node scripts/full-release-validation-at-sha.mjs \
  --sha <target-commit-SHA> \
  --target-ref release/YYYY.M.PATCH

Dispatched child sub-workflows:

Full standard CI matrix (including Android targets)
Plugin Prerelease: Plugin static release checks, extension shards, agentic plugin shards, plugin product Docker lanes
OpenClaw Release Checks: Install smoke tests, cross-OS release validation, Live/E2E suites, Docker release pipelines, OpenWebUI, QA Lab, fast Matrix tests, Telegram release lanes

Rerun group labels for post-fix targeted re-execution:

all, ci, plugin-prerelease, release-checks, install-smoke, cross-os, live-e2e, package, qa, qa-parity, qa-live, npm-telegram

2.8 Package Acceptance Workflow

Use this pipeline to validate installable production packages, distinct from source-code Vitest checks:

gh workflow run package-acceptance.yml --ref main \
  -f source=npm \
  -f workflow_ref=main \
  -f package_spec=openclaw@beta \
  -f suite_profile=product \
  -f telegram_mode=mock-openai

Supported package candidate sources:

source=npm: Official published versions openclaw@beta, openclaw@latest, or exact tagged releases
source=url: Remote tarball URL paired with package_sha256 hash
source=artifact: GitHub Actions build artifact tarball
source=ref: Unpublished source code candidates (trusted branch/SHA only)

Validation profiles:

smoke: Fast lightweight confidence check — tarball installation, single channel onboarding, one agent conversation turn

2.9 Plugin Package Validation

Separate two-stage validation for official or third-party plugin bundles to verify package integrity and security trust boundaries:

Local pre-release validation: Build plugin archive and install via openclaw plugins install npm-pack:<local-path> --force
Trusted official secondary validation: If functionality relies on bundled plugins or trusted official plugin permissions, re-validate via catalog-backed official installation or published package paths

Note: Local npm-pack archive validation alone is insufficient for privileged helper or trusted official scope logic.

3. Primary Use Cases

Select minimal-cost validation paths after code changes

Automatically route to lightweight local targeted tests or remote heavy suites based on trust and scope, avoiding reflexive full-suite executions
Debug CI pipeline failures

Retrieve granular logs for individual failing jobs via GitHub CLI instead of blind full CI reruns
Pre-release candidate validation

Execute Full Release Validation before Beta/Stable launches to verify all product functionality dimensions
Docker E2E pipeline testing

Run dry-run planning first, then isolate single broken lanes to avoid costly full Docker rebuilds
Package Acceptance verification

Validate end-to-end operability of published npm production packages
Plugin bundle validation

Audit structural integrity and security trust status of official and third-party plugin archives

4. Critical Guiding Principles

Validate the minimal affected surface first; avoid reflexively running full test suites
Route test workloads first by source trust level, then by validation scope
Trusted code may run targeted local tests; untrusted fork code must never execute arbitrary local tooling
Do not pre-provision remote backends for anticipated jobs; lazy-instantiate on first heavy command
Omit --no-sync when reusing backend instances unless explicitly re-testing unchanged synchronized tree states
Do not terminate unrelated background processes or test runners; assume other users/agents own concurrent workloads
Never run raw local pnpm test* commands inside Codex worktrees, linked checkouts, or sparse clones — use node scripts/run-vitest.mjs for scoped local validation
pnpm test:changed defaults to low-overhead execution; only set OPENCLAW_TEST_CHANGED_BROAD=1 when editing harness/config/package core files
Do not trigger full extension sweep tests for core code edits; if the fix targets a specific plugin bug, explicitly run only that plugin’s test suite
Untrusted contributors and fork code must use keyless fork CI or sanitized direct AWS Crabbox environments

Summary

OpenClaw Testing is the core test routing skill for the OpenClaw project, governed by the rule “route first by trust level, then by validation scale”. It empowers developers and maintainers to achieve high verification confidence with minimal resource expenditure. Its full spectrum of coverage spans lightweight local targeted tests, remote Blacksmith Testbox / AWS Crabbox execution, CI failure debugging, end-to-end Full Release Validation, Docker E2E pipelines, and Package Acceptance workflows. It serves as an indispensable quality assurance component within the OpenClaw development lifecycle.