OpenClaw QA Testing Skill

1. Skill Overview

This skill powers automated QA testing for the OpenClaw project, covering two core modules: qa-lab (QA Lab) and qa-channel (QA Channel). It leverages YAML-defined test scenarios to drive OpenClaw through predefined sequences of operations in simulated or real environments, then validates whether outputs and runtime behavior match expected specifications.

Core Principle: This is a repo-local QA tool for OpenClaw. All test scenarios and configurations are stored under the qa/ directory within the project repository.

2. Core Capabilities

2.1 QA Scenario Execution

Run full test suites via the openclaw qa suite command:

OPENCLAW_LIVE_OPENAI_KEY="${OPENAI_API_KEY}" \
pnpm openclaw qa suite \
  --provider-mode live-frontier \
  --model openai/gpt-5.4 \
  --alt-model openai/gpt-5.4 \
  --output-dir .artifacts/qa-e2e/run-all-live-frontier-

Two provider modes are supported:

mock-openai: Mocked OpenAI provider for rapid local development and regression testing
live-frontier: Live real OpenAI model endpoints for pre-release validation

Two artifacts are generated upon completion:

qa-suite-summary.json: Structured machine-readable test summary
qa-suite-report.md: Human-readable markdown test report

2. Model Policy

The skill enforces strict model usage rules:

Live OpenAI channel: openai/gpt-5.4
Fast inference mode enabled by default
Disallowed models: openai/gpt-5.4-pro, openai/gpt-5.4-mini

2.2 OTEL Observability Smoke Test

Supports local OpenTelemetry validation:

pnpm qa:otel:smoke

This command spins up a local OTLP/HTTP trace receiver, executes the otel-trace-smoke scenario, decodes protobuf span payloads, and validates exported trace names and privacy compliance contracts. No external Opik, Langfuse, or third-party collector credentials required.

2.3 Matrix Multi-Channel Testing

Supports matrix parallel testing across multiple channels including Telegram, WhatsApp, etc.:

OPENCLAW_QA_MATRIX_NO_REPLY_WINDOW_MS=3000 \
pnpm openclaw qa matrix --profile fast --fail-fast

Predefined test profiles:

fast: Critical transport contract tests for release gates; skips image generation and full deep E2EE recovery
transport, media, e2ee-smoke, e2ee-deep, e2ee-cli: Sharded full matrix coverage suites

2. Credential Management

All QA test credentials are managed via 1Password:

Credential Type	1Password Storage Path
Telegram E2E test accounts	vault:OpenClaw → item:Telegram E2E
Convex QA credentials	vault:OpenClaw → items prefixed with OPENCLAW_QA_CONVEX_*
WhatsApp QA	Dedicated test numbers with archived Baileys auth data

Credential handling rules:

Never guess missing values; consult maintainers or reference the exact 1Password item name
All credential secrets must never appear in repository code, logs, pull requests, or screenshots

2.4 Live Telegram Docker Testing

Supports end-to-end Telegram test runs inside Docker containers.

Direct Telegram environment variable mode

OPENCLAW_QA_TELEGRAM_GROUP_ID="..." \
OPENCLAW_QA_TELEGRAM_DRIVER_BOT_TOKEN="..." \
OPENCLAW_QA_TELEGRAM_SUT_BOT_TOKEN="..." \
OPENCLAW_QA_PROVIDER_MODE="mock-openai" \
OPENCLAW_NPM_TELEGRAM_PACKAGE_SPEC="openclaw@beta" \
pnpm test:docker:npm-telegram-live

Convex shared infrastructure mode (recommended for stable QA environments)

Rotating leased credential pools
Lightweight channel-specific wrapper utilities
CLI and management workflows built around pooled credentials

2.5 Character Evaluation

Evaluate consistency of tone, personality, and stylistic output across multiple LLMs:

pnpm openclaw qa character-eval \
  --model openai/gpt-5.4,thinking=xhigh \
  --model anthropic/claude-opus-4-6,thinking=high \
  --judge-model openai/gpt-5.4,thinking=xhigh,fast \
  --concurrency 16 \
  --output-dir .artifacts/qa-e2e/character-eval-

Key features:

Runs as local QA gateway child processes (no Docker required)
Model syntax format: provider/model,thinking=[xhigh/high/medium/low][,fast|,no-fast]
Default candidate models: OpenAI GPT-5.4/5.2/5, Claude Opus/Sonnet, GLM, Kimi, Gemini, and more
Outputs: judge model rankings, runtime statistics, latency metrics, full conversation transcripts
Raw judge response payloads are excluded from exports

2.6 Codex CLI Model Backend Testing

Supports Codex as an alternative model backend for QA validation:

pnpm openclaw qa suite \
  --provider-mode live-frontier \
  --model codex-cli/ \
  --alt-model codex-cli/ \
  --scenario <scenario-name> \
  --output-dir .artifacts/qa-e2e/codex-

Notes:

Exact Codex model identifiers are supplied via user configuration, not hardcoded in source or scenario files
Live QA workflows preserve CODEX_HOME while isolating HOME and OPENCLAW_HOME environment variables

2.7 Scenario Authoring

All test scenarios are written in YAML and stored under qa/scenarios/:

Structure: root index.yaml plus individual standalone *.yaml scenario files
Mandatory top-level fields: title, scenario; optional flow field
Forbidden format: fenced Markdown blocks labeled qa-scenario / qa-flow

3. Primary Use Cases

End-to-end functional regression testing

Run full QA suites after code changes to verify core functionality remains unbroken.
Pre-release validation

Validate critical scenarios against live real LLMs using live-frontier mode prior to Beta/Stable releases.
Multi-channel integration testing

Use matrix profiles to validate message streaming, E2EE encryption, and media delivery across Telegram, WhatsApp, and other channels.
Cross-model behavioral comparison

Character Eval quantifies consistency in style, personality, and task execution across multiple LLMs.
Observability compliance validation

OTEL smoke tests verify correct OpenTelemetry trace export and adherence to data privacy contracts.
Credential pool & infrastructure validation

Validate availability of Convex shared credential pools, Telegram Docker test environments, and WhatsApp test accounts.
Failed test triage

Upon scenario failures, resolve root causes in application code or test framework, then re-run the full test lane.

4. Critical Guiding Principles

Only adjust model policies when explicitly requested by stakeholders
Do not label RSS memory growth as a leak without supporting heap snapshot / retainer graph analysis
All credential secrets must be excluded from source code, logs, PR descriptions, and screenshots
Scenarios must use YAML exclusively; do not introduce Markdown qa-scenario / qa-flow fenced blocks
Character evaluation prompt rules: keep prompts natural and task-focused; avoid leading questions such as “How would you respond?” or disclosing the model is under evaluation
Every character eval suite must include at least one real actionable task (e.g., create/edit workspace artifacts) to capture authentic role behavior under normal tool usage
Duration metrics serve as baseline context reference only, not primary scoring signals

Summary

OpenClaw QA Testing is the core end-to-end automated testing skill for the OpenClaw project. It executes predefined YAML scenarios to drive and validate OpenClaw behavior in simulated or live environments. Its feature set covers matrix multi-channel testing, LLM character consistency evaluation, OTEL observability validation, Docker-based live Telegram testing, and Codex CLI backend integration, with centralized credential management via 1Password. This skill is intended for OpenClaw contributors and maintainers to conduct daily regression testing, pre-release validation, cross-model behavioral benchmarking, and shared QA infrastructure credential health checks.