# TestPion

> TestPion is a local-first desktop app and CLI for testing REST, GraphQL, WebSocket, MCP and AI/LLM systems, similar to Postman: collections, environments, Postman-compatible `pm.*` scripts, a Collection Runner, mock servers, a cookie jar and Markdown docs, plus evaluations for LLM, RAG and agent tests. Everything runs on the user's machine; secrets are stored in the OS credential store and redacted from logs, traces, reports and exports. The `testpion` CLI runs collections like Newman (`testpion run-collection`) and serves a workspace to AI agents over MCP (`testpion mcp-server`).

Key facts for agents:

- Workspaces are folders of JSON/YAML files (`workspace.json`, `collections/*.json`, `environments/*.json`, `tests/**/*.yaml`) that can be committed to git. Secret values are never in these files.
- Test types: http, graphql, grpc, websocket (or socketio / mqtt: connect, `send` messages / `{event,args,ack}` / `{topic,payload,qos,retain}` after `subscribe: [filters]`, assert on `$.received[n]`), mcp, llm, rag, agent.
- Same request, two environments: `testpion env diff <a> <b> --request <name> --json` or MCP `compare_request_across_environments` (response diff).
- Environment drift: `testpion env diff <a> <b> --json` (exit 1 when they differ) or MCP `compare_environments` (statuses only, no values).
- Proxy: the CLI honours HTTP_PROXY / HTTPS_PROXY / NO_PROXY (TESTPION_NO_PROXY=1 disables); the app has Settings ▸ Proxy (env, custom URL + bypass list, or off).
- Private / corporate CAs: app Settings ▸ Certificates (OS store, extra PEM); CLI TESTPION_USE_SYSTEM_CA=1, TESTPION_CA_FILE=path or NODE_EXTRA_CA_CERTS.
- Deleting a collection or environment moves it to the workspace trash for 30 days: `testpion trash list|restore <id>|empty --yes -w <ws>`.
- AI: the user's own Anthropic key (app Settings ▸ AI assistant, kept in the OS secret store; CLI/CI: TESTPION_SECRET_APP_ANTHROPIC_APIKEY) adds the provider `claude-app` (Claude Opus 5.5 / Sonnet 5.5 / Haiku 4.5) to every workspace.
- `testpion load --collection <name> -w <ws> [--folder <f>] [--warm-up]` load-tests a collection (each virtual user runs its requests in order; per-request latency in the results). gRPC: `testpion load host:port --grpc pkg.Service/Method -d '{json}'` (reflection or --proto). MCP `load_test` does the same for local hosts (≤ 50 users, ≤ 60 s). Thresholds: `--threshold "p95<500" "errors<1%" "rps>=50" "p99[Request]<800"` (exit 1 when one fails; MCP `thresholds`).
- `testpion vars usages <name> -w <ws> --json` and `testpion vars rename <from> <to> -w <ws>` find and rename a variable everywhere (MCP `variable_usages`, `rename_variable`).
- `testpion export <collection> --format openapi [--json]` writes an OpenAPI 3.1 document for a collection (MCP `collection_openapi`).
- `testpion record <target> [--port] [-w <ws> --collection <name>]` records traffic through a localhost reverse proxy and saves it as a collection (responses as examples, secrets as {{variables}}).
- `testpion lint <collection> -w <ws> [--fail-on high] [--json]` is a security review of a collection's requests (MCP `security_review`); the `security-headers` check tests responses.
- GraphQL subscriptions: `testpion graphql-subscribe <endpoint> -q <doc> [--max n] [--json]` (MCP `graphql_subscribe`), graphql-transport-ws and graphql-ws.
- GraphQL operation builder: `testpion graphql-op Query.field --endpoint <url> | --schema <file> [--depth n] [--json]` (MCP `graphql_operation`) returns a valid query with typed variables and placeholder values; in the app, **Build** on a root field of the schema explorer.
- `testpion send "<Collection/Request>"|<url> -w <ws> -e <env> [--json] [--fail]` sends one request (saved requests run their scripts and checks).
- `testpion env set <env> key=value … -w <ws> [--create]` / `env unset` / `env get` edit plain environment variables (MCP `set_environment_variable`); secrets are only set in the app.
- Newman command lines work with `newman run` replaced by `testpion run-collection` (same -e, -g, -d, -n, --folder, --env-var, --global-var, --export-environment, --export-globals, -k, --suppress-exit-code, --reporters, --reporter-junit-export). Scripts have lodash (`_`), `moment`, `tv4` and real `jsonSchema` checks, as in Postman.
- Dynamic variables use Postman's names: `{{$guid}}`, `{{$timestamp}}`, `{{$isoTimestamp}}`, `{{$randomInt(1,10)}}`, `{{$randomFirstName}}`, `{{$randomEmail}}` (example.test) and ~70 more `{{$random…}}` values; each use gives a new value.
- `testpion openapi-diff <old> <new> [--json] [--fail-on-breaking]` lists breaking changes between two OpenAPI versions (files or links); MCP `openapi_diff` does the same.
- `testpion load --saved <name> [-e env] [--threshold rule...]` runs a load test saved in the app (target, users, duration, rules); `{{variables}}` in any load URL resolve from `-e`.
- `testpion eval list [--json]` and `testpion eval run <name> [-e env] [--limit n]` list and run evaluations saved in the app (dataset × prompt × model × evaluators); MCP `list_evaluations` / `run_evaluation` do the same.
- `testpion coverage <spec> [--run <id...>] [--history [n]] [--min <pct>] [--json]` reports which OpenAPI operations and documented response codes the latest run (or given runs / request history) exercised, never-called operations and undocumented codes; MCP `api_coverage` returns the same JSON. Use it to find untested endpoints and error cases, then write tests for them.
- `testpion import <file|url|-> -w <ws> --json` imports from a file, stdin, or an http(s) link (OpenAPI URL, GitHub/GitLab/Bitbucket file page, Postman API link).
- .env files import as environments (secret-looking keys become secret variables) and environments export as .env.
- CI: `testpion ci <github|gitlab|azure|jenkins> -w <ws> [--suite s | --collection c [--folder f]] [-e env] --json` (or MCP `ci_config`) returns { path, content, secrets, command }; the pipeline installs the CLI from github.com/nasimuddin-dev/testpion at the version's tag (the CLI is not on npm).
- A complete example workspace against free public APIs is at `examples/public-workspace` (REST, GraphQL, gRPC, WebSocket, SSE, MCP, AI; suites `all` and `offline`); the desktop app opens it on first launch. Docs: /getting-started/examples
- OpenTelemetry: `testpion test|run-collection --trace all --otlp http://collector:4318` (or OTEL_EXPORTER_OTLP_ENDPOINT/HEADERS) sends traces as OTLP/HTTP JSON; MCP `export_traces`; Traces view ▸ Send to OpenTelemetry.
- Scripts may use top-level `await`: `const res = await pm.sendRequest(url)`; `await pm.vault.get('key')` and `{{vault:key}}` read (secret) variables.
- MCP client features: `type: mcp` tests answer server requests with `elicitation: { action, content }`, `sampling: "reply"`, `roots: [paths]`; the inspector shows elicitation forms and sampling review dialogs.
- Scripts have cheerio (`const $ = cheerio.load(pm.response.text()); $('title').text()`), lodash, moment, tv4, crypto-js, xml2Json.
- Script packages: `pm.require('@team/utils')` loads workspace file packages/@team/utils.js (CommonJS, module.exports), like Postman's package library.
- Save as test: REST tab menu / GraphQL, gRPC, WebSocket toolbar **Test** writes tests/<kind>/<name>.yaml; MCP `save_test` (collection + request).
- MCP `list_tests` / `run_tests` (paths, grep, tags, rerunFailed) run the YAML tests under tests/.
- Re-run only failures: `testpion test --rerun-failed [runId]` (default last run).
- Watch mode for local development: `testpion test --watch`, `testpion run-collection <c> --watch` re-run on file changes.
- Imports (app, `testpion import <file> -w <ws> --json`): OpenAPI 3 / Swagger 2, Postman v2.1, Insomnia v4/v5, Bruno (collection folder of .bru files, a .bru file, or JSON export; Bruno scripts run with the bru/req/res API and assertions become tests), Hoppscotch (export: `testpion export <c> --format bruno --out <dir>` writes a Bruno folder), WSDL 1.1 (SOAP 1.1/1.2 requests with SOAPAction and sample envelopes from the XSD; file or ?wsdl link), HAR, cURL/fetch/PowerShell snippets.
- Postman v2.1 collections and environments import and export losslessly where the formats overlap.
- `testpion collections --json` lists a workspace's collections (requests, gRPC calls, connections); `testpion requests <collection> --json` lists what one holds.
- To let an agent use a workspace, register the MCP server: `testpion mcp-server -w <workspace>` (stdio). Tools: `list_collections`, `list_requests`, `get_request`, `list_environments`, `collection_docs`, `send_request`, `run_collection`, `parse_request_snippet`, `save_request`, `import_definition` (from a url or text), `reorder_environments`. Add `--read-only` to disable sending and saving.
- A request copied as cURL (bash or cmd), fetch or PowerShell can be parsed (`parse_request_snippet`), sent (`send_request` with `snippet`) or saved (`save_request`, or CLI `testpion import <file|-> -w <ws> --json`). Secrets in it are replaced by `{{variables}}` listed in `placeholders`; they are never written to workspace files.
- Scripts use the Postman API as `tp.*` (TestPion's name) or `pm.*` (Postman's): the same object, both always work. Exports to Postman convert `tp.*` to `pm.*`; `testpion scripts convert --to tp|pm --json` rewrites a workspace's scripts.
- Test scripts can call `pm.visualizer.set(template, data)` (Handlebars); run results include the rendered `visualization.html` and `scriptLogs`.
- Monitors (collections on a schedule): `testpion monitor add <name> --collection <c> --every 15m`, `monitor list|results --json`, `monitor run --due` (cron) or `monitor start` (long-running); MCP `list_monitors` / `monitor_results` / `run_monitor`. Docs: /test-runner/monitors
- Response history: `testpion history list --request <name> --json` / `testpion history diff <before> <after> --json`, or MCP `request_history` / `compare_responses`, show what changed between two runs of a request (field-by-field JSON diff; secrets masked). `testpion history stats --request <name> --json` or MCP `response_time_stats` give its median / p95 / slowest response time and failure count.
- `testpion env list --json` and `testpion env order <names...>` read and set the environment order.
- CLI exit codes: 0 success, 1 test failure, 2 configuration error, 3 execution error.

## Getting started

- [Installation](https://nasimuddin-dev.github.io/testpion/getting-started/installation): install the desktop app or the CLI
- [Your first request](https://nasimuddin-dev.github.io/testpion/getting-started/first-request): send a request, add assertions, save it to a collection
- [Workspaces](https://nasimuddin-dev.github.io/testpion/getting-started/workspaces): switch, rename, duplicate, export and delete workspaces; `testpion workspace list|rename|delete --json`
- [Your first AI test](https://nasimuddin-dev.github.io/testpion/getting-started/first-ai-test): test an LLM prompt with evaluators

## API testing

- [REST and HTTP](https://nasimuddin-dev.github.io/testpion/api-testing/rest): requests, responses, `pm.*` scripts, console, tabs and history
- [WebSocket, Socket.IO and MQTT](https://nasimuddin-dev.github.io/testpion/api-testing/websocket): connect, send messages, emit Socket.IO events (with acknowledgements), subscribe and publish on MQTT brokers (3.1.1 and 5, QoS, retained); `testpion ws`, `testpion mqtt`; MCP tool `realtime_exchange` (`mode: mqtt`, `subscribe`); saved connections in folders; clear connection errors
- [Server-Sent Events](https://nasimuddin-dev.github.io/testpion/api-testing/rest#server-sent-events): text/event-stream responses parsed event by event (live, Stop keeps events); MCP send_request returns them
- [OpenAPI contract testing](https://nasimuddin-dev.github.io/testpion/test-runner/assertions#openapi-contract-testing): the `openapi` check validates status, content type and body schema against an OpenAPI 3 / Swagger 2 document (`spec: openapi.yaml`)
- [Authentication](https://nasimuddin-dev.github.io/testpion/api-testing/authentication): API key, Basic, Bearer, JWT, OAuth 1.0, OAuth 2.0, AWS Signature v4, Digest, mTLS
- [Environments and variables](https://nasimuddin-dev.github.io/testpion/api-testing/environments): scopes and precedence, secrets, quick look, export
- [Cookies](https://nasimuddin-dev.github.io/testpion/api-testing/cookies): the workspace cookie jar, `pm.cookies.jar()`, CLI cookie jar files
- [Mock servers](https://nasimuddin-dev.github.io/testpion/api-testing/mock-servers): serve saved examples on localhost
- [Collections](https://nasimuddin-dev.github.io/testpion/api-testing/collections): import, export to Postman, examples, documentation, Collection Runner

## GraphQL and MCP

- [GraphQL overview](https://nasimuddin-dev.github.io/testpion/graphql/overview)
- [GraphQL mock server](https://nasimuddin-dev.github.io/testpion/graphql/mocking): `testpion mock-graphql --schema schema.graphql` serves fake, typed, deterministic data for any valid query
- [GraphQL testing](https://nasimuddin-dev.github.io/testpion/graphql/testing)
- [gRPC](https://nasimuddin-dev.github.io/testpion/api-testing/grpc): call gRPC services from .proto files or server reflection (unary and streaming), metadata, TLS, deadlines; `type: grpc` tests with `grpc-status`; `testpion grpc`; MCP tool `grpc_call`
- [MCP overview](https://nasimuddin-dev.github.io/testpion/mcp/overview): test MCP servers (stdio, Streamable HTTP, SSE)
- [MCP tools](https://nasimuddin-dev.github.io/testpion/mcp/tools)
- [MCP mock server](https://nasimuddin-dev.github.io/testpion/mcp/mocking): fake MCP servers from `*.mcp-mock.yaml` (or recorded from a real server with Save as mock); `testpion mock-mcp <file>` serves one over stdio or `--http` for agents

## AI testing

- [AI testing overview](https://nasimuddin-dev.github.io/testpion/ai-testing/overview)
- [Evaluations](https://nasimuddin-dev.github.io/testpion/ai-testing/evaluations): deterministic, heuristic, embedding and LLM-as-judge checks
- [RAG](https://nasimuddin-dev.github.io/testpion/ai-testing/rag)
- [Agents](https://nasimuddin-dev.github.io/testpion/ai-testing/agents)
- [Safety](https://nasimuddin-dev.github.io/testpion/ai-testing/safety)

## Automation

- [Test runner](https://nasimuddin-dev.github.io/testpion/test-runner/overview): YAML test suites
- [Assertions](https://nasimuddin-dev.github.io/testpion/test-runner/assertions): all check types
- [CI/CD](https://nasimuddin-dev.github.io/testpion/test-runner/ci-cd): GitHub Actions and other CI, migrating from Newman
- [CLI reference](https://nasimuddin-dev.github.io/testpion/cli/reference): every command and option, including `run-collection`, `mock`, `docs`, `export` and `mcp-server`

## Optional

- [Security: secrets](https://nasimuddin-dev.github.io/testpion/security/secrets)
- [Privacy and redaction](https://nasimuddin-dev.github.io/testpion/security/privacy)
- [Load testing](https://nasimuddin-dev.github.io/testpion/performance/load-testing)
- [Architecture](https://nasimuddin-dev.github.io/testpion/architecture/overview)
- [Design](https://nasimuddin-dev.github.io/testpion/architecture/design): principles, brand, colour tokens, layout, components, core architecture and the request lifecycle
- [Changelog](https://nasimuddin-dev.github.io/testpion/changelog)