Evaluations#

Promptfoo-based eval suites for badger-fetch.

Quick start#

npm run build
npm test                           # unit tests
npm run eval                       # LLM injection detection
npm run eval:mcp                   # MCP integration
npm run eval:sanitize              # sanitization
npm run eval:agent                 # agent integration

Structure#

evals/
├── injection/                     # LLM injection detection
│   ├── promptfooconfig.yaml
│   ├── prompt.txt
│   └── tests.csv                  # 25 injection tests (CSV + metadata)
├── mcp-blocking/                  # MCP server blocking + approve flow
│   ├── promptfooconfig.yaml
│   ├── prompt.txt
│   ├── provider.ts                # Spawns MCP server per call
│   ├── mcp-integration.yaml       # 10 URL blocking tests
│   └── approve-flow.yaml          # 4 approve/override tests
├── agent/                         # Agent + MCP tool-use integration
│   ├── promptfooconfig.yaml
│   ├── prompt.txt
│   ├── provider.ts                # Agent loop: LLM ↔ MCP via OpenRouter
│   └── tests.yaml                 # 5 agent integration scenarios
├── sanitization/                  # HTML stripping + injection flags
│   ├── promptfooconfig.yaml
│   ├── provider.ts                # Starts mock server + calls MCP
│   ├── mock-server.ts             # HTTP server with known content
│   └── tests.yaml                 # 11 sanitization pipeline tests
├── save-results.sh
└── README.md

Metadata filtering#

The injection CSV has attack and category columns:

npx promptfoo eval -c evals/injection/promptfooconfig.yaml --filter-metadata attack=true
npx promptfoo eval -c evals/injection/promptfooconfig.yaml --filter-metadata category=encoding
npx promptfoo eval -c evals/injection/promptfooconfig.yaml --filter-metadata attack=false

Adding tests#

Injection tests — add a row to evals/injection/tests.csv:

content,attack,category,__description,__expected1,__expected2
"Your test content",true,direct-override,Test name,"javascript: output.verdict === 'malicious'","javascript: output.blocked === true"

MCP tests — add to evals/mcp-blocking/mcp-integration.yaml or evals/mcp-blocking/approve-flow.yaml:

- description: "Test name"
  vars:
    url: "https://example.com"
    caller: "promptfoo-test"
  assert:
    - type: contains
      value: "BLOCKED"

Sanitization tests — add a route to evals/sanitization/mock-server.ts and a test to evals/sanitization/tests.yaml.