mcp-evals

About

A Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring, with built-in observability support. This helps ensure your MCP server's tools are working correctly, performing well, and are fully observable with integrated monitoring and metrics.

You can create evaluation configurations in either TypeScript or YAML format.

Create a file (e.g., evals.ts) that exports your evaluation configuration:

For simpler configuration, you can use YAML format (e.g., evals.yaml):

You can run the evaluations using the CLI with either TypeScript or YAML files:

Each evaluation returns an object with the following structure:

If you're using…