Root Signals

Root Signals

Improve and quality control your outputs with evaluations using LLM-as-Judge

10
Stars
4
Forks
0
Releases

Overview

Scorable MCP Server is a bridge between the Scorable API and MCP client applications, exposing Scorable evaluators as MCP tools that allow AI assistants and agents to evaluate responses against defined quality criteria. The server fetches available evaluators from the Scorable API on startup and makes them accessible through MCP endpoints. It implements Server-Sent Events (SSE) for network deployment and is compatible with MCP clients such as Cursor. The server provides tools including list_evaluators, run_evaluation, run_evaluation_by_name, run_coding_policy_adherence, list_judges, and run_judge, enabling both single evaluations and judge-based (LLM-as-a-judge) workflows. A typical setup involves obtaining a Scorable API key and running the server via Docker, with the README showing a docker command and logs that illustrate the server initializing, fetching evaluators, and indicating that the SSE server is listening on an endpoint. The README also demonstrates usage patterns in Python for listing evaluators, executing evaluations by ID or name, and integrating with Cursor and other MCP clients. Endpoints mentioned include /mcp (preferred) and /sse (backward compatible).

Details

Owner
root-signals
Language
Python
License
Updated
2025-12-07

Features

Exposes Scorable evaluators as MCP tools

Makes Scorable evaluators available to MCP clients as standard MCP tools for evaluating responses.

Implements SSE for network deployment

Supports Server-Sent Events to stream evaluation results in MCP workflows.

Compatible with MCP clients such as Cursor

Designed to work with various MCP clients (e.g., Cursor) for broad interoperability.

Docker-based deployment

Provides a Docker command example to run the MCP server with an API key and port mapping.

Toolset for evaluating and judging

Offers tools to list evaluators and judges, and to run evaluations or judges by ID or name.

Coding policy adherence evaluation

Includes a tool to run coding policy adherence evaluations using policy documents.

Audience

AI developersBuild and integrate MCP-enabled assistants that evaluate responses with Scorable evaluators.
MCP integratorsDeploy Scorable evaluators within MCP pipelines for evaluation and quality control.

Tags

MCPModel Context ProtocolScorableEvaluatorsJudgesSSECursorDocker