Atla

Atla

Enable AI agents to interact with the Atla API for state-of-the-art LLMJ evaluation.

17
Stars
4
Forks
1
Releases

Overview

Atla MCP Server implements a standardized MCP interface that allows LLMs to query and receive evaluation feedback from the Atla API for state-of-the-art LLMJ evaluation. It provides two primary tools: evaluate_llm_response, which evaluates a single LLM response against a defined evaluation criterion and returns a dictionary with a numeric score and a textual critique; and evaluate_llm_response_on_multiple_criteria, which evaluates responses against multiple criteria and returns a list of score/critique dictionaries. Both tools rely on the Atla evaluation model to produce scoring and actionable feedback. The server is designed to be run in a Python environment with an Atla API key, and can be launched manually using uvx (via uvx atla-mcp-server). The README also documents client integration patterns for OpenAI Agents SDK, Claude Desktop, and Cursor, including configuration snippets showing how to start the MCP server and pass the ATLA_API_KEY environment variable. Note: the repository is archived, and the Atla API is no longer active.

Details

Owner
atla-ai
Language
Python
License
MIT License
Updated
2025-12-07

Features

Single-criterion evaluation (evaluate_llm_response)

Evaluates an LLM's response against a single evaluation criterion and returns a dictionary with a numeric score and a textual critique.

Multi-criteria evaluation (evaluate_llm_response_on_multiple_criteria)

Evaluates an LLM's response against multiple criteria and returns a list of dictionaries with scores and critiques.

Standardized MCP interface

Provides a standardized interface for LLMs to interact with the Atla API via MCP.

Atla evaluation model backend

Uses the Atla evaluation model to produce scoring and actionable feedback for LLM outputs.

API key required

Requires an Atla API key to operate the MCP server.

Local server startup via uvx

Can be started manually in a Python environment using uvx with the command pointing to atla-mcp-server.

OpenAI Agents SDK integration

Provides guidance and examples for connecting the MCP server to OpenAI Agents SDK clients.

Claude Desktop & Cursor configuration templates

Includes configuration snippets for Claude Desktop and Cursor to integrate the MCP server.

Audience

DevelopersIntegrate Atla-based LLM evaluation into MCP-enabled workflows for testing and benchmarking.

Tags

Atla APIMCP serverLLM evaluationLLMJ evaluationAtla evaluation modeluv/uvxOpenAI Agents SDKClaude DesktopCursorstate-of-the-art evaluation