Supplied ranking evaluation
Score ranked document IDs against caller-supplied graded judgments for up to 50 queries: precision/recall, reciprocal rank, truncated average precision and NDCG at k. Explicit gain, unjudged and zero-denominator conventions; evidence lists ranked judgments and missed relevant IDs. No search or relevance inference.
data-assurance · Operation ID: ranking-evaluate
Choose this operation when
- evaluate retrieval ranking ndcg map mrr precision recall at k
- score search results against supplied relevance judgments
- compare RAG retrieval quality using judged document ids
Outside this profile
- Search the web or determine semantic document relevance
- Claim benchmark completeness from partial judgments
Exact release references
Static JSON contract · Markdown reference · Fixed example response
Complete input schema
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"queries": {
"minItems": 1,
"maxItems": 50,
"type": "array",
"items": {
"type": "object",
"properties": {
"id": {
"type": "string",
"minLength": 1,
"maxLength": 80
},
"retrieved": {
"maxItems": 100,
"type": "array",
"items": {
"type": "string",
"minLength": 1,
"maxLength": 80
}
},
"judgments": {
"maxItems": 100,
"type": "array",
"items": {
"type": "object",
"properties": {
"id": {
"type": "string",
"minLength": 1,
"maxLength": 80
},
"grade": {
"type": "number",
"minimum": 0,
"maximum": 10
}
},
"required": [
"id",
"grade"
],
"additionalProperties": false
}
}
},
"required": [
"id",
"retrieved",
"judgments"
],
"additionalProperties": false
}
},
"k": {
"default": 10,
"type": "integer",
"minimum": 1,
"maximum": 100
},
"relevantAtLeast": {
"default": 1,
"type": "number",
"exclusiveMinimum": 0,
"maximum": 10
},
"gain": {
"default": "linear",
"type": "string",
"enum": [
"linear",
"exponential"
]
},
"unjudged": {
"default": "error",
"type": "string",
"enum": [
"error",
"nonrelevant"
]
}
},
"required": [
"queries"
],
"additionalProperties": false
}
Complete output-envelope schema
{
"type": "object",
"required": [
"operation",
"version",
"result",
"provenance"
],
"properties": {
"operation": {
"const": "ranking-evaluate",
"type": "string"
},
"version": {
"const": "0.29.0",
"type": "string"
},
"result": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"profile": {
"type": "string",
"const": "supplied-ranking-evaluation-v1"
},
"k": {
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
},
"queryCount": {
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
},
"queries": {
"type": "array",
"items": {
"type": "object",
"properties": {
"precisionAtK": {
"type": "number"
},
"recallAtK": {
"type": [
"number",
"null"
]
},
"reciprocalRankAtK": {
"type": [
"number",
"null"
]
},
"averagePrecisionAtK": {
"type": [
"number",
"null"
]
},
"ndcgAtK": {
"type": [
"number",
"null"
]
},
"id": {
"type": "string"
},
"relevantJudgmentCount": {
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
},
"retrievedAtKCount": {
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
},
"hitsAtK": {
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
},
"unjudgedAtKCount": {
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
},
"dcgAtK": {
"type": "number"
},
"idealDcgAtK": {
"type": "number"
},
"missedRelevantIds": {
"type": "array",
"items": {
"type": "string"
}
},
"ranking": {
"type": "array",
"items": {
"type": "object",
"properties": {
"rank": {
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
},
"id": {
"type": "string"
},
"grade": {
"type": [
"number",
"null"
]
},
"relevant": {
"type": "boolean"
},
"judged": {
"type": "boolean"
}
},
"required": [
"rank",
"id",
"grade",
"relevant",
"judged"
],
"additionalProperties": false
}
}
},
"required": [
"precisionAtK",
"recallAtK",
"reciprocalRankAtK",
"averagePrecisionAtK",
"ndcgAtK",
"id",
"relevantJudgmentCount",
"retrievedAtKCount",
"hitsAtK",
"unjudgedAtKCount",
"dcgAtK",
"idealDcgAtK",
"missedRelevantIds",
"ranking"
],
"additionalProperties": false
}
},
"macro": {
"type": "object",
"properties": {
"precisionAtK": {
"type": [
"number",
"null"
]
},
"recallAtK": {
"type": [
"number",
"null"
]
},
"reciprocalRankAtK": {
"type": [
"number",
"null"
]
},
"averagePrecisionAtK": {
"type": [
"number",
"null"
]
},
"ndcgAtK": {
"type": [
"number",
"null"
]
}
},
"required": [
"precisionAtK",
"recallAtK",
"reciprocalRankAtK",
"averagePrecisionAtK",
"ndcgAtK"
],
"additionalProperties": false
},
"definedQueryCounts": {
"type": "object",
"properties": {
"precisionAtK": {
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
},
"recallAtK": {
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
},
"reciprocalRankAtK": {
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
},
"averagePrecisionAtK": {
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
},
"ndcgAtK": {
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
}
},
"required": [
"precisionAtK",
"recallAtK",
"reciprocalRankAtK",
"averagePrecisionAtK",
"ndcgAtK"
],
"additionalProperties": false
},
"conventions": {
"type": "array",
"items": {
"type": "string"
}
}
},
"required": [
"profile",
"k",
"queryCount",
"queries",
"macro",
"definedQueryCounts",
"conventions"
],
"additionalProperties": false
},
"provenance": {
"type": "object",
"required": [
"inputSha256",
"outputSha256",
"deterministic",
"externalRequests"
],
"properties": {
"inputSha256": {
"type": "string",
"pattern": "^[a-f0-9]{64}$"
},
"outputSha256": {
"type": "string",
"pattern": "^[a-f0-9]{64}$"
},
"deterministic": {
"const": true
},
"externalRequests": {
"const": 0
}
}
}
},
"additionalProperties": false
}
Fixed example
One accepted fixed example, not a custom-input trial. No operation runs when this static page is requested.
Example input
{
"queries": [
{
"id": "q1",
"retrieved": [
"b",
"a",
"c"
],
"judgments": [
{
"id": "a",
"grade": 3
},
{
"id": "b",
"grade": 1
},
{
"id": "c",
"grade": 0
},
{
"id": "d",
"grade": 2
}
]
}
],
"k": 3
}
Example response
{
"operation": "ranking-evaluate",
"version": "0.29.0",
"result": {
"profile": "supplied-ranking-evaluation-v1",
"k": 3,
"queryCount": 1,
"queries": [
{
"precisionAtK": 0.6666666666666666,
"recallAtK": 0.6666666666666666,
"reciprocalRankAtK": 1,
"averagePrecisionAtK": 0.6666666666666666,
"ndcgAtK": 0.6074915180456525,
"id": "q1",
"relevantJudgmentCount": 3,
"retrievedAtKCount": 3,
"hitsAtK": 2,
"unjudgedAtKCount": 0,
"dcgAtK": 2.8927892607143724,
"idealDcgAtK": 4.7618595071429155,
"missedRelevantIds": [
"d"
],
"ranking": [
{
"rank": 1,
"id": "b",
"grade": 1,
"relevant": true,
"judged": true
},
{
"rank": 2,
"id": "a",
"grade": 3,
"relevant": true,
"judged": true
},
{
"rank": 3,
"id": "c",
"grade": 0,
"relevant": false,
"judged": true
}
]
}
],
"macro": {
"precisionAtK": 0.6666666666666666,
"recallAtK": 0.6666666666666666,
"reciprocalRankAtK": 1,
"averagePrecisionAtK": 0.6666666666666666,
"ndcgAtK": 0.6074915180456525
},
"definedQueryCounts": {
"precisionAtK": 1,
"recallAtK": 1,
"reciprocalRankAtK": 1,
"averagePrecisionAtK": 1,
"ndcgAtK": 1
},
"conventions": [
"Precision denominator is k even when fewer results are supplied. Recall and truncated AP denominators use all supplied relevant judgments; reciprocal rank is truncated at k.",
"DCG uses linear gain and log2(rank + 1) discount. Ideal DCG uses all supplied judgments sorted by grade, truncated at k.",
"No-relevant-judgment recall, reciprocal rank and AP are null; zero-ideal-gain NDCG is null. Macro averages exclude nulls and report denominators.",
"Evaluation uses supplied judgments only. It does not retrieve documents, judge semantic relevance, or establish complete relevance coverage."
]
},
"provenance": {
"inputSha256": "a830c056a36bdf25d24ce65b0f72f06975f4ac98f7a9cb4aa70c79f3c3fd8b4f",
"outputSha256": "948595d1d37750d1e18558475116ee9dd440ddde396ae7edc49b7d9f909ac4d0",
"deterministic": true,
"externalRequests": 0
}
}
Bounds and precision
JavaScript IEEE-754 numbers; use strings for large integer IDs/exact decimals where the schema accepts strings. No lossless numeric parsing.
{
"global": {
"requestBytes": 131072,
"responseBytes": 524288,
"jsonDepth": 32,
"jsonNodes": 20000,
"requestsPerMinute": 60,
"paidAttemptsPerMinute": 20,
"idempotencyHours": 24
},
"operation": {
"inputBytes": 100000,
"outputBytes": 400000,
"jsonNodes": 12000,
"depth": 20,
"evidence": 200,
"keyWorkBytes": 3000000,
"matchingSteps": 2000000,
"queries": 50,
"retrievedPerQuery": 100,
"judgmentsPerQuery": 100,
"maximumK": 100
}
}
Complete schemas, descriptions and cross-field validation may impose additional limits.
Proposed price and protocol definitions
{
"unit": "one successful operation call",
"proposedNominalUsd": "0.01",
"sixDecimalTokenBaseUnits": "10000",
"subscription": false,
"includesPayerWalletOrNetworkFees": false,
"liveQuoteVerified": false,
"condition": "Actual SDK challenge is authoritative only within the caller's explicit authorization; configured six-decimal token peg is an operator assertion, not a conversion guarantee."
}
Protocol definitions: x402, mpp. MPP uses Tempo charge. Paid MCP execution is unsupported. All runtime readiness is not evaluated in this build.
API path templates, not endpoints on this documentation host
{
"x402": "/v1/x402/ranking-evaluate",
"mpp": "/v1/mpp/ranking-evaluate"
}
Required headers
{
"Content-Type": "application/json",
"Idempotency-Key": "random 16–128 character operation identifier"
}
Actual SDK challenge amount, asset, network, recipient and wallet costs must pass independent authorization. Preserve identical key, body, protocol and credential on retries; on PAYMENT_UNCERTAIN stop and reconcile.
Execution profile and provider conditions
{
"deterministic": true,
"externalRequests": 0,
"maxExternalRequests": 0,
"resultSnapshotPersisted": false,
"fixedExampleIsIllustrativeSnapshot": false,
"requiresPayment": true,
"supportsMcpExecution": false
}
Deterministic supplied-input operation with no external requests or stored request/result bodies. Payment infrastructure retains payment metadata and hashes.
Failure handling
- HTTP 400: Malformed JSON, missing/invalid idempotency key, or payment identifier mismatch Correct the request before payment
- HTTP 402: Payment challenge or rejected payment Use official protocol SDK; inspect payment outcome before another payment
- HTTP 409: Idempotency conflict, duplicate proof, or PAYMENT_UNCERTAIN Keep original key, body, and proof; reconcile uncertainty with operator; never blindly repay
- HTTP 413: Input or generated output too large Reduce input; no payment attempted for validation failure
- HTTP 415: Unsupported media type or compression Send uncompressed application/json
- HTTP 422: Schema or service-specific semantic validation failure Correct input using returned error code; no payment attempted
- HTTP 429: Request/payment-attempt rate exceeded Wait for rate limit window; preserve existing payment identity
- HTTP 503: Payment configuration/provider/state unavailable, or live DNS preparation failed before settlement Check readiness; DNS preparation failures may retry the identical key/body/credential only; uncertainty requires reconciliation
Declared requirements
Before any paid call, refresh the live operation contract and POST the complete bounded budgeted plan to the separate API's /preflight. Unknown requirements block selection; compatible preflight is not permission to spend.