· 6 min read ·

Grounding, Schema Enforcement, and Error Design: Engineering Fixes From OpenEnv's Calendar Benchmark

Source: huggingface

Looking back a month at the OpenEnv Calendar Gym benchmark, most of the analysis has focused on what the results reveal about agent evaluation methodology. Less has gone into the engineering response: given what the benchmark found, what do you change about how you build agents that call external APIs.

This is worth thinking through concretely, because the failure modes the Calendar Gym surfaced are not obscure edge cases. They are the failures that appear in any production system where an agent translates natural language requests into structured API calls.

The Three Findings Worth Building From

The Calendar Gym, built by Turing Enterprises on the OpenEnv framework, tested agents against real Google Calendar APIs with real OAuth scopes and real permission enforcement. Three findings have direct engineering implications.

First, a 50-point success rate gap between tasks specified with explicit identifiers versus natural language descriptions. Agents succeed roughly 90% of the time when given a calendarId string; success drops to around 40% when working from a description like “my team’s shared calendar.”

Second, more than half of failures came from malformed arguments, not from wrong tool selection. Agents identified the correct tool but failed to populate it correctly: missing required fields, non-RFC3339 datetime formats, incorrect object nesting in the start/end structure.

Third, structured error payloads with actionable remediation information improved agent recovery rates compared to generic failure messages.

These three findings point to three different engineering layers: the grounding layer, the argument formation layer, and the error feedback layer.

The Grounding Layer

The 50-point gap is a grounding problem. Before an agent can form a valid API call from natural language, it needs to resolve the language into API-acceptable values. “My team’s shared calendar” needs to become a specific calendarId. “Next Tuesday at 3pm” needs to become an RFC3339 datetime string with an explicit timezone offset.

The typical mistake is treating grounding as implicit. The model reads the task description, reasons about it, and produces arguments. This works when the required values are in the task description. It fails when those values exist in an external system and need to be retrieved first.

The fix is to make grounding an explicit phase. Before calling the target tool, run a lookup step:

async def resolve_calendar_id(agent, description: str) -> str:
    """Fetch the calendar list and match the description against it."""
    calendars = await agent.call_tool("calendars_list", {})
    return await agent.resolve_entity(
        query=description,
        candidates=calendars["items"],
        match_field="summary",
        return_field="id"
    )

async def create_event(agent, task: dict):
    # Grounding phase: resolve all natural language references first
    calendar_id = await resolve_calendar_id(agent, task["calendar_description"])
    start_dt = normalize_datetime(task["start_description"], task["user_timezone"])
    end_dt = normalize_datetime(task["end_description"], task["user_timezone"])

    # Tool call phase: all values are now concrete and typed
    return await agent.call_tool("events_insert", {
        "calendarId": calendar_id,
        "start": {"dateTime": start_dt, "timeZone": task["user_timezone"]},
        "end": {"dateTime": end_dt, "timeZone": task["user_timezone"]},
    })

The entity resolution step is a mini-task in itself. It fetches a list of candidates and asks the model to match a natural language description against them. This is more reliable than asking the model to guess an identifier cold, and it produces a value from the actual system rather than a hallucinated one.

The same pattern applies to any API where identifiers exist in a system rather than in the user’s request. tau-bench’s customer service domain runs into an equivalent problem: user references to accounts, orders, and policies need to be resolved before the agent can call the right tools. The Calendar Gym made this quantifiable, but the underlying failure mode is general.

The Argument Formation Layer

More than half of Calendar Gym failures involved malformed arguments. The most common categories: missing required fields, datetime strings without timezone offsets, and incorrect object nesting in the Google Calendar event structure. These are serialization failures, not reasoning failures. The agent understood what it was trying to do and produced arguments that did not conform to the schema.

The direct mitigation is structured output generation. Instead of asking the model to produce arguments as free-form text that gets parsed into JSON, define the schema explicitly and enforce it at inference time.

Using Pydantic with structured completions:

from pydantic import BaseModel, Field

class EventDateTime(BaseModel):
    dateTime: str = Field(
        description="RFC3339 with timezone offset, e.g. 2026-02-15T10:00:00-08:00"
    )
    timeZone: str = Field(
        description="IANA timezone name, e.g. America/Los_Angeles"
    )

class CreateEventArgs(BaseModel):
    calendarId: str = Field(
        description="Calendar ID from calendars_list output, not a display name"
    )
    summary: str
    start: EventDateTime
    end: EventDateTime
    description: str = ""

args = await agent.structured_completion(
    prompt=f"Create event: {task_description}",
    output_schema=CreateEventArgs
)

The Field(description=...) annotations matter beyond documentation. When these schemas are injected into the model’s prompt, the field descriptions serve as targeted instructions co-located with the fields they govern. Telling the model “RFC3339 with timezone offset, e.g. 2026-02-15T10:00:00-08:00” in the schema is more reliable than including that instruction in a system prompt, because it appears next to the field where the model needs to apply it.

The Anthropic API supports structured outputs via tool use with defined input schemas; OpenAI’s Structured Outputs provides similar capability. Using explicit schemas for every tool call rather than only for “complex” ones eliminates the category of failures where the model knew what to do but serialized it incorrectly.

The Error Feedback Layer

The Calendar Gym finding on error feedback has a direct implementation implication: the error messages your tools return are part of the system’s capability.

When a Google Calendar API call fails with a 403, the raw response looks like this:

{
  "error": {
    "code": 403,
    "message": "The caller does not have permission",
    "status": "PERMISSION_DENIED"
  }
}

That tells an agent it failed but provides no path forward. A wrapper that converts this into something actionable changes the agent’s options on the next step:

{
  "error": "permission_denied",
  "calendarId": "team-shared@company.com",
  "required_scope": "https://www.googleapis.com/auth/calendar",
  "remediation": "Use a calendar ID with write access, or request elevated permissions.",
  "available_calendars_with_write_access": ["primary", "user-personal@example.com"]
}

The second response includes enough information for the agent to try a different calendar ID without restarting the task from the beginning. The Calendar Gym found that structured remediation information improved recovery rates. The translation is: write error wrappers for every API you integrate with agents, and include the information the agent would need to make a different decision.

This is not a novel idea in software engineering. Good error messages have always been more useful than bad ones. What the Calendar Gym adds is a quantified demonstration that this matters at the agent level too, where the “developer” reading the error is a model reasoning about what to try next.

What This Costs

These three layers, explicit grounding, schema-enforced argument formation, and structured error wrapping, add more structure than a minimal ReAct loop. Grounding requires knowing which tools return lookup data and sequencing them before the target call. Argument formation requires maintaining Pydantic models or JSON schemas for every tool. Error wrapping requires custom logic for every API you integrate.

The Calendar Gym’s numbers suggest this work is worth the cost. A 50-point gap between structured inputs and natural language inputs represents two operating regimes, and which one your users land in depends on how they phrase their requests. Closing that gap requires the grounding layer. Closing the argument malformation failures requires schema enforcement. Closing the error recovery failures requires better feedback design.

The OpenEnv framework makes it possible to test all three layers against real API behavior before deployment, using the same Gymnasium-compatible interface that connects to RL training frameworks like TRL and VeRL. That evaluation-to-training continuity is what makes the Calendar Gym’s findings actionable rather than just diagnostic. The failure modes it surfaces in evaluation are the same failure modes that appear in deployment, which means fixes that work in evaluation have a reasonable chance of working where it matters.

Was this interesting?