Built by Salesforce AI Research, GIFT-Eval-CTX measures how effectively LLMs and agentic AI convert forward-looking context, such as promotions or weather events, into precise forecasts. The benchmark demonstrates that feeding structured operational context into agentic AI drastically cuts error rates, turning static predictions into reliable, proactive planning.
Moving Beyond Historical Data
In enterprise operations, traditional time-series forecasting relies on a single premise: the past predicts the future.
Whether you are managing energy grid loads, optimizing logistics for global supply chains, scaling cloud infrastructure, or balancing retail inventory, traditional time series foundation models (TSFMs) analyze historical trends, seasonality, and variance to project what comes next.
However, the real world rarely operates on autopilot.
An energy operator knows a heatwave is coming, a supply chain director knows a road closure will divert freight, and an IT leader knows a software release will spike server load. In these moments, historical data falls short because the critical insights live in a forward-looking business context.
TSFMs, such as Moirai-2, learn from large collections of time series and can forecast new ones without additional training. But their standard interfaces primarily accept numerical history and predefined variables. Incorporating open-ended information from documents, plans, or expert judgment presents a different challenge.
Contextual forecasting brings that information into the prediction process. It requires understanding what the context implies about the future and translating those implications into numerical forecasts. Large language models (LLMs) offer a way to interpret diverse contextual information. But their ability to transform this knowledge into numerical predictions remains unclear.
To develop and compare LLM-based approaches, we need a consistent way to measure their abilities. GIFT-Eval-CTX asks:
Given reliable information about future changes, can forecasting systems turn it into accurate numerical forecasts?
Inside GIFT-Eval-CTX
Expanding on Salesforce’s flagship GIFT-Eval benchmark, GIFT-Eval-CTX evaluates models across 10,000 tasks from 20 real-world datasets spanning energy, logistics, weather, retail, and cloud operations.
Instead of relying on inconsistent free-form text, an automated pipeline extracts future operational data into standardized structured effects to measure how accurately forecasting systems turn context into numerical forecasts.
Structured Effects: Macro Trends & Micro Disruptions
GIFT-Eval-CTX categorizes future context into two actionable layers:
- Macro Effects (The Big Picture): Tracks horizon-wide behavior, including overall trends (rising, falling, flat), seasonal cycles, and value bounds.
- Micro Effects (Localized Events): Pinpoints specific time windows and scales for three event types:
- Excursions: Temporary spikes or drops (e.g., Black Friday traffic or server outages).
- Mean Shifts: Permanent baseline level changes (e.g., opening a new facility).
- Seasonal Changes: Shifts in recurring cycles (e.g., altered store hours).
Together, these layers give the AI the future’s shape and key events, leaving the model to fill in the exact step by step numbers.
Figure 1: Overview of the GIFT-Eval-CTX framework. Forward-looking operational context is decomposed into horizon wide Macro trends and localized Micro effects to guide forecasting systems.
How the Benchmark Is Constructed
GIFT-Eval-CTX specifically selects difficult test windows where traditional time-series models fail using past data alone. Following rigorous automated critique and human spot-checking, the final benchmark encompasses 23,476 validated Micro effects paired with corresponding Macro horizon annotations. The data annotation pipeline includes:
- Discovery Agents: Scan trends and actual outcomes to map Macro patterns and locate where traditional models missed local events.
- Code Validation: Agents run deterministic code tools to test annotations against raw data and fix mathematical errors.
- Joint Critic Agent: An independent reviewer inspects the raw data to validate, reject, or refine annotations before final inclusion.
Key Benchmark Takeaways
- Forward-Looking Context Significantly Cuts Error
While pure LLMs underperform traditional forecasting models when given history alone, adding structured future context reduces forecasting error dramatically: GPT-5.6 Sol’s error dropped by 40.7% and Claude Opus 4.8’s by 29.6%. - Agentic Systems Deliver The Best Accuracy
Combining an LLM with domain-specific models and code execution (MoiraiAgent) delivered the top performance across every forecast length. MoiraiAgent achieved an aggregate error score 36.3% lower than the best history only forecasting model, with its advantage widening on longer forecasts. - Micro Effects Improve Overall Forecast Structure
Beyond localized accuracy, providing event context helps agents navigate transitions between multiple events and preserve critical repeating cycles (e.g., daily electricity demand patterns) that standard models flatten over time.
Table 1: Comparative evaluation across forecasting systems measured by aggregated MASE scores (lower is better). Incorporating structured future context yields significant error reductions across frontier LLMs, with tool-augmented agentic architectures (MoiraiAgent) achieving superior accuracy across all forecast horizons.
Relying on Pure LLMs for Forecasting Is a Risk
Testing revealed a major flaw in relying on pure LLMs for raw forecasting: they struggle with numerical precision and output length.
- Without tools, GPT-5.6 Sol generated the correct forecast length in only 38.9%–40.5% of medium-to-long tasks, while Claude Opus 4.8 succeeded in less than 2.3%.
MoiraiAgent solves this through a two-step tool architecture:
- Foundation Service Call: Calls the Moirai-2 engine to establish a mathematically complete numerical baseline of the exact required length.
- Code Execution: Uses code tools to calculate exact mathematical targets, apply context adjustments, and verify constraint compliance before outputting final results.
The Future of Forecasting is here
The findings from GIFT-Eval-CTX highlight a critical milestone in AI-driven enterprise planning: feed AI systems structured future context, and forecasting accuracy leaps dramatically. To help researchers, developers, and enterprise teams evaluate and build the next wave of context aware forecasting systems, Salesforce AI Research has made the GIFT-Eval-CTX benchmark, dataset, and leaderboard fully open source. Explore our resources below and join us in advancing the future of intelligent operational planning.
