git404hub

what is forte fr?

agi-eval-official/forte — explained in plain English

Analysis updated 2026-05-18

17PythonAudience · developerComplexity · 4/5Setup · hard

tl;dr

A benchmark that tests AI agents on real office tasks across 15 professions, judged by an AI against expert written rubrics.

vibe map

mindmap
  root((FORTE))
    What it does
      Office task benchmark
      15 professions
      LLM judge scoring
    Tech stack
      Python
      Docker
      OpenClaw
    Use cases
      Benchmark agent models
      Test office file handling
      Compare model performance
    Audience
      AI researchers
      Agent developers
      Benchmark builders

Code map

Detail Auto

An interactive map of this repo's files and how they connect — its source is parsed live in your browser. Click Visualize to build it.

filefunction / class

what do people make with this?

VIBE 1

Benchmark an AI agent's ability to complete real office tasks like editing spreadsheets or slides

VIBE 2

Compare different AI models on standardized office productivity tasks

VIBE 3

Test how well an agent handles multi format files such as Word, Excel, and PDF

what's the stack?

PythonDockerOpenClaw

how it stacks up fr

agi-eval-official/forte0petru/sentimoalingalingling/akasha-wechat
Stars171717
LanguagePythonPythonPython
Setup difficultyhardmoderatehard
Complexity4/53/54/5
Audiencedeveloperdeveloperdeveloper

Figures from each repo's GitHub metadata at analysis time.

how do i run it?

Difficulty · hard time til it works · 1h+

Requires Docker and an OpenAI compatible API gateway key.

in plain english

FORTE is a benchmark for testing AI agents on everyday office work. Instead of measuring coding skill alone, it checks how well an agent can handle tasks that people in real jobs do every day, spread across 15 corporate professions such as marketing, sales, finance, HR, legal, and design. Each task comes as a single file describing what to do, along with a small office style environment that can include spreadsheets, Word documents, slide decks, PDFs, CSV files, images, or source code. An AI judge then reviews the finished work and checks it against a list of rubric items written by human experts. A task only counts as a pass if every rubric item is satisfied. The full benchmark has 180 tasks with at least 10 per profession, but this repository ships one demo task for each profession so people can try the setup before running the whole thing. One profession, SRE, is missing its supporting skills because they depend on internal systems the maintainers cannot share. The project also suggests installing general office skills, such as ones for reading and editing Word, PowerPoint, Excel, and PDF files, to help the agent work with these file types more reliably. To run FORTE you need Docker and Python 3.10 or later. The agent itself runs inside a Docker container using a tool called OpenClaw, and the benchmark script builds that container image before running tasks. Setup involves copying example config files, filling in an API gateway URL and key, then building the Docker image and running a script. Each task can be run multiple times, and results are saved as JSON and workspace files so you can inspect exactly what the agent did and how it was judged. Three metrics summarize performance across the three runs per task: an average score, a pass rate, and a strict pass rate requiring every run to succeed. A Chinese translation of the documentation is included.

prompts (copy fr)

prompt 1
Help me set up FORTE with Docker and run the demo task for the marketing profession.
prompt 2
Explain how FORTE's all or nothing scoring works for a single task.
prompt 3
Show me how to add the Anthropic docx and pptx skills to my FORTE agent container.
prompt 4
Walk me through writing a new FORTE task file with frontmatter and grading criteria.

Frequently asked questions

what is forte fr?

A benchmark that tests AI agents on real office tasks across 15 professions, judged by an AI against expert written rubrics.

What language is forte written in?

Mainly Python. The stack also includes Python, Docker, OpenClaw.

How hard is forte to set up?

Setup difficulty is rated hard, with roughly 1h+ to a first successful run.

Who is forte for?

Mainly developer.

peek the repo → explain another one

This repo across BitVibe Labs

double-check against the repo, no cap.