Skip to content

README.md

Latest commit

 

History

History
63 lines (47 loc) · 2.35 KB

README.md

File metadata and controls

63 lines (47 loc) · 2.35 KB

LLM Context Window Benchmark

A visual benchmark comparing how different Large Language Models (LLMs) handle complex coding prompts, particularly for games and interactive UI. This project serves as a "context-window-in-action" gallery.

🚀 How it Works

The project is a static site that aggregates benchmark results from various models (Gemini, Claude, GPT, Grok). Each benchmark is a directory containing:

  1. prompt.txt: The exact prompt given to the models.
  2. Sub-directories for each model (e.g., gemini, claude) containing the generated index.html.
  3. modelnames.json: Mapping internal IDs to human-readable names.

🏗 Directory Structure

.
├── create_config.sh    # Script to regenerate the gallery index
├── index.html          # Main gallery UI
├── flappy/             # Benchmark: Flappy Bird clone
│   ├── prompt.txt      # The prompt used
│   ├── gemini/         # Result from Gemini
│   │   └── index.html
│   └── claude/         # Result from Claude
│       └── index.html
└── platformer/         # Benchmark: Platformer game
    └── ...

🛠 Features

  • Side-by-Side Comparison: view model outputs for the same prompt in one interface.
  • Dynamic Config Generation: Just drop a new result folder and run create_config.sh.
  • Vanilla Implementation: No heavy frameworks, just fast, static HTML/JS.

🤝 How to Contribute

We welcome contributions of new benchmarks or new model results for existing benchmarks!

1. Adding a New Model Result

If you want to add a result for a model (e.g., "DeepSeek") to an existing benchmark (e.g., flappy):

  1. Create a folder named deepseek inside flappy/.
  2. Add the generated index.html file into flappy/deepseek/.
  3. (Optional) Add the model name to flappy/modelnames.json.
  4. Run ./create_config.sh to update the site.

2. Adding a New Benchmark

  1. Create a new root folder (e.g., tetris/).
  2. Add a prompt.txt with the prompt you used.
  3. Add folders for each model you tested.
  4. Run ./create_config.sh.

💻 Local Development

  1. Clone the repository.
  2. To view the site, you can use any static server, like npx phost or python -m http.server.
  3. After adding new folders or files, run: 
    bash create_config.sh

📜 License

See LICENSE.md for details.