Where to request benchmark allow-list access?

[lianghsun]

Hi! I've set up my dataset as a benchmark following the eval-results docs — eval.yaml is in the repo root. Could you add it to the benchmark allow-list?

Dataset: https://huggingface.co/datasets/lianghsun/tw-legal-benchmark-v1

It's a 209-question multiple-choice benchmark on Taiwan law (Traditional Chinese), using the inspect-ai framework. Thanks!

[julien-c]

do you have some model evals already?

cc @NathanHB

[lianghsun]

@julien-c Yes! Several Taiwan-focused models have already been evaluated on this benchmark:

yentinglin / Llama-3-Taiwan series

The Llama-3-Taiwan-70B-Instruct model card uses tw-legal-benchmark-v1 as a standard eval column (listed as "Legal Eval" with a direct link to the dataset):

Model	Legal Eval
Llama-3-Taiwan-70B-Instruct-DPO	70.33%
Llama-3-Taiwan-70B-Instruct	68.42%
Llama-3-Taiwan-8B-Instruct	53.11%
Claude-3-Opus	60.29%
GPT-4o	53.59%
Meta-Llama-3-70B-Instruct	52.63%

yentinglin also built the Open TW LLM Leaderboard which includes this benchmark as one of its standard columns.

twinkle-ai models

twinkle-ai evaluates all their Traditional Chinese models on this benchmark via Twinkle Eval (which lists tw-legal-benchmark-v1 as a supported benchmark):

Model	Taiwan Legal
gemma-3-4B-T1-it	44.18%
Llama-3.2-3B-F1-Instruct	35.24%
Llama-3.2-3B-F1-Reasoning-Instruct	34.92%

In total, 12+ models (including proprietary ones like Claude-3-Opus and GPT-4o) have published scores on this benchmark. It has become a de facto standard for evaluating Traditional Chinese legal capabilities in the Taiwan LLM community.

Happy to provide any additional information needed!