Skip to content

feat(evaluation): add AMD evalscope support #1049

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
luojun96 merged 2 commits into from
May 6, 2026
+133 −3
Merged

feat(evaluation): add AMD evalscope support #1049

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
63dec9d
feat(evaluation): add AMD evalscope support
ganisback May 6, 2026
28c5490
fix: use reflect.Pointer in internal message channel
May 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 102 additions & 0 deletions configs/evaluation/amd-evalscope.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
{
"engine_name": "amd-evalscope",
"enabled": 1,
"model_format": "safetensors",
"engine_images": [
{
"compute_type": "gpu",
"image": "opencsghq/evalscope-amd:1.6.0-rocm7.2.2",
"driver_version": "7.2",
"engine_version": "1.6.0"
}
],
"supported_archs": [
"AfmoeForCausalLM",
"AquilaForCausalLM",
"AquilaModel",
"ArcticForCausalLM",
"BaiChuanForCausalLM",
"BaichuanForCausalLM",
"BambaForCausalLM",
"BertModel",
"BloomForCausalLM",
"BartForConditionalGeneration",
"ChatGLMModel",
"ChatGLMForConditionalGeneration",
"CohereForCausalLM",
"Cohere2ForCausalLM",
"CwmForCausalLM",
"DbrxForCausalLM",
"DeciLMForCausalLM",
"DeepseekForCausalLM",
"DeepseekV2ForCausalLM",
"DeepseekV3ForCausalLM",
"ExaoneForCausalLM",
"FalconForCausalLM",
"FalconMambaForCausalLM",
"Gemma2Model",
"GemmaForCausalLM",
"Gemma2ForCausalLM",
"Gemma3ForCausalLM",
"GlmForCausalLM",
"Glm4ForCausalLM",
"GPT2LMHeadModel",
"GPTBigCodeForCausalLM",
"GPTJForCausalLM",
"GPTNeoXForCausalLM",
"GraniteForCausalLM",
"GraniteMoeForCausalLM",
"GraniteMoeSharedForCausalLM",
"GritLM",
"Grok1ModelForCausalLM",
"InternLMForCausalLM",
"InternLM2ForCausalLM",
"InternLM3ForCausalLM",
"JAISLMHeadModel",
"JambaForCausalLM",
"LlamaModel",
"LlamaForCausalLM",
"Llama4ForCausalLM",
"MambaForCausalLM",
"MiniCPMForCausalLM",
"MiniCPM3ForCausalLM",
"MiniMaxM1ForCausalLM",
"MiniMaxM2ForCausalLM",
"MistralModel",
"MistralForCausalLM",
"MixtralForCausalLM",
"MPTForCausalLM",
"NemotronForCausalLM",
"OLMoForCausalLM",
"OLMo2ForCausalLM",
"OLMoEForCausalLM",
"OlmoForCausalLM",
"Olmo2ForCausalLM",
"OlmoeForCausalLM",
"OPTForCausalLM",
"OrionForCausalLM",
"PhiForCausalLM",
"Phi3ForCausalLM",
"Phi3SmallForCausalLM",
"PhiMoEForCausalLM",
"PersimmonForCausalLM",
"Plamo2ForCausalLM",
"QWenLMHeadModel",
"Qwen2Model",
"Qwen2ForCausalLM",
"Qwen2MoeForCausalLM",
"Qwen3ForCausalLM",
"Qwen3MoeForCausalLM",
"RobertaModel",
"RobertaForMaskedLM",
"StableLmForCausalLM",
"Starcoder2ForCausalLM",
"SolarForCausalLM",
"TeleChat2ForCausalLM",
"TeleFLMForCausalLM",
"XverseForCausalLM",
"XLMRobertaModel",
"MiniMaxText01ForCausalLM",
"Zamba2ForCausalLM"
]
}
28 changes: 28 additions & 0 deletions docker/evaluation/Dockerfile.evalscope-amd
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
FROM docker.1ms.run/rocm/pytorch:rocm7.2.2_ubuntu22.04_py3.10_pytorch_release_2.10.0

RUN sed -i 's|http://archive.ubuntu.com/ubuntu|http://mirrors.aliyun.com/ubuntu|g' /etc/apt/sources.list && \
sed -i 's|http://security.ubuntu.com/ubuntu|http://mirrors.aliyun.com/ubuntu|g' /etc/apt/sources.list && \
sed -i 's|http://ports.ubuntu.com/ubuntu-ports|https://mirrors.aliyun.com/ubuntu-ports/|g' /etc/apt/sources.list && \
apt-get update && apt-get -y install dumb-init \
&& apt-get clean && rm -rf /var/lib/apt/lists/*

RUN pip install --no-cache-dir minio oss2 openpyxl csghub-sdk==0.7.2
RUN pip install -U --no-cache-dir evalscope==1.6.0 accelerate

# Download NLTK resources required for BLEU and other metrics
RUN python -c "from evalscope.utils.resource_utils import check_nltk_data; check_nltk_data('punkt_tab')"

COPY ./evalscope/ /etc/csghub/

RUN ln -sf /usr/bin/python3 /usr/bin/python && \
chmod +x /etc/csghub/*.sh && \
chmod +x /etc/csghub/*.py

WORKDIR /workspace/

ENV HUGGINGFACE_HUB_CACHE=/workspace/ \
HF_HUB_ENABLE_HF_TRANSFER=0 \
HF_HUB_DOWNLOAD_TIMEOUT=30

ENTRYPOINT [ "/usr/bin/dumb-init", "--" ]
CMD ["/etc/csghub/start.sh"]
4 changes: 2 additions & 2 deletions docker/evaluation/evalscope/start.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,8 @@ export HF_TOKEN=$ACCESS_TOKEN
mkdir -p /workspace/data

# Ensure NLTK resources are available for BLEU and other metrics
echo "Checking NLTK resources..."
python -c "import nltk; nltk.download('punkt_tab', quiet=True)" 2>&1 || echo "[WARNING] Failed to download NLTK punkt_tab resource"
#echo "Checking NLTK resources..."
#python -c "import nltk; nltk.download('punkt_tab', quiet=True)" 2>&1 || echo "[WARNING] Failed to download NLTK punkt_tab resource"

# Register custom datasets
echo "Registering custom datasets..."
Expand Down
2 changes: 1 addition & 1 deletion notification/notifychannel/channel/internalmsg/internalmsg.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ func (s *InternalMessageChannel) sendInternalMessageToUsers(ctx context.Context,

func extractMessageFromData(data any) (title, content string) {
v := reflect.ValueOf(data)
if v.Kind() == reflect.Ptr {
if v.Kind() == reflect.Pointer {
v = v.Elem()
}
if v.Kind() == reflect.Struct {
Expand Down
Loading