Gated MOE - 4 bit bug

[David-AU-github]

When attempting to create a GATED moe ("hidden") ; received a 4 bit error from bits and bytes.

Corrected via edit to:

https://github.com/arcee-ai/mergekit/tree/main/mergekit/moe/router.py

commented out lines:

122, and 123

Seems to be an issue with 4bit/8 bit defaults change(s).

Once commented out and regen MERGEKIT - > no issues.

[etemiz]

does that mean you are able to merge 4 bit bitsandbytes?
does mergekit work with those 4bit models that for example unsloth trains?

[Wojtekb30]

I confirm there is an error related to that, load_in_4bit and load_in_8bit are no more and mergekit crashes. Fixing that requires editing the router.py and moe.py to remove occurences of the parameters.

I could quickly provide a simple fix by adding yet another flag like --noquant which thru just an if statement would go thru the script but without ever using load_in_4bit nor load_in_8bit, resulting in no quantization being used but at least preventing the crash.