Support for InternVL

[chigkim]

New InternVL-Chat-V1.5 just came out, and the quality is really great, and the benchmark score is pretty high too. Possibly best open source vision language model yet?

Can we have llama.cpp to support it? @cmp-nct, @cjpais, @danbev, @monatis, has any of you tried it?

Demo: https://internvl.opengvlab.com/

[paryska99]

Would be great

[cjpais]

I am working on a few projects right now, but if I get a chance I will try to get support in (assuming it doesn't already work). I would also like to get moondream support in

[2132660698]

+1

[cjpais]

fwiw moondream support was merged in #6899, haven't had a chance to look at/try internvl

[sapere-aude-incipe]

I would really like to get InternVL support in llama.cpp.

I have tested the demo extensively and it is really good, so much so that I feel like it is a game changer in many ways. But running it on consumer hardware is not possible right now.

As noted here: InternLM/lmdeploy#1501 (comment)

architecture: InternViT-6B-448px-V1-5 + MLP + InternLM2-Chat-20B
I am afraid it cannot fit into A10 (24G) even though LLM weights are quantized into 4 bits.

Is it possible to GGUF the weights to allow for multi GPU splitting or splitting layers between CPU RAM and VRAM? Adding support for InternVL 1.5 would also (probably) make it easier to support future versions when they come out.

[Single430]

@cjpais Hello, may I ask what is the progress of internvl support now? We are looking forward to using it on llama.cpp.

[cjpais]

Hey I am quite busy with a few projects, it's on my list but just not very high priority at the moment. It's really only something I can do in my spare/free time

[Single430]

Hey I am quite busy with a few projects, it's on my list but just not very high priority at the moment. It's really only something I can do in my spare/free time

Thank you for your reply. Thank you for your hard work. Looking forward to your future work.

[chigkim]

Which one would be better to focus: CogVLM or InternVL?

I wish there is more resource/interest for language vision models among the llama.cpp community. Llama.cpp is the only hope to run newer language vision models on Apple Silicon. Especially since flash attention python library is not available for Apple Sillicon, you can't even run inference using Torch with MPS support. :(

[opisaac9001]

Which one would be better to focus: CogVLM or InternVL?

I wish there is more resource/interest for language vision models among the llama.cpp community. Llama.cpp is the only hope to run newer language vision models on Apple Silicon. Especially since flash attention python library is not available for Apple Sillicon, you can't even run inference using Torch with MPS support. :(

Please internVL,. In my tests it works better than CogVLM. Especially for stuff like receipts and documents.

[fzzylogic]

InternVL is quite good. Benchmarks, HF, Demo.

[DoiiarX]

how about now? any update?

[James4Ever0]

upvote for this