Understanding how a lot reminiscence it’s essential serve a VLM
vLLM is presently one of many quickest inference engines for giant language fashions (LLMs). It helps a variety of mannequin architectures and quantization strategies.
vLLM additionally helps vision-language fashions (VLMs) with multimodal inputs containing each photos and textual content prompts. As an illustration, vLLM can now serve fashions like Phi-3.5 Imaginative and prescient and Pixtral, which excel at duties reminiscent of picture captioning, optical character recognition (OCR), and visible query answering (VQA).
On this article, I’ll present you the way to use VLMs with vLLM, specializing in key parameters that influence reminiscence consumption. We’ll see why VLMs devour rather more reminiscence than commonplace LLMs. We’ll use Phi-3.5 Imaginative and prescient and Pixtral as case research for a multimodal software that processes prompts containing textual content and pictures.
The code for working Phi-3.5 Imaginative and prescient and Pixtral with vLLM is offered on this pocket book:
Get the pocket book (#105)
In transformer fashions, producing textual content token by token is sluggish as a result of every prediction is dependent upon all earlier tokens…
