There are a number of methods to find out the price of working a LLM (electrical energy use, compute value, and so forth.), nonetheless, for those who use a third-party LLM (a LLM-as-a-service) they sometimes cost you based mostly on the tokens you utilize. Completely different distributors (OpenAI, Anthropic, Cohere, and so forth.) have alternative ways of counting the tokens, however for the sake of simplicity, we’ll think about the fee to be based mostly on the variety of tokens processed by the LLM.
An important a part of this framework is the concept totally different fashions value totally different quantities. The authors of the paper conveniently assembled the beneath desk highlighting the distinction in value, and the distinction between them is important. For instance, AI21’s output tokens value an order of magnitude greater than GPT-4’s does on this desk!
As part of value optimization we at all times want to determine a method to optimize the reply high quality whereas minimizing the fee. Usually, larger value fashions are sometimes larger performing fashions, in a position to give larger high quality solutions than decrease value ones. The overall relationship may be seen within the beneath graph, with Frugal GPT’s efficiency overlaid on prime in purple.
Utilizing the huge value distinction between fashions, the researchers’ FrugalGPT system depends on a cascade of LLMs to provide the person a solution. Put merely, the person question begins with the most cost effective LLM, and if the reply is sweet sufficient, then it’s returned. Nonetheless, if the reply shouldn’t be adequate, then the question is handed alongside to the following most cost-effective LLM.
The researchers used the next logic: if a cheaper mannequin solutions a query incorrectly, then it’s possible {that a} dearer mannequin will give the reply appropriately. Thus, to attenuate prices the chain is ordered from least costly to costliest, assuming that high quality goes up as you get dearer.
This setup depends on reliably figuring out when a solution is sweet sufficient and when it isn’t. To unravel for this, the authors created a DistilBERT mannequin that may take the query and reply then assign a rating to the reply. Because the DistilBERT mannequin is exponentially smaller than the opposite fashions within the sequence, the fee to run it’s virtually negligible in comparison with the others.
One would possibly naturally ask, if high quality is most necessary, why not simply question the most effective LLM and work on methods to scale back the price of working the most effective LLM?
When this paper got here out GPT-4 was the most effective LLM they discovered, but GPT-4 didn’t at all times give a greater reply than the FrugalGPT system! (Eagle-eyed readers will see this as a part of the fee vs efficiency graph from earlier than) The authors speculate that simply as probably the most succesful individual doesn’t at all times give the best reply, probably the most advanced mannequin gained’t both. Thus, by having the reply undergo a filtering course of with DistilBERT, you’re eradicating any solutions that aren’t as much as par and rising the percentages of an excellent reply.
Consequently, this method not solely reduces your prices however also can improve high quality extra so than simply utilizing the most effective LLM!
The outcomes of this paper are fascinating to think about. For me, it raises questions on how we are able to go even additional with value financial savings with out having to spend money on additional mannequin optimization.
One such risk is to cache all mannequin solutions in a vector database after which do a similarity search to find out if the reply within the cache works earlier than beginning the LLM cascade. This might considerably scale back prices by changing a pricey LLM operation with a relatively cheaper question and similarity operation.
Moreover, it makes you marvel if outdated fashions can nonetheless be price cost-optimizing, as for those who can scale back their value per token, they will nonetheless create worth on the LLM cascade. Equally, the important thing query right here is at what level do you get diminishing returns by including new LLMs onto the chain.
