You’ll find the code on this GitHub repo:
https://github.com/amirarsalan90/personal_llm_assistant
The principle parts of the app embody:
Llama-cpp-python is a python binding for the good llama.cpp , which implements many Giant Language Fashions in C/C++ . Due to its broad adoption by open-source neighborhood, I made a decision to make use of it on this tutorial.
Be aware: I’ve examined this app on a system with Nvidia RTX4090 gpu.
Very first thing first, lets create a brand new conda atmosphere:
conda create --name assistant python=3.10
conda activate assistant
Subsequent we have to set up llama-cpp-python. As talked about in llama-cpp-python descriptions, llama.cpp helps plenty of {hardware} acceleration backends to hurry up inference. So as to leverage the GPU and run the LLM on GPU, we are going to construct this system with CUBLAS. I had some points with getting to dump the mannequin on GPU and I lastly discovered this put up on easy methods to correctly set up:
export CMAKE_ARGS="-DLLAMA_CUBLAS=on"
export FORCE_CMAKE=1
pip set up --upgrade --force-reinstall llama-cpp-python --no-cache-dir
pip set up llama-cpp-python[server]
