2024-02-13T18:59:00
Status: #blog
Tags: #ai #llm #nvidia
Links: [[Technology]] | [[AI]] | [[LLM]] | [[NVIDIA]]
# Chat with RTX
Running a LLM on your own PC is indeed a unique experience. It's one thing to chat with an abstract AI running somewhere in the cloud, such as [ChatGPT](https://chat.openai.com). And it is an entirely different feeling when you have a somewhat intelligent conversation with the GPU on your own graphics card inside your own laptop or desktop PC... 😮
![[nvidia-ai-on-rtx-owned-web-module-bb580_440-l.jpg]]
I have previously experimented with [LM Studio](https://lmstudio.ai/), of course, and ran several open source LLMs on my PC and even on my laptop in the last couple of months, but they were neither trained on my own data, nor were they very capable in terms of either performance or cognitive capabilities.
[Chat With RTX](https://www.nvidia.com/en-us/ai-on-rtx/chat-with-rtx-generative-ai/) is a new demo app from NVIDIA just released today that lets you personalize a GPT large language model ([[LLM]]) connected to your own content—docs, notes, videos, or other data. Leveraging [retrieval-augmented generation (RAG)](https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/), [TensorRT-LLM](https://developer.nvidia.com/blog/nvidia-tensorrt-llm-supercharges-large-language-model-inference-on-nvidia-h100-gpus/), and RTX acceleration, you can query a custom chatbot to quickly get contextually relevant answers. And because it all runs locally on your Windows RTX PC or workstation, you’ll get fast and secure results.
The download and installation takes a bit of time, because it is 35GB in size (since it includes the language model), so you need some patience to first retrieve, unzip, and then install it, with each step taking about 10 minutes or so. But it is well worth the wait!
The LLM being used at the core of Chat With RTX is appears to be a is a instruct fine-tuned version of the [Mistral 7B](https://mistral.ai/news/announcing-mistral-7b/) generative text model using a variety of publicly available conversation datasets (see https://catalog.ngc.nvidia.com/orgs/nvidia/models/mistral-7b-int4-chat).
Per default, the app comes with a dataset that includes several of the most recent blog posts from NVIDIA, but you can add your own data as .PDF, .TXT, or .DOC files. Once you have done that and reloaded the dataset, you can start talking to the AI. For my initial tests, I added all the governance documents of [Rotary International](https://rotary.org), including its bylaws, constitution, and code of policies...
> [!faq|] Question
> What are the qualification requirements for a candidate for district governor for Rotary?
> [!answer]
> The qualification requirements for a candidate for district governor for Rotary are as follows:
> 1. The candidate must have served a full term as a governor prior to being proposed as a candidate, with at least three years of time having elapsed since service as a governor.
> 2. The candidate must have attended at least two institutes and one convention in the 36-month period prior to being proposed.
> 3. The candidate must have professional experience with significant executive leadership accomplishments in business, government, philanthropy, or the nonprofit sector.
> 4. The candidate must have a commitment to and understanding of Rotary's members and beneficiaries.
> 5. The candidate must have diplomatic skills and a natural affinity for cultivating relationships and building consensus among diverse individuals.
> 6. The candidate must have personal qualities of integrity, credibility, and a passion for improving Rotary.
> 7. The candidate must have knowledge of Rotary International and The Rotary Foundation programs, finances, operations, and administration.
> 8. The candidate must have experience in serving on other governing bodies with specific focus on non-profit or charitable aims.
> 9. The candidate must have strong collaboration and negotiation skills.
> 10. The candidate must have effective communication skills (both verbal and written).
> 11. The candidate must be able to translate the organizational vision into an effective action plan during their year.
> 12. The candidate must be able to work collaboratively with predecessors and successors.
>
> In addition to these qualifications, the candidate must also serve as an ex-officio member of the district Youth Exchange committee.
>
> Reference files:
> [code_of_policies_rotary_international_en.pdf](https://my-cms.rotary.org/en/document/rotary-code-policies)
That is actually a pretty accurate and complete answer, and the speed in which the system responded was quite impressive. So my second test was a bit more of a delicate question:
> [!faq] Question
> Assume you have an elderly Rotary club member who makes inappropriate jokes that make a younger LGBTQ member uncomfortable. How would you counsel a Rotary Club President to handle this situation in accordance with Rotary's Diversity, Equity, and Inclusion guidelines?
> [!answer]
> As a helpful assistant, I would advise the Rotary Club President to handle the situation in accordance with Rotary's Diversity, Equity, and Inclusion guidelines. The President should first ensure the safety of the younger LGBTQ member and provide them with any necessary support. Then, the President should follow the steps outlined in the Rotary Code of Policies for reporting issues of harassment involving Rotarians or Rotaractors at meetings, events, or activities. This includes notifying a club officer, district leader, or zone leader, as well as reporting the incident to Rotary International's Club and District Support team. The President should also be mindful of Rotary's commitment to creating a welcoming and inclusive environment and celebrate diversity. It is important to remember that everyone has the right to feel respected and valued, regardless of their age, gender, sexual orientation, or any other characteristic.
>
> Reference files:
> [code_of_policies_rotary_international_en.pdf](https://my-cms.rotary.org/en/document/rotary-code-policies)
And it handled that pretty well, too. I'm actually quite impressed. And all of that running locally on my laptop using a GeForce RTX 4060 with 8GB RAM and with a response time that is comparable to ChatGPT.
---
# References
- https://www.nvidia.com/en-us/ai-on-rtx/chat-with-rtx-generative-ai/
- https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/
- https://developer.nvidia.com/blog/nvidia-tensorrt-llm-supercharges-large-language-model-inference-on-nvidia-h100-gpus/
- https://mistral.ai/news/announcing-mistral-7b/
- https://catalog.ngc.nvidia.com/orgs/nvidia/models/mistral-7b-int4-chat