Local Llms: Connecting Appsmith to Llama3 On an M1 Macbook

1. Appsmith Server

   For this guide, I'll be hosting Appsmith on my Mac using Docker. However, the same approach should work to connect to Ollama running on a Windows or Linux server. 

   Regardless of your OS, and server location (local or cloud), connecting from Appsmith to Ollama on the same machine is a similar process. 

   Note: This won't work with Appsmith's free cloud. You'll need your own server, where you can install Ollama on the same machine. See the Appsmith docs for details on self-hosting. 

2. Llama vs Ollama

   Before we get into installing, let's get this Llama-terminology sorted out. 

   Llama is an LLM model developed by Meta for natural language understanding and generation. It's a series of models with different versions, just like ChatGPT3, 4, o1, etc. In this guide, we'll be using the Llama3.2 model, which is a 2GB download, and contains 3 Billion parameters! 

   Ollama is a tool that allows users to run and manage Llama-based AI models locally on their devices. It has installers for Mac, Windows and Linux, and handles downloading and running different models. 
    

   Install Ollama and Download Llama3.2

   1. Download the Ollama installer, unzip it, and move the Ollama.app file to the Applications folder
   2. Open Ollama and complete the installer by granting drive access
   3. Copy the terminal command shown in the installer and run it in the terminal

   The Llama3.2 model will start downloading. Once it's finished, Ollama will run the model automatically and start a chat right in the terminal! 

   

   You can also check that the server is running by going to: 

   http://localhost:11434/

   

    

3. Appsmith API

   Next, open a new Appsmith app and go to the Query tab. Add a new API, with the following configuration: 

   Name	sendPrompt
   Method	POST
   URL	http://host.docker.internal:11434/api/generate
   Body type	JSON
   Body	{ "model": "llama3.2", "prompt": "Tell me about Appsmith", "stream": false }
   Timeout (Settings Tab)	60,000 (one minute)

   Notes: 

   1. To connect outside of the Docker container to other local resources, use host.docker.internal, instead of localhost. 
   2. The streaming=false parameter tells the server to wait until the model is done generating a response before replying, instead of sending multiple partial responses.

   Run the API, and you should get back a response from the model. 

   

    

4. Adding a Chat UI

   Now we'll add an interface to chat with the model. Drag in an input widget, button, and a text widget, and lay them out like a chat app with the input at the bottom. It helps if you set the input to multi-line, and the text for displaying the reply to fixed height. 

   Set the text widget to display the response from the model using: 

   {{sendPrompt.data.response}}

   Then set the button's onClick event to trigger the sendPrompt API. 

   Lastly, go back to the API, and replace the hard coded message with the value from the input. 

   {{
   {
    "model": "llama3.2",
    "prompt": Input1.text,
    "stream": false
   }
   }}

   Time to test it out! 

   Head back to the UI tab and enter a prompt then click the send button. 

   

Conclusion