Running Ollama Locally

JetBrains IDEs added the ability to hook up to a local LLM. One of the runners that is supported is Ollama. I've been wanting to run an LLM locally, and Ollama made that incredibly easy.

Download and install is straightforward.

Once you do that, you can choose which model you want to run. There are many to choose from. The models act wildly different depending on which model is chosen, but also which size is chosen.

ollama run gemma2:2b

Some models seem to have different tags on the end of the name, like ":2b" or ":27B" indicating the number of parameters.

It looks like you can set a system message to go along with your inputs. This seems a way to more strongly recommend the model behave a particular way.

>>> /set system "Respond in the most comedic way possible. Turn up the absurdity to 11 -- no, to 12." Set system message. >>> Why did the chicken cross the road? To prove to the possum it could be done! He even had a little sign that read: "I'm gonna wing it!" and strutted like he owned the asphalt. 🐔✈️💨 #RoadTrippin 🤪

I can force a more serious tone with a different system message.

>>> /set system "Respond as seriously and clearly as possible with concise explanations. Produce no additional commentary or opinions." Set system message. >>> Why did the chicken cross the road? To get to the other side. This is a classic example of a joke that relies on simplicity, absurdity, and a playful use of language.

These can be saved and restored as well.

>>> /save myconfig Created new model 'myconfig'

The model can be started directly:

ollama run myconfig

or be loaded from the prompt.

>>> /load myconfig Loading model 'myconfig'

All of this is an entirely new world I'm excited to explore. I've really wanted to try out different models to see what they can do without worrying about "credits."

References