Large Language Models¶

Warning

Starting in ChatterBot 1.2.7 experimental support for large language models is being added. This support is not yet complete and is not yet ready for general use beyond experimental purposes. The API will likely change in the future and the functionality may not be fully implemented.

LLM Roadmap¶

The following phases of development are the general roadmap for LLM support in ChatterBot. The goal is to provide a simple and consistent interface for LLM integration, and to make it easy to add support for new LLMs as they become available.

Note

Added April 1st, 2025
Last updated: April 8th, 2025

Phase 1:

Support for local and remote LLMs.

☑ Support for Ollama LLMs, which at the current time appear to be the easiest to set up and run on local hardware.
☑ Support for accessing LLMs that use the OpenAI API.

Phase 2:

☐ Streaming response support across features in ChatterBot.

Phase 3:

LLM integration with specific logic adapter features via RAG or similar approach.

☐ Mathematical operations MathematicalEvaluation via mathparse
☐ Date and time TimeLogicAdapter
☐ Unit conversion UnitConversion

One of the concepts / theories here that we want to evaluate is, for example, that it may be easier (and more efficient) to teach AI to use a calculator than it is to teach it the rules of mathematics. Whether this is because it lets us use smaller LLMs that don’t have a strong understanding of math, or because in general it allows us to offload processing of other complex tasks, there is likely a strong use case here.

Both interestingly and conveniently, ChatterBot’s existing architecture used for its logic adapters is already very similar to approaches used to provide additional tools to LLMs. The Model Context Protocol (MCP) supported by a range of MCP server implementations currently seems like a strong candidate for this.

Comparison of ChatterBot Architectures¶
Classic ChatterBot (Logic Adapters)	LLM with MCP

The choice to enforce one LLM per chat bot instance is currently based on the notion that one capable model per chat bot is likely simpler and more efficient than trying to reconcile the output of multiple models.

Phase 4:

☐ LLM integration with the ChatterBot training process

The ideal outcome for this phase would be the ability to use the existing training pipelines to fine-tune LLMs. It isn’t clear yet if this will be possible to do with common hardware, but right now this is the goal. An alternative may be to use a RAG approach to allow the LLM to access the chat bot’s database when generating responses.

Ollama Support¶

ChatterBot’s experimental support for using Ollama LLMs can be tested using the following setup:

Have Docker installed and running
Install ChatterBot and the Ollama client library
Use the following docker-compose file to run the Ollama server:

docker-compose.yml¶

services:

 # NOTE: This setup is for AMD GPUs
 ollama:
     image: ollama/ollama:rocm
     ports:
     - "11434:11434"
     volumes:
     - ./.database/ollama:/root/.ollama
     devices:
     - /dev/kfd
     - /dev/dri

The following commands can be used to download various Ollama models:

docker compose up -d

# Create a shell in the docker container
docker compose exec ollama bash

# Download and run the Gemma 3 model
ollama run gemma3:1b

More notes on the ollama container: https://hub.docker.com/r/ollama/ollama
Ollama model library: https://ollama.com/library

The following is an example of how to use the Ollama LLM in ChatterBot. Before running them you will need to install the ollama client library. This can be done directly using pip or by using the extra option from the ChatterBot package that includes it:

pip install chatterbot[dev]

examples/ollama_example.py¶

"""
EXPERIMENTAL: See https://docs.chatterbot.us/large-language-models/ for more information.
Example of using the Ollama API with the Ollama Python client.
"""
from chatterbot import ChatBot


# Create a new instance of a ChatBot
bot = ChatBot(
    'Ollama Example Bot',
    model={
        'client': 'chatterbot.llm.Ollama',
        'model': 'gemma3:1b',
        'host': 'http://localhost:11434'
    },
    stream=True  # Enable streaming responses
)

print('Type something to begin...')

# The following loop will execute each time the user enters input
while True:
    try:
        user_input = input()

        bot_response = bot.get_response(user_input)

        for part in bot_response:
            print(part, end='', flush=True)
        print()

    # Press ctrl-c or ctrl-d on the keyboard to exit
    except (KeyboardInterrupt, EOFError, SystemExit):
        break

Using the OpenAI client¶

The following is an example of how to use the OpenAI client in ChatterBot. Before running the example you will need to install the openai client library. This can be done directly using pip or by using the extra option from the ChatterBot package that includes it:

pip install chatterbot[dev] python-dotenv

Obtain an OpenAI API key: https://platform.openai.com/settings/organization/api-keys
Create a .env file to hold your API key in the parent directory from where your code is running.

../.env¶

OPENAI_API_KEY="API Key Here"

examples/openai_example.py¶

"""
EXPERIMENTAL: See https://docs.chatterbot.us/large-language-models/ for more information.
Example of using the OpenAI API with the OpenAI Python client.
"""
from chatterbot import ChatBot
from dotenv import load_dotenv

# Load the OPENAI_API_KEY from the .env file
load_dotenv('../.env')

# Create a new instance of a ChatBot
bot = ChatBot(
    'OpenAI Example Bot',
    model={
        'client': 'chatterbot.llm.OpenAI',
        'model': 'gpt-4o-mini',
    },
    stream=True  # Enable streaming responses
)

print('Type something to begin...')

# The following loop will execute each time the user enters input
while True:
    try:
        user_input = input()

        bot_response = bot.get_response(user_input)

        for part in bot_response:
            print(part, end='', flush=True)
        print()

    # Press ctrl-c or ctrl-d on the keyboard to exit
    except (KeyboardInterrupt, EOFError, SystemExit):
        break