Recommended way to use Quivr with LM Studio or Ollama for local chat + embeddings?

[byeugenius]

Hi,

I'm integrating Quivr into a Telegram support bot and want to run the stack locally.

Current setup:

• Python 3.12 on Windows
• local LM Studio server with an OpenAI-compatible endpoint
• fallback option: Ollama, if that is the recommended local path

Questions:

1. What is the recommended way to connect Quivr to LM Studio for both chat completions and embeddings?
2. Is the supported approach to pass a custom LLMEndpoint and a custom embedder to Brain.from_files(...), or should we rely on environment variables such as OPENAI_BASE_URL / OPENAI_API_KEY?
3. If LM Studio is not officially supported, is Ollama the preferred local provider for Quivr today?
4. Do you have a minimal working example for a fully local setup?

Any guidance would help. Thanks.

[dosubot[bot]]

Hi @byeugenius! I'm Dosu and I’m helping the quivr team.

Both LM Studio and Ollama can work with Quivr since they expose OpenAI-compatible endpoints. The recommended approach is to pass a custom LLMEndpoint and a custom embedder directly to Brain.from_files()—there is no OPENAI_BASE_URL environment variable in Quivr, so you must configure the base URL programmatically via LLMEndpointConfig [1].

Here's a minimal fully-local example:

from quivr_core import Brain
from quivr_core.llm import LLMEndpoint
from quivr_core.rag.entities.config import LLMEndpointConfig
from langchain_openai import OpenAIEmbeddings

# Configure local LLM (LM Studio or Ollama)
llm = LLMEndpoint.from_config(
    LLMEndpointConfig(
        model="llama2",               # your local model name
        llm_api_key="test",           # dummy key for local
        llm_base_url="http://localhost:8441"  # LM Studio default, or :11434 for Ollama
    )
)

# Configure local embeddings
embedder = OpenAIEmbeddings(
    model="your-embedding-model",
    api_key="test",
    base_url="http://localhost:8441"  # your local embeddings endpoint
)

brain = Brain.from_files(
    name="local_brain",
    file_paths=["document.pdf"],
    llm=llm,
    embedder=embedder
)

response = brain.ask("What is this document about?")
print(response.answer)

[2] [3] [4]

A few things to be aware of:

• LM Studio vs Ollama: Neither has deep first-class integration in Quivr's codebase—both work through the OpenAI-compatible endpoint path. Ollama is mentioned in the docs as supported but isn't in the DefaultModelSuppliers enum [5], so functionally they're on equal footing. Use whichever you prefer.
• Function calling limitation: Local models like llama2 and ollama3 are flagged as not supporting function calling [6], which may limit some RAG features that depend on structured tool use.
• Alternative approach: You can also pass any LangChain BaseChatModel directly to LLMEndpoint if you need more control [7]:

  from langchain_openai import ChatOpenAI
  custom_llm = ChatOpenAI(model="local-model", api_key="dummy", base_url="http://localhost:8441")
  llm_endpoint = LLMEndpoint(llm=custom_llm, llm_config=LLMEndpointConfig(model="local-model"))

Hope this helps with your Telegram bot setup! One thing to watch out for is the function calling limitation—depending on which local model you run, some RAG workflows may behave differently than with OpenAI models.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.