How to configure separate model for embedding and chat?

[dangnhdev]

Hello how can I configure separate model for embedding and chat? I want to use openai for chat only and another local model for embedding to save money.

[dosubot[bot]]

Hey @dangnhdev! 👋 I'm here to help you with any bugs, questions, or contributions while you're waiting for a human maintainer. Let's tackle this together!

To configure separate models for embedding and chat, using OpenAI for chat and a local model for embedding, you can modify the useLocalStorageChatConfig hook to differentiate between the two configurations. Here is an example of how you might achieve this:

import { getChatsConfigFromLocalStorage } from "@/lib/api/chat/chat.local";
import { useUserData } from "@/lib/hooks/useUserData";
import { BrainConfig } from "@/lib/types/BrainConfig";

// eslint-disable-next-line @typescript-eslint/explicit-module-boundary-types
export const useLocalStorageChatConfig = () => {
  const { userData } = useUserData();

  const chatConfig = getChatsConfigFromLocalStorage();

  const chatModel = (userData?.models ?? []).includes(chatConfig?.chatModel ?? "")
    ? chatConfig?.chatModel
    : undefined;

  const embeddingModel = (userData?.models ?? []).includes(chatConfig?.embeddingModel ?? "")
    ? chatConfig?.embeddingModel
    : undefined;

  return {
    chatConfig: {
      chatModel: chatModel,
      embeddingModel: embeddingModel,
      temperature: chatConfig?.temperature,
      maxTokens: chatConfig?.maxTokens,
    },
  };
};

In this example, chatConfig is expected to have separate properties for chatModel and embeddingModel. You would need to ensure that your local storage configuration includes these properties and that they are set appropriately. This way, you can use OpenAI for chat and a local model for embedding, helping you save on costs ^[1][2].

To continue talking to Dosu, mention @dosu.