Prerequisites
- Active TrueFoundry account with API Gateway configured.
- Installed instance of AnythingLLM (Desktop or Docker).
- TrueFoundry Base URL, Personal Access Token (API Key), and Model ID.
Configuration
- Open AnythingLLM and navigate to Settings > LLM Preference.
- Select Generic OpenAI as the LLM provider.
- Configure the connection settings:
| Parameter | Configuration / Value |
|---|
| Base URL | TrueFoundry Gateway base URL (e.g., https://<your-domain>/api/llm/api/inference/openai) |
| API Key | TrueFoundry Personal Access Token |
| Chat Model Name | Unified model identifier from TrueFoundry (e.g., openai-main/gpt-4o) |
| Token Context Window | Set according to model limits (e.g., 16000, 128000) |
| Max Tokens | Set desired output limit (e.g., 1024, 2048) |
- Save the configuration.
- Open or create a workspace and send a test message to verify the connection.