AnythingLLM

setup-guide

2 min read

1. Create a Router

  1. Navigate to AI Providers → Model Router.
  2. Click New Router and configure:
    • Name: Router identifier.
    • Description: Optional reference note.
    • Fallback Provider & Model: Default model used when no rules match, and the engine for evaluating LLM-classified rules.
    • Cache Cooldown (seconds): Sticky route duration (TTL) for conversation threads. Set to 0 to evaluate every message.

2. Rule Types & Syntax

Rules are evaluated sequentially from top to bottom (drag to reorder).

Calculated Rules

Non-LLM evaluation based on message metadata. Execution is instantaneous and free.

  • Rule Parameters:
    • Property: Prompt Content, Token Count, Message Count, Time of Day, Image Attachment.
    • Comparator: contains, equals, greater than, etc.
    • Value: Match criteria. (For contains, use a comma-separated, case-insensitive list. Matches on OR logic for any item in the list).
    • Route to: Target LLM Provider & Model.
  • Logic Operators: Combine multiple conditions using AND / OR toggles.

LLM-Classified Rules

Semantic classification evaluated by the designated fallback model. Incurs one classification LLM call per non-cached evaluation.

  • Rule Parameters:
    • Match Description: Plain-English definition of the target topic/intent.
    • Route to: Target LLM Provider & Model.

3. Deployment

  1. Open the workspace settings.
  2. Select the created Model Router from the LLM Provider dropdown.

4. Evaluation Flow & Cache Architecture

For every incoming message, the routing engine executes the following logic:

[Start]
   │
   ▼
[Step 1: Evaluate Calculated Rules] ──(Match Found)──► [Route to Target Model]
   │ (No Match)
   ▼
[Step 2: Check LLM Classify Cache] ──(Cache Hit)────► [Route to Cached Model / Skip]
   │ (Cache Miss)
   ▼
[Perform LLM Classification] ────────(Match Found)──► [Cache Match (5m) & Route]
   │ (No Match)
   ▼
[Cache "No Match" (30s)]
   │
   ▼
[Step 3: Check Sticky Route] ────────(Within Window)─► [Route to Sticky Model]
   │ (Expired / None)
   ▼
[Step 4: Fallback Model]

Caching and TTL Defaults

  • Sticky Window (Cooldown): Defaults to 5 minutes (configurable). Restarts on every message that utilizes the active sticky route.
  • LLM “Match” Cache: Caches successful semantic matches for the duration of the sticky window (5 minutes).
  • LLM “No Match” Cache: Caches negative semantic matches for 30 seconds to prevent classification spam on rapid inputs.

Graph View