Accurate answers from your own PDFs in an AnythingLLM workspace
Set up an AnythingLLM workspace for contracts, manuals or papers: embedder choice, chunk size, similarity threshold, reranking, pinning and costs.
· ranx.cloud
This is for someone with a few hundred pages of their own documents (contracts, equipment manuals, research papers, a policy handbook) who wants to ask questions and get answers that point to the page they came from. You end up with an AnythingLLM workspace whose retrieval settings you chose on purpose: an embedder that suits your plan and language, chunks sized for your documents, a threshold that keeps unrelated text out, reranking where it helps, and pinning for the one or two documents that need to be read whole. It's a workspace only you use; nobody else signs in.
Settings below were checked against AnythingLLM v1.19.0 (Docker).
How a question gets answered
When you upload a PDF, AnythingLLM extracts the text, splits it into chunks, turns each chunk into a vector with the embedder, and stores the vectors in LanceDB, the built-in vector database. When you ask a question, it embeds the question, finds the closest chunks, drops any below the similarity threshold, keeps the top few, and sends them to the LLM with your system prompt. The answer lists those chunks under Sources.
Each step has one setting that matters:
| Step | Setting | Where |
|---|---|---|
| Embedding | Embedder and model | Settings → AI Providers → Embedder |
| Splitting | Text Chunk Size, Text Chunk Overlap | Settings → AI Providers → Text Splitter & Chunking |
| Retrieval | Document similarity threshold, Max Context Snippets, Search Preference | Workspace gear → Vector Database |
| Answering | Chat mode, System Prompt, model | Workspace gear → Chat Settings |
The first two are instance-wide and apply when documents are embedded. The rest are per workspace and apply on every question.
Pick the embedder first
Changing the embedder later clears every workspace's embeddings; AnythingLLM warns you and you then re-embed everything. Choose before your first upload.
AnythingLLM Embedder (built in). Runs on your instance's CPU, no key, no bill. Three models:
| Model | Size | Languages | Max chunk |
|---|---|---|---|
| all-MiniLM-L6-v2 (default) | 23 MB | English | 1,000 characters |
| nomic-embed-text-v1 | 139 MB | English | 16,000 characters |
| multilingual-e5-small | 487 MB | 100+ | 1,000 characters |
It's realistic even on a 0.5 CPU, 2 GB instance for a few hundred pages; the trade-off is time. With 0.5 or 1 CPU, embedding a large batch takes noticeably longer than with 2 or 4, and the instance is slower to answer while it works. The larger models use more memory and CPU than the default. If your documents aren't in English, use multilingual-e5-small or an API embedder; the default model is English-only.
OpenAI. Settings → AI Providers → Embedder → OpenAI, paste a key from platform.openai.com/api-keys, and pick text-embedding-3-small under Model Preference. At $0.02 per million tokens (OpenAI pricing, November 2026), 500 pages at about 600 tokens each is 300,000 tokens, or $0.006. It's fast regardless of plan and handles most languages. Your document text goes to OpenAI to be embedded.
For a private archive on a small plan, the built-in default is fine if your documents are English and you can wait for the first import. Otherwise use text-embedding-3-small.
Chunk size
Settings → AI Providers → Text Splitter & Chunking. Defaults: Text Chunk Size 1,000 characters, Text Chunk Overlap 20 characters. The page shows the maximum your embedder allows; with the default built-in model that's 1,000.
What chunk size changes:
- Smaller chunks (500–1,000 characters) match specific facts well: a clause, a part number, a setting. Each answer sees less surrounding text.
- Larger chunks (2,000–4,000, with an API embedder or nomic-embed) keep a whole clause or procedure together. Each question sends more text to the LLM.
For contracts and manuals, where an answer usually lives in one numbered section, 1,500–2,000 characters with 200 overlap works well if your embedder allows it. For papers, the default is fine.
Changing these settings only affects documents embedded afterwards. To apply new sizes to existing documents, remove them from the workspace and add them back.
The workspace
Create a workspace, click the upload icon next to its name, drop your files in, select them under My Documents, Move to Workspace, Save and Embed. Scanned PDFs without a text layer go through OCR automatically, which is slow on small plans; if you have the original digital PDF, upload that instead.
Then the gear icon. Changes save as you edit.
Chat Settings
| Setting | Value |
|---|---|
| Workspace LLM Provider | Your provider, or leave on the system default |
| Chat mode | Query |
| LLM Temperature | 0 to 0.3 |
Query mode answers only when document context is found, and otherwise returns the Query mode refusal response. For document Q&A that's what you want: "not in your documents" beats an answer from the model's general knowledge that looks like it came from your contract.
System Prompt:
You answer questions about the user's own documents.
Use only the context provided. Each context excerpt starts with metadata
including the source document's title; name the document for every
claim you make.
If the context does not contain the answer, say so and say what you
looked for. Do not fill gaps from general knowledge.
If two documents disagree, say which document says what.
Quote exact wording for dates, amounts, limits and obligations.
AnythingLLM puts each excerpt's document title in the context it sends, so "name the document" works without extra setup.
Vector Database tab
| Setting | Start at | When to change |
|---|---|---|
| Max Context Snippets | 4 (recommended) | Raise to 6–8 if answers miss parts that are spread across sections. |
| Document similarity threshold | Low (≥ .25) | Medium if answers cite unrelated pages; No restriction if specific questions get refused. |
| Search Preference | Default | Accuracy Optimized if the right section exists but isn't among the sources. |
Accuracy Optimized re-ranks the candidates with a small cross-encoder model that runs on your instance (downloaded on first use). It usually puts the best passage first when many chunks look alike, as numbered clauses in a contract do. It adds a delay to every answer, more on a small CPU allowance. It's only offered with LanceDB, the default.
Test it with questions you know the answers to
Write down ten questions whose answers you can find yourself, including two that the documents don't answer. Ask each one, open Sources under the reply, and check the excerpt. Change one setting at a time:
- Right document, wrong section: chunks too big, or try Accuracy Optimized.
- Answer split across sections, half of it missing: raise Max Context Snippets.
- Sources from unrelated documents: raise the threshold.
- The two unanswerable questions got answers: you're not in Query mode, or the prompt needs the "do not fill gaps" line.
Pinning
Some documents should be read whole, not searched: a 10-page master agreement every question refers back to, a glossary, a one-page price list. In the workspace's document list, click the pin icon ("Pin to workspace"). AnythingLLM then puts the full text of that document into every prompt for the workspace, in addition to the search results.
Pinning trades money for completeness. If the pinned documents together don't fit in the model's context window, AnythingLLM skips the ones that don't fit. And every question pays for them.
Costs
Prices are Anthropic's list prices as of November 2026: Claude Haiku 5.5 at $0.10 / $0.50 per million input / output tokens (prompts up to 100,000 tokens), Claude Sonnet 5.5 at $2 / $10.
A question without pinning, using 1,500-character chunks: system prompt 150 tokens, 4 excerpts of ~400 tokens, some history, answer 300 tokens. Call it 2,500 in and 300 out.
Haiku 5.5: 2,500 × $0.10/M + 300 × $0.50/M = $0.00025 + $0.00015 = $0.0004
Sonnet 5.5: 2,500 × $2/M + 300 × $10/M = $0.005 + $0.003 = $0.008
40 questions a day for a month (1,200):
Haiku ≈ $0.48 Sonnet ≈ $9.60
With a 20,000-token agreement pinned, each question carries 22,500 input tokens:
Haiku 5.5: 22,500 × $0.10/M + 300 × $0.50/M ≈ $0.0024
Sonnet 5.5: 22,500 × $2/M + 300 × $10/M ≈ $0.048
1,200 questions: Haiku ≈ $2.90 Sonnet ≈ $58
Anthropic's Prompt Caching option (under Show advanced settings on the Anthropic LLM settings) doesn't rescue this. AnythingLLM sends the excerpts as part of the system message, so the cached part changes whenever the search results do.
For contract work, Sonnet-class models read obligations and exceptions more carefully; for lookups in manuals, Haiku is usually enough. A workspace can have its own model, so you can do both.
Plans and storage
Embedding and reranking run on the instance; the LLM doesn't. If you use an API embedder and leave Search Preference on Default, the instance mostly waits on network calls, and the smallest size is enough. With the built-in embedder and Accuracy Optimized, CPU is what you'll notice, and 2 or 4 CPUs make large imports and reranked answers faster. Uploaded files, extracted text and vectors all live on the instance's disk, so check the disk size against your archive. On ranx.cloud, daily backups run only on Pro and Power, so on Hobby or Starter keep your original files somewhere else as well.
What goes wrong
Every question gets the refusal text. The documents aren't embedded in this workspace (they're in My Documents but weren't moved in and embedded), or the threshold is too high. Set the threshold to No restriction temporarily; if answers appear, step back up to Low.
Answers cite a document that has nothing to do with the question. AnythingLLM also refills the excerpt list from documents cited earlier in the conversation, so a long thread drifts toward old sources. Start a new thread when you change subject.
A large upload takes forever or appears stuck. Embedding on a small CPU, or OCR on scanned pages. A user ingesting 4 GB of email was advised to upload .mbox files directly instead of converting them to huge Markdown files, because each email then becomes its own document (#6965). The same idea applies to PDFs: many normal-sized files embed and retrieve better than one giant one.
New chunk settings had no effect. They apply only to documents embedded after the change. Remove and re-add the documents.
Everything vanished after you changed the embedder. That's the documented behaviour: switching the embedding model clears all embeddings and removes documents from workspaces. The files stay under My Documents; move them back and embed again.
After pinning, answers lose facts from other documents. In v1.19.0 the search takes its top results first and only then drops the chunks that belong to pinned documents. When the pinned document matches the question best, it uses up the slots and nothing from your other documents is added (#6906). Upstream fixed this for Accuracy Optimized after v1.19.0 and doesn't plan to change the default search. With pinning, raise Max Context Snippets to 8 or so.
The threshold seemed to do nothing on an older version. LanceDB once scored some unrelated chunks as perfect matches, letting them past any threshold (#6255). That's fixed in v1.19.0; on something older, update before tuning.
A pinned document is ignored. The pinned documents together are larger than the model's context window, so the ones that don't fit are skipped. Unpin some, or use a model with a larger window.
