Keep an AnythingLLM workspace in sync with your docs site and GitHub repo
Import a docs site with the Bulk Link Scraper and a repo with the GitHub connector, then let AnythingLLM's document watching re-sync them every week.
· ranx.cloud
This is for whoever maintains a product's documentation, or an open-source project's README and docs/ folder, and wants an AnythingLLM workspace that answers from the current version of it rather than from whatever was uploaded once in spring. You end up with a workspace built from your docs website and your GitHub repository, with the important pages watched so AnythingLLM re-fetches them and re-embeds anything that changed, about once a week. The same workspace can sit behind a support widget on your site or serve your own questions.
Settings below were checked against AnythingLLM v1.19.0 (Docker). Document watching is still labelled experimental in that version.
What can be kept in sync, and how often
Watching works on content AnythingLLM can fetch again by itself. In the Docker version that means:
| Source | Watchable | How it got in |
|---|---|---|
| A web page | Yes | Fetch website, or the Bulk Link Scraper connector |
| A file in a GitHub, GitLab or Gitea repo | Yes | The repo connectors |
| A YouTube transcript | Yes | YouTube Transcript connector |
| Confluence, Drupal Wiki pages | Yes | Their connectors |
| A file you uploaded from your computer | No | The server can't read your disk again |
The schedule, from the live document sync docs and the source:
- A background job runs every hour and looks for watched documents that are due.
- A document is due seven days after its last sync. The interval isn't configurable from the UI.
- When a document is due, AnythingLLM fetches it from its source. If the text is unchanged, it just schedules the next check. If it changed, it replaces the document's vectors in every workspace that uses it.
- You watch individual documents, not folders.
So this keeps a workspace current within a week. If a pricing page changes and the bot must know today, refresh that page by hand (covered below).
Step 1: Turn on document watching
The switch is on a hidden settings page. Open settings, then hold down Ctrl (Cmd on a Mac) for three seconds; a toast says "Experimental feature previews unlocked!" and Experimental Features appears at the bottom of the settings sidebar. You can also go straight to https://<your instance>/settings/beta-features.
Under Live Document Sync, turn on Automatic Document Content Sync. The same panel links to Manage Watched Documents.
The upstream docs warn that this is a preview, that it can cost more if your embedder is a paid API, and they suggest watching a handful of files rather than everything. The cost section below puts a number on that.
Step 2: Scrape the docs site
Create or open the workspace, click the upload icon next to its name, and go to Data Connectors → Bulk Link Scraper.
| Field | What to enter |
|---|---|
| Website URL | The page to start from, e.g. https://docs.acme.example/ |
| Crawl Depth | How many links deep to follow, 1 to 5 |
| Maximum Pages | A cap on pages scraped (default 20) |
How the scraper decides what's in scope matters more than the numbers:
- It only follows links on the same origin as the start URL.
- It stays under the start URL's parent folder. Starting at
https://docs.acme.example/guides/introkeeps it inside/guides/; starting at the site root lets it roam the whole site. - It ignores
#anchordifferences, so a page full of table-of-contents links counts once.
For a typical docs site, start at the root with depth 2 and Maximum Pages set to roughly your page count plus some headroom (say 150 for a 120-page site). Depth 1 only gets pages linked from the start page; if your docs have a sidebar on every page, depth 2 usually reaches everything.
Each page is rendered in a headless browser before its text is extracted, so pages that build their content with JavaScript work. It's also why a 150-page scrape takes a while. Leave the modal open until it reports how many pages it collected.
The pages land in My Documents in a folder named after the site. Open the folder, skim the list, and delete what you don't want answered from: changelog archives, tag index pages, login pages, legal boilerplate.
Step 3: Import the GitHub repo
Still in Data Connectors, choose GitHub Repo.
| Field | What to enter |
|---|---|
| GitHub Repo URL | https://github.com/acme/acme-docs |
| GitHub Access Token | A fine-grained token (below) |
| File Ignores | Patterns in .gitignore format, one per entry |
| Branch | Loaded from the repo once the URL and token are filled in |
The token is not optional in practice. Without one, the connector only collects the repository's top-level files because of GitHub's API rate limits; the form says so. Create a fine-grained token at github.com/settings/personal-access-tokens/new: pick the repository under Repository access, give it Contents: Read-only and nothing else, and choose an expiry. Write the expiry date down; step 5 explains why.
File Ignores decides what the workspace knows. For a docs-only import of a code repo, ignore everything that isn't prose:
src/**
test/**
**/*.lock
package-lock.json
**/*.png
**/*.svg
**/*.min.js
.github/**
Press Enter after each entry. A docs workspace answers better from 80 Markdown files than from 80 Markdown files plus 2,000 source files that match half the keywords in every question.
Step 4: Embed and watch
In the document picker, select the scraped folder and the repo folder under My Documents, click Move to Workspace, then Save and Embed.
With watching turned on, the files in the workspace column on the right get an eye icon ("Watch document for changes") when their source can be re-fetched. Click it on each document you want kept current. The first time, a dialog explains watching; after that, each click is one document.
Because it's one click per file, pick deliberately:
- Watch pages that change: pricing, plans and limits, getting started, configuration reference, FAQ.
- Don't bother with pages that never change, or pages you'll delete and re-scrape anyway.
- Watching applies to the document, not the workspace: it stays watched in every workspace you add it to, and one sync updates all of them.
Step 5: Check on it
Experimental Features → Manage Watched Documents lists each watched document with when it was last synced and when it will next be refreshed. Clicking the trash icon on a row stops watching it; the document stays in the workspace.
Look at this page a week after setup. Every row should show a recent last sync. A document that keeps failing to refresh is retried each hour and, after five failures in a row, removed from the watch list without any notice in the UI. The usual cause is the next entry.
Refreshing something today
Nothing in the UI forces a watched document to sync early. For a page that must be current now:
- In the workspace's document list, remove the old copy of the page from the workspace.
- In Documents, paste the URL and click Fetch website (or re-run the connector for a repo).
- Move the new copy into the workspace, Save and Embed, and click its eye icon if it should stay watched.
Remove the old copy first. If both are in the workspace, the bot sees two versions of the page and may quote either.
Costs
The sync itself only costs money if your embedder is a paid API, and only for documents whose text changed.
With OpenAI's text-embedding-3-small at $0.02 per million tokens (OpenAI pricing, November 2026), and assuming a docs page is about 2,000 tokens:
Initial import, 150 pages: 150 × 2,000 = 300,000 tokens × $0.02/M = $0.006
A week where 20 pages change: 20 × 2,000 = 40,000 tokens × $0.02/M = $0.0008
Embedding is cheap enough to ignore at docs-site scale. Unchanged pages are fetched but not re-embedded.
With the built-in AnythingLLM Embedder there's no bill, but the work runs on your instance's CPU. On ranx.cloud's smaller plans (0.5 or 1 CPU) a big first import takes noticeably longer than on Pro or Power; the weekly re-syncs are small.
Changing the embedder later wipes every workspace's embeddings, so choose before you import. The PDF Q&A post covers that choice.
What goes wrong
Repo files stop syncing a few weeks in. The GitHub token expired. The token is stored with each imported file and reused on every re-sync, so when it expires, each watched file fails five hourly retries and drops off the watch list. Create a new token, delete the repo's documents, run the GitHub connector again with the new token, and re-watch the files you care about.
The scraper reports success but the folder is nearly empty. The start page's links are outside the scraper's scope (another subdomain, or a folder above the start URL's), or the navigation uses buttons and click handlers instead of ordinary <a href> links, which the scraper doesn't follow. Start from a page that links to everything with plain links, such as a sitemap-style index page, or scrape sections separately. Older releases also reported success when nothing was imported (#6773).
Maximum Pages ran out on duplicate pages. On versions before v1.18.0, every #anchor link counted as a separate page (#6618), and relative links on deeper pages were resolved against the start URL (#6511). On v1.19.0 both are fixed; if you're on an older version, raise Maximum Pages.
One broken document stopped all syncing. An unreadable watched document used to stop the rest of the queue (#6803), and a document that failed to re-embed could be marked as synced and never retried (#6700). Both are fixed in recent releases. If Manage Watched Documents shows a stale row on v1.19.0, unwatch it and watch it again.
The bot quotes an old version of a page. There are two copies in the workspace, usually one from Fetch website and one from the scraper. Remove the older one.
The eye icon is missing. Watching is off, the document was uploaded from your computer, or it came from a connector that isn't in the table at the top of this post.
A sync was due while the instance was down. The upstream docs note that the sync worker doesn't run while the container is stopped. After a restart (an update, for example), documents that came due in the meantime are picked up by the next hourly check.
