Usage

Indexing Content

The chatbot and semantic search answer from a vector index stored in your Turso database. This page explains what goes into the index and how to keep it current.

What is indexed

Content Text used Conditions Link shown in results
Items Title plus every literal and URI property value Skipped when an Item set filter is set and the item is in none of the selected sets /s/{site}/item/{id}
Media Title plus the literal and URI property values of the media record Always The parent item's page
Site pages Page title plus the text of every block, as Omeka's own full-text indexing extracts it The page and its site must both be public /s/{site}/page/{slug}
Blog posts (Blog module) Title, excerpt and content with HTML removed Status published and assigned to at least one site /s/{site}/blog/post/{slug}

Media file contents (PDF text, transcripts and so on) are not read. Only the metadata values on the media record are indexed.

Each resource's text is split into chunks of Chunk size characters, with Chunk overlap characters shared between neighbours. Every chunk is sent to OpenAI for an embedding and stored in Turso together with the resource type, id, title, URL and the ids of the sites the resource belongs to. Resources with no text are skipped.

Site membership matters. Chat and search on a site only return chunks whose resource belongs to that site. An item that is in no site is stored with an empty site list, links to its admin page, and never appears in site-scoped results.

Automatic indexing

With Auto-index items on save enabled (the default), the module listens for create, update and delete events on items, media, site pages and blog posts:

  • A create or update queues an IndexItemJob background job. The job removes the resource's old chunks, embeds the new text and rebuilds the vector index.
  • A delete removes the resource's chunks from Turso immediately, in the same request, whether or not auto-indexing is enabled.

Jobs are only queued when Turso is reachable; otherwise the save completes normally and nothing is indexed. Failures are written to the module log under the dispatch.error and index.embed events.

Turn auto-indexing off before a large CSV import or OAI-PMH harvest so the job queue is not flooded with one job per record, then turn it back on and run a full re-index.

Re-indexing everything

  1. Open Admin > Chatbot > Settings and select the Diagnostics tab.
  2. Under Content types to re-index, tick Items, Blog Posts and/or Pages.
  3. Click Re-index Now and confirm.

A BulkIndexJob is dispatched. For each selected type it first deletes every existing chunk of that type, then indexes the resources (items in pages of 100, honouring the Item set filter), and finally rebuilds the DISKANN vector index once. The success message links to the job in Admin > Jobs; per-resource failures appear in the job log and in the module log.

Media is not part of a bulk re-index. Media chunks are created and refreshed only by the automatic indexing that runs when a media record is saved.

The job checks for a stop request between resources, so it can be stopped from the Jobs page. A stopped run has already deleted the old chunks of that type and has not rebuilt the vector index, so run it again to completion.

Re-indexing selected items

  1. Go to Admin > Items.
  2. Tick the items you want to refresh.
  3. In the Batch actions dropdown choose Re-index selected and click Go.

The same dropdown has a Re-index all option that queues a full item re-index after a confirmation. Both options index items only.

Purging the index

Purge All Index Data on the Diagnostics tab drops the chunks table in Turso and recreates it empty with the current Embedding dimensions. Use it after changing the embedding model or dimensions, or to start over. Run Re-index Now afterwards.

Checking the index

The settings page header shows how many chunks are indexed and when the last one was written. Run Test Search on the Diagnostics tab embeds a query and lists the closest chunks with their distance, which is the quickest way to confirm the pipeline end to end. See Dashboard and Logs.

Running jobs synchronously

For local development, add this to config/local.config.php:

'chatbot' => ['sync_jobs' => true],

Automatic index jobs, bulk re-index and log purge then run inside the web request instead of through Omeka's job queue, and the settings page reports "completed (ran synchronously)". Do not use this on a production site: a full re-index can take longer than a web request allows.

Log in for Support