Full-text Search Plugin

sqliteFullTextSearch indexes every heading as its own searchable section and adds content.search(sources, query, opts?) and content.resetSearchIndex() to the Content instance. Built on top of SQLite FTS5, with BM25 ranking and snippet support.

Usage

import { comarkContent } from 'comark-content'
import sqlite from 'comark-content/database/sqlite-node'
import sqliteFullTextSearch from 'comark-content/plugins/sqlite-full-text-search'

const database = sqlite()

const content = comarkContent({
  source: fs('./content'),
  plugins: [sqliteFullTextSearch({ database })],
})

const hits = await content.search(['default'], 'streaming markdown', {
  limit: 10,
  snippet: { columns: ['title', 'content'], around: 30, tag: 'mark' },
  weights: { title: 10, content: 5, heading: true },
  minTermLength: 2,
})

API

sqliteFullTextSearch(options)

Returns a ContentPlugin that maintains an FTS5 index of your content and adds content.search() and content.resetSearchIndex() to the Content instance. Throws if options.database is missing.

Parameters:

• options: see Options.

Returns: ContentPlugin

content.search(sources, query, opts?)

Added by the plugin. Searches the given sources for query and returns ranked sections. The first search() per source builds the index lazily.

Parameters:

• sources: string[], source names to search across.
• query: string, the search query. Terms are split on whitespace.
• opts: SearchOptions, see Options.

Returns: Promise<SearchResult[]>

content.resetSearchIndex()

Added by the plugin. Drops the entire __fts_search table and clears the in-memory indexed-source set. The next search() rebuilds the index.

Returns: Promise<void>

Options

The opts argument to content.search() is a SearchOptions object:

limit

Caps the number of returned sections. Defaults to 50.

fields

Restricts matching to specific FTS columns ('title', 'content'). When omitted, terms match either column.

minTermLength

Drops query terms shorter than this length before searching. Defaults to 1. If every term is dropped, search() returns [].

weights

BM25 weighting per column.

• weights.title (number, default 10), BM25 weight for the title column.
• weights.content (number, default 5), BM25 weight for the content column.
• weights.heading (boolean, default true), divide rank by heading level, boosting pages over deep H6s.

snippet

Highlighted snippet configuration.

• snippet.columns (('title' | 'content')[]), columns to return as highlighted snippets. When snippet is set without columns, content is used.
• snippet.around (number, default 30), tokens of context around each match.
• snippet.tag (string, default 'mark'), HTML tag used to wrap matches.

Searching

Section model

The plugin walks each document's parsed AST and produces one section per heading:

Search results are returned at section granularity: a single page can yield several hits for different headings. Only entries whose meta.kind is document (and that have parsed nodes) are indexed.

Index management

The plugin maintains a __fts_search virtual FTS5 table and hooks into the Content instance lifecycle automatically:

Indexing is lazy: the first content.search() for a source builds its sections on demand, so no explicit boot step is required.

// First call builds the index for 'default', then queries it.
const hits = await content.search(['default'], 'getting started')

Query syntax

The plugin escapes each term and applies a wildcard suffix, so "foo bar" matches foo* AND bar*. To restrict matching to a column, pass fields:

const hits = await content.search(['default'], 'getting', {
  fields: ['title'],
})

For FTS5 advanced syntax (phrase queries, NEAR, boolean ops), pre-process your query string and pass it verbatim: see the FTS5 reference.

Snippets and weights

Return highlighted snippets and tune BM25 ranking per column:

const hits = await content.search(['default'], 'streaming markdown', {
  limit: 10,
  weights: { title: 10, content: 5, heading: true },
  snippet: { columns: ['title', 'content'], around: 40, tag: 'mark' },
})

hits[0].snippets?.content   // '<mark>streaming</mark> markdown ...'
hits[0].rank                // BM25 rank, lower is better