Typesense DocSearch Scraper

The typesense-docsearch-scraper scans your documentation site and indexes its content into a Typesense collection, enabling fast, full-text search for your docs. You’ll typically run it whenever you update your documentation content. To set it up, follow the instructions in the official Typesense Docs.

DocSearch Scraper Config File

Here is the template for the Scraper Config File that you can quickly copy into your Starlight project:

docsearch.config.json

{
  "index_name": "YOUR_TYPESENSE_COLLECTION_NAME", // make sure this matches your `typesenseCollectionName` in `astro.config.ts`, the plugin uses it to query the same Typesense collection that the scraper indexes.
  "start_urls": [
    {
      "url": "YOUR_DOCUMENTATION_SITE_URL"
    }
  ],
  "selectors": {
    "default": {
      "lvl0": ".main-pane h1",
      "lvl1": ".main-pane h2",
      "lvl2": ".main-pane h3",
      "lvl3": ".main-pane h4",
      "lvl4": ".main-pane h5",
      "text": ".main-pane p, .main-pane ul li, .main-pane table tbody tr"
    }
  },
  "strip_chars": " .,;:#",
  "scrape_start_urls": false
}

Tip

If you are running Starlight site locally and you’re using Docker to run the scraper, update the start_urls to use your machine’s Network IP address instead of localhost, and include that IP address in allowed_domains. You can find the IP by starting your Starlight dev server with --host, e.g. npm run dev -- --host.

docsearch.config.json

{
  "allowed_domains": ["123.456.7.89"], // replace with your machine's network IP address
  "start_urls": [
    {
      "url": "http://123.456.7.89:4321/"
    }
  ]
  // ... rest of the config
}

Multilingual sites

If your site has multiple languages, the scraper indexes all of them in the same collection. To only show results in the language of the current page:

1. Enable filterByLanguage in the plugin configuration.

   astro.config.mjs

   starlightDocSearchTypesense({
     typesenseCollectionName: 'docs',
     typesenseServerConfig: {
       /* ... */
     },
     filterByLanguage: true,
   }),

   The plugin adds a <meta name="docsearch:language"> tag with the page language (e.g. en, fr) to every page. The scraper stores it in the language field of each record, and the search bar filters results with language:=<lang>.

2. Add a start URL for each language. Starlight’s language picker is not a link, so the scraper cannot discover the pages of the other languages by itself. Use a page that has a sidebar, e.g. a guide, so the scraper can follow the sidebar links:

   docsearch.config.json

   {
     "start_urls": [
       { "url": "https://docs.example.com/" },
       { "url": "https://docs.example.com/fr/getting-started/" }
     ],
     "scrape_start_urls": true
     // ... rest of the config
   }

   Alternatively, set your Astro site to generate a sitemap, and use it with "sitemap_urls": ["https://docs.example.com/sitemap-index.xml"].

3. Deploy your site, then run the scraper again so all records have a language field.

Caution

Records indexed before the docsearch:language tag was added have no language field and won’t be found when filterByLanguage is enabled. Re-run the scraper after deploying the site with this plugin version.

Tip

The built-in indexer creates one collection per language instead, each tokenized with the right Typesense locale, and doesn’t need filterByLanguage.

Running the Scraper

Here is how to run the Typesense DocSearch Scraper in your Starlight project.

Run the scraper locally

After completing the steps in the official Typesense Docs, add this script to your Starlight site’s package.json:

package.json

{
  "scripts": {
    "scrape": "docker run -it --env-file=./.env -e \"CONFIG=$(cat docsearch.config.json | jq -r tostring)\" typesense/docsearch-scraper:0.12.2"
  }
}

You can then run the scraper with:

npm run scrape # yarn, pnpm, bun equivalently

Run the scraper in GitHub Actions

Here is an example GitHub Actions workflow that runs the Typesense DocSearch Scraper Docker image whenever documentation content is updated in the docs/src/content folder, or when manually triggered. Store your Typesense connection details as repository secrets named TYPESENSE_API_KEY, TYPESENSE_HOST, TYPESENSE_PORT and TYPESENSE_PROTOCOL:

name: typesense-docsearch-scraper

on:
  push:
    branches:
      - master
    paths:
      - 'docs/src/content/**'
  workflow_dispatch:

permissions:
  contents: read

concurrency:
  cancel-in-progress: true
  group: ${{ github.workflow }}

jobs:
  scrape:
    name: Scrape documentation and update Typesense
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v6

      - name: Run DocSearch Scraper
        env:
          TYPESENSE_API_KEY: ${{ secrets.TYPESENSE_API_KEY }}
          TYPESENSE_HOST: ${{ secrets.TYPESENSE_HOST }}
          TYPESENSE_PORT: ${{ secrets.TYPESENSE_PORT }}
          TYPESENSE_PROTOCOL: ${{ secrets.TYPESENSE_PROTOCOL }}
        run: |
          docker run --rm \
            -e TYPESENSE_API_KEY \
            -e TYPESENSE_HOST \
            -e TYPESENSE_PORT \
            -e TYPESENSE_PROTOCOL \
            -e CONFIG="$(jq -r tostring docs/docsearch.config.json)" \
            typesense/docsearch-scraper:0.12.2

Once the scraper finishes, your Typesense collection will be populated and ready to use with the Starlight Typesense plugin.