[{"id":"cG9zdDo3MTg=","slug":"how-to-search-in-the-jam","title":"How to search in the JAM","content":"\n

So a lot (me included) now are building JAM stack landing pages, shops, full-stack apps, etc. and while you can have a backend of course not all of them have. For those who don’t: How do we search?

\n\n\n\n

So there is the obvious Google Programmable Search Engine but that looks bad and it not really customizable. The results are very good, it’s google after all. However for those who want something more custom: Here’s one way how.

\n\n\n\n

A working example can found right here nicco.io/search 😉
We will look at the following:

\n\n\n\n

How to implement the search
Search Accuracy & Precision
Performance & Size

\n\n\n\n

\"Telescope\" — Photo by Uriel Soberanes on Unsplash

\n\n\n\n

We can’t rely on a backend as discussed above, so the magic will happen at build time, like everything in the JAM-verse.

\n\n\n\n

I’ve decided to go with the free and open source lunr.js which is a simple but still quite powerful search engine that can run in the client.

\n\n\n\n

const idx = lunr(function () {\n  this.ref('name')\n  this.field('text')\n  this.add({ name: 'A', text: 'Lorem...' })\n})\nconst results = idx.search('Lor*')

\n\n\n\n

The first question that probably will pop out in your head is: “How will lunr be able to know what is on our website?” Here is where our work begins.

\n\n\n\n

Roadmap

\n\n\n\n

Aggregate all the data of your site
Prebuild the index and make it available as static JSON
Load lunr.js into your site and start searching

\n\n\n\n

Preparing the Index

\n\n\n\n

So I’m using Sapper for this blog so the examples will be based on it, but the same principle applies to all JAM tech.

\n\n\n\n

First we need to aggregate all our data. In my case this means all the single pages, blog entries, projects and works. So I created a /src/routes/search.json.js file and got to work.

\n\n\n\n

import lunr from 'lunr'\n\nimport { getAll } from '../lib/wp'\n\nfunction removeHTML(s) {\n  return s.replace(/<.*?>|\\s+|&#\\d+;/g, ' ').trim()\n}\n\nasync function convertForIdx(type, fields = []) {\n  // Load the data from WordPress\n  const items = await getAll(type)\n  // Map only the fields we need and are relevant\n  const defaults = ['title', 'content', 'slug']\n  return items.map((item) => ({\n    url: `${item.type}/${item.slug}`,\n    data: [...defaults, ...fields].map((field) => removeHTML(item[field])).join(' '),\n  }))\n}\n\nexport async function get(req, res) {\n  const all = await Promise.all([\n    convertForIdx('projects', ['description']),\n    convertForIdx('pages'),\n    convertForIdx('posts'),\n    convertForIdx('works', ['role']),\n  ])\n\n  const idx = lunr(function () {\n    this.ref('url')\n    this.field('data')\n\n    all.flat().forEach((doc) => this.add(doc))\n  })\n  res.setHeader('Content-Type', 'application/json')\n  res.end(JSON.stringify(idx))\n}

\n\n\n\n

First I get all the data from the WordPress backend and for each item I select at least the title and content as I want them to be searchable. Then we remove any html tags with a dirty regexp and finally we build the index.

\n\n\n\n

When we call JSON.stringify(idx) the precomputed index will be serialized to JSON. Otherwise every client would had to compute that on their CPU, wasting cycles and possibly battery. We don’t want that.

\n\n\n\n

Now I have the “search model” ready. You can have a look: nicco.io/search.json

\n\n\n\n

Integrating the search

\n\n\n\n

It’s time to integrate the search into the actual website 🚀

\n\n\n\n

<script context=\"module\">\n  export async function preload() {\n    const prebuilt = await this.fetch(`/search.json`).then((res) => res.json())\n    return { prebuilt }\n  }\n</script>\n\n<script>\n  import lunr from 'lunr'\n\n  import SearchResult from '../components/SearchResult.svelte'\n\n  export let prebuilt\n\n  let needle\n  let results = []\n\n  async function search(needle) {\n    if (!needle || !idx) {\n      results = []\n    } else {\n      let found = idx.search(needle + '~1')\n      if (!found.length) found = idx.search(needle + '*')\n      results = found.slice(0, 20)\n    }\n  }\n\n  $: idx = lunr.Index.load(prebuilt)\n  $: search(needle)\n</script>\n\n<input bind:value={needle} placeholder=\"needle\" />\n<ul>\n  {#each results as result (result.ref)}\n    <SearchResult {result} />\n  {/each}\n</ul>

\n\n\n\n

The first thing we do is load our preloaded /search.json and loading into an instance of lunr. This only need to happen once, once the index is loaded we ready to go.

\n\n\n\n

const idx = lunr.Index.load(prebuilt)

\n\n\n\n

For the searching itself lunr has quite a few options. The most relevant for me where the wildcard and fuzzy search. While wildcard is good for when we don’t have completed a word yet, fuzzy helps us with typos.

\n\n\n\n

const fuzzy = idx.search(needle + '~1') // foo~1

\n\n\n\n

While not explicitly said in the docs I’m guessing they use the Levenshtein Distance, which means ~1 will replace at most 1 char.

\n\n\n\n

const wildcard = idx.search(needle + '*') // fo*

\n\n\n\n

Wildcard are straight forward. lunr supports any kind: *oo, f*o and fo*.

\n\n\n\n

The result is an array with the ref field so you can find the related item and a score. They are already sorted by score, so basically you just need to write a for loop.

\n\n\n\n

Search Quality

\n\n\n\n

Now the accuracy and precision are of course on par with Google, but way good enough for a blog or a smaller site. However in 1h you can add search to your JAM site without much work and you stay google free.
Also this approach gives you all the artistic liberties over the design.

\n\n\n\n

Performance & Size

\n\n\n\n

Since we are prebuilding and packaging the whole site into one big JSON file it’s worth taking a look at the size of the index.

\n\n\n\n

For this I took the Iliad by Homer and slitted it up into different amount of pieces to simulate the amount of pages. At the same tame, the more pieces, the smaller the single content on one “page”.

\n\n\n\n

Please not that it’s ~1mb of plain text so it’s quite a lot.
You can get the source code for the “test” here.

\n\n\n\n

Variable size documents

\n\n\n\n

\n\n\n\n

As you can see, with 1000 each around 1.15k in size we end up with a compressed size of 563 KiB which starts to get big.

\n\n\n\n

A more real example

\n\n\n\n

Here is an example where each document is around 10k in size. Roughly double the text amount needed for this blog post. Then we add an ever increasing amount of documents to the index and watch it grow.

\n\n\n\n

\"\" — Chart of 10k sized documents building the index.

\n\n\n\n

The results are very different of course. Please note the the second graph has a logarithmic scale! If we compare the compressed size at 500 documents we have 494kb @2.3k/doc vs 1.09MiB @10k/doc. Basically double, which is not that bad if we consider that the documents are around 5 times bigger.

\n\n\n\n

Hope you enjoyed and learned something, take care ❤️

\n","status":"publish","date":"2020-12-06T17:27:22","modified":"2021-05-29T16:21:36","tags":{"nodes":[{"id":"dGVybTo2Mg==","slug":"jam-stack","name":"jam-stack","count":1},{"id":"dGVybTo2Mw==","slug":"lunr","name":"lunr","count":1},{"id":"dGVybTo2MQ==","slug":"search","name":"search","count":1},{"id":"dGVybTo2MA==","slug":"svelte","name":"svelte","count":2}]},"post":{"featured":{"srcSet":"https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash-290x300.jpg 290w, https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash-991x1024.jpg 991w, https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash-145x150.jpg 145w, https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash-768x794.jpg 768w, https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash-1487x1536.jpg 1487w, https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash-484x500.jpg 484w, https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash-774x800.jpg 774w, https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash-1161x1200.jpg 1161w, https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash-1452x1500.jpg 1452w, https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash-1936x2000.jpg 1936w, https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash.jpg 1955w","altText":"","sourceUrl":"https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash.jpg"}}}]