[{"id":"cG9zdDo3MTg=","slug":"how-to-search-in-the-jam","title":"How to search in the JAM","content":"\n

So a lot (me included) now are building JAM stack landing pages, shops, full-stack apps, etc. and while you can have a backend of course not all of them have. For those who don’t: How do we search?

\n\n\n\n

So there is the obvious Google Programmable Search Engine but that looks bad and it not really customizable. The results are very good, it’s google after all. However for those who want something more custom: Here’s one way how.

\n\n\n\n

A working example can found right here nicco.io/search 😉
We will look at the following:

\n\n\n\n

How to implement the search
Search Accuracy & Precision
Performance & Size

\n\n\n\n

\"Telescope\" — Photo by Uriel Soberanes on Unsplash

\n\n\n\n

We can’t rely on a backend as discussed above, so the magic will happen at build time, like everything in the JAM-verse.

\n\n\n\n

I’ve decided to go with the free and open source lunr.js which is a simple but still quite powerful search engine that can run in the client.

\n\n\n\n

const idx = lunr(function () {\n  this.ref('name')\n  this.field('text')\n  this.add({ name: 'A', text: 'Lorem...' })\n})\nconst results = idx.search('Lor*')

\n\n\n\n

The first question that probably will pop out in your head is: “How will lunr be able to know what is on our website?” Here is where our work begins.

\n\n\n\n

Roadmap

\n\n\n\n

Aggregate all the data of your site
Prebuild the index and make it available as static JSON
Load lunr.js into your site and start searching

\n\n\n\n

Preparing the Index

\n\n\n\n

So I’m using Sapper for this blog so the examples will be based on it, but the same principle applies to all JAM tech.

\n\n\n\n

First we need to aggregate all our data. In my case this means all the single pages, blog entries, projects and works. So I created a /src/routes/search.json.js file and got to work.

\n\n\n\n

import lunr from 'lunr'\n\nimport { getAll } from '../lib/wp'\n\nfunction removeHTML(s) {\n  return s.replace(/<.*?>|\\s+|&#\\d+;/g, ' ').trim()\n}\n\nasync function convertForIdx(type, fields = []) {\n  // Load the data from WordPress\n  const items = await getAll(type)\n  // Map only the fields we need and are relevant\n  const defaults = ['title', 'content', 'slug']\n  return items.map((item) => ({\n    url: `${item.type}/${item.slug}`,\n    data: [...defaults, ...fields].map((field) => removeHTML(item[field])).join(' '),\n  }))\n}\n\nexport async function get(req, res) {\n  const all = await Promise.all([\n    convertForIdx('projects', ['description']),\n    convertForIdx('pages'),\n    convertForIdx('posts'),\n    convertForIdx('works', ['role']),\n  ])\n\n  const idx = lunr(function () {\n    this.ref('url')\n    this.field('data')\n\n    all.flat().forEach((doc) => this.add(doc))\n  })\n  res.setHeader('Content-Type', 'application/json')\n  res.end(JSON.stringify(idx))\n}

\n\n\n\n

First I get all the data from the WordPress backend and for each item I select at least the title and content as I want them to be searchable. Then we remove any html tags with a dirty regexp and finally we build the index.

\n\n\n\n

When we call JSON.stringify(idx) the precomputed index will be serialized to JSON. Otherwise every client would had to compute that on their CPU, wasting cycles and possibly battery. We don’t want that.

\n\n\n\n

Now I have the “search model” ready. You can have a look: nicco.io/search.json

\n\n\n\n

Integrating the search

\n\n\n\n

It’s time to integrate the search into the actual website 🚀

\n\n\n\n

<script context=\"module\">\n  export async function preload() {\n    const prebuilt = await this.fetch(`/search.json`).then((res) => res.json())\n    return { prebuilt }\n  }\n</script>\n\n<script>\n  import lunr from 'lunr'\n\n  import SearchResult from '../components/SearchResult.svelte'\n\n  export let prebuilt\n\n  let needle\n  let results = []\n\n  async function search(needle) {\n    if (!needle || !idx) {\n      results = []\n    } else {\n      let found = idx.search(needle + '~1')\n      if (!found.length) found = idx.search(needle + '*')\n      results = found.slice(0, 20)\n    }\n  }\n\n  $: idx = lunr.Index.load(prebuilt)\n  $: search(needle)\n</script>\n\n<input bind:value={needle} placeholder=\"needle\" />\n<ul>\n  {#each results as result (result.ref)}\n    <SearchResult {result} />\n  {/each}\n</ul>

\n\n\n\n

The first thing we do is load our preloaded /search.json and loading into an instance of lunr. This only need to happen once, once the index is loaded we ready to go.

\n\n\n\n

const idx = lunr.Index.load(prebuilt)

\n\n\n\n

For the searching itself lunr has quite a few options. The most relevant for me where the wildcard and fuzzy search. While wildcard is good for when we don’t have completed a word yet, fuzzy helps us with typos.

\n\n\n\n

const fuzzy = idx.search(needle + '~1') // foo~1

\n\n\n\n

While not explicitly said in the docs I’m guessing they use the Levenshtein Distance, which means ~1 will replace at most 1 char.

\n\n\n\n

const wildcard = idx.search(needle + '*') // fo*

\n\n\n\n

Wildcard are straight forward. lunr supports any kind: *oo, f*o and fo*.

\n\n\n\n

The result is an array with the ref field so you can find the related item and a score. They are already sorted by score, so basically you just need to write a for loop.

\n\n\n\n

Search Quality

\n\n\n\n

Now the accuracy and precision are of course on par with Google, but way good enough for a blog or a smaller site. However in 1h you can add search to your JAM site without much work and you stay google free.
Also this approach gives you all the artistic liberties over the design.

\n\n\n\n

Performance & Size

\n\n\n\n

Since we are prebuilding and packaging the whole site into one big JSON file it’s worth taking a look at the size of the index.

\n\n\n\n

For this I took the Iliad by Homer and slitted it up into different amount of pieces to simulate the amount of pages. At the same tame, the more pieces, the smaller the single content on one “page”.

\n\n\n\n

Please not that it’s ~1mb of plain text so it’s quite a lot.
You can get the source code for the “test” here.

\n\n\n\n

Variable size documents

\n\n\n\n

\n\n\n\n

As you can see, with 1000 each around 1.15k in size we end up with a compressed size of 563 KiB which starts to get big.

\n\n\n\n

A more real example

\n\n\n\n

Here is an example where each document is around 10k in size. Roughly double the text amount needed for this blog post. Then we add an ever increasing amount of documents to the index and watch it grow.

\n\n\n\n

\"\" — Chart of 10k sized documents building the index.

\n\n\n\n

The results are very different of course. Please note the the second graph has a logarithmic scale! If we compare the compressed size at 500 documents we have 494kb @2.3k/doc vs 1.09MiB @10k/doc. Basically double, which is not that bad if we consider that the documents are around 5 times bigger.

\n\n\n\n

Hope you enjoyed and learned something, take care ❤️

\n","status":"publish","date":"2020-12-06T17:27:22","modified":"2021-05-29T16:21:36","tags":{"nodes":[{"id":"dGVybTo2Mg==","slug":"jam-stack","name":"jam-stack","count":1},{"id":"dGVybTo2Mw==","slug":"lunr","name":"lunr","count":1},{"id":"dGVybTo2MQ==","slug":"search","name":"search","count":1},{"id":"dGVybTo2MA==","slug":"svelte","name":"svelte","count":2}]},"post":{"featured":{"srcSet":"https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash-290x300.jpg 290w, https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash-991x1024.jpg 991w, https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash-145x150.jpg 145w, https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash-768x794.jpg 768w, https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash-1487x1536.jpg 1487w, https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash-484x500.jpg 484w, https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash-774x800.jpg 774w, https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash-1161x1200.jpg 1161w, https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash-1452x1500.jpg 1452w, https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash-1936x2000.jpg 1936w, https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash.jpg 1955w","altText":"","sourceUrl":"https://api.nicco.io/wp-content/uploads/2020/12/uriel-soberanes-gCeH4z9m7bg-unsplash.jpg"}}},{"id":"cG9zdDo1NjQ=","slug":"why-i-think-svelte-is-the-next-big-thing-a-reacts-lover-view","title":"Why I think svelte is the next big thing: a reacts lover view","content":"\n

The first thing I thought when I heard there is this new framework called “Svelte” was: F*** no, please, not again. Why? Why do we need yet another framework ffs. React is amazing, don’t try to reinvent the wheel.

\n\n\n\n

I was so wrong…

\n\n\n\n

So maybe one of you is thinking the same thing. Why Svelte? What is all the fuzz about? Well…. let me tell you 🙂

\n\n\n\n

This is not a tutorial, for that check the amazing official tutorial which teaches you everything from the basics to more advanced stuff.

\n\n\n\n

\"\" — Photo by Alessandra Caretto on Unsplash

\n\n\n\n

Why is there a wheel? Well quite simply: Svelte IS reinventing the wheel.
You see: what are most of the frameworks doing nowadays? They provide a way to code user interfaces with a component approach. This makes a lot of sense because we can reuse them, they are boxed items that stand for them self. Modularizing and splitting up concerns make big apps easier to write and afterwards maintain.

\n\n\n\n

What sets Svelte apart from a tecnical standpoint ist that it’s foremost a compiler as opposed to a runtime like React or Vue. What does it mean? It means that all the reactivity is done at compile time and not in the browser. Svelte knows before hand what property can change and what doesn’t. That’s very powerful.

\n\n\n\n

Easy of use and simplicity

\n\n\n\n

This in my opinion might be the biggest driver for svelte. The official tutorial is great and well thought out. What stood out to me was how easy it was for non web people I know to get up and running in the matter of a day.
Simplicity is really hard however Svelte abtracts everything you need away and breaks it down to simple html, js and css. The learning curve is… well more like a flat line you just need to step on.

\n\n\n\n

Taking the fun back to the web

\n\n\n\n

Personally the greatest feat Svelte achieves is staing out of the way while supporting you, the developer in everything you need to do. It is never in the way, it always makes your life easier.

\n\n\n\n

Vue and React had a child

\n\n\n\n

It is very clear that Svelte was inspired by both React and Vue picking the best features of both worlds. This is probably why Svelte is so amazing. It has the luxury to build on years of experience of what works, and what does not.

\n\n\n\n

Inspired by React:

\n\n\n\n

Components are JS Objects

\n\n\n\n

Inspired by Vue:

\n\n\n\n

2 Way binding
Syling
Making ecosystem first party
Single file components

\n\n\n\n

Moslty “Unique” features:

\n\n\n\n

Class toggles
Stores

\n\n\n\n

Styling

\n\n\n\n

Styling in react sucks. There is no way around it. In Vue that problem is solved by simply having scoped css built-in. No need for preprocessors, since we have CSS Variables and namespacing is no longer an issue.

\n\n\n\n

<script>\n\tlet title = 'Nietzsche Ipsum'\n</script>\n\n<style>\n\th1 {\n\t\tcolor: #05f;\n\t}\n\t\n\tp {\n\t\tfont-family: monospace;\n\t}\n\t\n\ta {\n\t\tcolor: initial;\n\t\ttext-decoration: underline;\n\t}\n</style>\n\n<h1>{title}</h1>\n\n<p>\n\tVirtues sexuality philosophy law chaos society evil strong self christianity sexuality truth. Revaluation convictions decrepit snare passion oneself decieve oneself. Right suicide grandeur fearful play. Right joy merciful transvaluation good truth derive evil contradict intentions. Self salvation faithful disgust marvelous transvaluation aversion.\n</p>\n\n<a href=\"http://nietzsche-ipsum.com/\">Generate more!</a>

\n\n\n\n

No need to worry that our styles will affect some other stuff and we don’t need to pass around classNames from other files.

\n\n\n\n

Single File Components

\n\n\n\n

.svelte files are just like .vue files and they contain everything: code (js/ts), markup (html) and styling (css). No need to separate files for styling which bloats every React project that does not want to rely on css-in-js which in my opinion is just garbage.

\n\n\n\n

2 way binding

\n\n\n\n

This alone will save you tons of time and makes implementing stuff more intuitive and fun.

\n\n\n\n

<script>\n\tlet form = {\n\t\tusername: '',\n\t\tpassword: ''\n\t}\n</script>\n\n<input bind:value={form.username} />\n<input bind:value={form.password} />

\n\n\n\n

Yes, it will sync the UI to the variables, but both ways. In react you need to either use some 3rd party lib or juggle value and onChange events.

\n\n\n\n

This also works for parent – child props, so no need to write verbose update functions to be called from the children. Everyone hates those.

\n\n\n\n

Stores

\n\n\n\n

So I truly hate Redux. It is the reason I’ve quit companies and unfortunately it is the way to go in react. In svelte we have build it stores and they are simply put: amazingly powerful and easy.

\n\n\n\n

// store/todos.js\nimport { writable } from 'svelte/store'\nimport axios from 'axios'\n\nexport const data = writable([])\n\nexport async function load() {\n  const { data: d } = await axios({\n    method: 'get',\n    url: '/api/todos/',\n  })\n  data.set(d)\n}

\n\n\n\n

<script>\n  import { onMount } from 'svelte'\n\n  import Todo from '../components/Todo.svelte'\n  import { data, load } from '../store'\n\n  onMount(() => {\n    load()\n  })\n</script>\n\n{#each $data as todo}\n  <Todo {todo} />\n{/each}\n

\n\n\n\n

With a simple $ in front of a store svelte will update the UI according to the store content. And we simply update it. No complex bootstrapping with context, dispatcher, acitons and so on.

\n\n\n\n

Class toggles

\n\n\n\n

This is a feature I use all the time and it is just so useful. You can basically bind a value to toggle a css class, in this case done. No need to to string magic or, again, import some 3rd party module to manage our class names.

\n\n\n\n

<script>\n\texport let done = true;\n</script>\n\n<style>\n\t.done {\n\t\ttext-decoration: line-through;\n\t}\n</style>\n\n<div class:done>\n\tSome task\n</div>

\n\n\n\n

First party support

\n\n\n\n

Another thing that I cannot stand in React and Svelte does right in my opinion is that in the react world everything is basically up to the community. Even the typing! That means that while there is more choice in theory, in practise it leads to a fragmented unmaintained ecosystem.

\n\n\n\n

The Svelte is not afraid of incorporating tools from the community into the officials repos. I strongly believe that it is in everybodies interest that tooling and core functionality should always be first party code. Much like Vue does it.

\n\n\n\n

Concluding thoughts

\n\n\n\n

First of all lets all thank Rich Harris which is the creator of Svelte.

\n","status":"publish","date":"2020-10-29T19:13:26","modified":"2020-11-19T10:07:05","tags":{"nodes":[{"id":"dGVybTo0Nw==","slug":"react","name":"react","count":3},{"id":"dGVybTo2MA==","slug":"svelte","name":"svelte","count":2}]},"post":{"featured":{"srcSet":"https://api.nicco.io/wp-content/uploads/2020/03/alessandra-caretto-cAY9X4rPG3g-unsplash-300x199.jpg 300w, https://api.nicco.io/wp-content/uploads/2020/03/alessandra-caretto-cAY9X4rPG3g-unsplash-1024x680.jpg 1024w, https://api.nicco.io/wp-content/uploads/2020/03/alessandra-caretto-cAY9X4rPG3g-unsplash-150x100.jpg 150w, https://api.nicco.io/wp-content/uploads/2020/03/alessandra-caretto-cAY9X4rPG3g-unsplash-768x510.jpg 768w, https://api.nicco.io/wp-content/uploads/2020/03/alessandra-caretto-cAY9X4rPG3g-unsplash-1536x1020.jpg 1536w, https://api.nicco.io/wp-content/uploads/2020/03/alessandra-caretto-cAY9X4rPG3g-unsplash-2048x1361.jpg 2048w, https://api.nicco.io/wp-content/uploads/2020/03/alessandra-caretto-cAY9X4rPG3g-unsplash-500x332.jpg 500w, https://api.nicco.io/wp-content/uploads/2020/03/alessandra-caretto-cAY9X4rPG3g-unsplash-800x531.jpg 800w, https://api.nicco.io/wp-content/uploads/2020/03/alessandra-caretto-cAY9X4rPG3g-unsplash-1200x797.jpg 1200w, https://api.nicco.io/wp-content/uploads/2020/03/alessandra-caretto-cAY9X4rPG3g-unsplash-1500x997.jpg 1500w, https://api.nicco.io/wp-content/uploads/2020/03/alessandra-caretto-cAY9X4rPG3g-unsplash-2000x1329.jpg 2000w","altText":"","sourceUrl":"https://api.nicco.io/wp-content/uploads/2020/03/alessandra-caretto-cAY9X4rPG3g-unsplash.jpg"}}}]