Build a fast search UI with Meilisearch, HTMX, and Django

In this tutorial, we will build an airbnb-like search UI for property listings. We will be using Meilisearch as our search engine. Meilisearch is open-source, fast, and hyper-relevant search engine that can be easily added to your Django app. The frontend is Django templates, HTMX, and only 10 lines of javascript!

Prerequisites

  • Basic Knowledge of Django & HTMX
  • Docker & Docker Compose

LEVEL - 💻 💻 Intermediate.

Basic knowledge of Django and HTMX is assumed. The codebase for the tutorial can be found here. Clone the repository if you want to follow along. This post builds on some of the concepts from the Django htmx server-side data tables article. We suggest you check that out if you haven't already.

What we are building

Project overview & folder structure

|django-htmx 
├── config                  --> app configuration
├── apps 
|   ├── mybnb      --> main app and focus of this article
├── static
|   ├── js                  --> includes htmx scripts
|   ├── css                 --> site css
├── templates
|   ├── base.html           --> base template, includes htmx scripts
|   └── common
|      └── django_tables2.html
|   └── mybnb
|      ├── htmx             --> keep htmx templates in separate folders 
|         ├── gallery.html
|      ├── components
|      ├── search.html
└── .pre-commit-config.yml  --> linting & formatting tools.
└── Makefile                --> shortcuts for convenience
└── requirements.txt        --> project dependencies
└── manage.py

Notes

  • apps/mybnb app - focus of this article
  • Makefile - Makefile with shortcut commands for convenience. For example make run_d to run docker-compose up.
  • .pre-commit-config.yml - includes pre-commit configuration for linting, formatting the code base with flake8, black, isort and . This is the most important framework for automating & managing the various code quality tools in the project.

Installation with Docker

We want to get up and running as quickly as possible. For that we will use Docker and docker-compose. Docker helps you build containers easily. Docker-compose is used for building multi-container applications. Our application will use two containers, one for the app and another for the search engine. To keep things simple, we will not be using Postgres as the database.

Let's create a Docker file that will be used for building an image for the Django app. We will use the simplest configuration for this, taken directly from the Docker docs.

Dockerfile

FROM python:3
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
WORKDIR /app
COPY requirements.txt /app/
RUN pip install -r requirements.txt
COPY . /app/

Notes

  • FROM python:3 - This tells docker to download the parent image from ducker
  • RUN pip install -r requirements.txt - install requirements
  • COPY . /app/ copy the app to the container

docker-compose.yml

version: '3'

volumes:
  meili-search: {}

services:
  django:
    build: .
    volumes:
      - .:/app:z
    env_file:
      - ./.env
    ports:
      - "8000:8000"
    command: python3 manage.py runserver 0.0.0.0:8000


  meilisearch:
    image: getmeili/meilisearch
    environment:
      - MEILI_MASTER_KEY=TESTKEYLOCAL
    volumes:
      - meili-search:/data.ms
    ports:
      - "7700:7700"

Notes

  • image: getmeili/meilisearch - we are using the official meilisearch image.
  • MEILI_MASTER_KEY=TESTKEYLOCAL - In production environments, make sure to keep this key secure.
  • meili-search:/data.ms - we are using a named volume to persist data generated by the container. data.ms is the default folder where the database and indexes are stored.

Django and Meilisearch

Our Django app needs to be able to talk to our search engine. This is done via the Meilisearch rest API. Writing out the requests manually would be a lot of work. Fortunately, there is a python wrapper around the API that will save us the trouble.

requirements.txt

django==4.0.3
django-htmx==1.9.0
...

meilisearch==0.18.2 

config/settings.py

....
MEILISEARCH_URL = os.getenv("MEILISEARCH_URL", "http://meilisearch:7700/")
MEILISEARCH_API_TOKEN = os.getenv("MEILISEARCH_API_TOKEN ", "TESTKEYLOCAL")


Notes

  • meilisearch==0.18.2 - add the meilisearch-python client to our requirements
  • MEILISEARCH_URL - this is the url for the meilisearch, this is pointing to our docker service. In production, this would point to where your search engine instance is hosted.
  • MEILISEARCH_API_TOKEN - this is the value you defined earlier in the docker-compose file

At this stage, you can already interact with the search engine. Here is a short example of what you can already do:

import meilisearch

client = meilisearch.Client('http://127.0.0.1:7700', 'masterKey')

# An index is where the documents are stored.
index = client.index('books')

documents = [
      { 'id': 1, 'title': 'Test'},
      { 'id': 2, 'title': 'Other test',},
]
index.add_documents(documents)

This is convenient, out of the box we can already create indexes, add documents and perform search queries. We can improve our code by placing logic for these actions in one place. Let's add a search_index.py file to the mybnb app.

search_index.py

import meilisearch
from django.conf import settings


class SearchIndex:
    index = None

    class Indexes:
        """Available indexes"""
        app = "app"

    def __init__(self, index=None):
        self.index = index or self.Indexes.app

    def get_index(self):
        """Retrieve index"""
        client = meilisearch.Client(
            settings.MEILISEARCH_URL, settings.MEILISEARCH_API_TOKEN
        )
        return client.index(self.index)

    def delete_doc(self, doc):
        self.get_index().delete_document(doc)

    def clear_index(self):
        self.get_index().delete_all_documents()


search_index = SearchIndex().get_index()

Notes

  • class SearchIndex - we create a new object that abstracts away the details of connecting to the search engine, selecting the index, etc. The advantage of this approach is that your code is simpler, and easier to maintain. For example, you could switch to another search engine by only changing this class. Leaving the rest of your code undisturbed.
  • class Indexes: - An index is an equivalent of a table in relational databases or a collection of documents in document-based databases like MongoDB or firebase. Meilisearch supports multiple indexes. We added one index to the class but you could add multiple indexes within one instance.
  • get_index - use this method to retrieve the index after instantiating the class. For example, you could in your codebase you could use the class to create or fetch a new index like this
    • test_index=SearchIndex(index='test').get_index()

Models, Views, and Routing

This next section is all standard Django models, views, and urls. We will only focus on the important details in each file.

Let's create the model for storing data about our homes.

models.py

from django.db import models
from model_utils.models import UUIDModel


class HomeManager(models.Manager):
    def get_index_objects(self):
        """Objects formatted for indexing"""
        return [h.dict() for h in self.get_queryset()]

    def get_filter_attributes(self):
        """A dict of filterable attributes"""
        qs = self.get_queryset()
        return {
            "countries": list(set(qs.values_list("country", flat=True))),
            "cities": list(set(qs.values_list("city", flat=True))),
        }


class Home(UUIDModel):
    address = models.CharField(max_length=300)
    price = models.DecimalField(decimal_places=2, max_digits=7, default=100)
    city = models.CharField(max_length=200)
    country = models.CharField(max_length=200)
    image_url = models.CharField(max_length=500, default="")

    objects = HomeManager()

    def dict(self):
        return {
            "id": str(self.id),
            "address": self.address,
            "price": float(self.price),
            "city": self.city,
            "country": self.country,
            "image_url": self.image_url,
        }

Notes

We defined a model for storing Homes in the database. We also include a custom model manager for 'table-level functions like fetching all the homes for indexing.

  • get_index_objects - convenience method that will return a list of formatted documents for indexing.
  • get_filter_attributes - This method returns the list of cities and countries that you can filter on. This is relevant for later.
  • def dict - returns a dictionary of the home data for indexing.

views.py

@require_http_methods(["POST", "GET"])
def search(request):

    context, query_dict = {}, {}
    
    # use template partial for htmx requests
    template_name = "mybnb/search.html"
    if request.htmx:
        template_name = "mybnb/htmx/gallery.html"
    else:
        context.update(Home.objects.get_filter_attributes())

    # fetch and format search query parameters
    query_dict = request.GET if request.method == "GET" else request.POST
    opt_params = get_opt_params(query_dict)
    query = query_dict.get("query", None)

    # fetch results from the index and add them to the context
    results = search_index.search(query=query, opt_params=opt_params)
    context.update(
        {
            "homes": results["hits"],
            "total": results["nbHits"],
            "processing_time": results["processingTimeMs"],
            "offset": opt_params.get("offset", 0)
        }
    )

    return render(request, template_name, context)


def preview_home(request, doc_id):
    home = search_index.get_document(doc_id)
    template_name = "mybnb/htmx/preview.html"
    return render(request, template_name, {"home": home})

Notes

The view will handle both, POST and GET request methods. The first request when a user lands on the search page will be a GET request. Subsequent requests for filtering, full-text search, and loading objects dynamically in response to a user scrolling the page will be POST requests.

  • @require_http_methos(..) - the view will only support GET, POST requests
  • template_name - use a template partial for htmx request
  • context.update(Home..) - we need to populate our frontend template with values we can filter on. We use the get_filter_attributes method we defined earlier in the model manager.
  • query_dict - this view supports filtering using either GET, POST requests. The query_dict is passed to the get_opt_params function. More on that later.
  • results = search_index.search() - finally, we fetch the results from the index and add them to the context.
  • preview_home - modal view for previewing a home.

urls.py

from django.urls import path

from .views import search, preview_home

app_name = "mybnb"
urlpatterns = [
    path("search/", search, name="search"),
    path("preview_home/<str:doc_id>", preview_home, name="preview"),
]

Notes

Wire up the views to urls.py, preview_home will accept a string representing a document id as a parameter.

Business Logic

Create a new file called business_logic.py. This file will contain the logic for formatting search option parameters and a few other utility functions for interacting with Meillisearch.

business_logic.py

DEFAULT_FILTER_ATTRS = ["country"]
DEFAULT_SORT_ATTRS = ["price"]
DEFAULT_SEARCH_ATTRS = ["city", "address"]


def format_search_str(param):
    """Enclose in quotes if there is a space"""
    return f'"{param}"' if " " in param else param


def format_sort_params(query_dict, sort_attrs):
    """Format sort attrs for meilisearch"""
    formatted_sort = []
    for sort_attr in sort_attrs:
        s = query_dict.getlist(sort_attr, None)
        if len(s) > 0:
            formatted_sort.extend(s)
    return formatted_sort


def format_filter_params(query_dict, filter_attrs):
    """
    Formats filter attrs for meilisearch
    """
    filters = []
    for filter_attr in filter_attrs:
    
        filters.append([f'{filter_attr} = {format_search_str(f)}' \
        for f in query_dict.getlist(filter_attr)])
    return filters


def get_opt_params(query_dict, filter_attrs=None, sort_attrs=None):
    """
    Returns dictionary of formatted search options for meili
    """
    filter_attrs = filter_attrs or DEFAULT_FILTER_ATTRS
    sort_attrs = sort_attrs or DEFAULT_SORT_ATTRS
    
    opt_params = {}
    opt_params.update({"filter": format_filter_params(query_dict, filter_attrs)})
    opt_params.update({"sort": format_sort_params(query_dict, sort_attrs)})

    apply_offset = query_dict.get("apply_offset", False)
    opt_params["offset"] = int(query_dict.get("next_offset", 0)) \
                           if apply_offset == 'true' else 0
    
    return opt_params

def setup_attributes():
    """Setup index attrs"""
    search_index.update_filterable_attributes(DEFAULT_FILTER_ATTRS)
    search_index.update_searchable_attributes(DEFAULT_SEARCH_ATTRS)
    search_index.update_sortable_attributes(DEFAULT_SORT_ATTRS)

def index_homes():
    ...

def clear_index():
    ...

Notes

Most of the logic in this module is for converting query parameters in the request into a format that meilisearch can understand.

  • format_sort_params, format_filter_params , format_search_str- formats search and sort filters for meilisearch. For more information on this, check out the meilisearch docs.
  • get_opt_params - this function returns the option parameters used by meilisearch for filtering, sorting, and pagination. It takes in a dictionary of all the request parameters and formats them into search option parameters. In this example, we are filtering on country and sorting on price. You can extend the functionality to include faceted search attributes for example.
  • setup_attributes - before you can search, filter, and sort on any attributes, you need to configure the search engine. Use this method to configure your default attributes for sorting, filtering, and searching.
  • def index_homes & def clear index convenience methods you can use to index all the homes in your database & to clear the search index.

Django templates

On to the fun part. In this section, we will set up the frontend templates. Our focus will be on htmx and hyperscript and how we can leverage these

base.html

....
<script src="https://cdn.tailwindcss.com"></script>

<!--In production save these to your static files -->
<script defer src="https://unpkg.com/[email protected]/dist/cdn.min.js"></script>
<script src="https://unpkg.com/[email protected]"></script>

<script src="{% static 'js/htmx/htmx.min.js' %}" defer></script>
...

Notes

  • tailwindcss - include tailwindcss for styling. In production, you should minify your stylesheets via a build step.
  • scripts - Include alpinejs, hyperscript and htmx as dependencies.

mybnb/search.html

{% extends "base.html" %}

{% block content %}
    <div class="flex flex-col mx-auto container">
        <!--header section -->
        <div class="group flex mt-12 pt-2 justify-between bg-slate-100 sticky top-0 z-40">
            <!--logo, left aligned -->
            {% include "mybnb/components/header.html" %}
            <!--filter/search form, right aligned -->
            <form id="searchForm" class="flex space-x-2 opacity-100 items-center"
                  hx-post="{% url 'mybnb:search' %}"
                  hx-target="#search-results"
              >
                {% include "mybnb/components/filters.html" %}
                {% include "mybnb/components/sort.html" %}
                {% include "mybnb/components/offset.html" %}
                {% include "mybnb/components/search_field.html" %}
            </form>
        </div>
        <div class="flex justify-end mb-2 bg-slate-100 sticky top-12 z-40">
            {% include "mybnb/htmx/stats.html" %}
        </div>
    

        <!--Content-->
        <div class="mx-auto container mt-5">
            <section class="mt-8 pb-16" aria-labelledby="gallery-heading">
                <ul id="search-results" role="list"
                    class="grid grid-cols-2 gap-x-4 gap-y-8 sm:grid-cols-3 sm:gap-x-6 md:grid-cols-4 lg:grid-cols-3 xl:grid-cols-4 xl:gap-x-8">
                    {% include "mybnb/htmx/gallery.html" %}
                </ul>
            </section>
        </div>
    </div>
{% endblock %}

{% block footer_js %}
  {# todo #}
{% endblock

Notes

The template is small, less than 40 lines. The structure of the page is immediately clear at first glance. The most interesting bit on this page is the search form.

  • <form id="searchForm"..> the main search for the page, includes filters, sort, offset and a search field.
  • hx-post="{% url 'mybnb:search' %}", hx-target="#search-results" - on submit, htmx will make an ajax request to the search endpoint and swap out the inner HTML of the search-results element.

Components

mybnb/components/filters.html

<div id="filters" class="relative"
     x-data="{open: false }">
    <div>
        <button type="button"
            .....
            <span>Filters</span>
        </button>
    </div>
    <div
        x-show="open" x-cloak
        class="absolute z-10 left-1/2 transform -translate-x-2/3 mt-3 px-2 w-64 max-w-md sm:px-0">
        <div
            @click.away="open = false"
            class="rounded-lg shadow-lg ring-1 ring-black ring-opacity-5 overflow-hidden"  >
           ....
            <div class="border-t border-b border-gray-200 divide-y divide-gray-200 bg-white h-56 overflow-y-auto">
                {% for country in countries|slice:"1:10" %}
                    {% include "mybnb/components/_filter_checkbox.html" with prefix='country' name=country %}
                {% endfor %}
            </div>
            <div class="p-1 bg-gray-50">
                <div class="-m-3 p-3 flex justify-end rounded-md hover:bg-gray-100 transition ease-in-out duration-150">
                    <button type="submit"
                        @click="open = false">
                        <span>Apply</span>
                    </button>
                </div>
            </div>
        </div>
    </div>
</div>

Notes:

  • x-data= "{open: false}" - this dropdown component uses alpine to show a list of filters when (currently country) when clicked.
  • {% for country in countries|slice..%} - this will loop through the list of countries and include a checkbox for each country. You will remember that we included this list of filterable attributes earlier when we were creating the view.
  • <button type="submit"..> this component includes a button which will trigger a submit event on the #searchForm form when clicked.

mybnb/components/search_field.html

<div class="max-w-lg w-full lg:max-w-xs">
    <label for="search" class="sr-only">Search</label>
    <div class="relative">
        <div class="flex flex-col">
            <div class="relative">
             ...
            </div>
            <input class=""
              type="search"
              name="query" placeholder="Search for homes"
              @search="$event.prevent"
              hx-post="{% url 'mybnb:search' %}"
              hx-trigger="keyup changed delay:500ms, search"
              hx-target="#search-results"/>
            </div>
        </div>

    </div>
</div>

Notes:

This component implements the active search pattern from the htmx docs, with a slight variation.

  • hx-trigger="keyup changed delay:500ms, search"- this means, 'on keyup, if the value of the search box is changed, trigger a search event after 500ms.
  • @search="$event.prevent" - on search events, use alpinejs to prevent the defaultPOST event. Instead, the enclosing form will be submitted. This is necessary to maintain the state of the other filters when searching. If the post request is sent directly from the input, any filters on price or country would be discarded.

mybnb/htmx/gallery.html

{% load humanize %}
{% for home in homes %}
    <li class="relative">
        <!--like button-->
        ....

        <!--Image-->
        <div class="focus-within:ring-2 focus-within:ring-offset-....">
            .....
            <button type="button"
                    hx-get="{% url "mybnb:preview" home.id %}"
                    hx-target="body"
                    hx-swap="beforeend"
                    class="absolute inset-0 focus:outline-none z-30" >
                <span class="sr-only">{{ home.address }}</span>
            </button>
        </div>
        <!--Details-->
        <div class="flex justify-between">
          ...
        </div>
            {# infinite scroll only if 20 or more items in search query #}
            {% if forloop.last and total > 19 %}
               <div
               _="on intersection(intersecting)
                 if intersecting remove me then call infiniteScroll()">
              </div>
            {% endif %}
    </li>
{% endfor %}

{# include htmx oob template to show search stats under the search button #}
{% if request.htmx %}
    {% include "mybnb/htmx/stats.html" %}
    {% include "mybnb/components/offset.html" %}
{% endif %}

Notes

This template displays the homes.

  • {% for home in homes %} - loop through the homes context variable that we included in the view.
  • hx-get="{% url "mybnb:preview" home.id %}" - The image section includes a button that will fetch a modal that displays a preview of the home when clicked.
  • {% if forloop.last and total > 19 %} - This is how we implement infinite scrolling. By default, meilisearch returns 20 items per search request. If there are 20 or more homes for this search query, this element will be included. The element enters the viewport, hyperscript will call the infiniteScroll() function (which we still have to create) to load more events. The element is removed from the DOM to prevent further calls to the backend.
  • {% if request.htmx %} - since this template is rendered each time a search request is made we can piggyback information about search stats and page offset to each response and update the DOM with the results.

Let's extend the search.html template with the only javascript that your have to write.

mybnb/search.html

....
{% block footer_js %}
    <script>
        function infiniteScroll(){
            const form = document.getElementById("searchForm")
            const formData = new FormData(form)
            
            formData.append('apply_offset', true);
            let data = {}
            formData.forEach(function(value, key){
                data[key] = value;
            });

            data.country = formData.getAll('country')
            const context = {
              target:'#search-results', swap:'beforeend', values: data
            }
            htmx.ajax('POST', '{% url 'mybnb:search' %}', context)
        }
    </script>
{% endblock %}

Notes

This is the most 'complex' part. The search form stores the current filtering state of the pages, e.g. search query, active country filter, or sort attribute. When we fetch the next 20 results, the current filters should be applied to the request.

  • ..formData.forEach(..) - fetch form attributes and pass them as values to our htmx ajax request.
  • const context = {... - here we instruct htmx to append the response from the server before the end of the search results list.

mybnb/htmx/preview.html

{% load humanize %}
<div class="fixed z-50 inset-0 overflow-y-auto" role="dialog" aria-modal="true" id="previewContainer">
  <div class="flex min-h-screen text-center md:block md:px-2 lg:px-4" style="font-size: 0">

    <div class="flex text-base text-left transform transition w-full md:inline-block md:max-w-2xl md:px-4 md:my-8 md:align-middle lg:max-w-4xl">
      <div class="w-full relative flex items-center bg-white px-4 pt-14 pb-8 overflow-hidden shadow-2xl sm:px-6 sm:pt-8 md:p-6 lg:p-8">

          <button type="button"
                onclick="document.getElementById('previewContainer').remove()"
                class="absolute top-4 right-4 text-gray-400 hover:text-gray-500 sm:top-8 sm:right-6 md:top-6 md:right-6 lg:top-8 lg:right-8">
          <span class="sr-only">Close</span>
          <svg class="h-6 w-6" xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke="currentColor" aria-hidden="true">
            <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12" />
          </svg>
        </button>

        <div class="w-full grid grid-cols-1 gap-y-8 gap-x-6 items-start sm:grid-cols-12 lg:gap-x-8">
         ...

        </div>
      </div>
    </div>
  </div>
</div>

Notes

This is the preview container for displaying summary information about the home. This modal is rendered by the server and inserted into the DOM using htmx.

  • onclick="document.getElementById('previewContainer').remove()" - when dismissed, the element is removed from the DOM. The next preview will just be rendered by the server and appended to the DOM.

Conclusion

That's it. You should have a search UI complete with instant search, infinite scrolling, filtering, and sorting.

Bonus Tasks

This is a basic example, you can improve this by adding extending functionality to filter on multiple attributes.

Utilities

There are management commands included for seeding the db with sample data, setting up, updating, and clearing the index on your meilisearch instance. Check the make file on how to run them.

Further reading

  • htmx examples - checkout the examples section of the htmx docs. Demonstrates a number of common patterns such as active search, infinite scrolling, inline updates. Useful if you are starting with htmx.
  • meilisearch docs - see what meilisearch is capable of, including examples of how you can perform different actions, indexing, searching, filtering, etc from your favorite language.
  • instant-search - meilisearch includes an instant frontend library you can integrate with your stack. This is worth looking at if you will be exposing your instance directly to your frontend app.

Copyright © 2022 www.advantch.com