Ocyan Plugin: Oscar Elasticsearch

Installation

Add ocyan.plugin.oscar_elasticsearch to your projects dependencies.

The supported runtime is Python 3.10–3.12 with the documented MandelBlogStack Django/Oscar stack. Install the certified compatibility adapter ocyan.compat.django-oscar-elasticsearch>=3.0.2,<4; the plugin is packaged as an ordinary wheel or sdist (editable installs are for local development only).

Usage

This plugin adds search capabilities through elasticsearch. Elasticsearch provides scored search results, facetting, suggestions and autocomplete.

Configuration

Field that need to be treated as facets can be defined in webgui or ocyan.json. Configuration

All django settings that can be configured:

• **OSCAR_ELASTICSEARCH_HANDLE_STOCKRECORD_CHANGES: Enables handling of stock record changes automatically. Default is True.
• **OSCAR_ELASTICSEARCH_MIN_NUM_BUCKETS: Minimum number of buckets for search facets. Default is 2.
• **OSCAR_ELASTICSEARCH_FILTER_AVAILABLE: Filters products based on availability status. Default is False.
• **OSCAR_ELASTICSEARCH_DEFAULT_ITEMS_PER_PAGE: Number of items displayed per page. Defaults to OSCAR_PRODUCTS_PER_PAGE.
• **OSCAR_ELASTICSEARCH_ITEMS_PER_PAGE_CHOICES: Options for items per page settings. Default is [DEFAULT_ITEMS_PER_PAGE].
• **OSCAR_ELASTICSEARCH_MONTHS_TO_RUN_ANALYTICS: Defines months to run analytics queries. Default is 3.
• **OSCAR_ELASTICSEARCH_FACETS: Customizable search facets for filtering.
• **OSCAR_ELASTICSEARCH_SUGGESTION_FIELD_NAME: Field name used for suggestions. Default is "search_title".
• **OSCAR_ELASTICSEARCH_AUTOCOMPLETE_STATUS_FILTER: Status filter for search autocomplete, default depends on availability settings.
• **OSCAR_ELASTICSEARCH_AUTOCOMPLETE_CONTEXTS: Contexts for autocomplete suggestions.
• **OSCAR_ELASTICSEARCH_AUTOCOMPLETE_SEARCH_FIELDS: Fields used in autocomplete search. Default is ["title", "upc"].
• **OSCAR_ELASTICSEARCH_SEARCH_FIELDS: Specifies fields used for general search queries.
• **OSCAR_ELASTICSEARCH_SEARCH_QUERY_TYPE: Type of query used in search; default is "most_fields".
• **OSCAR_ELASTICSEARCH_SEARCH_QUERY_OPERATOR: Logical operator for search queries. Default is "or".
• **OSCAR_ELASTICSEARCH_NUM_SUGGESTIONS: Maximum number of suggestions returned. Default is 20.
• **OSCAR_ELASTICSEARCH_SERVER_URLS: Elasticsearch server URLs. Default is ["http://127.0.0.1:9200"].
• **OSCAR_ELASTICSEARCH_INDEX_PREFIX: Prefix used for Elasticsearch indices. Default is "django-oscar-elasticsearch".
• **OSCAR_ELASTICSEARCH_SORT_BY_CHOICES_SEARCH: Sorting options for search results.
• **OSCAR_ELASTICSEARCH_SORT_BY_MAP_SEARCH: Maps sort options to actual query parameters.
• **OSCAR_ELASTICSEARCH_SORT_BY_CHOICES_CATALOGUE: Sorting options specific to the catalog view.
• **OSCAR_ELASTICSEARCH_SORT_BY_MAP_CATALOGUE: Maps catalog sort options to query parameters.
• **OSCAR_ELASTICSEARCH_DEFAULT_ORDERING: Default ordering setting for searches.
• **OSCAR_ELASTICSEARCH_FACET_BUCKET_SIZE: Sets the size of facet buckets. Default is 10.
• **OSCAR_ELASTICSEARCH_INDEXING_CHUNK_SIZE: Defines chunk size for batch indexing operations. Default is 400.
• **OSCAR_ELASTICSEARCH_PRIORITIZE_AVAILABLE_PRODUCTS: Prioritizes available products in search results. Default is True.
• **OSCAR_ELASTICSEARCH_ALLOW_INSECURE_HTTP: Explicitly allow plain HTTP Elasticsearch URLs for disposable development only. Keep False in shared or production environments.
• **OSCAR_ELASTICSEARCH_FEED_USERNAME / OSCAR_ELASTICSEARCH_FEED_PASSWORD: Optional credentials for the staff-only feed endpoint. Configure both through a secret manager; neither has a package default.

The plugin does not contain credentials and will fail closed when only one feed credential is configured. Elasticsearch URLs should use HTTPS with a CA/API-key or other managed credential in deployed environments.

The cleanup_indices command is a dry-run by default. Removing stale indices requires both --remove-stale and the explicit --confirm flag, and only indices under the active configured prefix are eligible for deletion.

Facet types

Currently 2 facet types are supported

1. term, facets are treated as keywords and counted and matched as such.
2. range, facets are treated as integer ranges, the ranges parameter must de defined for type range. It can be used to segment the range, eg. [10, 100, 100] will yield 4 filters ranges, 0-9, 10-99, 100-999 and 1000+

Facet formatters

You can change the way your indexed data will be displayed by specifying a formatter in the facet definition::

{
    'name': 'price',
    'label': 'Prijs',
    'type': 'range',
    'formatter': 'ocyan.plugin.oscar_elasticsearch.search.format.currency',
    'ranges': [25, 100, 500, 1000]
}

The formatter can be specified my providing the module path to the function. Some useful formatters can be found in the module oscar_elasticsearch.search.format

Facet ordering

this is kinda broken now, needs fixing here: https://app-eu.wrike.com/open.htm?id=1673683285

The default ordering for results within a facet is alpahnumerically ({"-key", "asc"}). The number of facets returned can be changed with the ocyan parameter facet_bucket_size. By default only 10 facets will be returned. If there are a lot more facets then 10 and you do not want to increase the number of facets it can make a lot of sense to order by the number of occurrences, this will select the most useful facets. The ordering can be changed in the facet definition::

{
  "label": "Brand",
  "name": "brand",
  "type": "term",
  "order": { "_count" : "desc" }
}

Now the most popular brands will be shown. For more info, please read https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-order

Quicksearch filter

By default the quicksearch is configured to return all public products, or if you've chosen to show only available product, all available products.

Quicksearch can be configured with 2 django settings.

You can change the default filtering with OSCAR_ELASTICSEARCH_SUGGESTION_STATUS_FILTER

Choices are:

1. p - p means public, it will return any public product

2. a - a means available, it will return any available product

3. b - b means browsable, it will return exactly the same as your product listing pages, so no child products.

   from oscar_elasticsearch.search.constants import ES_CTX_PUBLIC, ES_CTX_AVAILABLE, ES_CTX_BROWSABLE, ES_CTX_NOT_PUBLIC OSCAR_ELASTICSEARCH_SUGGESTION_STATUS_FILTER = ES_CTX_NOT_PUBLIC

You can change the field name that get's searched on by altering the OSCAR_ELASTICSEARCH_SUGGESTION_FIELD_NAME setting. Default value is "search_title". The field name is field name that's added to the index. After changing this field name you should always reindex!

OSCAR_ELASTICSEARCH_SUGGESTION_FIELD_NAME = "my_custom_suggestion_search_field"

Autocomplete functionality is created using Elasticsearch's Completion suggester: https://www.elastic.co/docs/reference/elasticsearch/rest-apis/search-suggesters#completion-suggester

Which is optimised for speed and works completely different from the normal search API. First read the documentation before promising features to clients. If a customer wants features that are impossible, we could make an actual search preview with full search queries, this does increase the server load and dedicated hosting is strongly recommended.

Setting up items per page filters

To avoid unneeded load on the website, by default the user has no control over how many results are show on listing pages. However it is possible to enable such functionality by adding a setting to your oscar_elasticsearch config in ocyan.json under the items_per_page_choices key. This should be a list and MUST include the value oscar.dashboard_items_per_page::

"oscar_elasticsearch": {
    "items_per_page_choices": [21, 50, 100]
}

Boosting fields

Boosting field relevance is done by changing the settings on the index. You can add a ^ after you field name and add the boost after that:

Either change the Django settings like:

OSCAR_ELASTICSEARCH_SEARCH_FIELDS = [
    "_all_text",
    "code",
    "search_title^1",
    "search_title.reversed^0.8",
]

Or alter the search fields directly on the wanted search API: https://github.com/django-oscar/django-oscar-elasticsearch/blob/master/oscar_elasticsearch/search/api/search.py#L255

Limiting number of query results

Some people find that the search engine returns to many irrelevant results. To make the search more restrictive, such that there is only a match if ALL entered terms match.

We can alter 2 settings: OSCAR_ELASTICSEARCH_SEARCH_QUERY_TYPE OSCAR_ELASTICSEARCH_SEARCH_QUERY_OPERATOR

More info on the different multi matches: https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-multi-match-query#multi-match-types

More info on the default operator:

default_operator (Optional, string) Default boolean logic used to interpret text in the query string if no operators are specified. Valid values are:

OR (Default) For example, a query string of capital of Hungary is interpreted as capital OR of OR Hungary. AND For example, a query string of capital of Hungary is interpreted as capital AND of AND Hungary.

https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-top-level-params

If the customer requests to change from OR to AND, the following settings are recommended:

OSCAR_ELASTICSEARCH_SEARCH_QUERY_TYPE = "cross_fields"
OSCAR_ELASTICSEARCH_SEARCH_QUERY_OPERATOR = "and"

Customizing search

To customize the search in a project you need to create a new extension on ocyan.plugin.oscar_elasticsearch.search

for example: https://git.mandelblog.com/mandel/partswise/commit/a9a7d69a9e1fe3a1371d67649d59cd967ef22d13

• You can add fields to the mapping or change them in indexing/settings.py
• You could alter the odin mapping from a product to elasticsearch to change the content of the fields
• You can add new fields to the resource so they actually get picked up by elasticsearch
• You can add this new field as a filter, or as a new search field

Adding facets not based on attributes

To add a facet that's not based on an attribute you need to add a new field in the index/settings.py

from ocyan.main.loading import get_baseclass
base_get_products_index_mapping = get_baseclass("search.indexing.settings", "get_products_index_mapping", "myproject.apps.myproject_es_extension")

def get_products_index_mapping():
    base_settings = base_get_products_index_mapping()
    base_settings["properties"]["new_field"] = {"type": "text"}
    return base_settings

Then we add it to the resource in mappings/products/resources.py

from ocyan.main.loading import get_baseclass
BaseProductElasticSearchResource = get_baseclass(
    "search.mappings.products.resources",
    "ProductElasticSearchResource",
    "myproject.apps.myproject_es_extension",
)


class NewProductElasticSearchResource(BaseProductElasticSearchResource):
    new_field: str

After that we can fill the field through our odin mapping in mappings/products/mappings.py

import odin

from oscar_odin.resources.catalogue import (
    Product as ProductResource,
)

from ocyan.main.loading import get_baseclass

from .resources import NewProductElasticSearchResource

BaseProductMapping = get_baseclass(
    "search.mappings.products.mappings",
    "ProductMapping",
    "myproject.apps.myproject_es_extension",
)


class ProductMapping(BaseProductMapping):
    from_resource = ProductResource
    to_resource = NewProductElasticSearchResource
    
    @odin.assign_field()
    def new_field(self):
        return self.source.attributes.get("theattribute")

After this we update the index and we have our new field available in elasticsearch.

You can search on the new field by adding it to the search fields: OSCAR_ELASTICSEARCH_SEARCH_FIELDS You can add the field to the facet filters by adding a new definition in: OSCAR_ELASTICSEARCH_FACETS You can sort on it by changing the OSCAR_ELASTICSEARCH_SORT_BY_MAP_CATALOGUE and OSCAR_ELASTICSEARCH_SORT_BY_CHOICES_CATALOGUE

Modifying data before insertion into elasticsearch

This can be done by changing the odin mapping like explained in the above example. You can also edit existing fields instead of adding new ones

Sometimes public, sometimes not

import random

class ProductMapping(BaseProductMapping):
    from_resource = ProductResource
    to_resource = NewProductElasticSearchResource
    
    @odin.map_field(from_field="is_public")
    def is_public(self, is_public):
        if random.randint(0, 10) == 10:
            return False
        
        return is_public

Totally custom search

Sometimes you need to do something that just isn't supported by this plugin. We can make search engines out of any model or any data for that matter.

Here is an example of completely custom search engine:

https://git.mandelblog.com/mandel/filterwebshop/src/branch/master/filterwebshop/search.py

We can also do a lookup filter for creating personalised assortment based on the customers custom database tables

https://git.mandelblog.com/mandel/gvgoliehandel/src/branch/master/gvgoliehandel/index.py#L60

Or do some really extensive UI changes by figuring out the deselect url for each facet:

https://git.mandelblog.com/mandel/kroon/src/branch/master/kroon/apps/kroon_search_extension/views/base.py

Update index secretly

To update the index via the website go to url : /search/update_oscar_index

You will get prompted with a username and password (set these in your environment/settings).

You HAVE to be logged in as staff to do this. If you're not you will be send to a login screen and after logging in de update wil automatically go.