Ocyan Plugin: Oscar Elasticsearch
=================================

Installation
------------
Add `ocyan.plugin.oscar_elasticsearch` to your projects dependencies.
------------

Usage
-----
This plugin adds search capabilities through elasticsearch.
Elasticsearch provides scored search results, facetting, suggestions and
autocomplete.

Configuration
-------------
Field that need to be treated as facets can be defined in webgui or ocyan.json.
Configuration
-------------

All django settings that can be configured:

- **`OSCAR_ELASTICSEARCH_HANDLE_STOCKRECORD_CHANGES`: Enables handling of stock record changes automatically. Default is True.
- **`OSCAR_ELASTICSEARCH_MIN_NUM_BUCKETS`: Minimum number of buckets for search facets. Default is 2.
- **`OSCAR_ELASTICSEARCH_FILTER_AVAILABLE`: Filters products based on availability status. Default is False.
- **`OSCAR_ELASTICSEARCH_DEFAULT_ITEMS_PER_PAGE`: Number of items displayed per page. Defaults to OSCAR_PRODUCTS_PER_PAGE.
- **`OSCAR_ELASTICSEARCH_ITEMS_PER_PAGE_CHOICES`: Options for items per page settings. Default is [DEFAULT_ITEMS_PER_PAGE].
- **`OSCAR_ELASTICSEARCH_MONTHS_TO_RUN_ANALYTICS`: Defines months to run analytics queries. Default is 3.
- **`OSCAR_ELASTICSEARCH_FACETS`: Customizable search facets for filtering.
- **`OSCAR_ELASTICSEARCH_SUGGESTION_FIELD_NAME`: Field name used for suggestions. Default is "search_title".
- **`OSCAR_ELASTICSEARCH_AUTOCOMPLETE_STATUS_FILTER`: Status filter for search autocomplete, default depends on availability settings.
- **`OSCAR_ELASTICSEARCH_AUTOCOMPLETE_CONTEXTS`: Contexts for autocomplete suggestions.
- **`OSCAR_ELASTICSEARCH_AUTOCOMPLETE_SEARCH_FIELDS`: Fields used in autocomplete search. Default is ["title", "upc"].
- **`OSCAR_ELASTICSEARCH_SEARCH_FIELDS`: Specifies fields used for general search queries.
- **`OSCAR_ELASTICSEARCH_SEARCH_QUERY_TYPE`: Type of query used in search; default is "most_fields".
- **`OSCAR_ELASTICSEARCH_SEARCH_QUERY_OPERATOR`: Logical operator for search queries. Default is "or".
- **`OSCAR_ELASTICSEARCH_NUM_SUGGESTIONS`: Maximum number of suggestions returned. Default is 20.
- **`OSCAR_ELASTICSEARCH_SERVER_URLS`: Elasticsearch server URLs. Default is ["http://127.0.0.1:9200"].
- **`OSCAR_ELASTICSEARCH_INDEX_PREFIX`: Prefix used for Elasticsearch indices. Default is "django-oscar-elasticsearch".
- **`OSCAR_ELASTICSEARCH_SORT_BY_CHOICES_SEARCH`: Sorting options for search results.
- **`OSCAR_ELASTICSEARCH_SORT_BY_MAP_SEARCH`: Maps sort options to actual query parameters.
- **`OSCAR_ELASTICSEARCH_SORT_BY_CHOICES_CATALOGUE`: Sorting options specific to the catalog view.
- **`OSCAR_ELASTICSEARCH_SORT_BY_MAP_CATALOGUE`: Maps catalog sort options to query parameters.
- **`OSCAR_ELASTICSEARCH_DEFAULT_ORDERING`: Default ordering setting for searches.
- **`OSCAR_ELASTICSEARCH_FACET_BUCKET_SIZE`: Sets the size of facet buckets. Default is 10.
- **`OSCAR_ELASTICSEARCH_INDEXING_CHUNK_SIZE`: Defines chunk size for batch indexing operations. Default is 400.
- **`OSCAR_ELASTICSEARCH_PRIORITIZE_AVAILABLE_PRODUCTS`: Prioritizes available products in search results. Default is True.

Facet types
-----------

Currently 2 facet types are supported
1. ``term``, facets are treated as keywords and counted and matched as such.
2. ``range``, facets are treated as integer ranges, the ``ranges`` parameter
   must de defined for type range. It can be used to segment the range, eg.
   ``[10, 100, 100]`` will yield 4 filters ranges, 0-9, 10-99, 100-999 and 1000+


Facet formatters
----------------

You can change the way your indexed data will be displayed by specifying a
formatter in the facet definition::

    {
        'name': 'price',
        'label': 'Prijs',
        'type': 'range',
        'formatter': 'ocyan.plugin.oscar_elasticsearch.search.format.currency',
        'ranges': [25, 100, 500, 1000]
    }

The formatter can be specified my providing the module path to the function.
Some useful formatters can be found in the module ``oscar_elasticsearch.search.format``

Facet ordering
--------------

this is kinda broken now, needs fixing here: https://app-eu.wrike.com/open.htm?id=1673683285

The default ordering for results within a facet is alpahnumerically  (``{"-key", "asc"}``).
The number of facets returned can be changed with the ocyan parameter ``facet_bucket_size``.
By default only 10 facets will be returned. If there are a lot more facets then
10 and you do not want to increase the number of facets it can make a lot of sense
to order by the number of occurrences, this will select the most useful facets.
The ordering can be changed in the facet definition::

    {
      "label": "Brand",
      "name": "brand",
      "type": "term",
      "order": { "_count" : "desc" }
    }

Now the most popular brands will be shown.
For more info, please read https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-order


Quicksearch filter
------------------

By default the quicksearch is configured to return all public products, or if
you've chosen to show only available product, all available products.

Quicksearch can be configured with 2 django settings.

You can change the default filtering with OSCAR_ELASTICSEARCH_SUGGESTION_STATUS_FILTER

Choices are:
1. p - p means public, it will return any public product
2. a - a means available, it will return any available product
3. b - b means browsable, it will return exactly the same as your product listing
   pages, so no child products.

    from oscar_elasticsearch.search.constants import ES_CTX_PUBLIC, ES_CTX_AVAILABLE, ES_CTX_BROWSABLE, ES_CTX_NOT_PUBLIC
    OSCAR_ELASTICSEARCH_SUGGESTION_STATUS_FILTER = ES_CTX_NOT_PUBLIC

You can change the field name that get's searched on by altering the OSCAR_ELASTICSEARCH_SUGGESTION_FIELD_NAME setting. Default value is "search_title".
The field name is field name that's added to the index. After changing this field name you should always reindex!

    OSCAR_ELASTICSEARCH_SUGGESTION_FIELD_NAME = "my_custom_suggestion_search_field"

Autocomplete functionality is created using Elasticsearch's Completion suggester: https://www.elastic.co/docs/reference/elasticsearch/rest-apis/search-suggesters#completion-suggester

Which is optimised for speed and works completely different from the normal search API. First read the documentation before promising features to clients.
If a customer wants features that are impossible, we could make an actual search preview with full search queries, this does increase the server load and dedicated hosting is strongly recommended.

Setting up items per page filters
---------------------------------

To avoid unneeded load on the website, by default the user has no control over
how many results are show on listing pages. However it is possible to enable
such functionality by adding a setting to your oscar_elasticsearch config in
ocyan.json under the ``items_per_page_choices`` key. This should be a list and
MUST include the value ``oscar.dashboard_items_per_page``::

    "oscar_elasticsearch": {
        "items_per_page_choices": [21, 50, 100]
    }

Boosting fields
---------------

Boosting field relevance is done by changing the settings on the index. You can add a ^ after you field name and add the boost after that:

Either change the Django settings like:

    OSCAR_ELASTICSEARCH_SEARCH_FIELDS = [
        "_all_text",
        "code",
        "search_title^1",
        "search_title.reversed^0.8",
    ]


Or alter the search fields directly on the wanted search API: https://github.com/django-oscar/django-oscar-elasticsearch/blob/master/oscar_elasticsearch/search/api/search.py#L255

Limiting number of query results
--------------------------------

Some people find that the search engine returns to many irrelevant results.
To make the search more restrictive, such that there is only a match if ALL entered terms match.

We can alter 2 settings:
    OSCAR_ELASTICSEARCH_SEARCH_QUERY_TYPE
    OSCAR_ELASTICSEARCH_SEARCH_QUERY_OPERATOR

More info on the different multi matches:
https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-multi-match-query#multi-match-types

More info on the default operator:

default_operator
(Optional, string) Default boolean logic used to interpret text in the query string if no operators are specified. Valid values are:

OR (Default)
    For example, a query string of `capital of Hungary` is interpreted as capital OR of OR Hungary.
AND
    For example, a query string of `capital of Hungary` is interpreted as capital AND of AND Hungary.

https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-top-level-params

If the customer requests to change from OR to AND, the following settings are recommended:

    OSCAR_ELASTICSEARCH_SEARCH_QUERY_TYPE = "cross_fields"
    OSCAR_ELASTICSEARCH_SEARCH_QUERY_OPERATOR = "and"

Customizing search
------------------

To customize the search in a project you need to create a new extension on ocyan.plugin.oscar_elasticsearch.search

for example:
https://git.mandelblog.com/mandel/partswise/commit/a9a7d69a9e1fe3a1371d67649d59cd967ef22d13

- You can add fields to the mapping or change them in indexing/settings.py
- You could alter the odin mapping from a product to elasticsearch to change the content of the fields
- You can add new fields to the resource so they actually get picked up by elasticsearch
- You can add this new field as a filter, or as a new search field



Adding facets not based on attributes
-------------------------------------

To add a facet that's not based on an attribute you need to add a new field in the index/settings.py

    from ocyan.main.loading import get_baseclass
    base_get_products_index_mapping = get_baseclass("search.indexing.settings", "get_products_index_mapping", "myproject.apps.myproject_es_extension")
    
    def get_products_index_mapping():
        base_settings = base_get_products_index_mapping()
        base_settings["properties"]["new_field"] = {"type": "text"}
        return base_settings

Then we add it to the resource in mappings/products/resources.py

    from ocyan.main.loading import get_baseclass
    BaseProductElasticSearchResource = get_baseclass(
        "search.mappings.products.resources",
        "ProductElasticSearchResource",
        "myproject.apps.myproject_es_extension",
    )
    
    
    class NewProductElasticSearchResource(BaseProductElasticSearchResource):
        new_field: str
    

After that we can fill the field through our odin mapping in mappings/products/mappings.py

    import odin
    
    from oscar_odin.resources.catalogue import (
        Product as ProductResource,
    )
    
    from ocyan.main.loading import get_baseclass
    
    from .resources import NewProductElasticSearchResource
    
    BaseProductMapping = get_baseclass(
        "search.mappings.products.mappings",
        "ProductMapping",
        "myproject.apps.myproject_es_extension",
    )
    
    
    class ProductMapping(BaseProductMapping):
        from_resource = ProductResource
        to_resource = NewProductElasticSearchResource
        
        @odin.assign_field()
        def new_field(self):
            return self.source.attributes.get("theattribute")

After this we update the index and we have our new field available in elasticsearch.

You can search on the new field by adding it to the search fields: OSCAR_ELASTICSEARCH_SEARCH_FIELDS
You can add the field to the facet filters by adding a new definition in: OSCAR_ELASTICSEARCH_FACETS
You can sort on it by changing the OSCAR_ELASTICSEARCH_SORT_BY_MAP_CATALOGUE and OSCAR_ELASTICSEARCH_SORT_BY_CHOICES_CATALOGUE

Modifying data before insertion into elasticsearch
--------------------------------------------------

This can be done by changing the odin mapping like explained in the above example. You can also edit existing fields instead of adding new ones

Sometimes public, sometimes not

    import random
    
    class ProductMapping(BaseProductMapping):
        from_resource = ProductResource
        to_resource = NewProductElasticSearchResource
        
        @odin.map_field(from_field="is_public")
        def is_public(self, is_public):
            if random.randint(0, 10) == 10:
                return False
            
            return is_public

Totally custom search
---------------------

Sometimes you need to do something that just isn't supported by this plugin.
We can make search engines out of any model or any data for that matter.

Here is an example of completely custom search engine:

    https://git.mandelblog.com/mandel/filterwebshop/src/branch/master/filterwebshop/search.py

We can also do a lookup filter for creating personalised assortment based on the customers custom database tables

    https://git.mandelblog.com/mandel/gvgoliehandel/src/branch/master/gvgoliehandel/index.py#L60

Or do some really extensive UI changes by figuring out the deselect url for each facet:

    https://git.mandelblog.com/mandel/kroon/src/branch/master/kroon/apps/kroon_search_extension/views/base.py

Update index secretly
---------------------

To update the index via the website go to url : /search/update_oscar_index

You will get prompted with a username and password (set these in your environment/settings).

You HAVE to be logged in as staff to do this. If you're not you will be send to a login screen and after logging in de update wil automatically go.
