Search API Extension

[psilabs-dev]

Issue: #1226

(this might take a while)

Generalization of existing search API to support OR/NOT clauses. Due to the UI design, the PR feature is limited to category AND filter composition, with filters supporting OR composition (at least, that's what I find visually sensible w/o cluttering the UI in an unseemly way, might need to try it out more). However, the API should support a wide variety of use cases, which third-party clients may readily take advantage of if their UI layer accomodates this customization.

This PR will be broken down to several phases:

1. core search functionality will be extracted; do_composite_search_inner and search_core become supersets of existing search functionality, will be unit-tested and checked for regressions.
2. API contract design, we figure out how the API should look for composite search. This wouldn't fit into "/api/search" or "/search", so I've opted for "/api/search/composite" and be free of existing limits. This means we can also start integration testing contract for correctness and functionality immediately.
3. initial UI design (category AND, search filter OR extension), tentatively category NOT.
4. Caching design. Caching logic will be taken out of LRR, and then gradually re-introduced as tests are confident.

"/api/search" and "/search" are subsets of "/api/search/composite", so we should be able to swap their internal logic with the composite search algorithm and find no behavioral difference.

I consider the important parts of the PR to be the API schema (which decides how everything will be implemented), and what is expected of search functionality. If the API changes, then all UI based on this API will change. From a UI perspective, will have to think about how to design this in a way that is still aesthetically acceptable.

Given the presence of JSON filtering data I've opted to switch to POST.

Regarding the core search functionality, you'll notice that the search algorithm does not concern itself with categories, as they are logically reducible.

• If we AND a static category, then we introduce a list of archives for further filtering.
• If we NOT a static category, then we remove filtered archives which are members.
• If we AND/NOT a dynamic category, then we decompose to tokens (and optionally flip isneg) pre-filtering.

So category operations live around the search algorithm, in the form of pre/post-search filtering.

A proof-of-concept UI: ability to select multiple categories in AND composition, combined with OR composition of multiple search filters (which by themselves are AND clauses). On re-draw, the UI joins the categories into each OR part.

Still eye-balling #1438, as discussions there affect the API design. The current direction is probably still safe in terms of tank-related extensibility.