Revamp text search within Ambra.

if we ever need to search annotations, one path will be to index annotations with lucene, requiring some interaction between mulgara and lucene.

unknown when/if this will happen.

Just to add to the discussion. With the checkin on the backend with regards to 'blobs', we can potentially mark a blob content for Lucene indexing and searching. One of the many way we are thinking full-text search can potentially be improved. Not for this release of course, but for future reference.

Lucene Annotations

I would point to compass instead - I think that provides more features for us. Also, it's not just the blob but also various fields that need to be indexed by lucene, so I think it will be more than just blob triggered.

Increasing priority for next release.

We also need to merge simple search and advanced search internally into a single function and not two separate functions. From an end-user perspective there still might be two visible ways of searching, but internally it should map to one and the same.

We also need to make this 'journal' aware if necessary. At this stage Rich and I both agree that journal should be one more condition that the user can apply. I am making this the overall ticket related to text search and closing others as duplicate.

For next release just to remove complexity related to ingestion.

Note: need to make sure we fix #1030 and make sure we got fields mentioned in #723 when we upgrade Ambra to the new search stuff. This may require pulling out more metadata during ingest than we currently do.

There are also a number of fields that are not being indexed during ingest but should be. See Search Use Cases - Additional Fields

r6667 addresses this ticket.

(In [6771]) Switch Ambra to use the new OTM full-text search support. On the front-end side things still use and build lucene queries; these are then parsed and translated into an OQL query. This way nothing really changes from the users point of view.

The search-server is now no longer necessary and has been removed; and bringing up ambra with jetty and embedded mulgara/blob-store now supports searching too. The configuration for using mulgara's distributed resolver to split mulgara into two instances, with and without the lucene queries and storage, has not been done yet.

A couple things don't work quite yet either: there are some bugs with the -foo (i.e. the "without words") translation in complex cases, and the boost factors are not applied. Most everything else should work, though.

The hits are evaluated lazily (i.e. the loading of the article info is done only when those hits are viewed). Also, this is done in a way such that multiple clients running the same query (most likely a spider) will queue up as necessary and get those results as soon as they are ready without resorting to re-running the query if things time out.

Regarding caching, the query results are cached but the cache is never explicitly invalidated; instead it is assumed the ttl on the cache (currently configured to 10min) will be sufficient. The problem is it's basically impossible to figure out when an article is published whether it would show up in the search without re-running the query.

Lastly, because the query is filtered just like other OQL queries, the returned results are now already limited to the journal, so the total-results estimate is now more likely to be close to the real value (unpublished articles will still cause the results to be further filtered on demand, though).

Addresses #744.

(In [6854]) Add full support for -foo in search queries. This closes #744.

Milestone 0.9.1 deleted