About Ranking

No Ranking

With No ranking, the term score is always 0, for all alphanumeric nodes of the query.

When term scoring is not really required, disabling term.score significantly improves hit matching performance (by up to +30%).

Rank

The Rank term score uses only the statically defined rank of each word. In the index, each word can have a rank for each document. The term.score value is rank * w.

w is a special node option, which can be set on each alphanumeric value, both in ELLQL and in UQL:

• in UQL: a OR b{w=2}
• in ELLQL: #alphanum{w=2}(text, "a")

The default value of w is 1.0

Use w to increase, decrease, or cancel the importance of the presence of one word with a specific rank.

For example, for query a AND b we have two matching documents:

• doc 1: a[rank=4] b[rank=6]
• doc 2: a[rank=6] b[rank=4]

With the default configuration, both doc1 and doc2 have term.score=10.

With the query a AND b{w=2}:

• doc1 has term.score=16
• doc2 has term.score=14

You can also use w to ignore a given word for the textual relevance calculation, by setting w=0.

Rank IDF

The Rank IDF term score adds the notion of IDF, or Inverse Document Frequency.

The IDF represents the relative rarity of a word in a corpus. The more frequently the word appears in the corpus, the lower its IDF. The idea behind this algorithm, is the rarer the word, the greater its importance.

For example, on query the OR economy, we want the documents matching economy first, because they are more specific.

For a given word, IDF(word) = 1 + log( number of docs in corpus / number of docs containing this word)

The term.score of one word with this algorithm is: rank * w * idf * 10000

IDF is a positive double above 1.0 (for a word that is in all documents).

For example, for a word present in only one document out of a corpus of one million, IDF = 20.9

Rank TF-IDF

The Rank TF-IDF term score adds the notion of Term Frequency. To represent the importance of a term within a document, it takes into account term density instead of term occurrences.

For example, we have the query: iphone and the following documents:

• Doc1: {iPhone}
• Doc2: {iPhone accessories}

Both have the same number of iphone occurrences, but doc1 is more dense with iphone and intuitively a better match. We consider that the number of occurrences is not as meaningful as the term density.

To use this algorithm, go to Data Model > Advanced Schema, click the index field to modify, select Compute TF, and click Apply.

For a word w in a document d, a simple version of TF would be:

SimpleTF(w, d) = (number of occurrences of w in d).

To avoid overranking documents where a word occurs frequently, Exalead CloudView uses a more advanced version of the formula:

TF(w, d) = (2.2 * SimpleTF(w, d) / (1.2 + SimpleTF(w,d))

The term.score of one word with this algorithm is: rank * w * tf * idf * 10000

TF varies between 1 (for a word present only once) to 2.2. Therefore, the term.score varies between rank * w * 10000 and rank * w * 10000 * 2.2.

BM25F

TF-IDF does not use the length of the document to normalize the term frequency. The BM25 term score uses a more complete version the TF formula:

SimpleTF(w, d) = (number of occurrences of w in d) * (length of the document) / 
(average length of all the documents)

As for TF-IDF, this SimpleTF is normalized to avoid overranking. Moreover, Exalead CloudView combines this term frequency with the TF-IDF value, using the following formula:

The term.score of one word with this algorithm is: rank * w * tf * idf * 10000.

TF varies between almost 0 (for a word that occurs once in a very large document) and 2.2 (for a word that occurs once in a small document and where all the other documents are large).

Custom

You can define your own custom ranking by selecting the Custom scoring algorithm and defining a formula.