The Analyzer Property

Doradus Spider uses analyzers to determine how to store and index scalar fields. The analyzer controls two aspects of scalar field management:

•	Indexing: The analyzer determines whether or not the field’s value are indexed.

•	Term extraction: If values are indexed, the analyzer generates one or more terms from each field value that are used to create index records. In other words, the analyzer tokenizes the field for indexing.

Every scalar field defined in the schema is assigned a default analyzer that is optimized for the scalar’s field type. Optionally, the field can explicitly declare an analyzer property that assigns the default or an alternate analyzer as allowed by the chart below:

Field Type	Default Analyzer	Alternate Analyzers
Text	TextAnalyzer	NullAnalyzer, OpaqueTextAnalyzer, HTMLAnalyzer
Integer	IntegerAnalyzer	NullAnalyzer
Long	IntegerAnalyzer	NullAnalyzer
Boolean	BooleanAnalyzer	NullAnalyzer
Timestamp	DateAnalyzer	NullAnalyzer
Binary	NullAnalyzer

 

Every scalar field type can declare the NullAnalyzer to prevent the field from being indexed. Text fields can assign the OpaqueTextAnalyzer to index the entire field as a single value instead of being tokenized into terms, or they can assign the HTMLAnalyzer to index the field as HTML text. Binary fields are not indexed and must always use the NullAnalyzer.

When a scalar field is MV, all values are indexed. For example, consider the following declaration:

<field name="Quotes" type="Text" collection="true" analyzer="TextAnalyzer"/>

Suppose an object is inserted with the following three Quotes values:

{"What time is it?", "Life is a Beach", "It's Happy Hour!"}

Using the TextAnalyzer, each unique term is indexed once (“a”, “happy”, “is”, etc.), and each whole value (“life is a beach”) is also indexed. Therefore phrase clauses such as Quotes:Happy will select the object, as will equality searches such as Quotes="life is a beach". If the analyzer is set to OpaqueTextAnalyzer, only whole values are indexed, hence the equality clause will select the object but the term clause won’t. Details of each analyzer are described in the next sections.