Logging Service Features

Doradus uses a NoSQL database such as Cassandra for persistence. Each storage service organizes data into rows and columns using techniques beneficial to specific application types. The unique features of the Logging service are summarized below:

•	Designed for immutable time-series data: The Logging service is designed for time-stamped objects that are not modified once added. Special storage techniques are used that leverage each object’s timestamp and immutability, yielding fast loading, fast time-based queries, and dense storage.

•	Irregular data: Logging application objects can be highly irregular, using different fields for each object. This is useful, for example, when log records have a variable format or when logs from multiple user applications are stored in the same table.

•	Space usage: The Logging service “hyper compresses” data using special storage and compression techniques. For example, a batch of 64,000 objects that requires 10MB uncompressed and 1MB using GZIP compression requires only 250KB when stored in the Logging service.

•	Fast loading: The Logging service can load data up to 500K objects per second on a single node.

•	Queries: Object and aggregate queries can scan millions of objects per second without indexes. Query processing is designed to make time-constrained queries especially efficient. Even with irregular objects, all fields are queryable.

•	Live data: All data can be queried as soon as it is stored. Multiple applications can store objects with overlapping and out-of-sequence timestamps simultaneously.

•	Data aging: A Logging application schema can define automatic data aging, causing expired objects to be deleted automatically.

Because of its focus, the Logging service imposes restrictions compared to other Doradus storage services. In addition to requiring immutable, time-stamped objects, the Logging service has these restrictions:

•	No object IDs: Logging service objects do not require nor use object IDs.

•	SV scalar fields only: Link fields are not supported; each consists of single-valued (SV) scalar fields only.

•

Text field semantics: Except for the required Timestamp field, which is a timestamp value, queries treat all fields like text fields even if they contain data that is numeric, Boolean, etc. Queries that use numeric literals will work in some cases such as the clause EventID IN [500 TO 600]. However, the clause EventID IN [500 TO 1000] will not work as expected since EventID and the values 500 and 1000 are treated as text strings.

•	Metric functions: In aggregate queries, the Logging service currently only supports the metric function COUNT(*). Similarly, metric expressions (e.g., COUNT(*)+10) are not supported.

•	Single-level grouping: In aggregate queries, only a single grouping expression can be provided. Also, some grouping functions such as BATCH and SETS are not yet supported.

Most of these restrictions will be lifted as the Logging service evolves.