OLAP REST Commands : Shard Management Commands : Merge Shard
Merge Shard
All batches that have been loaded for a given shard are merged with the following REST command:
POST /{application}/_shards/{shard}[?{params}]
where:
•
{application} is the owning application's name.
•
{shard} is the name of the shard to be merged.
•
{params} are optional, comma-separated parameters that set properties on or control the merging of the shard. The available options are:
•
expire-date=<timestamp>: When specified, assigns an expiration date to the shard. The specified timestamp value must be in the format YYYY-MM-DD. A background merge task periodically checks for and deletes expired shards. If this option is not set, the shard is set to not expire even if it previously was assigned an expire-date. Also, see the Set Shard Properties REST command.
•
timeout=<seconds>: When specified, this value indicates the time in seconds to wait after the shard is merged before deleting the obsolete segment data. This can be used to allow executing queries to finish in case they are using the pre-merged segments. Normally, if a shard is merged while a query is executing, the query will abort and restart using the new segment data. Setting a timeout helps prevent ongoing queries from being restarted.
•
force-merge=<true|false>: This option forces the shard to be merged even if there are no new update batches for it. One reason for re-merging a shard is to store the shard’s data with a new compression ratio, defined by the doradus.yaml file’s olap_compression_level option.
The command requires no input entity. It merges the shard’s data synchronously, returning when the merge is complete. If the command is successful, a 200 OK response is returned, and all updates will be visible to queries.
The Merge Shard REST command is compatible with the auto-merge application option. The Merge Shard command is affected by the presence of the auto-merge option as summarized below:
•
Not set: If auto-merge is not defined, the Merge Shard command executes immediately. If another request is already performing a merge for the same shard via the same Doradus process, the command will immediately abort with a 404 error, indicating that the shard is already being merged. However, Doradus cannot detect if another merge is being performed via another server instance. Merging the same shard via multiple instances can result in corruption of the shard. Hence, is auto-merge is not set, all merge requests should be send to the same Doradus instance.
•
auto-merge set: If auto-merge is defined, the Merge Shard command uses the Task Manager service to perform an immediate “merge shard” task. This mechanism ensures that only one instance of the task is executing among all Doradus instances. If the shard is already being merged, the Merge Shard will (1) wait until the existing task finishes, then (2) perform an additional check to see if the shard has new, unmerged data, and if it does, (3) perform an additional merge operation for the shard. The Merge Shard command returns when the merge is complete.
Setting the auto-merge option adds an extra second or two to the Merge Shard command due to the interaction with the Task Manager service. However, this option ensures that a shard is not merged simultaneously by multiple tasks even in a multi-node environment.