Replication internals

Replication is still very crudely implemented. The slave gets all the existing data on the first connection, which is less than optimal. As everything here, use it at your own risk.

Master

Master can bind a replication TCP server to a designed port (port 5293 by default).

The master server sits listens to command requests from slaves.

For now, the only supported command is 'SYNC'.

When it receives a SYNC command, the master transmits:

the metadata key map (containing the current key maps and indexes)
the exisiting data

and then it starts transmitting live data.

Format

All transmissions use the JSON format, separated by line feeds.

Internals internals

For each slave connection, the master buffers the transmission data using temporary files. These temporary files roll-over when they reach a configurable maximum.

Slave

A slave connects to the master and sends a SYNC command.

As it receives action commands, it starts applying them to the local database.

Transmission command structure

meta commands:

{m: 'meta', command: <command_name>, arguments: <arguments>}

data commands:

{m: <key_map_name>, k: <key>, v: <value>}

Valid meta commands are:

attach_key_map (key_map_name, options)

options:

type: 'cached_key_map' (default) or 'indexed_key_map'
compact_interval: in miliseconds. Defaults to 1 hour.

detach_key_map (key_map_name)

add_index (key_map_name, index_name, options, transform_function)