Registering a Shardman Cluster

To register a Shardman cluster in the etcd store, run the following command:

shardman-ladle [common_options] init [-f|--spec-file spec_file_name]|spec_text

You must provide the string with the cluster specification. You can do it as follows:

In the init mode, shardman-ladle reads the cluster specification, processes it and saves to the etcd store as parts of two JSON documents: ClusterSpec — as part of shardman/cluster0/clusterdata and LadleSpec — as part of shardman/cluster0/ladledata (cluster0 is the default cluster name used by Shardman utilities). Common options related to the etcd store, such as --store-endpoints, are also saved to the etcd store and pushed down to all Shardman services started by shardman-bowl. See sdmspec.json for the description of the Shardman initialization file format.

Adding Nodes to a Shardman Cluster

To add nodes to a Shardman cluster, run the following command:

shardman-ladle [common_options] addnotes -n|--nodes nodes_names [--no-rebalance]

You must specify the -n (--nodes) option to pass the comma-separated list of nodes to be added. Since all nodes are referred by their hostnames, these hostnames must be correctly resolved on all nodes.

With the default clover placement policy, nodes are added to a cluster by clovers. Each node in a clover runs the primary DBMS instance and perhaps several replicas of other nodes in the clover. The number of replicas is determined by the Repfactor configuration parameter. So, each clover consists of Repfactor + 1 nodes and can stand loss of Repfactor nodes.

shardman-ladle performs the addnodes operation in several steps. The command:

If addnodes command fails during execution, use the cleanup command to fix possible cluster configuration issues.

Performing Cleanup

To perform cleanup after failure of the addnodes command or shardmanctl rebalance command, run the following command:

shardman-ladle [common_options] cleanup [-p|--processrepgroups]

Final changes to the etcd store are done at the end of the command execution. This simplifies the cleanup process. During cleanup, incomplete clover definitions and definitions of the corresponding replication groups are removed from the etcd metadata. Definitions of the corresponding foreign servers are removed from the DBMS metadata of the remaining replication groups. Since the cleanup process can be destructive, by default it operates in the report-only mode: the tool only shows actions to be done during the actual cleanup. To execute the actual cleanup, use the -p (--processrepgroups) option.

Removing Nodes from a Shardman cluster

To remove nodes from a Shardman cluster, run the following command:

shardman-ladle [common_options] rmnodes -n|--nodes nodes_names

Use the -n (--nodes) option to pass the comma-separated list of nodes to be removed. This command removes clovers containing the specified nodes from the cluster. The last clover in the cluster cannot be removed. If the cluster has any global tables whose one of the main replication groups belongs to a clover, such a clover cannot be removed either. Any data (such as partitions of sharded relations) on removed replication groups is migrated to the remaining replication groups using logical replication, and all references to the removed replication groups (including definitions of foreign servers) are removed from the metadata of the remaining replication groups. Finally, the metadata in etcd is updated.

To remove a replication group containing global table data, first, disassemble the global table with the shardman.make_table_local() extension function. To remove all nodes in a cluster and not care about the data, just reinitialize the cluster. If a removed replication group contains local (non-sharded and non-global) tables, the data is silently lost after the replication group removal.

Getting the Health Status of Cluster Subsystems

To display the health status of Shardman cluster subsystems, run the following command:

shardman-ladle [common_options] status [-f|--format text|json]

To get the report in plain-text or JSON format, pass the value of plain or json through the -f (--format) option. Plain-text format is used by default. The command checks the availability of all etcd cluster nodes, consistency of metadata stored in etcd, correctness of replication group definitions, availability of shardman-bowl daemons and of all DBMS instances in the cluster. Each detected issue is reported as an unknown status, warning, error or fatal error. The tool can also report an operational error, which means that there was an issue during the cluster health check. When the command encounters a fatal or operational error, it stops further diagnostics. An error is considered fatal if it impacts higher-level subsystems. For example, an inconsistency in etcd metadata does not allow correct cluster operations and must be handled first, so there is no point in further diagnostics.

Backing up a Shardman Cluster

To backup a Shardman cluster, run the following command:

shardman-ladle [common_options] backup --datadir directory [--maxtasks number_of_tasks]

Use the following options:

A backup consists of a directory with base backups of all replication groups and WAL files needed for recovery. etcd metadata is saved to the etcd_dump file. The backup_info file is created during a backup and contains the backup description.

For details of the backup command logic, see Cluster Backup Process.

Restoring a Shardman Cluster

To restore a Shardman cluster from a backup created by the backup command, run the following command:

shardman-ladle [common_options] recover [--info file] [--dumpfile file] [--metadata-only] [--timeout seconds]

Use the following options:

Before running the recover command, specify DataRestoreCommand and RestoreCommand in the backup_info file. DataRestoreCommand fetches the base backup and restores it to the Stolon data directory. RestoreCommand fetches the WAL file and saves it to Stolon pg_wal directory. These commands can use the following substitutions:

stolon-keeper runs both commands on each node in the cluster. Therefore:

These examples show how to specify RestoreCommand and DataRestoreCommand:

For details of the recover command logic, see Recovery from Shardman Backups.

Initializing the Cluster

To initialize a Shardman cluster that has the cluster0 name, uses an etcd cluster consisting of n1,n2 and n3 nodes listening on port 2379, ensure proper settings in the spec file sdmspec.json and run:

$ shardman-ladle --store-endpoints http://n1:2379,http://n2:2379,http://n3:2379 init -f sdmspec.json

Adding Nodes to the Cluster

To add n1,n2, n3 and n4 nodes to the cluster, run:

$ shardman-ladle --store-endpoints http://n1:2379,http://n2:2379,http://n3:2379 addnodes -n n1,n2,n3,n4

Removing Nodes from the Cluster

To remove n1 and n2 nodes, along with clovers that contain them, from the cluster0 cluster, run:

$ shardman-ladle --store-endpoints http://n1:2379,http://n2:2379,http://n3:2379 rmnodes -n n1,n2

Getting the Cluster Status

Here is a sample status output from shardman-ladle:

$ shardman-ladle --store-endpoints http://n1:2379,http://n2:2379,http://n3:2379 status

=== Store status ===
  STATUS    MESSAGE    REPLICATION GROUP  NODE
  OK      Store is OK
=== Metadata status ===
  STATUS     MESSAGE      REPLICATION GROUP  NODE
  OK      Metadata is OK
=== Bowls status ===
  STATUS         MESSAGE        REPLICATION GROUP  NODE
  OK      Bowl on node n1 is OK                    n1
  OK      Bowl on node n2 is OK                    n2
  OK      Bowl on node n3 is OK                    n3
  OK      Bowl on node n4 is OK                    n4
=== Replication Groups status ===
  STATUS             MESSAGE             REPLICATION GROUP  NODE
  OK      Replication group clover-1-n1  clover-1-n1
          is OK
  OK      Replication group clover-1-n2  clover-1-n2
          is OK
  OK      Replication group clover-2-n3  clover-2-n3
          is OK
  OK      Replication group clover-2-n4  clover-2-n4
          is OK
=== Dictionary status ===
  STATUS             MESSAGE             REPLICATION GROUP  NODE
  OK      Replication group clover-1-n1  clover-1-n1
          dictionary is OK
  OK      Replication group clover-1-n2  clover-1-n2
          dictionary is OK
  OK      Replication group clover-2-n3  clover-2-n3
          dictionary is OK
  OK      Replication group clover-2-n4  clover-2-n4
          dictionary is OK

Performing Backup and Recovery

To create a backup of the cluster0 cluster using etcd at etcdserver listening on port 2379 and store it in the local directory /var/backup/shardman, run:

$ shardman-ladle --store-endpoints http://etcdserver:2379 backup --datadir=/var/backup/shardman

Assume that you are performing a recovery from a backup to the cluster0 cluster using etcd at etcdserver listening on port 2379 and you take the backup description from the /var/backup/shardman/backup_info file. Edit the /var/backup/shardman/backup_info file, set DataRestoreCommand, RestoreCommand as necessary and run:

$ shardman-ladle --store-endpoints http://etcdserver:2379 recover --info /var/backup/shardman/backup_info

For metadata-only recovery, run:

$ shardman-ladle --store-endpoints http://etcdserver:2379 recover --metadata-only --dumpfile /var/backup/shardman/etcd_dump