backup, restore, merge, delete, catchup, archive-push, archive-get, and validate processes can be executed on several parallel threads. This can significantly speed up pg_probackup3 operation given enough resources (CPU cores, disk, and network bandwidth).
Parallel execution is controlled by the
-j/--threads,
--num-write-threads, and --num-validate-threads
command-line options. These options must be non-negative integers.
The restore command does not support these options yet.
The number of threads will match the number of segments in the backup.
Parallel restore applies only to copying data from the backup catalog to the data directory of the cluster. When Postgres Pro server is started, WAL records need to be replayed, and this cannot be done in parallel.
If these options are not specified or set to zero, pg_probackup3 defaults to the number of CPU cores. If the core count cannot be determined, a single thread will be used.
When specified, --num-write-threads and
--num-validate-threads override -j.
If the requested threads exceed the system limit (e.g., from
/proc/sys/kernel/threads-max), a warning will be
displayed, and the system limit value will be used instead. If no limit is
found, the value specified by the user will be applied.
The thread limit can also be controlled by the PG_PROBACKUP_MAX_THREADS
environment variable. If the -j option is not specified,
but PG_PROBACKUP_MAX_THREADS is set, the thread count is
automatically determined based on available CPU cores but is capped by the
PG_PROBACKUP_MAX_THREADS value.
In the PRO mode, the number of read threads must be less than the value of
the max_wal_senders server parameter.
For example, to create a backup using four parallel threads, run the following command:
pg_probackup3 backup -Bbackup_dir--instance=instance_name-b FULL -j 4
The archive-push command is called by
Postgres Pro as part of the
archive_command
parameter and archive-get — as part of
restore_command.
Although triggered for one WAL file, a single pg_probackup3
process can handle multiple WAL files (scanning the pg_wal
directory in case of archive-push and copying from the
backup catalog in case of archive-get) in parallel using the
specified number of threads. You can also specify the --batch-size
option to process WAL segments in batches of the specified size. For more
details, refer to the archive-push and archive-get
command descriptions.