How to use the command 'fpsync' (with examples)

fpsync is a powerful command-line utility used for synchronizing files and directories across different locations. It can perform file synchronization locally as well as over remote systems through SSH connections. What distinguishes fpsync from other synchronization tools is its ability to execute multiple parallel synchronization jobs simultaneously, offering increased speed and efficiency, especially beneficial when dealing with large data transfers. Here we explore several use cases of fpsync, providing practical examples and explanations for each.

Use case 1: Recursively synchronize a directory to another location

Code:

fpsync -v /path/to/source/ /path/to/destination/

Motivation:

This basic use case is perfect for users who need to keep data in two directories consistent without any complex operations. It’s akin to using cp, but with added benefits like progress monitoring and error handling, especially for large tasks.

Explanation:

• -v: This flag stands for ‘verbose’ and is used to output detailed information during the synchronization process, allowing a user to track the progress and catch any potential issues.
• /path/to/source/: The path to the directory whose contents are being copied. This can be any local directory on the user’s system.
• /path/to/destination/: The path to the directory where the contents are replicated. It can be local or mounted as a network drive.

Example output:

Synchronizing /path/to/source/ with /path/to/destination/
Synchronization complete. All files are up-to-date.

Use case 2: Recursively synchronize a directory with the final pass

Code:

fpsync -v -E /path/to/source/ /path/to/destination/

Motivation:

This use case is geared towards ensuring the destination directory is an exact mirror of the source by using rsync’s --delete option. It’s particularly crucial for environments where obsolete or moved files should not clutter the destination.

Explanation:

• -E: This option flags fpsync to execute a final pass, which incorporates rsync’s --delete functionality. This ensures that any files in the destination not present in the source are removed.

Example output:

Files successfully synchronized and non-existent files in the destination removed.
Synchronization complete.

Use case 3: Recursively synchronize a directory using 8 concurrent synchronization jobs

Code:

fpsync -v -n 8 -E /path/to/source/ /path/to/destination/

Motivation:

For those dealing with large datasets or directories, increasing the number of concurrent jobs can significantly accelerate the synchronization process. This particular configuration makes efficient use of system resources by spreading the workload across multiple threads.

Explanation:

• -n 8: This argument specifies the number of concurrent jobs; in this case, 8. Increasing parallel operations decreases synchronization time.
• As before, -E facilitates synchronization by ensuring all extraneous files in the destination are deleted.

Example output:

Starting 8 synchronization jobs...
All jobs completed successfully. Destination updated accurately.

Use case 4: Recursively synchronize using 8 concurrent synchronization jobs over two remote workers

Code:

fpsync -v -n 8 -E -w login@machine1 -w login@machine2 -d /path/to/shared/directory /path/to/source/ /path/to/destination/

Motivation:

Deploying tasks over multiple remote workers is indispensable for data centers or distributed storage nodes. This method leverages remote systems to distribute the synchronization load, enabling swift and efficient data management across different network locations.

Explanation:

• -w login@machine1 and -w login@machine2: These options specify the remote workers to distribute synchronization tasks via SSH. Each worker contributes to the overall synchronization, speeding up the process.
• -d /path/to/shared/directory: This indicates a shared working directory accessible by the remote machines, necessary for coordinating synchronization tasks.

Example output:

Distributing jobs to machine1 and machine2...
All tasks executed remotely. Synchronization complete.

Use case 5: Recursively synchronize using 4 local workers with transfer limits

Code:

fpsync -v -n 4 -f 1000 -s $((100 * 1024 * 1024)) /path/to/source/ /path/to/destination/

Motivation:

Imposing limits on the number of files and the size of data handled per job can help manage system resources effectively, preventing potential bottlenecks or overloads. This approach is beneficial for systems with limited hardware resources.

Explanation:

• -n 4: Directs the command to use 4 local workers.
• -f 1000: Restricts each job to transfer a maximum of 1000 files.
• -s $((100 * 1024 * 1024)): Limits each job to transfer data up to 100 MB. The calculation converts 100 MB to bytes.

Example output:

Running 4 local jobs with 1000 file and 100 MB maximums...
Synchronization jobs completed within limits.

Use case 6: Recursively synchronize excluding specific .snapshot* files

Code:

fpsync -v -O "-x|.snapshot*" /path/to/source/ /path/to/destination/

Motivation:

Excluding specific files or patterns ensures that unwanted data isn’t transferred, saving bandwidth and storage. This use case is particularly useful for environments that auto-generate temporary files that shouldn’t be synchronized.

Explanation:

• -O "-x|.snapshot*": The -O option allows integrating rsync options directly into fpsync execution. The pattern -x|.snapshot* ensures all files matching .snapshot* are excluded from synchronization.

Example output:

Ignoring files matching pattern .snapshot*
Synchronized without excluded files.

Conclusion:

fpsync is a versatile and powerful tool for file synchronization, offering a range of options to speed up and manage complex synchronization processes effectively. Its ability to conduct parallel jobs, manage remote workers, apply transfer limitations, and exclude files based on patterns allows users to customize synchronization tasks to meet their particular needs. Whether maintaining simplicity or addressing complex requirements, fpsync provides the functionality and efficiency expected in today’s fast-paced digital environments.