I'm proud to announce the release of our newest package Laravel Backup Server.

This paid package can backup several servers at once. When a backup contains files also present in a previous backup, deduplication using hard links will be performed. Even though you will see full backups in the filesystem, only changed files will take up disk space.

In this blog post, I'd like to introduce this package to you.

Alternatively, you can watch this video, where I'll give you a quick introduction to the package.

Introducing Laravel Backup Server

You can think of Laravel Backup Server as the spiritual successor to Laravel Backup. Let's explain the main difference first.

Laravel Backup is installed into a production Laravel app. That app can then copy itself and a dump of the database over to a remote filesystem.

Laravel Backup Server is installed into a separate Laravel application hosted on a server with lots of storage available. That application will ssh to all servers that need to be backed up.

When installing Laravel Backup Server, three important tables will be created. In backup_server_sources you can define which applications should be backed up. It contains the hostname and ssh configuration that should be used. There are attributes on the table, includes and excludes that can be used to determine which files need to be copied over. In the pre_backup_commands and post_backup_commands you can specify commands that should be run on the server you will back up. You can use these to dump your database to the filesystem, so it gets included in the backup.

In the backup_server_destinations table, you can specify where backups should be stored. A destination has a disk_name, which should be filled with one of the disks configured in the filesystems.php config file. Due to how file deduplication works (more on this fantastic space-saving feature later), you can only use local disks.

After you've defined a source and a destination, backups can be created. The most common way to do this is by hourly scheduling the backup-server:dispatch-backups command as shown in the installation instructions.

The backup_server_backups table is used to track all made backups. We use this to quickly calculate how much storage is used and how fast this backup was created.

The backup process

Under the hood, rsync is used to copy all files and directories specified in the includes array on the Source. Files and directories in the excludes array of the Source will be excluded.

If the backup isn't the initial one for a particular source, then rsync will take the previous backup into account. Only files that are new or changed will be copied to the destination directory of the backup. Files that are identical to the ones in the previous backup will not be copied. Instead, rsync will put a hard link to the file in the previous backup. This means that, even though a file will appear in your filesystem, no real disk space will be used.

When the previous backup is deleted, the hard links will still work. The filesystem is smart enough to know that there is still an entry in the filesystem pointing to the hard disk space and that space will not be freed.

This all has the fantastic benefit that you can just browse each backup. It contains all the files you expect. But under the hood, only new files in the backup will eat disk space.

Take a look at this screenshot, where we list a summary of all backups. For spatie.be, you can see that even though the total backup size is 13.41 GB, only 3.35 GB is used on disk.

In this screenshot, you'll see that each backup still contains all files.

The cleanup process

Over time the number of backups and the storage required to store them will grow. At some point, you will want to clean up backups.

To clean up all backups, this artisan command can be performed:

php artisan backup-server:cleanup

As mentioned in the installation instructions, it is recommend scheduling this command to run daily.

For each separate source, this command will dispatch a cleanup job that will delete all "old" backups. You can define a strategy class in the config file that will determine which backups are considered old. Laravel Backup Server ships with a default strategy called... 🥁 DefaultStrategy. This is how it works:

• Rule #1: it will never delete the latest backup regardless of its size or age
• Rule #2: it will keep all backups for the number of days specified in keep_all_backups_for_days
• Rule #3: it will only keep daily backups for the number of days specified in keep_daily_backups_for_days for all backups older than those covered by rule #2
• Rule #4: it will only keep weekly backups for the number of months specified in keep_weekly_backups_for_weeks for all backups older than those covered by rule #3
• Rule #5: it will only keep weekly backups for the number of months specified in keep_monthly_backups_for_months for all backups older than those covered by rule #4
• Rule #6: it'll only keep yearly backups for the number of years specified in keep_yearly_backups_for_years for all backups older than those covered by rule #5
• Rule #7: it will start deleting old backups until the volume of storage used is lower than the amount specified in delete_oldest_backups_when_using_more_megabytes_than.

Those keep-* and delete_oldest_backups_when_using_more_megabytes_than attributes are present on the source and on the destination of backup and in the cleanup.default_strategy key of the backup-server config file.

The monitoring process

The package can check the health of backups for every source, and the destination is healthy. A source is considered healthy if a backup was recently made, and not too much disk space is used for all backups for this source. A destination is considered healthy if it doesn't take up too much space.

This monitoring happens by executing the backup-server:monitor command that should be scheduled to run daily.

This command will send out notifications for each (un)healthy source and destination. It can notify you when a backup failed, when a source/destination is unhealthy, ... You can customize which notifications should be sent.

Because Laravel's notification system is used, you can use any of the channels you'd like: mail, slack, ...

Picking up a license

Because we invested quite some time in polishing this package, we've decided to make it a paid one. If this package is useful for you, consider picking up a license in our store at spatie.be.

Even if you're not in the market for a backup solution, purchasing a license can still be worthwhile. The code contains a couple of cool things for you to discover. In this video, which I made a couple of months ago, you'll already see a couple of neat things.

In closing

At Spatie, we manage 100+ servers for our clients. We've been using Laravel Backup Servers ourselves for the past several months to back them all up. Here's a screenshot of our destination. As you can see, the backup process's hard links save us an incredible amount of disk space.

Be sure also to take a look at our other paid products and this big list of free open source packages.

If you like our open source efforts, considering sponsoring us through GitHub. Sponsers will get access to new stuff earlier. Every now and then we'll send coupons with a deep discount for our paid products. In fact, all people that sponsored us yesterday morning got a 100% discount on backup server! You missed that, but by sponsoring us now you'll be one of the lucky ones the next time we send a coupon.