Scout APM is PHP application performance monitoring designed for developers. With tracing logic that ties issues back to the line of code causing them, you can pinpoint n+1 queries, memory leaks, and other abnormalities in real time so you can knock them out and get back to building a great product. Start your free 14-day trial today and get the performance insight you need in less than 4 minutes.

Introducing Laravel Backup Server

Original – by Freek Van der Herten – 8 minute read

I'm proud to announce the release of our newest package Laravel Backup Server.

This paid package can backup several servers at once. When a backup contains files also present in a previous backup, deduplication using hard links will be performed. Even though you will see full backups in the filesystem, only changed files will take up disk space.

In this blog post, I'd like to introduce this package to you.

Alternatively, you can watch this video, where I'll give you a quick introduction to the package.

Introducing Laravel Backup Server

You can think of Laravel Backup Server as the spiritual successor to Laravel Backup. Let's explain the main difference first.

Laravel Backup is installed into a production Laravel app. That app can then copy itself and a dump of the database over to a remote filesystem.

Laravel Backup Server is installed into a separate Laravel application hosted on a server with lots of storage available. That application will ssh to all servers that need to be backed up.

When installing Laravel Backup Server, three important tables will be created. In backup_server_sources you can define which applications should be backed up. It contains the hostname and ssh configuration that should be used. There are attributes on the table, includes and excludes that can be used to determine which files need to be copied over. In the pre_backup_commands and post_backup_commands you can specify commands that should be run on the server you will back up. You can use these to dump your database to the filesystem, so it gets included in the backup.

screenshot

In the backup_server_destinations table, you can specify where backups should be stored. A destination has a disk_name, which should be filled with one of the disks configured in the filesystems.php config file. Due to how file deduplication works (more on this fantastic space-saving feature later), you can only use local disks.

screenshot

After you've defined a source and a destination, backups can be created. The most common way to do this is by hourly scheduling the backup-server:dispatch-backups command as shown in the installation instructions.

The backup_server_backups table is used to track all made backups. We use this to quickly calculate how much storage is used and how fast this backup was created.

screenshot

The backup process

Under the hood, rsync is used to copy all files and directories specified in the includes array on the Source. Files and directories in the excludes array of the Source will be excluded.

If the backup isn't the initial one for a particular source, then rsync will take the previous backup into account. Only files that are new or changed will be copied to the destination directory of the backup. Files that are identical to the ones in the previous backup will not be copied. Instead, rsync will put a hard link to the file in the previous backup. This means that, even though a file will appear in your filesystem, no real disk space will be used.

When the previous backup is deleted, the hard links will still work. The filesystem is smart enough to know that there is still an entry in the filesystem pointing to the hard disk space and that space will not be freed.

This all has the fantastic benefit that you can just browse each backup. It contains all the files you expect. But under the hood, only new files in the backup will eat disk space.

Take a look at this screenshot, where we list a summary of all backups. For spatie.be, you can see that even though the total backup size is 13.41 GB, only 3.35 GB is used on disk.

screenshot

In this screenshot, you'll see that each backup still contains all files.

screenshot

The cleanup process

Over time the number of backups and the storage required to store them will grow. At some point, you will want to clean up backups.

To clean up all backups, this artisan command can be performed:

php artisan backup-server:cleanup

As mentioned in the installation instructions, it is recommend scheduling this command to run daily.

For each separate source, this command will dispatch a cleanup job that will delete all "old" backups. You can define a strategy class in the config file that will determine which backups are considered old. Laravel Backup Server ships with a default strategy called... 🥁 DefaultStrategy. This is how it works:

  • Rule #1: it will never delete the latest backup regardless of its size or age
  • Rule #2: it will keep all backups for the number of days specified in keep_all_backups_for_days
  • Rule #3: it will only keep daily backups for the number of days specified in keep_daily_backups_for_days for all backups older than those covered by rule #2
  • Rule #4: it will only keep weekly backups for the number of months specified in keep_weekly_backups_for_weeks for all backups older than those covered by rule #3
  • Rule #5: it will only keep weekly backups for the number of months specified in keep_monthly_backups_for_months for all backups older than those covered by rule #4
  • Rule #6: it'll only keep yearly backups for the number of years specified in keep_yearly_backups_for_years for all backups older than those covered by rule #5
  • Rule #7: it will start deleting old backups until the volume of storage used is lower than the amount specified in delete_oldest_backups_when_using_more_megabytes_than.

Those keep-* and delete_oldest_backups_when_using_more_megabytes_than attributes are present on the source and on the destination of backup and in the cleanup.default_strategy key of the backup-server config file.

The monitoring process

The package can check the health of backups for every source, and the destination is healthy. A source is considered healthy if a backup was recently made, and not too much disk space is used for all backups for this source. A destination is considered healthy if it doesn't take up too much space.

This monitoring happens by executing the backup-server:monitor command that should be scheduled to run daily.

This command will send out notifications for each (un)healthy source and destination. It can notify you when a backup failed, when a source/destination is unhealthy, ... You can customize which notifications should be sent.

Because Laravel's notification system is used, you can use any of the channels you'd like: mail, slack, ...

Picking up a license

Because we invested quite some time in polishing this package, we've decided to make it a paid one. If this package is useful for you, consider picking up a license in our store at spatie.be.

Even if you're not in the market for a backup solution, purchasing a license can still be worthwhile. The code contains a couple of cool things for you to discover. In this video, which I made a couple of months ago, you'll already see a couple of neat things.

In closing

At Spatie, we manage 100+ servers for our clients. We've been using Laravel Backup Servers ourselves for the past several months to back them all up. Here's a screenshot of our destination. As you can see, the backup process's hard links save us an incredible amount of disk space.

screenshot

Be sure also to take a look at our other paid products and this big list of free open source packages.

If you like our open source efforts, considering sponsoring us through GitHub. Sponsers will get access to new stuff earlier. Every now and then we'll send coupons with a deep discount for our paid products. In fact, all people that sponsored us yesterday morning got a 100% discount on backup server! You missed that, but by sponsoring us now you'll be one of the lucky ones the next time we send a coupon.

Stay up to date with all things Laravel, PHP, and JavaScript.

Follow me on Twitter. I regularly tweet out programming tips, and what I myself have learned in ongoing projects.

Every two weeks I send out a newsletter containing lots of interesting stuff for the modern PHP developer.

Expect quick tips & tricks, interesting tutorials, opinions and packages. Because I work with Laravel every day there is an emphasis on that framework.

Rest assured that I will only use your email address to send you the newsletter and will not use it for any other purposes.

Comments

Webmentions

David Hallin liked on 28th October 2020
Luis Pesaressi liked on 25th October 2020
Roman Pronskiy liked on 24th October 2020
nero liked on 24th October 2020
Web Developer MF liked on 24th October 2020
Web Developer MF liked on 24th October 2020
Abhishek Jain liked on 24th October 2020
Mueez Sattar Khan liked on 24th October 2020
jellydn liked on 24th October 2020
Abu Hurairah liked on 24th October 2020
Ricardo Vargas / Web Developer liked on 23rd October 2020
João Machado liked on 23rd October 2020
Probir Dey liked on 23rd October 2020
geraldmuvengei liked on 23rd October 2020
Ciro Lo Sapio liked on 23rd October 2020
Ciro Lo Sapio liked on 23rd October 2020
Nam Dau liked on 23rd October 2020
Nelson Martell ネルソン liked on 23rd October 2020
ahmed ali liked on 23rd October 2020
undefined liked on 23rd October 2020
م. إسماعيل عاشور liked on 23rd October 2020
Stefan Galescu liked on 23rd October 2020
Stefan Tanevski liked on 23rd October 2020
Spatie retweeted on 23rd October 2020
Aleksandr liked on 23rd October 2020
Abhishek Jain liked on 23rd October 2020
Dannie L. liked on 23rd October 2020
Haz liked on 23rd October 2020
이현석 Hyunseok Lee liked on 23rd October 2020
Alejandro Pérez liked on 23rd October 2020
Júnior liked on 23rd October 2020
Ibnu Halim Mustofa liked on 23rd October 2020
Yogi Prasetyawan liked on 23rd October 2020
Daniel Rodríguez liked on 23rd October 2020
Fernando Sedrez liked on 23rd October 2020
Jake Casto liked on 23rd October 2020
アレハンドロ。⬢ liked on 23rd October 2020
Wyatt liked on 23rd October 2020
Alexandru Bucur liked on 23rd October 2020
Helder Willian liked on 23rd October 2020
Ilija Tatalovic liked on 22nd October 2020
EquiPC liked on 22nd October 2020
Borislav Borissov liked on 22nd October 2020
Salman Zafar liked on 22nd October 2020
Daniel Bailey liked on 22nd October 2020
Joao Pinell liked on 22nd October 2020
ダビッド トレス liked on 22nd October 2020
Ivan Radunovic liked on 22nd October 2020
FoFa liked on 22nd October 2020
Max Vogl liked on 22nd October 2020
FORMGEFLECHT liked on 22nd October 2020
Hicaro Alfonso Rios liked on 22nd October 2020
SARSMUSTEND liked on 22nd October 2020
Jorge González liked on 22nd October 2020
Wahala for who get drip liked on 22nd October 2020
Sergio Cardenas liked on 22nd October 2020
Daniel liked on 22nd October 2020
सोझो केटो liked on 22nd October 2020
Vitor liked on 22nd October 2020
Tony Messias liked on 22nd October 2020
Roberto B 🚀 liked on 22nd October 2020
Oilmone liked on 22nd October 2020
Groovix liked on 22nd October 2020
Aryeh Feld liked on 22nd October 2020
Guilherme Fontenele 🐘 liked on 22nd October 2020
Michael Aguiar liked on 22nd October 2020
Scorp974 liked on 22nd October 2020
Tuginho liked on 22nd October 2020
Salman Ahmed Rajput liked on 22nd October 2020
Filippo Fortino liked on 22nd October 2020
Manuel Pirker-Ihl liked on 22nd October 2020
Omar Andrés Barbosa Ortiz liked on 22nd October 2020
Scott Zirkel liked on 22nd October 2020
Sonny Spaan liked on 22nd October 2020
Travis Elkins liked on 22nd October 2020
Songhua Hu liked on 22nd October 2020
Tim ⛈ liked on 22nd October 2020
Farid Mammadov 🇦🇿 liked on 22nd October 2020
Edwin I Arellano liked on 22nd October 2020
Ken V. liked on 22nd October 2020
Ken V. replied on 22nd October 2020
Once again, big ups! <3
Moses liked on 22nd October 2020
Florian Voutzinos ⚡ liked on 22nd October 2020
Tauseef shah liked on 22nd October 2020
Derick Alangi © liked on 22nd October 2020
José Cage liked on 22nd October 2020
The gift that keeps on giving!! Thank you and the @spatie_be team for the excellent packages, knowledge sharing that makes #Laravel developers such haPHPy people #ILovePHP #PHP
Miguel Piedrafita 🚀 liked on 22nd October 2020
Rocco Howard liked on 22nd October 2020
Peter Fox liked on 22nd October 2020
Juan Carlos liked on 22nd October 2020
Cruorzy liked on 22nd October 2020
Paulius Jasiulis liked on 22nd October 2020
Florian Weinrich liked on 22nd October 2020
ali ali liked on 22nd October 2020
Smith Junior liked on 22nd October 2020
Guillermo Antony Cava Nuñez liked on 22nd October 2020
Cyril de Wit liked on 22nd October 2020
Nick Stanko liked on 22nd October 2020
Román Miranda liked on 22nd October 2020
Ross Bookmarks liked on 22nd October 2020
Justin ! liked on 22nd October 2020
AHMED 🏴‍☠️𓂀 liked on 22nd October 2020
Atiq Samtia liked on 22nd October 2020
Pelumi Ezekiel liked on 22nd October 2020
Kaka liked on 22nd October 2020
Raju Mandapati liked on 22nd October 2020
Anh Tran liked on 22nd October 2020
Emre liked on 22nd October 2020
Omanof liked on 22nd October 2020
Johan Alvarez liked on 22nd October 2020
Tiagosimoes liked on 22nd October 2020
/e/halil 💻🎧📌 retweeted on 22nd October 2020
/e/halil 💻🎧📌 liked on 22nd October 2020
Haneef Ansari 🍭 liked on 22nd October 2020
Rahul Chauhan liked on 22nd October 2020
Tudor 🏳️‍🌈 liked on 22nd October 2020
Parthasarathi G K liked on 22nd October 2020
Anthony liked on 22nd October 2020
Fabricio Garcia liked on 22nd October 2020
Baspa retweeted on 22nd October 2020
Davide Bellini liked on 22nd October 2020
Oluwatobi Samuel Omisakin liked on 22nd October 2020
Diego Lorenzo liked on 22nd October 2020
Jong Z liked on 22nd October 2020
Onno Lissenberg liked on 22nd October 2020
Ernest Chiang liked on 22nd October 2020
Lucas Yang liked on 22nd October 2020
Mark Walet retweeted on 22nd October 2020
Angad Dubey liked on 22nd October 2020
🇦🇷 Carolina 🇦🇷 liked on 22nd October 2020
KGG liked on 22nd October 2020
Baspa liked on 22nd October 2020
Cristóbal Ugarte liked on 22nd October 2020
Aidan Casey liked on 22nd October 2020
Swapnil Bhavsar liked on 22nd October 2020
Vaggelis Yfantis liked on 22nd October 2020
Peter Štovka retweeted on 22nd October 2020
Spatie retweeted on 22nd October 2020
बेद retweeted on 22nd October 2020
Cauan Cabral liked on 22nd October 2020
Brenier Arnaud liked on 22nd October 2020
Niels liked on 22nd October 2020
Tobias Schulz liked on 22nd October 2020
Denis Žoljom liked on 22nd October 2020
Florian Wartner liked on 22nd October 2020
Jose Alberto Lopez liked on 22nd October 2020
Astrotomic liked on 22nd October 2020
Mattia Migliorini liked on 22nd October 2020
Tobias Schulz replied on 22nd October 2020
Awesome! Thanks 😊
Do Hoang Dinh Tien retweeted on 22nd October 2020
Anne Koepcke retweeted on 22nd October 2020
Nehal Hasnayeen 🛍 retweeted on 22nd October 2020
kronos_I retweeted on 22nd October 2020
Afzal Abbas liked on 22nd October 2020
Nanderson Castro retweeted on 22nd October 2020
Nehal Hasnayeen 🛍 liked on 22nd October 2020
Ahmed Selim Refaat liked on 22nd October 2020
Jigal Sanders liked on 22nd October 2020
Tom Green liked on 22nd October 2020
Mike liked on 22nd October 2020
Cássio Almeida liked on 22nd October 2020
Nanderson Castro liked on 22nd October 2020
Mohammad Nur liked on 22nd October 2020
Do Hoang Dinh Tien liked on 22nd October 2020
Anne Koepcke liked on 22nd October 2020
Lennart Fischer liked on 22nd October 2020
Yaniv Harush liked on 22nd October 2020
Mohammad kaab liked on 22nd October 2020
Gytis Karčiauskas liked on 22nd October 2020
Ar 👨‍💻 liked on 22nd October 2020
Tauseef shah liked on 22nd October 2020
Helmer liked on 22nd October 2020
Peter Brinck 🤘 liked on 22nd October 2020
JOHN DOE liked on 22nd October 2020
Paulo Costa liked on 22nd October 2020
Bajram Emini liked on 22nd October 2020
Amit Merchant liked on 22nd October 2020
Nattapon Pondongnok 🚀 retweeted on 22nd October 2020
Nick retweeted on 22nd October 2020
Dries Vints retweeted on 22nd October 2020
Tony retweeted on 22nd October 2020
NUNO MADURO retweeted on 22nd October 2020
Bernhard Krämer retweeted on 22nd October 2020
Derick Alangi © retweeted on 22nd October 2020
Syafiq ▲ retweeted on 22nd October 2020
MadLipz Kenya😅 liked on 22nd October 2020
Divan liked on 22nd October 2020
Mike Vosters liked on 22nd October 2020
Tauseef shah liked on 22nd October 2020
NUNO MADURO liked on 22nd October 2020
Bernhard Krämer liked on 22nd October 2020
Nattapon Pondongnok 🚀 liked on 22nd October 2020
Peter Brinck 🤘 liked on 22nd October 2020
Daniel Logvin liked on 22nd October 2020
José Cage liked on 22nd October 2020
Jānis Lācis liked on 22nd October 2020
Philo Hermans liked on 22nd October 2020
Derick Alangi © liked on 22nd October 2020
Felix Schmid liked on 22nd October 2020
Dries Vints liked on 22nd October 2020
Apfel Frisch liked on 22nd October 2020
René Sinnbeck liked on 22nd October 2020
Moinuddin S. Khaja liked on 22nd October 2020
Robin Dirksen liked on 22nd October 2020
Bajram Emini liked on 22nd October 2020
Tony liked on 22nd October 2020
René Sinnbeck replied on 22nd October 2020
Works now
Freek Van der Herten replied on 22nd October 2020
Fixed! 😅
Freek Van der Herten replied on 22nd October 2020
Thank you!
Victor replied on 22nd October 2020
Thank you so much for your packages! If I see "spatie" in github, I know IT WORTH IT
James Brooks replied on 22nd October 2020
There's a 404 on your blogpost.
Sasha replied on 22nd October 2020
What are you using to document everything? Btw. nice package as always :)
Apfel Frisch replied on 22nd October 2020
Nice one, time to migrate my rsnapshot setup :)
Freek Van der Herten replied on 22nd October 2020
Described here: freek.dev/1795-selling-d…