On our site you'll find the documentation of our bigger bigger packages, like laravel-medialibrary, laravel-backup, laravel-event-sourcing, ... Let's take a look at how that works.

The basis of this documentation is a collection of markdown files that is included in the various packages. Here's the docs folder of laravel-medialibrary where those markdowns reside.

The markdown files of all those GitHub repositories are imported using the ImportDocsFromRepositoriesCommand, which is executed every 5 minutes.

Let's take a look at the handle method of that command.

public function handle()
{
    $this->info('Importing docs...');

    $loop = Factory::create();

    $updatedRepositoriesValueStore = UpdatedRepositoriesValueStore::make();

    $updatedRepositoryNames = $updatedRepositoriesValueStore->getNames();

    $this
       ->convertRepositoriesToProcesses($updatedRepositoryNames, $loop)
      ->pipe(fn (Collection $processes) => $this->wrapInPromise($processes));

    $loop->run();

    $updatedRepositoriesValueStore->flush();

    $this->info('All done!');
}

There are several interesting things at play here. The ImportDocsFromRepositoriesCommand command does not import every repository with docs that we have, but only repositories that were recently updated. The names of those updated repositories are in the $updatedRepositoryNames array.

The contents of $updatedRepositoriesValueStore is what we call a value store. A value store is nothing more than a class that wraps a JSON file on disk with some nice write/read methods. If you want to use a value store in your project, take a look at the spatie/valuestore package.

On each of the repositories on GitHub with docs, we configured a webhook to https://spatie.be/webhooks/github. The webhook is executed whenever a file in the repo is changed. In the HandleGitHubWebhookController the incoming webhook is handled. Here's the __invoke` method of that controller. This method will verify that request is a valid request from GitHub. It will determine the repo name where the request originates from, and it will save that repo name in the value store.

public function __invoke(Request $request)
{
    $this->ensureValidRequest($request);

    $payload = json_decode($request->getContent(), true);

    $updatedRepositoryName = $payload['repository']['full_name'] ?? null;

    if ($updatedRepositoryName === null) {
        return;
    }

    UpdatedRepositoriesValueStore::make()->store($updatedRepositoryName);
}

Now that you know how we determine for which repos we should import docs, let's go back to the ImportDocsFromRepositoriesCommand and focus on these lines:

$this
   ->convertRepositoriesToProcesses($updatedRepositoryNames, $loop)
   ->pipe(fn (Collection $processes) => $this->wrapInPromise($processes));

$loop->run();

For each of the updated repositories, we are going to create a process that will copy the docs markdown files of the repo to the storage directory of our Laravel app.

The cool thing to note here is that all those processes will be registered at a loop. When the loop is started, this happens here, all process will execute simultaneously. This will significantly improve the time needed to fetch all docs.

All the processes are wrapped in a promise by the all function. This allows us to execute code when all processes are finished in the then method.

protected function wrapInPromise(Collection $processes): void
{
    all($processes->toArray())
        ->then(function () {
            $this->info('Fetched docs from all repositories.');

            $this->info('Caching Sheets.');

            $pages = app(Sheets::class)->collection('docs')->all()->sortBy('weight');

            cache()->store('docs')->forever('docs', $pages);

            $this->info('Done caching Sheets.');
        })
        ->always(function () {
            File::deleteDirectory(storage_path('docs-temp/'));
        });
}

When the callable in then is executed, all repositories will have been fetched to the storage_path('docs') directory. Our homegrown spatie/sheets package can parse a directory with markdown files to HTML. It will also wrap each fetched document in an Eloquent-like class so you can easily get the title, content, and things defined in the Markdown front matter. To know more about this powerhouse package, read its readme on GitHub.

The sheets package is configured in such a way that all files on the docs disk (which has its root set to storage_path('docs')) are being handled. All docs are sorted by weight (which is defined in the front matter). Sorting the collection of docs is an expensive operation, so we cache the result.

The DocsController is responsible for displaying the docs on our site. It will use those docs we cached previously.

This series is continued in part 8: Mailing updates and news using Mailcoach.