Cloud filesystem (s3, spaces) optimizations

[Plytas]

First of all, thank you for this package, really amazing work you put in so far.

That said, using cloud filesystem (s3, spaces) requires some optimizations, because right now it makes multiple requests for every LfmItem when trying to fill() the attributes. I've been playing around and managed to get the /jsonitems request with 20 folders and 1 image from over 1 min (it would timeout after 1 min) down to around 2 seconds. Here are my insights:

Avoid extra isDirectory() and isFile() requests

Right now isDirectory() does this weird check:

public function isDirectory()
{
    $working_dir = $this->path('working_dir');
    $parent_dir = substr($working_dir, 0, strrpos($working_dir, '/'));

    $parent_directories = array_map(function ($directory_path) {
        return app(static::class)->translateToLfmPath($directory_path);
    }, app(static::class)->dir($parent_dir)->directories());

    return in_array($this->path('url'), $parent_directories);
}

Since it's already making 2 different calls to filesystem to fetch folders and files separately, it can be implemented like this:

LfmPath.php

public function folders()
{
	$all_folders = array_map(function ($directory_path) {
		return $this->pretty($directory_path, true);
	}, $this->storage->directories());

	$folders = array_filter($all_folders, function ($directory) {
		return $directory->name !== $this->helper->getThumbFolderName();
	});

	return $this->sortByColumn($folders);
}

public function files()
{
	$files = array_map(function ($file_path) {
		return $this->pretty($file_path);
	}, $this->storage->files());

	return $this->sortByColumn($files);
}

public function pretty($item_path, bool $isDirectory = false)
{
	return Container::getInstance()->makeWith(LfmItem::class, [
		'lfm' => (clone $this)->setName($this->helper->getNameFromPath($item_path)),
		'helper' => $this->helper,
		'isDirectory' => $isDirectory
	]);
}

Note: I only added $isDirectory variable to pretty() function and passing true from folders() function.

LfmItem.php

class LfmItem
{
    private $lfm;
    private $helper;
    private $isDirectory;

    private $columns = ['name', 'url', 'time', 'icon', 'is_file', 'is_image', 'thumb_url'];
    public $attributes = [];

    public function __construct(LfmPath $lfm, Lfm $helper, bool $isDirectory = false)
    {
        $this->lfm = $lfm->thumb(false);
        $this->helper = $helper;
        $this->isDirectory = $isDirectory;
    }

...

    public function isDirectory()
    {
    	return $this->isDirectory;
    }

    public function isFile()
    {
        return ! $this->isDirectory();
    }

Folders do not need an isImage() check

This can be easily avoided adding an isFile() check first:
LfmItem.php

public function isImage()
{
	return $this->isFile() && starts_with($this->mimeType(), 'image');
}

Avoid extra mimeType() requests

Mime time request gets called at least 3 times when trying to fill is_mage, icon and thumb_url.
This can be optimized by including another private property $mimeType and requesting mimeType like this:

public function mimeType()
{
	if (is_null($this->mimeType))
		$this->mimeType = $this->lfm->mimeType();

	return $this->mimeType;
}

Note: I left out uploaded file comment since it was not implemented yet.

I've also removed time property (lastModified()) since I don't need it and it isn't visible in thumbnails mode.

These optimizations made this package somewhat usable with cloud disks. Further improvements are needed with files though, because they are still making a lot of requests for icons, thumbs and urls. IMO these can be optimized using lazy loading, by showing a generic file on load and then doing ajax requests and loading extra data separately. Also cache would be a nice boost in user experience, because right now when you navigate back to a folder you already were before, it has to load the same stuff again.

[damienfournier]

Hello,

thank you @Plytas

I think these optimizations should be implemented in the library asap.
How can we speed up things ?

There is a commit made by @vmgr in a fork...

It's not impacting only cloud filesystems, it's the same on a local disk.
On my dev machine, using local disk, time for the call to jsonItems went from 2.5s to 0.4s for 57 photos, just with the first optimization (the "isDirectory" variable).

On a production server, the package is unusable with 300 photos (timeout each time).

Greetings,

[Plytas]

Hello @damienfournier

sadly I'm not a maintainer of this package. These were only my insights that I could find. If someone has time to create a PR, maybe maintainers would be willing to accept it?
@albertcht @FreedomKnight @storyn26383 @streamtw @youchenlee

[damienfournier]

Hello,

thanks Plytas, I know you're not a maintainer, but you've found the solution to the problem I had.

Btw, I've given up of using this package... it seems that the maintainers don't have the time for it, and there are big bugs like this one (which make the package unusable with > 100 images) that aren't fixed for over a year.. so I've started to rewrite it for myself in a simpler way.

Thanks

[Plytas]

I'm interested in seeing your spin on this. If it gets to see the public, please let me know ;)

[damienfournier]

Well, it won't be public because it will be part of something else...

but basically, same principles but:

• works only with ckeditor, and on linux server
• store file & image info (time, size, mime) in database instead of scanning it each time
• no user folder, only shared
• virtual folders or tags to filter them (using database)

[youchenlee]

Are you interested in joining the author/maintainer of this package? @Plytas
It would be great to have this package keep improving.

[krokovicsmate]

Can someone implement this change? I tried in local and it's lot more faster on local disk too and works perfectly!

[mafftor]

I cloned this package and started to maintain it and fix a lot of old bugs https://github.com/mafftor/laravel-file-manager
You can test it if you want! I've already done some cool features and redesigned it a bit.
Soon I will implement this optimizations as well.
Press Star if you want to support it

[krokovicsmate]

@mafftor Thanks your package is awesome and is start using it, but i really need these fix because not opening the folders with more files thank ~300. So i check really closely this fix, i need really this. :)

[mafftor]

@krokovicsmate Use it and tomorrow or the day after I will publish new release with speed improvements. Now I'm waiting the amazon's approval for creating my own S3 and testing

[krokovicsmate]

@mafftor Okay i integrated, and i excited to use it :)

[streamtw]

Thanks to @Plytas for the detailed suggestion! I have fixed these issues in v2.1.0.