Skip to content

feat(cache): add iterate method to list files from cache #57911

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Altahrim wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

feat(cache): add iterate method to list files from cache #57911

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions apps/files_sharing/lib/External/Cache.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,4 +48,13 @@
}
return $results;
}

#[Override]

Check failure on line 52 in apps/files_sharing/lib/External/Cache.php

View workflow job for this annotation

GitHub Actions / static-code-analysis

UndefinedAttributeClass 

apps/files_sharing/lib/External/Cache.php:52:4: UndefinedAttributeClass: Attribute class OCA\Files_Sharing\External\Override does not exist (see https://psalm.dev/241)
public function iterateFolderContentsById(int $fileId, bool $includeMetadata = false, bool $sortByName = false): iterable {
$displayId = $this->cloudId->getDisplayId();
foreach (parent::iterateFolderContentsById($fileI, $includeMetadata, $sortByName) as $entry) {

Check failure on line 55 in apps/files_sharing/lib/External/Cache.php

View workflow job for this annotation

GitHub Actions / static-code-analysis

UndefinedVariable 

apps/files_sharing/lib/External/Cache.php:55:46: UndefinedVariable: Cannot find referenced variable $fileI (see https://psalm.dev/024)
$entry['displayname_owner'] = $displayId;
yield $entry;
}
}
}
3 changes: 2 additions & 1 deletion apps/files_trashbin/lib/Helper.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,8 @@ public static function getTrashFiles($dir, $user, $sortAttribute = '', $sortDesc
$internalPath = $mount->getInternalPath($absoluteDir);

$extraData = Trashbin::getExtraData($user);
$dirContent = $storage->getCache()->getFolderContents($mount->getInternalPath($view->getAbsolutePath($dir)));
$dirId = $storage->getCache()->getId($mount->getInternalPath($view->getAbsolutePath($dir)));
$dirContent = $storage->getCache()->iterateFolderContentsById($dirId);
foreach ($dirContent as $entry) {
$entryName = $entry->getName();
$name = $entryName;
Expand Down
28 changes: 28 additions & 0 deletions lib/private/Files/Cache/Cache.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
use OCP\FilesMetadata\IFilesMetadataManager;
use OCP\IDBConnection;
use OCP\Util;
use Override;
use Psr\Log\LoggerInterface;

/**
Expand Down Expand Up @@ -238,6 +239,33 @@ public function getFolderContentsById($fileId) {
return [];
}

#[Override]
public function iterateFolderContentsById(int $fileId, bool $includeMetadata = false, bool $sortByName = false): iterable {
if ($fileId < 1) {
return [];
}

$query = $this->getQueryBuilder()
->selectFileCache()
->whereParent($fileId)
->whereStorageId($this->getNumericStorageId());
if ($includeMetadata === true) {
$metadataQuery = $query->selectMetadata();
}
if ($sortByName) {
$query->orderBy('name', 'ASC');
}

$result = $query->executeQuery();
foreach ($result->iterateAssociative() as $row) {
if ($includeMetadata === true) {
$row['metadata'] = $metadataQuery->extractMetadata($row)->asArray();
}
yield self::cacheEntryFromData($row, $this->mimetypeLoader);
}
$result->closeCursor();
}

/**
* insert or update meta data for a file or folder
*
Expand Down
5 changes: 5 additions & 0 deletions lib/private/Files/Cache/FailedCache.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,11 @@
return [];
}

#[Override]

Check failure on line 62 in lib/private/Files/Cache/FailedCache.php

View workflow job for this annotation

GitHub Actions / static-code-analysis

UndefinedAttributeClass 

lib/private/Files/Cache/FailedCache.php:62:4: UndefinedAttributeClass: Attribute class OC\Files\Cache\Override does not exist (see https://psalm.dev/241)
public function iterateFolderContentsById(int $fileId, bool $includeMetadata = false, bool $sortByName = false): iterable {
return [];
}

public function put($file, array $data) {
}

Expand Down
2 changes: 1 addition & 1 deletion lib/private/Files/Cache/Watcher.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ public function needsUpdate($path, $cachedData) {
* @param string $path
*/
public function cleanFolder($path) {
$cachedContent = $this->cache->getFolderContents($path);
$cachedContent = $this->cache->iterateFolderContentsById($this->cache->getId($path));
foreach ($cachedContent as $entry) {
if (!$this->storage->file_exists($entry['path'])) {
$this->cache->remove($entry['path']);
Expand Down
7 changes: 7 additions & 0 deletions lib/private/Files/Cache/Wrapper/CacheWrapper.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,13 @@
return array_map([$this, 'formatCacheEntry'], $results);
}

#[Override]

Check failure on line 113 in lib/private/Files/Cache/Wrapper/CacheWrapper.php

View workflow job for this annotation

GitHub Actions / static-code-analysis

UndefinedAttributeClass 

lib/private/Files/Cache/Wrapper/CacheWrapper.php:113:4: UndefinedAttributeClass: Attribute class OC\Files\Cache\Wrapper\Override does not exist (see https://psalm.dev/241)
public function iterateFolderContentsById(int $fileId, bool $includeMetadata = false, bool $sortByName = false): iterable {

Check failure on line 114 in lib/private/Files/Cache/Wrapper/CacheWrapper.php

View workflow job for this annotation

GitHub Actions / static-code-analysis

InvalidReturnType 

lib/private/Files/Cache/Wrapper/CacheWrapper.php:114:115: InvalidReturnType: The declared return type 'iterable<mixed, OCP\Files\Cache\ICacheEntry>' for OC\Files\Cache\Wrapper\CacheWrapper::iterateFolderContentsById is incorrect, got 'Generator<int, OCP\Files\Cache\ICacheEntry|false, mixed, void>' (see https://psalm.dev/011)
foreach ($this->getCache()->iterateFolderContentsById($fileId, $includeMetadata, $sortByName) as $entry) {
yield $this->formatCacheEntry($entry);
}
}

/**
* insert or update meta data for a file or folder
*
Expand Down
4 changes: 2 additions & 2 deletions lib/private/Files/ObjectStore/ObjectStoreStorage.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ public function rmdir(string $path): bool {
}

private function rmObjects(ICacheEntry $entry): bool {
$children = $this->getCache()->getFolderContentsById($entry->getId());
$children = $this->getCache()->iterateFolderContentsById($entry->getId());
foreach ($children as $child) {
if ($child->getMimeType() === ICacheEntry::DIRECTORY_MIMETYPE) {
if (!$this->rmObjects($child)) {
Expand Down Expand Up @@ -263,7 +263,7 @@ public function opendir(string $path) {

try {
$files = [];
$folderContents = $this->getCache()->getFolderContents($path);
$folderContents = $this->getCache()->iterateFolderContentsById($this->getCache()->getId($path), false, true);
foreach ($folderContents as $file) {
$files[] = $file['name'];
}
Expand Down
2 changes: 1 addition & 1 deletion lib/private/Files/View.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1507,7 +1507,7 @@ public function getDirectoryContent($directory, $mimetype_filter = '', ?\OCP\Fil
}

$folderId = $data->getId();
$contents = $cache->getFolderContentsById($folderId); //TODO: mimetype_filter
$contents = $cache->iterateFolderContentsById($folderId, false, true); // TODO Mimetype filter

$sharingDisabled = \OCP\Util::isSharingDisabledForUser();
$permissionsMask = ~\OCP\Constants::PERMISSION_SHARE;
Expand Down
5 changes: 5 additions & 0 deletions lib/private/Lockdown/Filesystem/NullCache.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,11 @@
return [];
}

#[Override]

Check failure on line 52 in lib/private/Lockdown/Filesystem/NullCache.php

View workflow job for this annotation

GitHub Actions / static-code-analysis

UndefinedAttributeClass 

lib/private/Lockdown/Filesystem/NullCache.php:52:4: UndefinedAttributeClass: Attribute class OC\Lockdown\Filesystem\Override does not exist (see https://psalm.dev/241)
public function iterateFolderContentsById(int $fileId, bool $includeMetadata = false, bool $sortByName = false): iterable {
return [];
}

public function put($file, array $data) {
throw new \OC\ForbiddenException('This request is not allowed to access the filesystem');
}
Expand Down
10 changes: 10 additions & 0 deletions lib/public/Files/Cache/ICache.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,16 @@ public function getFolderContents($folder);
*/
public function getFolderContentsById($fileId);

/**
* Get an iterator on files stored in folder
*
* Only returns files one level deep, no recursion
*
* @return iterable<ICacheEntry>
* @since 34.0.0
*/
public function iterateFolderContentsById(int $fileId, bool $includeMetadata = false, bool $sortByName = false): iterable;
Copy link
Contributor

@artonge artonge Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that getFolderContentsById() does not support those parameters, are $includeMetadata and $sortByName really useful?

Also, you touched a comment suggesting that having a mimetype parameter would be nice.

Copy link
Collaborator Author

@Altahrim Altahrim Jan 30, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These parameters are optimisations for the underlying query. If we don't use metadata, better avoid the join. If we don't need sorted elements, better leave default order.

From what I've found, the mimetype filter is not really used. That's why I didn't implemented it, but I can :)

Copy link
Collaborator Author

@Altahrim Altahrim Jan 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should think about another kind of API… Something like:

// Similar to getFolderContentsById()
$cache->iterateById($folderId)->withMetadata()->sortBy(ICache::SORT_NAME)->exec();
// Unsorted list of ID
$cache->iterateById($folderId)->asType(ICache::RESULT_ID)->exec();

Copy link
Contributor

@artonge artonge Jan 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another though, adding those kinds of arguments/filters sounds more like a search than a listing. So if those are actually useful, maybe we could name the new method something like searchFolderContentsById.


/**
* store meta data for a file or folder
* This will automatically call either insert or update depending on if the file exists
Expand Down
Loading