fix: replace assert(offset==0) with early return in readdir

[Yike-Ye]

On macOS with macFUSE 5.3.x (FSKit backend) and macOS 27 Golden Gate beta, Finder/Spotlight may call readdir with a non-zero offset even when fuse_fill_dir is always called with offset=0 (old-style readdir). This triggers the assert and aborts the sshfs process.

Replace assert(offset == 0) with if (offset != 0) return 0 in:

• sftp_readdir_async() in sshfs.c
• sftp_readdir_sync() in sshfs.c
• cache_readdir() in cache.c

Since all entries are returned in the first (offset=0) call, returning 0 on subsequent calls is semantically correct and prevents the crash.

Fixes: #338
Tested: macOS 27 Golden Gate beta 2, macFUSE 5.3.2

[dtgriscom]

Why assert that offset must be 0, and then handle the case where it isn't?

[Yike-Ye]

Because previously sshfs uses buf_get_entries (sshfs.c: 985) with offset=0, and assert(offset=0) is then used in sftp_readdir_async / sftp_readdir_sync for bug detection. In this process, all directory entries are returned from the server within one step. If the offset is non-zero, something might be wrong. See the notes in fuse.h:

	/** Read directory
	 *
	 * The filesystem may choose between two modes of operation:
	 *
	 * 1) The readdir implementation ignores the offset parameter, and
	 * passes zero to the filler function's offset.  The filler
	 * function will not return '1' (unless an error happens), so the
	 * whole directory is read in a single readdir operation.
	 *
	 * 2) The readdir implementation keeps track of the offsets of the
	 * directory entries.  It uses the offset parameter and always
	 * passes non-zero offset to the filler function.  When the buffer
	 * is full (or an error happens) the filler function will return
	 * '1'.
	 *
	 * When FUSE_READDIR_PLUS is not set, only some parameters of the
	 * fill function (the fuse_fill_dir_t parameter) are actually used:
	 * The file type (which is part of stat::st_mode) is used. And if
	 * fuse_config::use_ino is set, the inode (stat::st_ino) is also
	 * used. The other fields are ignored when FUSE_READDIR_PLUS is not
	 * set.
	 */
	FUSE_DARWIN_EXTEND_OPERATION(
		readdir,
		int (*) (const char *, void *, fuse_fill_dir_t, off_t,
			 struct fuse_file_info *, enum fuse_readdir_flags),
		int (*) (const char *, void *, fuse_darwin_fill_dir_t, off_t,
			 struct fuse_file_info *, enum fuse_readdir_flags)
	)

However, the updated MacFuse uses the new FSKit backend instead of kernel extension backend. This will sometimes deliver a non-zero offset, and thus lead to abort and disconnection.

[dtgriscom]

But won't the assert cause the code to bail on a non-zero offset? Do you perhaps want to #ifdef the assert so it isn't active on macOS?

[Yike-Ye]

I have replaced the assert(offset == 0) with if (offset !=0) return 0, which will return early instead of aborting the sshfs process. Although this may affect the FSKit backend behavior I mentioned, it works well for me so far. By the way, the #ifdef flag suggestion is a good point. I will update the PR shortly.

[Yike-Ye]

Oh, I got what you meant in the beginning. There was a typo and have been fixed in the update =)

[dmik]

Please see macfuse/macfuse#1131 (comment) for some (failing) test results.

[Yike-Ye]

@dmik Thanks for testing and for the very clear repro — you were right, PR #371 only removed the assertion, it didn't fix the underlying problem.

Root cause. sshfs uses old-style readdir on top of a single-pass SFTP directory handle: it reads the handle once, streaming entries to FUSE. On macOS with macFUSE 5.3.x (the FSKit backend), the kernel re-issues readdir on the same handle — either with a non-zero offset, or an offset==0 rewind. By then the SFTP handle is exhausted, so the re-read yields nothing. PR #371 turned that into return 0, i.e. an empty listing. And because sshfs has a directory cache (cache.c, default 20s), that empty result gets cached — so the whole directory looks empty for ~20s until the cache expires. That matches exactly what you saw: contents vanish under load and reappear once the pressure (and the cache) goes away.

Fix. Instead of streaming from the single-pass handle, snapshot the entire directory into memory on the first readdir and serve every subsequent call — any offset, including rewinds — from that snapshot. This makes readdir idempotent: no matter how often or with what offset the kernel re-issues it, it always gets the complete listing, so the cache never stores a partial/empty result.

Testing. I reproduced your scenario on a real remote mount (an HPC filesystem over a ProxyJump) and drove it hard with VS Code (large project + .git, global search, file watchers), with logging added to the cache layer. With the return 0 build the directory goes empty; with the snapshot build every readdir returns the full listing, there are zero empty results, and the directory never disappears under sustained load.

Branch: https://github.com/Yike-Ye/sshfs/tree/fix/readdir-full-snapshot

@h4sh5 — happy to open a separate PR for this if you'd like to take it. Otherwise I'm fine leaving #371 as-is for now: macFUSE is actively reworking this area for macOS 27 support, so things may shift and I'd rather avoid conflicts until that settles. Just let me know which you prefer.

[dmik]

@Yike-Ye thanks for the proper fix! I tested it with my stress test and it passes. The directory contents survives and everything else seems to function well. A few notes:

1. I test everything with the kernel backend, not FSKit (I don't use it due to various limitations). But I guess that it doesn't matter since the underlying logic changes that bring offset != 0 live in the macOS kernel and affect both backends.

2. I had to update to macFUSE 5.3.3 (released just yesterday) because of common daemonization (fork) issues after touching new ObjC machinery via MFMount.framework (i.e. using libfuse before the first fork I suppose). With 5.3.2, running sshfs without -f would eventually crash in the ObjC/fork guard. I think sshfs daemonization should be changed so that it happens before triggering MFMount use because macFUSE 5.3.3 now always forces sshfs to run in foreground (fuse: daemonize requested after mount started; continuing in foreground) to avoid this crash.

3. While you mention that vanished directories could be related to caching, it was not the case here. First, I use the cache timeout of 1 second (not the default 20) and second, I tried to disable caching completely while testing (didn't help with the VS Code test case).

[Yike-Ye]

@dmik Thanks for testing and confirming it passes — great to have independent verification on the kernel backend. And you're right that the offset != 0 behaviour comes from the macOS kernel, so it affects both backends.

Thanks especially for the correction on caching — you're right, it's not the cause. The real cause is readdir itself returning empty: sshfs reads the directory over a single-pass SFTP handle, and when the kernel re-issues readdir on that already-exhausted handle (non-zero offset, or an offset==0 rewind), the re-read yields nothing. The cache sits above readdir and just re-serves its answer, so lowering the timeout or disabling it entirely doesn't help — the underlying readdir keeps returning empty. That matches your result. The snapshot fix addresses it at the source: the whole directory is captured in memory on the first readdir and every later call is served from that snapshot, so readdir is idempotent and never returns empty, regardless of cache settings.

On daemonization — agreed, and macFUSE 5.3.3 already moved that way on the libfuse side: it defers the actual mount until first session use and avoids forking in fuse_daemonize() after the mount has started, which is why sshfs is now forced to foreground. That's a libfuse-level change, independent of this fix.

Framing for anyone following: #371 and the snapshot branch are two versions of the same readdir fix, not separate issues.

• fix: replace assert(offset==0) with early return in readdir #371 (return 0) is a minimal stopgap — it stops the crash (the old assert(offset==0) aborting under the new kernel behaviour) but returns an empty listing, so directories still vanish under load.
• The snapshot branch is the proper fix — readdir becomes idempotent and always returns the full listing. It supersedes fix: replace assert(offset==0) with early return in readdir #371.

@h4sh5 two ways to land this, your call:

• Preferred: I replace this PR (or open a fresh one) with the snapshot fix.
• Or: merge fix: replace assert(offset==0) with early return in readdir #371 as the interim crash-stop, and I'll open the snapshot fix as a follow-up.

For clarity, this is entirely within sshfs's readdir callback (sshfs.c) and doesn't touch libfuse or macFUSE. macFUSE isn't at fault — 5.3.x legitimately started issuing readdir with non-zero offsets / rewinds (which FUSE has always permitted), and that simply exposed a long-standing assumption in sshfs that offset is always 0.

[bfleischer]

When enumerating a directory, the FSKit API does not provide the number of directory entries it expects us to return (or a buffer size for us to fill). This means macFUSE does not know how many directory entries to request from libfuse (and by extension the file system server). This means the macFUSE FSKit backend may request more directory entries than FSKit can handle during a particular call. This results in the next readdir call re-requesting directory entries that have already been returned in the previous response. This might be related to the issue.

[Yike-Ye]

@bfleischer Thanks, that's a really helpful explanation — it accounts for the non-zero offset we see once a request goes through the FSKit layer. As mentioned in my previous comment, I've addressed this by snapshotting the whole directory in memory and serving every readdir from it, so the listing stays correct no matter how often or at what offset it gets re-requested.

Since this stems from the FSKit API itself, I'd imagine it won't need major changes on the macFUSE side, so the snapshot commit should be able to serve as a long-term fix here. Either way, we'll see how the sshfs maintainer wants to proceed.

[bfleischer]

@Yike-Ye I have not had time to look into this more closely yet because I have been busy with macFUSE 5.3, but it looks like either macFUSE or sshfs is breaking the FUSE contract here.

The fact that sshfs is affected could mean that other file systems are affected as well. If so, I may need to add a workaround in macFUSE. However, if sshfs is the one breaking the FUSE contract and the issue is isolated to sshfs, then a workaround on the macFUSE side may not be necessary.

[dmik]

@Yike-Ye Some more details. It appears that something's still wrong with your latest fix. Under my stress test (VS Code running and open on a large .git project), the directory contents is reported by sshfs inconsistently. First, newly created files don't appear in the listing (although they do get created just fine on the remote). Second, the size of a file touched by VS Code alternates between consequent directory reloads in far2l from a real value to a smaller one. In VS Code this looks like the file gets suddenly (externally) truncated. Reloading it there restores the contents. The file is definitely not changed on the remote side. The moment I close VS Code this weird behavior disappears and newly created files appear as well.

When I get back to stock sshfs-3.7.6 (i.e. without your patches at all) directory contents expectedly disappears, but I don't see this weird file size behavior. I can freely edit files in VS Code (files themselves are readable and writable, it's only the directory listing that gets empty) and their sizes remain consistent.

PS. Forgot to mention that I somehow don't see offset == 0 assertions in stock sshfs-3.7.6 with macFUSE 5.3.3 at all. Maybe there are some related changes that affect that but still, it's strange. We are missing something.

PPS. Using sshfs 2.10 with macFUSE 5.3.3 shows nor crashes neither missing directory contents. And no weird file size behavior when editing files in VS Code. So the problem of sshfs 3.x may be bigger than just this offset issue… I recalled now that I tried sshfs 3.x some months ago (before I switched to macFUSE 5.2 and even before switching to macOS 26 at all) and I got annoyed by disappearing directories so I went back to sshfs 2.10 back then (until the recent round).

[Yike-Ye]

@dmik Thanks for the follow-up — that repro pinned it down. Both symptoms trace back to my first patch: it snapshotted the directory once per opendir and then served every readdir from that frozen snapshot. While an app keeps the directory open (VS Code's watcher), the snapshot was never refreshed — so new files never showed up, and the stale stat sizes in it fought the live sizes from getattr, which is exactly the truncation / size flip-flop you saw. Once VS Code is closed the handle is dropped, the next opendir builds a fresh snapshot, and everything reappears.

I've fixed this in 30bf21b by making the snapshot per-enumeration instead of per-opendir. On offset == 0 (a fresh listing or a rewinddir) it now reopens a fresh SFTP handle and re-reads the directory from the server, so new files and updated attributes show up; offset != 0 is still served from the snapshot taken at offset 0, so the listing never comes back empty on an already-exhausted handle.

I tested it on localhost with a process holding a single directory handle open (to mimic the VS Code case): new files appear and file sizes update within the same open session, and concurrent listings stay complete and stable. Could you run it through your stress test?

Branch: https://github.com/Yike-Ye/sshfs/tree/fix/readdir-full-snapshot

Let me know if there is any issue on this fix.

[Yike-Ye]

@dmik On macFUSE 5.3.3 I think the crash this PR guards against and the recent fuse_daemonize() change are one root cause with two symptoms.

macFUSE 5.3.3 now refuses to fork inside fuse_daemonize() once the mount has started (daemonize requested after mount started; continuing in foreground), as noted in the 5.3.3 release notes. The reason is that the subsystems the mount brings up — XPC, DiskArbitration, CoreFoundation, and libdispatch — are fork-unsafe on macOS. Following sshfs's upstream order mount → connect → daemonize, fuse_daemonize()'s fork() carries that already-initialized state into the daemon child in an undefined post-fork state. The FSKit→FUSE enumeration bridge running on top of that dispatch/XPC machinery then loses its cookie/verifier bookkeeping and either forwards a bogus non-zero offset down to sshfs's old-style readdir (the crash this PR handles) or aborts outright. The 5.3.3 notes point right at this: the mount is now deferred until after daemonization specifically to keep "XPC, DiskArbitration, CoreFoundation, and MFMount state from being created before daemonization."

In other words, the problem was never being in the background — it's forking a process that already holds the mount state. As long as the fork happens before fuse_mount() brings those subsystems up, the offset stays 0 and nothing crashes.

To keep sshfs backgrounding as before, I reordered the macOS-only path (#ifdef __APPLE__) to connect → daemonize → mount:

• ssh_connect() runs first, in the foreground, so the interactive password/passphrase prompt still reaches the terminal;
• fuse_daemonize() forks next, while the process is still clean;
• fuse_mount() runs last, so XPC/CF/FSKit are initialized fresh in the final daemon process rather than inherited across the fork.

This still daemonizes — it just forks before the mount, so those subsystems are never crossed by a fork. Same principle as 5.3.3 deferring the mount until after daemonization. The non-Apple path is left untouched to avoid any Linux regression.

Confirmed working on macOS 27 + macFUSE 5.3.3: two mounts (one key-auth, one keyboard-interactive/password) both daemonize to the background as before, keep their SSH connections alive, and enumerate directories with no offset crash — stable across 2h+ of uptime.

Change: Yike-Ye@b85d744

[bfleischer]

Regarding the assert(offset == 0) or if (offset != 0):

sshfs is breaking the FUSE contract here. When using readdir option 1, sshfs may be called with non-zero offsets, but sshfs may chose to ignore the offset and return the whole directory contents in one go by passing 0 as offset to the filler function.

	 * 1) The readdir implementation ignores the offset parameter, and
	 * passes zero to the filler function's offset.  The filler
	 * function will not return '1' (unless an error happens), so the
	 * whole directory is read in a single readdir operation.

assert(offset == 0) or simply returning 0 on offset == 0 (without any directory entries) is a bug or at the very least a wrong assumption in sshfs. This is true for macOS and Linux. On Linux the issue might never be triggered, though.

Regarding macFUSE 5.3.3:

macFUSE 5.3 uses an XPC connection as channel to exchange FUSE messages with the FSKit backend. Unlike file descriptors, XPC connections do not survive a fork().

The FSKit→FUSE enumeration bridge running on top of that dispatch/XPC machinery then loses its cookie/verifier bookkeeping and either forwards a bogus non-zero offset down to sshfs's old-style readdir (the crash this PR handles) or aborts outright.

This is not correct. The XPC connection is severed by forking the process. That's the main reason libfuse in macFUSE 5.3.3 delays mounting the volume (and by extension establishing the XPC connection). FSKit is not affected by sshfs forking and there is no bogus data being sent to sshfs.

The FUSE contract states that calling readdir with offset != 0 is valid. The two readdir modes are local to libfuse, which means the macOS and Linux kernels, as well as FSKit, do not know which mode a particular file system chooses.

libfuse sits between the kernel, or FSKit, and the file system process, and it handles both modes internally. From the kernel’s and FSKit’s perspective, readdir looks the same either way.

It is valid for libfuse to pass readdir calls with offset != 0 to sshfs, and sshfs must handle them correctly. If passing offset != 0 to sshfs wasn't legal, this would rather be a libfuse bug than a kernel or FSKit bug.

@Yike-Ye

To keep sshfs backgrounding as before, I reordered the macOS-only path (#ifdef APPLE) to connect → daemonize → mount:

This should not be necessary. fuse_main() internally calls fuse_daemonize() after fuse_mount(). fuse_daemonize() after fuse_mount() is the de facto standard. This way the mount parameters are checked and errors can be printed before daemonizing the process.

Not having to reorder fuse_daemonize() and fuse_mount() in every file system is the whole point of delayed mounting in macFUSE 5.3.3.

Where does this leave us?

This PR and more precisely "replace assert(offset==0) with early return in readdir" does not fix the real issue. That's why I'm closing it.

Snapshotting or simply re-reading the whole directory might be better options. Let's continue the discussion in #338.