SSH key manager

[krokicki]

This PR adds an SSH key manager to the profile menu in the upper right. This lets you create SSH keys and authorize them so that you can seamlessly move between cluster nodes. The main purpose for this is to allow easy on boarding for our Seqera Platform instance.

I tried to be careful about making backups of files in ~/.ssh before changing them, but would welcome feedback especially if there are any edge cases I didn't think of. If you are testing this, I would recommend making a backup of your ~/.ssh directory first, just in case.

@JaneliaSciComp/fileglancer @StephanPreibisch

[mkitti]

I think we should consider treating .ssh as read-only as much as possible. Keys generated by fileglancer could be stored elsewhere.

The main file that may need to be written to here might be .ssh/authorized_keys . We should consider if .ssh/authorized_keys2 could be used or if the sshd configuration could be modified to recognize an authorized_keys file exclusively managed by fileglancer.

Here is the documentation for OpenSSH sshd:
https://man.openbsd.org/sshd_config#AuthorizedKeysFile

[krokicki]

I added a VITE_ENABLE_SSH_KEYS feature toggle, and enabled both this and VITE_ENABLE_TASKS in the CI build. I also modified the implementation to do as little as possible with ~/.ssh.

[krokicki]

@mkitti As discussed, we agree about "treating .ssh as read-only as much as possible". To minimize complexity and because for now this feature has a very small user base, I've modified the GUI to do the most minimal thing possible to ~/.ssh, namely: if id_ed25519 does not exist, this tool can create one for you and add it to authorized_keys using cat. I've removed functionality around managing multiple keys, and all functionality that deletes keys. This minimizes risks to user's existing SSH keys and setups.

[mkitti]

In a few places we still read the private key into resident memory of the Python process as an immutable Python string. This is not good. Core dumps of the Python process may contain the contents of the private key.

Rather than loading file contents into an immutable string, we should read the contents into a mutable bytearray and then explicitly overwrite the contents of the bytearray when the contents are no longer needed.

import os

key_path = "private_key.pem"
file_size = os.path.getsize(key_path)

# 1. Pre-allocate mutable memory
sensitive_data = bytearray(file_size)

# 2. Read directly into the pre-allocated buffer
with open(key_path, "rb") as f:
    f.readinto(sensitive_data)

# Use the bytes as needed

# 3. Securely overwrite the memory with zeros
for i in range(len(sensitive_data)):
    sensitive_data[i] = 0

# 4. Now safe to remove the reference
del sensitive_data

References:

1. https://stackoverflow.com/questions/728164/securely-erasing-password-in-memory-python#:~:text=The%20correct%20solution%20is%20to,bytearray%22%20instead%20of%20python%20strings.
2. https://softwareengineering.stackexchange.com/questions/270113/using-a-bytearray-rather-than-a-string-to-store-password-in-memory#:~:text=Using%20a%20bytearray%20datatype%20to,in%20python%203.4%20using%20bytearray?

[mkitti]

Consider using a bytearray and perhaps a context manager to minimize time that a private key spends in resident memory of the Python process.

Ref: https://softwareengineering.stackexchange.com/a/270159

[mkitti]

Consider making the private key contents a mutable bytearray that we can explicitly overwrite to clear from memory. Avoid converting the bytearray to an immutable string.

[mkitti]

Perhaps we could make SSHKeyContent follow the context manager protocol so we can use a with statement. On __exit__ the context manager would then make sure to shred the key contents in memory by zeroing out the resident bytearray. You could use contextlib to help with this.

I also suggest centralizing the the potentially sensitive reading implementation here.

1. The class would only store a string or Path containing the location of the key file.
2. The context manager __enter__ would read the file contents into a bytearray via readinto and then return that bytearray
3. The context manager __exit__ would then explicitly overwrite the bytearray.

This will minimize the time the any sensitive secret spends in resident memory.

[mkitti]

Consider making the passphrase a mutable bytearray that we can explicitly overwrite to clear from memory. Avoid converting the bytearray to an immutable string.

[mkitti]

Use readinto to read into a mutable bytearray.

[mkitti]

Shred the key contents in a bytearray after usage. Ensure the contents get shreded even if an error occurs. Consider using a context manager here.

[mkitti]

Could we add a restrict keyword here? Does Seqera need an interactive terminal or port forwarding capabilities?

https://manpages.debian.org/experimental/openssh-server/authorized_keys.5.en.html#restrict

[mkitti]

Perhaps we could also restrict what domain names or IP addresses this key can be used from?
https://manpages.debian.org/experimental/openssh-server/authorized_keys.5.en.html#from=_pattern-list_

[mkitti]

At the end could we add a comment as the last space delimited field that says this was added by fileglancer for seqera?

[mkitti]

If I understand correctly, the exclusive user of the private key should be Seqera alone. The private key does not need to be used by the user for any other purpose. Why do we need to store the private or public key to disk at all other than adding it to the authorized_keys file?

Rather we only need to transiently show the private key to the user via the web interface to copy into Seqera. If they fail to do so, then we just generate another private key anew the next time.

That process would work more like how Github personal access tokens work. Take a look at how the user interface works for tokens.

If you push the "Generate a new token" button and fill out the form, it will show you the token once in this example (I deleted the key before posting the screenshot):

It warns that you will not be able to see the personal access token again. Similarly, we can issue the same warning when showing the user the private key for Seqera. If they lose the key, we can just generate another private key.

In summary, there is no need to write the private key to ~/.ssh/id_ed25519 or the public key to ~/.ssh/id_ed25519.pub. Rather we can generate the private key and the public key at some temporary file location or just just store it transiently in memory. All we need to do is add the public key to the ~/.ssh/authorized_keys file, preferably with restrictions and a comment, and show the private key to the user once via the web interface. The private key never needs to be seen again.

[krokicki]

I implemented the bytearray clearing. The rest are also good ideas but will be deferred to a future release.

[mkitti]

Decoding this to a string does defeat the point of using a bytearray. If we could avoid that somehow, that would be better. The problem is that we do not know when the string will be garbage collected and we cannot control when the string will be overwritten.

If elimanating the use of string is not possible, minimizing the number of references to the content would be helpful. In this case, just return directly rather than adding a named reference in result.

Suggested change

	result = sensitive_buffer.decode('utf-8')
	return result
	return sensitive_buffer.decode('utf-8')

[mkitti]

Suggested change

	key_content = _read_file_secure(private_key_path)
	return SSHKeyContent(key=key_content)
	return SSHKeyContent(
	key=_read_file_secure(private_key_path)
	)

Remove extra references to the string.

[mkitti]

Decoding the bytearray to a string largely defeats the point of using the bytearray. The problem is that we cannot control when the string will be cleared from memory. At minimum, we could try to reduce references to the string to increase the chances that it can be immediately garbage collected.

Gemini suggests that it might be possible to keep using a bytearray and avoid the string by using a custom Response class.

from fastapi import FastAPI, Response
import gc

app = FastAPI()

class SecureResponse(Response):
    def __init__(self, content: bytearray, *args, **kwargs):
        super().__init__(content=content, *args, **kwargs)
        # Store a reference to the mutable bytearray to clear it later
        self._sensitive_content = content

    async def __call__(self, scope, receive, send):
        # 1. Standard response logic sends the data to the client
        await super().__call__(scope, receive, send)
        
        # 2. Immediately zero out the sensitive data after sending
        for i in range(len(self._sensitive_content)):
            self._sensitive_content[i] = 0
            
        # 3. Explicitly clear the reference and suggest garbage collection
        del self._sensitive_content
        gc.collect()

@app.get("/secure-data")
async def get_secure_data():
    # sensitive_data must be a mutable bytearray
    sensitive_data = bytearray(b"SENSITIVE_SECRET_2026")
    
    return SecureResponse(
        content=sensitive_data, 
        media_type="application/octet-stream"
    )

I would consider a try-finally around the call to super().__call__.

try:
    await super().__call__(scope, receive, send)
finally:
    # clean-up     

https://share.google/aimode/d54nOXW54RdNTA5aT

[mkitti]

Using the following procedure I was able to identify multiple copies of id_ed25519 resident in the memory of the Python process via gef on Linux.

sudo setcap cap_sys_ptrace=eip /usr/bin/gdb
bash -c "$(curl -fsSL https://gef.blah.cat/sh)"
gdb -p <PYTHON_PID>
gef> search-pattern <CONTENTS_OF_id_ed25519>

I then got the following results, showing two copies.

[+] Searching '<CONTENTS_OF_id_ed25519>' in memory
[+] In (0x78102c000000-0x78102cbd3000), permission=rw-
  0x78102cb04bd5 - 0x78102cb04c0c  →   "<CONTENTS_OF_id_ed25519>[...]" 
  0x78102cb7ad94 - 0x78102cb7adcb  →   "<CONTENTS_OF_id_ed25519>[...]" 

This confirms to me that Python is hanging on to the contents of the private key in memory.

While I think there may be ways to eliminate the private key from Python's memory, I am questioning if ssh key generation even needs to be done on the server-side at all. There appears to be at least one project, js-keygen, that is meant for creating ssh keys.

The LLMs suggest that the Web Crypto API is now sufficient for the purpose of key generation.

// Native Ed25519 generation in modern browsers (2025/2026)
async function generateNativeEd25519() {
  const keyPair = await window.crypto.subtle.generateKey(
    { name: "Ed25519" }, 
    true, // Must be extractable to export for Seqera
    ["sign", "verify"]
  );

  // To turn this into an SSH key, we must export the raw bytes
  const pubRaw = await window.crypto.subtle.exportKey("raw", keyPair.publicKey);
  const privRaw = await window.crypto.subtle.exportKey("pkcs8", keyPair.privateKey);

  return { pubRaw, privRaw };
}

The bulk of the remaining code is reformatting the output of this function into OpenSSH format.

Thus my current suggestion is that we generate the key pairs on the client side in the browser. In this case the only thing to keep here is code to add the public key to the authorized_keys file. Only the public key needs to be transmitted to the server for that purpose.

[krokicki]

Thus my current suggestion is that we generate the key pairs on the client side in the browser. In this case the only thing to keep here is code to add the public key to the authorized_keys file. Only the public key needs to be transmitted to the server for that purpose.

That's for sure an interesting idea, though we need to think through the implications of adding this kind of complexity to the client. Additional dependencies will bloat the downloads required to have the page load. Use of bleeding edge APIs could create browser compatibility issues.

More broadly, it feels worth thinking about all this in context of our threat model. What are we actually defending against here? Fileglancer runs on the internal network, and host access is restricted to a small number of admins. Given that context, how much value does each of these additional security features provide, when you weigh each one against its cost?