Dev/sftp start: Implementing basic SFTP functionality#29
Dev/sftp start: Implementing basic SFTP functionality#29jubeormk1 wants to merge 404 commits intomkj:dev/sftp-startfrom
Conversation
|
After that I have:
As a minor change I have clarified that the module will follow SFTP v3 specifications in 12 |
|
We have a complete, not complaining sftp/src/proto.rs. However I restricted the lifetime for Apart from that, I have avoided the use or There is a number of TODOs in the file, mainly regard to:
|
Fixed according to the indications in the next discussion: mkj#29 (comment) Adding the expanded version for reference too Thanks
|
Hi again, I am starting to process the sftp subsystem, I have deserialized I have made mayor changes in The use of the macro has changed as well and maybe for good. |
|
I am working on providing basic functionality to handle the sftp initialisation and adjusting the sftpserver trait to expose as little as possible protocol details to the user. I also have to handle properly errors inside sftphandle. The code at this point is able to start an sftp session after running |
|
At this point, I need to handle the case where after extracting from the I am exploring storing the fragment of the request in an auxiliar buffer to make sense of it after the next buffer read. Did you have to deal with this situation at a different level @mkj? For now, I am going to use an auxiliary buffer to assemble the request. More coming soon... |
Yeah I think a buffer is probably necessary (if SSHWire had partial decoding it'd still need to store intermediate parts in memory somewhere). As an optimisation it might be possible to special_case |
|
Thanks for the ideas. For now I am writing a helper where I can take enough data from buffer in to get the size and packet type packet type. In the case of SSH_FXP_WRITE I copying enough data to create a multi part write of the content reusing the current technique for it. SSH_FXP_NAME is an interesting case. And FXP_DATA will also require some thinking, since the sink length may limit the sftpserver implementer and would create a bottleneck reading. |
|
Thanks to @mkj original design of structures in proto.rs, there is only one entity that for now cannot be used in The reason is simple and @mkj mentioned before. The response for Since reading a directory is part of the basic features that I have proposed in the library roadmap, I think that I should address it. I would like to share the approach that I am starting to put together but I am open to more ideas. I am considering processing the data element by element in the directory to resolve the arbitrary number of elements with new traits and structures for collections of SSH_FXP_READPATH responses. I would like to capture the next workflow:
I have a lifetime challenge here because the I would like to keep those internal details for the the Does anybody have a simpler or better idea? Maybe there is a better pattern to use here that I am overlooking. |
|
One way to resolve lifetime problems could be to have the |
|
Thanks @mkj! We've been inspecting this piece of code (and your advice), for readdir and we have a few questions: We assume that the developer would eventually implement We are a bit confused by the What is We think that As in: we noticed the |
|
I'm currently travelling, should be able to take a look later next week.
|
|
Hi again @mkj, I think that I finally understand what you where suggesting that could work. A visitor pattern to avoid borrowing/lifetime issues while allowing iteratively process the directory elements. @brainstorm I am implementing a
For now I have a mock structure that simulates sending data. Soon I will push it. |
… generate encoded values? To fix the issue with the message type encoding I have done a manual implementation of SSHEncode for `StatusCode`. It is tested in a proto_test mod. From last commit: sshwire-derive lib.rs defines the rules to encode enums in `encode_enum` and the encoding decision is not to encode the value. See [implementation here](https://github.com/mkj/sunset/blob/8e5d20916cf7b29111b90e4d3b7bb7827c9be8e5/sshwire-derive/src/lib.rs#L287) This could be addressed by introducing a parameter for enum fields where the encoded data type is declared (or not). There is probably a good reason for this behavior.
Not ready yet. In order to answer EOF to a SSH_SXP_READDIR after all the dir items have been already sent. I am storing a `sftpserver::ReadStatus` passed as a result on `SftpServer.dirread()` with PendingData or EndOfFile. This would not work well in scenarios where there is more than one read transaction in course. I have decided that I am going to make the SftpServer implementators the responsibility to send the EOF status message but I will help them by providing a detailed trait documentation. To test this listing you may run the demo/sftp/std and run the script testing/test_read_dir.sh mind that the hard link counter is not recovered as it is not part of SFTP V3 implementation
jubeormk1
changed the title
|
In the commit Fixed Status Encoding..and the previous one, I came across with the fact that sshwire-derive does not encode enum values. Therefore, SSHEncoding structures with enum fields would not encode enum fields. To solve this situation I implemented the SSHEncode for I identified the relevant function and point in sshwire-derive/src/lib.rs where automatic encoding of numeric enum fields could be done, but the enum data type would need to be provided somehow, maybe as a struct Attribute, to automate the numerical value encoding. @mkj Do you recognise that implementing an automatic enum value encoding would be a good use of time? |
Will be used for helpers only available to Std compilations
This way we keep from std users some of the protocol details
Ah I see, that's a bit unfortunate, sorry! sshwire-derive should either implement it or fail if sunset/sshwire-derive/src/lib.rs Lines 285 to 288 in 351e2ef In SSH base protocol there are only a couple of places that use numeric enums 1. Are there many places you'd use it in SFTP? I think if sshwire-derive were to implement it we'd want to check that all the variants are the same simple value form, and maybe require an attribute like Footnotes |
sftpserver trait to remove the `std` dependency It splits the responsibility of encoding/decoding a SSH_FXP_NAME into Name and NameEntry<'a> avoiding the Vector or any fixed memory allocation for the list NameEntry. That required refactoring SftpServer.realpath() and SftpHandler::handle_general_request() to adopt this approach of sending a header and after that one or more NameEntry items Also documenting and reducing the number of warnings
sunset-sftp and sunset-demo-sftp-std crates version have been increased and now match. Documentation has been updated, An Producer Consumer subsystem has been introduced to allow sending responses exceeding the output buffer length without introducing lifetimes and borrowing issues. proto.rs has been refactored to eliminate all std dependencies. helper structures and modules have been added to assist the SftpServer` implementer. a new crate feature `std` has been introduced to allow accessing std helper structures that reduce the exposure of sftp protocol details to the `SftpServer` implementer. The std example has been refactored. While still quite complex it delegates some pretty specific tasks to the std helpers.
|
In response to mkj today
No worries! It gave me the excuse to look at sshwire-derive again and learn something. I am only encoding an enum to return Sorry about not reading you earlier and keeping pushing commits. I am finally finishing the Thanks to @mkj @brainstorm and @Autofix for listening, helping and discussing details of the implementation. |
It was introduced for read_packet_from_file.rs and it is no longer needed
At this point I can see that on stall, the backpressure from the ChanOut blocks the data put in the pipe. I took measures in a previous commit (5d1c1f9) to guarantee that for SSH_FXP_DATA and SSH_FXP_NAME the announced length and real length matches. - SftpServer Unit tests to check the length in header and data sent - Runtime checks for ReadDir, Read and PathInfo requests I am going to start digging in the ChanOut Write process
This gives us visibility on when the ChanOut won't take all the bytes. It hapens regularly and that is expected. Does not provide extra visibility on the problem. I will leave it like this for now
|
One question @mkj: Looking at async channels, the first thing that I have noticed is that I might not be using ChanOut as intended. According to ChanOut docs
And later adds
Does this means that only one instance should be used for writing and only one (other) instance should be used for reading or only one instance should be used for read and writing? I believe it is the first interpretation, given the 'each' word, but please correct me if I am wrong and will refactor accordingly. |
|
I have notice something important. The On every call of The only point in the code where the windows length is increased is in the dispatch_inner match packet case with But Checking why they stopped being sent by the client I realised that they where sent. Here goes an extract of the client vs server logs: |
Yes, one reading instance and one writing instance. If you want they can be the same (ChanInOut).
|
|
I'm not certain, but it looks like it might be a problem with OpenSSH's SFTP read request pipelining. The server is processing read requests sequentially, so it has to send all response channel data for a read request before proceeding to the next packet. If it runs out of channel window while there are outstanding read requests, it will get stuck. The Window Adjust is queued after the outstanding read requests.
By default the sftp client has 64 requests pending, each of size 32768. The default channel window is 64*32768 (maybe deliberate?). That doesn't have any leeway for SFTP headers. The lock always seems to occur when the gap between read requests and sent data reaches the 2MB gap (target window size). An example running log_get_single_long.sh with https://github.com/mkj/sunset/tree/dev/matt-sftp-debug (changed debugging a bit), the last request received by sunset-sftp is offset 4751360: In the client log, that's around the point the amount of outstanding data crosses the window size: 4751360 - 2654208 = 2097152 (64*32 kB) |
|
That is a very interesting point and all your analysis adds up. Thanks for your insights! Now that you have pointed out to the client logs, I have noticed something that I find peculiar. Doing a grep to select Here we have eight requests of 32kB each follow by a window adjust of 98816 bytes. This adjust Counting the 32kB request in my log files and adding up all the window adjustment sent by the client I can summarise this deficit: which accounts a total of 2,089,984 less bytes for adjust messages than request sent. Is it not strange? |
The reason is that the adjust occurs after the read response is received by the client, and that may be some time later than the read request. So when the deficit reaches 2097152 (64*32kB) it can't send any more data, the deadlock occurs. The exact window adjust sizes are kind of arbitrary (since data packets aren't necessarily a nice multiple), the client will send those whenever the window drops below 2097152 I think. I added some logging here to openssh client, the window adjustments matched what was being done in sunset. |
|
You are right, the receiver window adjustments are not preemptive but need to follow the reception of bytes from the sending peer. So really, the window is use to prevent deadlocks from a peer sending more bytes than the receiver can process. In this case the client is sending more bytes than the server can process. Without a full understanding on the SSH Connection protocol, my intuition tells me that maybe we should slow down the number of request that the client sends by delaying the server window reception adjustment. Looking at I believe that since sending a 32kB data block in a SSH_FXP_DATA response is orders of magnitud slower than receiving short sftp requests ideally the SFTP subsystem should slow the request rate. How to do this is tricky since the SSH Connection Protocol layer is not concern about the subsystem nature. @mkj: Would you consider an optional Maybe a transparent way to do this is using the I have tried with different thresholds in check_window_adjust and could not find a "sweet spot" to prevent the deadlock. |
Those are currently ignored, so fail if they are present. If needed they could be implemented in future. Discussion in #29 (comment)
|
Implementing the ssh-stamp OTA functionality I identified a possible bug where the SftpServer sending a failure over a write operation is not handled properly by the client. I will follow this up. |
…c move block) I have done this in the view that all that operations might need to run async code. Demo changed accordingly and test passing
Provides an overview of what you find in this demo and instructions on how to use it. It also includes some warnings about security risks. The SftpServer trait operations offer some info logs to illustrate the flow of the operations.
By default sunset-sftp will only allow paths up to 256 character long but this can be adjusted with the mention flags. The motivation for this is keeping a small RequestHandler in the SftpHandler
Fixing a number of issues to get a baseline to make an example SFTP. More changes will be required. I will use the sftp/std example to explore the implementation of the SFTP protocol and to test the code. error fix on casting SftpNum to u8 Fixing error message `casting `&SftpNum` as `u8` is invalid: needs casting through a raw pointer firstrust-analyzerE0606` fixing error in sftpmessages macro_rules SSHDecode implementation I believe that $SSH_MESSAGE_NAME should be $SSH_FXP_NAME for this macro to work and that it was intended for this match ty expansion. fixing use of sftp proto Adding some extra uses in proto.rs making proto StatusCode pub Fixing sftpmessages SftpNum impl issue providing the correct data type to contains (a reference to a u8) removing reduntant use in proto.rs Added variant to proto StatusCode cleaning redundant comments the commit 7028e03 covers this Removing some unused lifetimes that caused issues They can be added when needed Chasing SSHEncode,SSHDecode issues Adding the derive for Filename and Attrs Adding empty vanilla DirReply struct as ReadReply Adding Vanilla ChanOut to finish the placeholders for Dir and Read Reply Extra fix for SftpNum to avoid moving issues WIP: Replacing sunset::Result with proto::Result Fixing issues with error handling is far from finished sunset::Result cannot take the new Errors definded in proto.rs therefore I have added error conversion for sftp errors to Sunset global error. commenting out sftpmessages macro rules to focus on the foundational SFTP data types commenting out Result and ane extra SunsetError implementation too Removing phantomData for the moment to avoid SSHEncode/Decode issues Implementing SSHEncode/Decode for Attrs To do so, I introduced the method flags(), and the enum AttrsFlags to serialise and deserialise the flags Added a response variant "ResponseAttributes" to complete all server responses SFTP packets Feel free to rename it Fixed typo and change other variant string to ssh_fx_other isolating proto.rs issues. Commenting out sftpserver Modifying packets.rs ParseContext for allowing proto.rs StatusCode to use #[sshwire(unknown)] Without changing some visiblitities, the code generated by SSHDecode (SSHDecodeEnum) for proto.rs would try accessing s.ctx().seen_unknown and Unknown::new() throwing errors. Am I doing using SSHDecode wrong? proto.rs StatusCode SSHEncode expanded version seems incorrect. I made a StatusCode version with lifetime (to be used by Other) and StatusCode SSHEncode autogenerated enc() function (L584) looks wrong since it is not performing any ::sunset::sshwire::SSHEncode::enc() Am I using SSHEncode wrong for this enum? As per the pull request fails because of the edition, I am downgrading it 2024->2021 Clarifying that this implementation will use SFTP version 3 As it is the most common version in use today, ensuring broad compatibility. Support for other versions may be considered in the future. Uncommenting macro_rules sftpmessages partially until problems arose with `Name` Implementing SSHEncode and SSHDecode for Name Name has changed. It no longer contains count, but it Encodes it as the Names length adding macro_rules! sftpmessages closing curly brackets StatusCode: SSHEncode and SSHDecode Adding num_enum to sftp to handle the u32 to enum expansion moving sftp edition to 2024 WIP: Starting to implement SftpPacket modifying SftpNum - modified macro pattern messsage_num from literal to tt (token tree) - lifetimes for SftpPacket are back - Added dependency to crate paste for pattern modification (to lower case but can be use for many other things) - I had to duplicate InitVersion to avoid repetitions of implementations in the macro expansion Adding SSHEncode and SSHDecode for SftpPacket Updating cargo-expand-sftp.rs too Added past dependency in Cargo.lock and as use Restricting SSHDecode for SftpPacket lifetimes so they match It is convoluted, changing unifying lifetimes for this implementation to `'a` would work as well cargo check passing and all the code uncommented. However... I have changed the signature for methods `decode_request` and `decode_response` to avoid the dyn SSHSource which was causing a problen during calls to dec(s)? Since it does not have a known size at compile time. To do so I have included the `'de` a lifetime for the method parameter s and made the structure SftpPacket lifetime `'a` and `'de` be equal, as I did in the previous commit fab7996 Will this be an issue? Last, the checks to find out if the decoded packet is a response or a request in `decode_response` and `decode_request` now return WireError::PacketWrong instead of error::SSHProto.fail() or Error::bug(). This might need to change Fixing Avoidable warnings: - visibility for `FileHandle` - removing unused uses - removing unused local result and error removing outdated TODOs in sftp/src/proto.rs Fix from comment @mkj comment on Name Fixed according to the indications in the next discussion: mkj#29 (comment) Adding the expanded version for reference too Thanks reordering dependencies in sftp to match other workspace members
- modified macro pattern message_num from literal to tt (token tree) - lifetimes for SftpPacket are back - Added dependency to crate paste for pattern modification (to lower case but can be use for many other things) - I had to duplicate InitVersion to avoid repetitions of implementations in the macro expansion Adding SSHEncode and SSHDecode for SftpPacket Updating cargo-expand-sftp.rs too Added past dependency in Cargo.lock and as use Restricting SSHDecode for SftpPacket lifetimes so they match It is convoluted, changing unifying lifetimes for this implementation to `'a` would work as well cargo check passing and all the code uncommented. However... I have changed the signature for methods `decode_request` and `decode_response` to avoid the dyn SSHSource which was causing a problen during calls to dec(s)? Since it does not have a known size at compile time. To do so I have included the `'de` a lifetime for the method parameter s and made the structure SftpPacket lifetime `'a` and `'de` be equal, as I did in the previous commit fab7996 Will this be an issue? Last, the checks to find out if the decoded packet is a response or a request in `decode_response` and `decode_request` now return WireError::PacketWrong instead of error::SSHProto.fail() or Error::bug(). This might need to change Fixing Avoidable warnings: - visibility for `FileHandle` - removing unused uses - removing unused local result and error removing outdated TODOs in sftp/src/proto.rs Fix from comment @mkj comment on Name Fixed according to the indications in the next discussion: mkj#29 (comment) Adding the expanded version for reference too Thanks reordering dependencies in sftp to match other workspace members Starting a new sftp-std demo taking the basic std demo as model It has some very basic code proving that when an sftp connection is accepted the version handshake starts SSH_FXP_INIT -> <- SSH_FXP_VERSION etc adding mod sftpserver as I am going to start working on it next Adding PathInfo SFTP Packet as it is requested by sftp clients after initialisation adding some Doc comments exposing sftp entities in lib.rs fixing impl From<SftpNum> for u8 captures the number in Other SftpNum Added log to sftp Big changes: Fixed decoding SftpPacket This changes the sftpmessge parameters to handle the three classes of SFTP Packets: Initialization, Requests and Responses. The motivation is to add a mechanism to capture the Request Ids. Since only requests and responses contain that field but they do not include in their "inner" sftp Packets definitions (See Read, Write, etc) I thought that it was a good idea keeping them out of them. To store them away from the Inner packets I decided adding extra parameters to the SftpPacket enum. To do so the parameters have been reestructured so each class of sftp packet can be expanded with the required data. As a side effect, the sftpmessages packets definition readability has been improved. Certainly this adds complexity to the sftpmessages adding from str for Filename changing ranges for is_init, is_request, is_response fixing decode_request Fixing the exposed items in SFTP library. More changes will come added sunset-sftp to sftp demo adding log dep to sunset-sftp WIP: Adding demosftpserver.rs This will be the local implementation to serve the requests from the client. For now just structure and a mock realpath adding the module demosftpserver to the demo. Reordering uses Reordering uses to clarify internal and external dependencies tyding up dependencies adding trait associated function realpath and struct ItemHandle Still deciding on how to use ItemHandle WIP Added sftphandle.rs This is the "traffic coordinator" for SFTP. - Handles incoming packets deserialisation - Handle the initialisation - Calls SftpServer trait to handle client requests - Handles incoming packets serialisation The current implementation tries limiting its dependencies to the minimum. Therefore it does not handle the SSH Channels itself but only buffers. This also allow the user to make decisions on the buffers. the sftp demo has been updated to use this, delegating the sftp details to the sftphandle and demosftpserver Adding as_str for Filename Just gets the inner BinString as_str Handling the conditions where the SFTP protocol fails, setting things up so it can be handle and will wait for a new channel on the pipe from the program loop removing std uses Reformed sftpserver - flags are no longer required - removed ItemHandle - Added Into and TryFrom File Handle for SftpServer trait - Added lifetime 'a to Handle and the trait associated function taking or returning the type minor changes in proto: added clone, copy and compare implements to FileHandle and TextString Bug fix in SftpPacket.encode_request enconde+request was encoding sftp number and the request id, which is incoded in self.enc(c) call. Fixed and removed req_id from parameter list Working on sftphandle - handling unsupported packet - fixing calls to SftpPacket.encode_request - adding 'a lifetime to SftpHandler to handle a Vec of FileHandle<'a> sftp.rs - removed ItemHandle; + Adding StatusCode + FileHandle minor fix in decode_request return value changes in demosftpserver very unstable at this point handling and logging unsupported or uninitialised messages fix bug encoding_request->encoding_response Adding trait implementation - requires instance: WIP - catches default behaviour: Unsupported Simplifying SftpServer trait async and the Handle are out for now I would like to keep those but since I want a structured of SftpServer to keep it's own internal state, I could not use async or a Handle with a trait definition. It is unfortunately unstable. Still a WIP but now a client can PUT a small file and its content will be displayed in debug. For now the buffers overflow on long SSH_FXP_WRITE packets and fail after trying to decode in the next loop iteration more of the same packet content. The previous sftpserver trait has been stored in asyncsftpserver.rs Simple random data files generation for big write testing Adding SFTP Packet Header and SSH_FXP_WRITE offset definitions Simplified Main Demo SFTP loop and reduced the SFTP buffers size WIP: Working on long WRITE packets processing WIP: Processing long Write Requests For now the flow of receiving SFTP Write request longer than the buffer is going in the right direction. I have been able to process up to 64kB of data. Still this is not complete since I need to: - Write the data received and check that the data sent and received has not been altered - Issue: Writes with files over 64kB fail - Reshape the file handle exchanged from the demoserver so it has a fixed length and it is compact - Refactor sftphandle Adding basic obscuring of handler This forces the SftpServer to use obscured handles that are size constrain. It also introduces the handle manager, that is a helper that the SFTP server implementer can use for dealing with the obscured handle and a private handle, that can contain all the information that the local server requires. Minimal SFTP Server implementation of write 64kB write pass the test File sent and written are identical. That is a good start. Next I will find what is wrong passed 64kB
Those are currently ignored, so fail if they are present. If needed they could be implemented in future. Discussion in mkj#29 (comment)
Fixing a number of issues to get a baseline to make an example SFTP. More changes will be required. I will use the sftp/std example to explore the implementation of the SFTP protocol and to test the code. error fix on casting SftpNum to u8 Fixing error message `casting `&SftpNum` as `u8` is invalid: needs casting through a raw pointer firstrust-analyzerE0606` fixing error in sftpmessages macro_rules SSHDecode implementation I believe that $SSH_MESSAGE_NAME should be $SSH_FXP_NAME for this macro to work and that it was intended for this match ty expansion. fixing use of sftp proto Adding some extra uses in proto.rs making proto StatusCode pub Fixing sftpmessages SftpNum impl issue providing the correct data type to contains (a reference to a u8) removing reduntant use in proto.rs Added variant to proto StatusCode cleaning redundant comments the commit 7028e03 covers this Removing some unused lifetimes that caused issues They can be added when needed Chasing SSHEncode,SSHDecode issues Adding the derive for Filename and Attrs Adding empty vanilla DirReply struct as ReadReply Adding Vanilla ChanOut to finish the placeholders for Dir and Read Reply Extra fix for SftpNum to avoid moving issues WIP: Replacing sunset::Result with proto::Result Fixing issues with error handling is far from finished sunset::Result cannot take the new Errors definded in proto.rs therefore I have added error conversion for sftp errors to Sunset global error. commenting out sftpmessages macro rules to focus on the foundational SFTP data types commenting out Result and ane extra SunsetError implementation too Removing phantomData for the moment to avoid SSHEncode/Decode issues Implementing SSHEncode/Decode for Attrs To do so, I introduced the method flags(), and the enum AttrsFlags to serialise and deserialise the flags Added a response variant "ResponseAttributes" to complete all server responses SFTP packets Feel free to rename it Fixed typo and change other variant string to ssh_fx_other isolating proto.rs issues. Commenting out sftpserver Modifying packets.rs ParseContext for allowing proto.rs StatusCode to use #[sshwire(unknown)] Without changing some visiblitities, the code generated by SSHDecode (SSHDecodeEnum) for proto.rs would try accessing s.ctx().seen_unknown and Unknown::new() throwing errors. Am I doing using SSHDecode wrong? proto.rs StatusCode SSHEncode expanded version seems incorrect. I made a StatusCode version with lifetime (to be used by Other) and StatusCode SSHEncode autogenerated enc() function (L584) looks wrong since it is not performing any ::sunset::sshwire::SSHEncode::enc() Am I using SSHEncode wrong for this enum? As per the pull request fails because of the edition, I am downgrading it 2024->2021 Clarifying that this implementation will use SFTP version 3 As it is the most common version in use today, ensuring broad compatibility. Support for other versions may be considered in the future. Uncommenting macro_rules sftpmessages partially until problems arose with `Name` Implementing SSHEncode and SSHDecode for Name Name has changed. It no longer contains count, but it Encodes it as the Names length adding macro_rules! sftpmessages closing curly brackets StatusCode: SSHEncode and SSHDecode Adding num_enum to sftp to handle the u32 to enum expansion moving sftp edition to 2024 WIP: Starting to implement SftpPacket modifying SftpNum - modified macro pattern messsage_num from literal to tt (token tree) - lifetimes for SftpPacket are back - Added dependency to crate paste for pattern modification (to lower case but can be use for many other things) - I had to duplicate InitVersion to avoid repetitions of implementations in the macro expansion Adding SSHEncode and SSHDecode for SftpPacket Updating cargo-expand-sftp.rs too Added past dependency in Cargo.lock and as use Restricting SSHDecode for SftpPacket lifetimes so they match It is convoluted, changing unifying lifetimes for this implementation to `'a` would work as well cargo check passing and all the code uncommented. However... I have changed the signature for methods `decode_request` and `decode_response` to avoid the dyn SSHSource which was causing a problen during calls to dec(s)? Since it does not have a known size at compile time. To do so I have included the `'de` a lifetime for the method parameter s and made the structure SftpPacket lifetime `'a` and `'de` be equal, as I did in the previous commit fab7996 Will this be an issue? Last, the checks to find out if the decoded packet is a response or a request in `decode_response` and `decode_request` now return WireError::PacketWrong instead of error::SSHProto.fail() or Error::bug(). This might need to change Fixing Avoidable warnings: - visibility for `FileHandle` - removing unused uses - removing unused local result and error removing outdated TODOs in sftp/src/proto.rs Fix from comment @mkj comment on Name Fixed according to the indications in the next discussion: mkj#29 (comment) Adding the expanded version for reference too Thanks reordering dependencies in sftp to match other workspace members
This was an automatic resolution error fix on casting SftpNum to u8 Fixing error message `casting `&SftpNum` as `u8` is invalid: needs casting through a raw pointer firstrust-analyzerE0606` fixing error in sftpmessages macro_rules SSHDecode implementation I believe that $SSH_MESSAGE_NAME should be $SSH_FXP_NAME for this macro to work and that it was intended for this match ty expansion. fixing use of sftp proto Adding some extra uses in proto.rs making proto StatusCode pub Fixing sftpmessages SftpNum impl issue providing the correct data type to contains (a reference to a u8) removing reduntant use in proto.rs Added variant to proto StatusCode cleaning redundant comments the commit 7028e03 covers this Removing some unused lifetimes that caused issues They can be added when needed Chasing SSHEncode,SSHDecode issues Adding the derive for Filename and Attrs Adding empty vanilla DirReply struct as ReadReply Adding Vanilla ChanOut to finish the placeholders for Dir and Read Reply Extra fix for SftpNum to avoid moving issues WIP: Replacing sunset::Result with proto::Result Fixing issues with error handling is far from finished sunset::Result cannot take the new Errors definded in proto.rs therefore I have added error conversion for sftp errors to Sunset global error. commenting out sftpmessages macro rules to focus on the foundational SFTP data types commenting out Result and ane extra SunsetError implementation too Removing phantomData for the moment to avoid SSHEncode/Decode issues Implementing SSHEncode/Decode for Attrs To do so, I introduced the method flags(), and the enum AttrsFlags to serialise and deserialise the flags Added a response variant "ResponseAttributes" to complete all server responses SFTP packets Feel free to rename it Fixed typo and change other variant string to ssh_fx_other isolating proto.rs issues. Commenting out sftpserver proto.rs StatusCode SSHEncode expanded version seems incorrect. I made a StatusCode version with lifetime (to be used by Other) and StatusCode SSHEncode autogenerated enc() function (L584) looks wrong since it is not performing any ::sunset::sshwire::SSHEncode::enc() Am I using SSHEncode wrong for this enum? As per the pull request fails because of the edition, I am downgrading it 2024->2021 Clarifying that this implementation will use SFTP version 3 As it is the most common version in use today, ensuring broad compatibility. Support for other versions may be considered in the future. Uncommenting macro_rules sftpmessages partially until problems arose with `Name` Implementing SSHEncode and SSHDecode for Name Name has changed. It no longer contains count, but it Encodes it as the Names length adding macro_rules! sftpmessages closing curly brackets StatusCode: SSHEncode and SSHDecode Adding num_enum to sftp to handle the u32 to enum expansion Bumping CI minimum version to 1.85 and sftp to 2024 edition More on this version [here](https://blog.rust-lang.org/2025/02/20/Rust-1.85.0/) moving sftp edition to 2024 WIP: Starting to implement SftpPacket modifying SftpNum - modified macro pattern message_num from literal to tt (token tree) - lifetimes for SftpPacket are back - Added dependency to crate paste for pattern modification (to lower case but can be use for many other things) - I had to duplicate InitVersion to avoid repetitions of implementations in the macro expansion Adding SSHEncode and SSHDecode for SftpPacket Updating cargo-expand-sftp.rs too Added past dependency in Cargo.lock and as use Restricting SSHDecode for SftpPacket lifetimes so they match It is convoluted, changing unifying lifetimes for this implementation to `'a` would work as well cargo check passing and all the code uncommented. However... I have changed the signature for methods `decode_request` and `decode_response` to avoid the dyn SSHSource which was causing a problen during calls to dec(s)? Since it does not have a known size at compile time. To do so I have included the `'de` a lifetime for the method parameter s and made the structure SftpPacket lifetime `'a` and `'de` be equal, as I did in the previous commit fab7996 Will this be an issue? Last, the checks to find out if the decoded packet is a response or a request in `decode_response` and `decode_request` now return WireError::PacketWrong instead of error::SSHProto.fail() or Error::bug(). This might need to change Fixing Avoidable warnings: - visibility for `FileHandle` - removing unused uses - removing unused local result and error removing outdated TODOs in sftp/src/proto.rs Fix from comment @mkj comment on Name Fixed according to the indications in the next discussion: mkj#29 (comment) Adding the expanded version for reference too Thanks reordering dependencies in sftp to match other workspace members Starting a new sftp-std demo taking the basic std demo as model It has some very basic code proving that when an sftp connection is accepted the version handshake starts SSH_FXP_INIT -> <- SSH_FXP_VERSION etc adding mod sftpserver as I am going to start working on it next Adding PathInfo SFTP Packet as it is requested by sftp clients after initialisation adding some Doc comments exposing sftp entities in lib.rs fixing impl From<SftpNum> for u8 captures the number in Other SftpNum Added log to sftp Big changes: Fixed decoding SftpPacket This changes the sftpmessge parameters to handle the three classes of SFTP Packets: Initialization, Requests and Responses. The motivation is to add a mechanism to capture the Request Ids. Since only requests and responses contain that field but they do not include in their "inner" sftp Packets definitions (See Read, Write, etc) I thought that it was a good idea keeping them out of them. To store them away from the Inner packets I decided adding extra parameters to the SftpPacket enum. To do so the parameters have been reestructured so each class of sftp packet can be expanded with the required data. As a side effect, the sftpmessages packets definition readability has been improved. Certainly this adds complexity to the sftpmessages adding from str for Filename changing ranges for is_init, is_request, is_response fixing decode_request Fixing the exposed items in SFTP library. More changes will come added sunset-sftp to sftp demo adding log dep to sunset-sftp WIP: Adding demosftpserver.rs This will be the local implementation to serve the requests from the client. For now just structure and a mock realpath adding the module demosftpserver to the demo. Reordering uses Reordering uses to clarify internal and external dependencies tyding up dependencies adding trait associated function realpath and struct ItemHandle Still deciding on how to use ItemHandle WIP Added sftphandle.rs This is the "traffic coordinator" for SFTP. - Handles incoming packets deserialisation - Handle the initialisation - Calls SftpServer trait to handle client requests - Handles incoming packets serialisation The current implementation tries limiting its dependencies to the minimum. Therefore it does not handle the SSH Channels itself but only buffers. This also allow the user to make decisions on the buffers. the sftp demo has been updated to use this, delegating the sftp details to the sftphandle and demosftpserver Adding as_str for Filename Just gets the inner BinString as_str Handling the conditions where the SFTP protocol fails, setting things up so it can be handle and will wait for a new channel on the pipe from the program loop removing std uses Reformed sftpserver - flags are no longer required - removed ItemHandle - Added Into and TryFrom File Handle for SftpServer trait - Added lifetime 'a to Handle and the trait associated function taking or returning the type minor changes in proto: added clone, copy and compare implements to FileHandle and TextString Bug fix in SftpPacket.encode_request enconde+request was encoding sftp number and the request id, which is incoded in self.enc(c) call. Fixed and removed req_id from parameter list Working on sftphandle - handling unsupported packet - fixing calls to SftpPacket.encode_request - adding 'a lifetime to SftpHandler to handle a Vec of FileHandle<'a> sftp.rs - removed ItemHandle; + Adding StatusCode + FileHandle minor fix in decode_request return value changes in demosftpserver very unstable at this point handling and logging unsupported or uninitialised messages fix bug encoding_request->encoding_response Adding trait implementation - requires instance: WIP - catches default behaviour: Unsupported Simplifying SftpServer trait async and the Handle are out for now I would like to keep those but since I want a structured of SftpServer to keep it's own internal state, I could not use async or a Handle with a trait definition. It is unfortunately unstable. Still a WIP but now a client can PUT a small file and its content will be displayed in debug. For now the buffers overflow on long SSH_FXP_WRITE packets and fail after trying to decode in the next loop iteration more of the same packet content. The previous sftpserver trait has been stored in asyncsftpserver.rs Simple random data files generation for big write testing Adding SFTP Packet Header and SSH_FXP_WRITE offset definitions Simplified Main Demo SFTP loop and reduced the SFTP buffers size WIP: Working on long WRITE packets processing WIP: Processing long Write Requests For now the flow of receiving SFTP Write request longer than the buffer is going in the right direction. I have been able to process up to 64kB of data. Still this is not complete since I need to: - Write the data received and check that the data sent and received has not been altered - Issue: Writes with files over 64kB fail - Reshape the file handle exchanged from the demoserver so it has a fixed length and it is compact - Refactor sftphandle Adding basic obscuring of handler This forces the SftpServer to use obscured handles that are size constrain. It also introduces the handle manager, that is a helper that the SFTP server implementer can use for dealing with the obscured handle and a private handle, that can contain all the information that the local server requires. Minimal SFTP Server implementation of write 64kB write pass the test File sent and written are identical. That is a good start. Next I will find what is wrong passed 64kB Troubleshooting the 65kB limit: There are two write requests I was assuming that the client would wait for a `SSH_FXP_STATUS` Ok, but it does not. I need to handle a second request segment in the same SftpSource WIP: Refactoring opaque_file_handle -> obscure_file_handle and adding traits Will not compile ATM First, I am still working on the details to make the use of traits to define the DemoSftpServer flexible. In this commit there is too much rigidity in the DemoSftpServer which does not allow my intended vision. Second, there is an unrelated refactoring here where SFTPSink and SftpSource have been extracted from sftphandle.rs to their own files for clarity WIP: Fixed DemoSftpServer definition to use a custom implementation This allows the library user to define both a private and opaque file handle WIP: Minimal implementation for DemoSftpServer Still a work in progress, but extract the definition of: - OpaqueFileHandle trait struct: to allow library users to define the OpaqueFileHandle at will - OpaqueFileHandle trait struct: A proposed structure trait to allow the users defining its own OpaqueFileHandle and InternalHandle Manager This increase the flexibility of the library allowing users taking decisions of their application rather than forcing them to use predefined elements Next step: Handle consecutive requests that might be received in the same buffer opaque_file_handle not obscure_file_handle Letting the `sftp_loop` and `prog_loop` finish DemoServer.run() This way the common server closes the socket and SFTP clients do not get hanged on failure or bye Tiding up Added change to Cargo.lock is part of commit a19b0c6 Removed cargo-example-sftp.rs, used for macro_rules troubleshooting not required at the moment Bact to SftpHandle: Process failing as soon as possible on short packets Added extra docs to SFTP proto decode_request Broken: Ugly experimentation with multi part consecutive packets Somehow I am sending an ok different from the expected by the client decoding request len in SftpPacket::decode_request Handing consecutive request in a buffer with some problems The issue that is left to fix is what happen when the second request is trunked and cannot be decoded. I am contemplating creating a "truncated" request buffer that would be used when the request cannot be decoded because requires extra data to be received in the next process call. Apart from this, the current code is able to process fragmented and consecutive SSH_FXP_WRITE for files with occasional problems caused by the issue detailed. WIP: Modifying an error condition to RanOut and adding peak_packet_len WIP: Restructuring SftpHandle. Adding a FSM Not finished, but this structure prevents borrowing issues WIP: handling ok request in FragmentedRequestState::ProcessingClippedRequest This does not cover what happens with NoRoom requests such as Write request WIP: Removing dead code WIP: replacing WireErrors and minor renaming WIP: Fixing bugs RequestHolder Transition logic WIP: Successfully received 100MB and 1024MB It is time to tidy up SFTP Roadmap added to docs The crate doc has been rendered and improved. I propose a roadmap for the development of teh SFTP crate WIP: Added SftpError::FileServerError(StatusCode) + others - Reasonable warning removal - Adding comments - Removing dead code - More refactoring WIP: sftphandle->sftphandler fixing sftperror statuscode casting WIP: Removing dead code and adding comments TODO: Use heapless::Vec instead of Vec for Name as it breaks the no_std premisses WIP: fixed bad tag ids WIP: More docs WIP: Refactoring SFTP lib and added comments from flat structure to a more conceptual structure Also added a brief header doc to show what is the intent and the status of the module. Partial functionality and refactor completed In this commit we have a working server that can receive files of arbitrary size. This can be tested running demo/sftp/std/ and running `demo/sftp/std/testing/test_long_write_requests.sh` Significative documentation has been added. SftpHandler relies on a FSM to process fragmented and long packet. [ci skip] Adding no_std, removing unnecessary use of String [ci skip] WIP: Added OpenDir operation SftpServer trait and DemoSftpServer modified to allow the use of Directories [skip ci] WIP: Added OpenDir and ReadDir requests I am at the point where I can start implementing the way I am going to put all the data in the channel. Not trivial [skip ci] WIP: moving main loop inside process_loop to have stdio in scope [skip ci] WIP: commenting out the no_std restriction for now [skip ci] WIP: I missed out adding items to Cargo.lock [skip ci] WIP: Exploring the iterator option [skip ci] WIP: Experimenting with SftpServer DirReply structure For now I am only checking borrowings [skip ci] WIP: SftpSink playload_slice() [skip ci] WIP: Adding DirReply Visitor to `ReadDir` and `DirEntriesResponseHelpers` to sftpserver.rs These will help me standardize other implementations. Also, I have proven that the Visitor pattern can be used to process each directory entry without borrowing or lifetime issues [skip ci] WIP: Refactored constant [skip ci] WIP: Reworking a wrapper for Sink and ChanOut I want to use them grouped to later pass them to DirReply Also I might be able to use better the buffer out [skip ci] WIP: fixing some comments and warnings [skip ci] WIP: Ugly explorative refactoring [skip ci] WIP: Ugly explorative refactoring now working It is really ugly [skip ci] WIP: Refactoring. Creating a mod for sftphandler [skip ci] WIP: Refactoring. Normalizing push function Unifying around push_status [skip ci] WIP: Refactoring. Integrating push functions in OutputWrapper - push_status - push_packet [skip ci] WIP: Refactoring. Moving SftpOutputChannelWrapper (aka OutputWrapper) to its own module cleaning up sftphandle.rs [skip ci] WIP: Refactoring - SftpHandler: estate machine restructuring - SftpServer: WIP for DirReply - SftpOutputChannelWrapper: WIP refactoring [skip ci] WIP: Adding Producer and Consumer to avoid mutable references writing to ChanOut [skip ci] WIP: Removing DirReplay wrong lifetime parameters [skip ci] WIP: Improved SftpSource readability [skip ci] WIP: Removing excessive lifetimes to build successfully [skip ci] WIP: Improved readability in requestholder.rs [skip ci] WIP: Fixing the Pipe mutex to main crate SunsetRawMutex, solve send_packet bug and added debugging [skip ci] WIP: Fixing Bug where an error was thrown instead of trying to encode a packet [skip ci] WIP: Refactor sftphandler.rs and main.rs to accommodate the new Producer-Consumer for ChanOut It fixes the mutable borrow problems but I am getting the client disconnected with writing request: ``` Received message too long 1572864 Ensure the remote shell produces no output for non-interactive sessions. ``` [skip ci] WIP: Looking for the error. Made some small readability changes [skip ci] WIP: looking for traces explaining where the length [0,0,0,18] becames [18,0,0,0] Increasing verbosity level globally Still I cannot find where it is happening if it happens in Sunset SSH [skip ci] WIP: Tyding up: Removing deprecated parts of code ,commenting out others and adding comments [skip ci] WIP: write request fails but will be parked for now I am going to focus on listing folder files [skip ci] WIP: Some issues in DemoSftpServer DirEntriesCollection but ready to make the SftpServer trait functions async The length calculation seems erroneous but I am more worried about the async trait [skip ci] WIP: Progress with readdir. Needs tyding up Issue with pipe Looks like the number of sent items in the pipe does not match received items. I am going to add a send/receive total bytes counter. It might also be the cause of the write errors [skip ci] WIP: Cargo lock update long forgotten. Apologies [skip ci] WIP: FIX read operation Looping to write all bytes. The counters confirm that there is no missing bytes as they match. Tested with a 100MB writing operation: ./test_long_write_requests.sh I am confident that this the write issue is solved [skip ci] WIP: Hack readdir response was not adding header field length packet type + request id + items (1 + 4 + 4 bytes) hardcoded but will refactor [skip ci] WIP: BUG in sftphandler process keeps responding to the same read dir request [skip ci] WIP: BUG **Sending loop includes sftp client** - the ReqId increases - The loop stop in a client interruption [ctl]+C I am going to need more visibility on why the client keeps on sending new request for the same files [skip ci] WIP: BUG I was not following V3 specification as did not send EOF https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-02#section-6.7 Next issue is that I am sending the relative path and not just the filename. Will fix next [skip ci] WIP: BUG The EOF needs to be sent once the readdir has sent all the items and is asked again. Anything else fails This is point I have not considered. I could have sent one item at a time, with different challenges. In my opinion I should add this behavior to the SftpServer implementation, but that passes the responsibility to the library user and the idea is making the user not needing to care too much about the protocol implementation [skip ci] WIP: Typo and cleanup commented out code [skip ci] WIP: Added a simplistic last read status approach Found a bug sending back the status code. It is always Ok! [skip ci] WIP: Temporary ways to track completed Read operations I will add a test to understand the unexpected behaviour in response status packet [skip ci] WIP: Missing StatusCode field in Status derived from SSHEncode implementation for enum sshwire-derive lib.rs defines the rules to encode enums in `encode_enum` and the encoding decision is not to encode the value. See [implementation here](https://github.com/mkj/sunset/blob/8e5d20916cf7b29111b90e4d3b7bb7827c9be8e5/sshwire-derive/src/lib.rs#L287) This could be addressed by introducing a parameter for enum fields where the encoded data type is declared (or not). There is probably a good reason for this behavior. [skip ci] Fixed Status Encoding. Shall I extend the sshwire-derive to generate encoded values? To fix the issue with the message type encoding I have done a manual implementation of SSHEncode for `StatusCode`. It is tested in a proto_test mod. From last commit: sshwire-derive lib.rs defines the rules to encode enums in `encode_enum` and the encoding decision is not to encode the value. See [implementation here](https://github.com/mkj/sunset/blob/8e5d20916cf7b29111b90e4d3b7bb7827c9be8e5/sshwire-derive/src/lib.rs#L287) This could be addressed by introducing a parameter for enum fields where the encoded data type is declared (or not). There is probably a good reason for this behavior. [skip ci] WIP: First ls command successful Not ready yet. In order to answer EOF to a SSH_SXP_READDIR after all the dir items have been already sent. I am storing a `sftpserver::ReadStatus` passed as a result on `SftpServer.dirread()` with PendingData or EndOfFile. This would not work well in scenarios where there is more than one read transaction in course. I have decided that I am going to make the SftpServer implementators the responsibility to send the EOF status message but I will help them by providing a detailed trait documentation. To test this listing you may run the demo/sftp/std and run the script testing/test_read_dir.sh mind that the hard link counter is not recovered as it is not part of SFTP V3 implementation [skip ci] Implementing EOF responsibility in SftpServer and documenting [skip ci] Added SunsetSftp Feature: Std Will be used for helpers only available to Std compilations [skip ci] Extracted DirEntriesCollection and added to std helpers This way we keep from std users some of the protocol details [skip ci] Refactor proto.rs Name structure and sftpserver trait to remove the `std` dependency It splits the responsibility of encoding/decoding a SSH_FXP_NAME into Name and NameEntry<'a> avoiding the Vector or any fixed memory allocation for the list NameEntry. That required refactoring SftpServer.realpath() and SftpHandler::handle_general_request() to adopt this approach of sending a header and after that one or more NameEntry items Also documenting and reducing the number of warnings Basic Features Directory read is now complete sunset-sftp and sunset-demo-sftp-std crates version have been increased and now match. Documentation has been updated, An Producer Consumer subsystem has been introduced to allow sending responses exceeding the output buffer length without introducing lifetimes and borrowing issues. proto.rs has been refactored to eliminate all std dependencies. helper structures and modules have been added to assist the SftpServer` implementer. a new crate feature `std` has been introduced to allow accessing std helper structures that reduce the exposure of sftp protocol details to the `SftpServer` implementer. The std example has been refactored. While still quite complex it delegates some pretty specific tasks to the std helpers. [skip ci] Extending SftpSource and Tests Now we can peak the total packet length and also if the contained packet fits within the buffer. Tests for all of that [skip ci] SftpOutputPipe checks if already splits removed TODO, not a good idea [skip ci] intercepting incomplete UnknownPacket in decode request to help flushing it It is important that we flush the rest of an incomplete unknown packet. So until it is completed, we will return a RanOut so we can read it full and then flush it Handling properly unknown packets This way: - We send the right ReqId for Unsupported status - Flush out the complete packet to avoid desynchronizing packet decoding One more time I missed the Cargo.lock changes refactored std helper to expose get_file_attrs completed getting file stats Tested with `test_get.sh` Ticked off from the roadmap [skip ci] Fixed misrepresentation of SSH_FXP_OPEN packet also fixing an error with no_std for std helper function Removed doc orphan document comments Fixing SftpTrait and implementations to use the proper Open mode (PFlags) Tested with `test_get.sh` Next step: Implement SSH_FXP_READ WIP: Added SFTP Get capability. Chasing unexpected locks After sending a full SSH_FXA_DATA response, sometimes the process stalls and there is no further request from the client. Looks like this happens when the server answer one response behind the last SSH_FXP_READ request. See client verbose output ``` sftp> get ./65kB_random debug3: Looking up ./demo/sftp/std/testing/out/./65kB_random debug3: Sent message fd 3 T:7 I:21 debug3: Received stat reply T:105 I:21 F:0x000f M:100644 Fetching ./demo/sftp/std/testing/out/./65kB_random to 65kB_random debug2: do_download: download remote "./demo/sftp/std/testing/out/./65kB_random" to local "65kB_random" debug2: Sending SSH2_FXP_STAT "./demo/sftp/std/testing/out/./65kB_random" debug3: Sent message fd 3 T:17 I:22 debug3: Received stat reply T:105 I:22 F:0x000f M:100644 debug2: Sending SSH2_FXP_OPEN "./demo/sftp/std/testing/out/./65kB_random" debug3: Sent remote message SSH2_FXP_OPEN I:23 P:./demo/sftp/std/testing/out/./65kB_random M:0x0001 65kB_random 0% 0 0.0KB/s --:-- ETAdebug3: Request range 0 -> 32767 (0/1) debug2: channel 0: window 1998336 sent adjust 98816 debug3: Received reply T:103 I:24 R:1 debug3: Received data 0 -> 32767 debug3: Request range 32768 -> 65535 (0/2) debug3: Request range 65536 -> 98303 (1/2) debug3: Received reply T:103 I:25 R:2 debug3: Received data 32768 -> 65535 debug3: Finish at 98304 ( 1) ``` SFTP Read stuck: Failing to receive a a read request? Anyone can take a look at this? Following the analysis in the previous commit, it happens that from time to time there is no more data to be read from the SSH Chan_In while the client is waiting for a response. I have documented this with some logs extracts See files failing-get-client.log, failing-get-server.log (sucess-get{client, server}.log provided for reference). In those failing examples, the server looks stuck waiting to receive new bytes from the channel (SFTP: About to read bytes from SSH Channel) During another run, while in the deadlock, I have manually killed the sftp client (`pkill sftp`) and the server logs receives an rx complete and listen again for new connections (get-deadlock-client-killed.log) [skip ci] small changes to make visible SftpOutputPipe total bytes communicated There is no data lost in these communications [skip ci] Removing outdated check in SftpHandler.process The fragmented packet handling will take care of that situation [skip ci] Removed nested enum in Sftp process FSM [skip ci]some tiding up adding large file 2MB to test_get.sh [skip ci] Bug? Large write buffers cause for Chan_out cause ssh.server.progress fail: NoRoom Running `sunset_demo_sftp_std` and calling test_get.sh triggers this error from time to time [skip ci] Debugging runner.read_channel and .input Still haven't found where the missing requests are. If I am sending bad data, I am still expecting to receive the data already sent according to the SFTP client New logs can be found running sftp_std demo and ./test_get.sh [skip ci] RequestHolder function visibility reduced and capacity() added I added capacity for uses when I want to check if a given slice fits in the holder [skip ci] Removing outdated logs [skip ci] Adding test_get_single.sh and simplifying test_get.sh The upload process has been removed to reduce verbosity and test time. Also keeps the test focus on dowloading items. test_get_single.sh only performs a get transaction of a long file (4MB) Tests will keep failing check files for further inspection [skip ci] Refactor packet handling: Packets will have fixed length part and extra data This normalize the way of treating packets while decoding and completing fragments. Processing long write requests is simplified with a new state `HandlerState::ProcessWriteRequest`, that keeps the progress and simply consume data for the the extra data that is no longer part of the proto definition of Write. requestholder.rs has new methods that simplify its usage and makes it generic. sftpsource.rs has been simplified most warnings have been corrected [skip ci] Adding more unit tests. Noted that it would be nice improving encoding of SSH_FXP_DATA and SSH_FXP_NAME packages [skip ci] Clearing RequestHolder on reset() and updating field len on each SftpSink push call clearing the reset holder makes it easier to debug it finalizing every time data is serialized in the sink avoid usability by not relying on users finalizing the sink [skip ci] caught a condition where an unknown packet would not be flush [skip ci] Checking the errors getting files Results of a simple test, repeated 10 times. Blocked receiving data, with our without error. Next I will add more visibility to the base sunset library [skip ci] Fixed test to accommodate that sinks don't need to be finalized [skip ci] Adding checks around sftpserver read and readdir Since in production the library will not be in control of the way the SftpServer trait implementer will do with the {Dir/Data}Reply parameters checking that the data announced to be sent and the data really sent would prevent running into invalid packet. Until further changes data length sent != data length announced will trigger a runtime bug. The trait documentation has been updated accordingly [skip ci] some changes to the testing sh. -o LogLevel=[Error->Debug] [skip ci] extra verbosity strace runs in sftp get tests I have rocorded packets (encrypted) and straces for exec and test script finding: - invalid packets reading from the channel - No data reading from channel after sending EOF - demo-sftp-std server stucked reading from file [skip ci] Starting logging stuck communication The next files have been updated to facilitate the collection of data. The data collected will include: - Output of trace level of the sunset-demo-sftp-std - strace for sunset-demo-sftp-std - Output for log_get_single.sh script [skip ci] Minor change regarding buffer names
3 participants
Hi there @mkj,
This is an early draft of a pull request to help with the SFTP subsystem for sunset. I have been reading and working on
proto.rsand I would like you to see my commits and discuss problems and further steps. If you prefer other channels to discuss this I am happy to do it in nay other way.To begin with, there were a number of build errors and to make incremental steps I opted to comment out and out
mod sftpserverfromlib.rsand themacro_rules! sftpmessagesin recent commits (1,2,3). This way I could focus and use cargo-expand to analyse the SSHEncode/SSHDecode macro expansions.The 20th of August I was trying to fix small things without getting into how the sftp module would work In the commits from the 20th of August I fixed some minor problems and is likely that I will have to revert some changes later on, like 4, 5.
Before working on the big entity in
proto.rs,sftpmessages!, I need to fix some issues withStatusCode. I would like to ask you if I am using SSHEncode/SSHDecode macros correctly forStatusCodesince I felt forced to add to it a lifetime which might not be necessary. Also I believe that SSHEncodeEnum is not working as expected (6, 7). I addedsftp/out/cargo-expand-sftp.rsas reference to the expanded SSHEncode.I also would like to mention that I had to modify some files outside of the sftp module as you will see in Files changed tab to implement or alter visibility of elements (
src/packets.rs,src/sshwire.rs)