Skip to content

Webdav upload to S3 external storage not in chunk v2 #64885

Description

@antoinetran

⚠️ This issue respects the following points: ⚠️

  • This is not a troubleshooting question, general support matter, or webserver/proxy problem, but likely a bug (if unsure, ask the Community Help Forum).
  • This issue is not already reported on Github OR solved at the Community Help Forum (I've searched!).
  • I'm using a maintained major version of Nextcloud Server and tested against the latest patch level. (Supported major versions and current patch levels).
  • I agree to follow Nextcloud's Code of Conduct.
  • I've tried my best to provide clear reproduction steps that someone unfamiliar with this bug could use to reproduce it.

Bug description

When uploading a large file as chunked v2 mode, with a nextcloud server using S3 as external storage (not primary storage), the server does not proceed to chunk v2 and fallback to chunk v1. Thus the file is in the server disk twice the size (one as chunks in "upload" of the user data, one for aws-php-s3 in /tmp to upload to s3).

Steps to reproduce

  1. deploy nextcloud 34.0.1 with nginx and fpm
  2. use rclone fork that enabled chunk v2 (https://github.com/antoinetran/rclone/tree/fix-8321-nextcloud-chunk-v2-upload)
  3. upload using rclone image

Expected behavior

When uploading as chunk v2 mode (see https://docs.nextcloud.com/server/latest/developer_manual/client_apis/WebDAV/chunking.html), the chunks should be uploaded directly to s3 backend instead of disk as intermediate data.

Nextcloud Server version

34

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.5

Web server

Nginx

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Fresh Nextcloud Server install

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

  • Default user-backend (database)
  • LDAP/ Active Directory
  • SSO - SAML
  • Other

Configuration report

{
    "system": {
        "htaccess.RewriteBase": "\/",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "openmetrics_allowed_clients": [
            "127.0.0.1",
            "10.42.0.0\/16",
            "10.43.0.0\/16"
        ],
        "check_data_directory_permissions": false,
        "forwarded_for_headers": [
            "HTTP_X_FORWARDED_FOR"
        ],
        "skeletondirectory": "\/dev\/null",
        "filelocking.enabled": false,
        "debug": false,
        "enable_previews": false,
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "upgrade.disable-web": true,
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "my-nextcloud.globalstorage-nextcloud",
            "globalstorage-nextcloud-dcc-fi-csc-okd-pm-sys2.2.rahtiapp.fi"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "34.0.1.2",
        "overwrite.cli.url": "http:\/\/localhost",
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory"
    }

List of activated Apps

Enabled:
  - appstore: 1.0.0
  - cloud_federation_api: 1.18.0
  - dav: 1.39.0
  - federatedfilesharing: 1.24.0
  - files: 2.6.0
  - files_external: 1.26.0
  - files_sharing: 1.26.0
  - lookup_server_connector: 1.22.0
  - oauth2: 1.22.0
  - profile: 1.3.0
  - provisioning_api: 1.24.0
  - serverinfo: 6.0.0
  - settings: 1.17.0
  - theming: 2.9.0
  - twofactor_backupcodes: 1.23.0
  - user_ldap: 1.25.0
  - viewer: 7.0.0-dev.0
  - workflowengine: 2.16.

Nextcloud Signing status

Results
=======
- core
	- EXTRA_FILE
		- tmp/.cache/fontconfig/7ef2298fde41cc6eeb7af42e48b7d293-le64.cache-9
		- tmp/.cache/fontconfig/CACHEDIR.TAG
		- tmp/.cache/fontconfig/3830d5c3ddfd5cd38a049b759396e72e-le64.cache-9
		- tmp/.cache/fontconfig/4c599c202bc5c08e2d34565a40eac3b2-le64.cache-9
		- tmp/.cache/fontconfig/d589a48862398ed80a3d6066f4f56f4c-le64.cache-9
		- customconfig.log

Nextcloud Logs

No relevant log

Additional info

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions