scitags: no fireflies for webdav

[marian-babik]

Trying to issue a firefly on a sample webdav transfer doesn't produce anything.

This is on dCache 11.2 with the following configuration:
pool.enable.firefly=true
pool.firefly.destination=some.remote.host

The command to reproduce is below, the expected header "SciTag" is passed in the HTTP request, but no firefly is issued.

curl --location-trusted -L -v -H "SciTag: 313" --capath /etc/grid-security/certificates/ -E $X509_USER_PROXY https://head01.aglt2.org:2880/atlasdatadisk/rucio/mc15_13TeV/6c/7e/TXT.42054890._014847.tar.gz.1 --output test3.dat

• SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
• ALPN, server did not agree to a protocol
• Server certificate:
  ...
• TLSv1.2 (OUT), TLS header, Unknown (23):
  } [5 bytes data]

GET /atlasdatadisk/rucio/mc15_13TeV/6c/7e/TXT.42054890._014847.tar.gz.1?dcache-http-uuid=e0c7f931-c00d-46a2-a256-33b9f012a4f0&dcache-http-ref=https%3A%2F%2Fhead01.aglt2.org%3A2880 HTTP/1.1
Host: umfs51.aglt2.org:25541
User-Agent: curl/7.76.1
Accept: /
SciTag: 313

The relevant codebase is at:

dcache/modules/dcache-webdav/src/main/java/org/dcache/webdav/DcacheResourceFactory.java

Line 1712 in a93e7cd

_transferTag = request.getHeader("SciTag");

[ShawnMcKee]

More information from our testing. We set this in all the layout files with webdav:

[root@head01 ~]# grep -C4 redirect /etc/dcache/layouts/head01.conf
[webdav-${host.name}Domain]
dcache.java.memory.heap=4g
dcache.java.memory.direct=4g
[webdav-${host.name}Domain/webdav]
webdav.redirect.on-read=true
webdav.redirect.on-write=true
webdav.authz.anonymous-operations=NONE
webdav.authn.protocol=https
webdav.authn.accept-client-cert=true
webdav.authn.require-client-cert=false

The full curl is below.

[smckee@umt3int05 ~]$ curl --location-trusted -L -v -H "SciTag: 313" --capath /etc/grid-security/certificates/ -E $X50>
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 2001:48a8:68f7:1:192:41:230:44:2880...

• Connected to head01.aglt2.org (2001:48a8:68f7:1:192:41:230:44) port 2880 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• CAfile: /etc/pki/tls/certs/ca-bundle.crt
• CApath: /etc/grid-security/certificates/
• TLSv1.0 (OUT), TLS header, Certificate Status (22):
  } [5 bytes data]
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
  } [512 bytes data]
• TLSv1.2 (IN), TLS header, Certificate Status (22):
  { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Server hello (2):
  { [122 bytes data]
• TLSv1.2 (IN), TLS header, Finished (20):
  { [5 bytes data]
• TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
  { [1 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  { [36 bytes data]
• TLSv1.3 (IN), TLS handshake, Request CERT (13):
  { [7876 bytes data]
• TLSv1.3 (IN), TLS handshake, Certificate (11):
  { [6225 bytes data]
• TLSv1.3 (IN), TLS handshake, CERT verify (15):
  { [520 bytes data]
• TLSv1.3 (IN), TLS handshake, Finished (20):
  { [52 bytes data]
• TLSv1.2 (OUT), TLS header, Finished (20):
  } [5 bytes data]
• TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  } [1 bytes data]
• TLSv1.2 (OUT), TLS header, Unknown (23):
  } [5 bytes data]
• TLSv1.3 (OUT), TLS handshake, Certificate (11):
  } [6380 bytes data]
• TLSv1.2 (OUT), TLS header, Unknown (23):
  } [5 bytes data]
• TLSv1.3 (OUT), TLS handshake, CERT verify (15):
  } [264 bytes data]
• TLSv1.2 (OUT), TLS header, Unknown (23):
  } [5 bytes data]
• TLSv1.3 (OUT), TLS handshake, Finished (20):
  } [52 bytes data]
• SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
• ALPN, server did not agree to a protocol
• Server certificate:
• subject: DC=org; DC=incommon; C=US; ST=Michigan; O=University of Michigan; CN=head01.aglt2.org
• start date: Nov 7 00:00:00 2025 GMT
• expire date: Nov 7 23:59:59 2026 GMT
• subjectAltName: host "head01.aglt2.org" matched cert's "head01.aglt2.org"
• issuer: C=US; O=Internet2; CN=InCommon RSA IGTF Server CA 3
• SSL certificate verify ok.
• TLSv1.2 (OUT), TLS header, Unknown (23):
  } [5 bytes data]

GET /atlasdatadisk/rucio/mc15_13TeV/6c/7e/TXT.42054890._014847.tar.gz.1 HTTP/1.1
Host: head01.aglt2.org:2880
User-Agent: curl/7.76.1
Accept: /
SciTag: 313

• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  { [12896 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• Mark bundle as not supporting multiuse
  < HTTP/1.1 302 Found
  < Date: Thu, 12 Feb 2026 09:14:22 GMT
  < Server: dCache/11.2.0
  < Location: https://umfs51.aglt2.org:25541/atlasdatadisk/rucio/mc15_13TeV/6c/7e/TXT.42054890._014847.tar.gz.1?dcache-http-uuid=86e43766-7703-49f3-82eb-53125ae5b506&dcache-http-ref=https%3A%2F%2Fhead01.aglt2.org%3A2880
  < Content-Length: 0
  <
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
• Connection #0 to host head01.aglt2.org left intact
• Issue another request to this URL: 'https://umfs51.aglt2.org:25541/atlasdatadisk/rucio/mc15_13TeV/6c/7e/TXT.42054890._014847.tar.gz.1?dcache-http-uuid=86e43766-7703-49f3-82eb-53125ae5b506&dcache-http-ref=https%3A%2F%2Fhead01.aglt2.org%3A2880'
• Trying 2001:48a8:68f7:1:192:41:230:189:25541...
• Connected to umfs51.aglt2.org (2001:48a8:68f7:1:192:41:230:189) port 25541 (pnfsmanager: Fix permission check on root directory #1)
• ALPN, offering h2
• ALPN, offering http/1.1
• CAfile: /etc/pki/tls/certs/ca-bundle.crt
• CApath: /etc/grid-security/certificates/
• TLSv1.0 (OUT), TLS header, Certificate Status (22):
  } [5 bytes data]
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
  } [512 bytes data]
• TLSv1.2 (IN), TLS header, Certificate Status (22):
  { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Server hello (2):
  { [122 bytes data]
• TLSv1.2 (IN), TLS header, Finished (20):
  { [5 bytes data]
• TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
  { [1 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  { [10 bytes data]
• TLSv1.3 (IN), TLS handshake, Certificate (11):
  { [2088 bytes data]
• TLSv1.3 (IN), TLS handshake, CERT verify (15):
  { [520 bytes data]
• TLSv1.3 (IN), TLS handshake, Finished (20):
  { [52 bytes data]
• TLSv1.2 (OUT), TLS header, Finished (20):
  } [5 bytes data]
• TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  } [1 bytes data]
• TLSv1.2 (OUT), TLS header, Unknown (23):
  } [5 bytes data]
• TLSv1.3 (OUT), TLS handshake, Finished (20):
  } [52 bytes data]
• SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
• ALPN, server did not agree to a protocol
• Server certificate:
• subject: DC=org; DC=incommon; C=US; ST=Michigan; O=University of Michigan; CN=umfs51.aglt2.org
• start date: Sep 2 00:00:00 2025 GMT
• expire date: Sep 2 23:59:59 2026 GMT
• subjectAltName: host "umfs51.aglt2.org" matched cert's "umfs51.aglt2.org"
• issuer: C=US; O=Internet2; CN=InCommon RSA IGTF Server CA 3
• SSL certificate verify ok.
• TLSv1.2 (OUT), TLS header, Unknown (23):
  } [5 bytes data]

GET /atlasdatadisk/rucio/mc15_13TeV/6c/7e/TXT.42054890._014847.tar.gz.1?dcache-http-uuid=86e43766-7703-49f3-82eb-53125ae5b506&dcache-http-ref=https%3A%2F%2Fhead01.aglt2.org%3A2880 HTTP/1.1
Host: umfs51.aglt2.org:25541
User-Agent: curl/7.76.1
Accept: /
SciTag: 313

• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  { [198 bytes data]
• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  { [198 bytes data]
• old SSL session ID is stale, removing
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• Mark bundle as not supporting multiuse
  < HTTP/1.1 200 OK
  < Accept-Ranges: bytes
  < Content-Length: 889614
  < Content-Disposition: attachment;filename=TXT.42054890._014847.tar.gz.1
  < Content-Location: https://head01.aglt2.org:2880/atlasdatadisk/rucio/mc15_13TeV/6c/7e/TXT.42054890._014847.tar.gz.1
  < Server: dCache/11.2.0
  <
  { [8192 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
• TLSv1.2 (IN), TLS header, Unknown (23):
  { [5 bytes data]
  100 868k 100 868k 0 0 8195k 0 --:--:-- --:--:-- --:--:-- 8195k
• Connection pnfsmanager: Fix permission check on root directory #1 to host umfs51.aglt2.org left intact

[marian-babik]

Trying to replicate this on 11.2.0 using system-test module has failed as I get fireflies for both GET and PUT requests.

The following two tests have both produced fireflies:
curl -vv -s -f -L -H "SciTag: 313" -o /tmp/dcache.test20260213-399872-eor4uj -u root:password http://etf-cs9.cern.ch:2880/disk/test-1770973895-1

as well as
curl -vv -s -f -o /dev/null -H "SciTag: 313" -u root:password -L -T /bin/bash http://etf-cs9.cern.ch:2880/disk/test-1770973895-1

This was IPv4-only and non-authenticated.

[marian-babik]

Trying to replicate via https with:
curl -vv --insecure --capath /vol1/dcache/packages/system-test/target/dcache/etc/grid-security/certificates -H "SciTag: 313" -s -L -f -u usersroot:password -T /bin/bash https://etf-cs9.cern.ch:2881/test-1770973895-4

curl -vv --insecure --capath /vol1/dcache/packages/system-test/target/dcache/etc/grid-security/certificates -H "SciTag: 313" -s -L -f -u usersroot:password -T /bin/bash https://etf-cs9.cern.ch:2881/test-1770973895-4

produces fireflies that have both src-ip and dst-ip set to localhost, i.e.
<134>1 2026-02-13T09:42:27.810743410Z 127.0.0.1 dCache - firefly-json - {"flow-id":{"dst-ip":"127.0.0.1","src-ip":"127.0.0.1","protocol":"tcp","afi":"ipv4","src-port":0,"dst-port":20011},"flow-lifecycle":{"start-time":"2026-02-13T09:42:27.810653628Z","state":"start","current-time":"2026-02-13T09:42:27.810683551Z"},"usage":{},"context":{"application":"http","activity-id":57,"experiment-id":4},"version":1}

curl commands were run on a remote host and fireflies were sent to a remote host, so localhost in src/dst is incorrect.

[marian-babik]

The culprint is changing settings to:
webdav.redirect.on-read=true
webdav.redirect.on-write=true

This stops all fireflies for https (ip4 and ip6), http still works fine. So this is easy to replicate on 11.2.0.

[kofemann]

Looks like the fix was trivial... @marian-babik, could you please check that it works for you?

[marian-babik]

Thanks, this fix is so simple it’s actually insulting :-)
Must be a new record in: "time spent in investigation" vs "lines of code changed"

I have redone the tests and it all works fine now, so I think we can close this.