Query timeout when going through HTTP Proxy since 1.4.4

[eledhwen]

Hi 👋

We're having issues with our duckdb-java setup after bumping the patch version from 1.4.2 to 1.4.4 (so technically the issue could be post-1.4.3).

Our setup is as follows:

• Java app in a docker container deployed on K8S
• cluster is air-gapped so we package the plugins (namely httpfs) in the docker image and deactivate remote plugin download
• queries are done against an S3-like server, through an HTTP proxy

Behaviour before 1.4.4 was fine, basic queries without filtering took a few seconds to run.

After 1.4.4 we observe the following:

• it works seemingly as well as before when querying the S3 server directly (same data, outside K8S)
• it times out when querying the S3 server through the HTTP proxy

The error is:

Caused by: java.sql.SQLException: IO Error: Connection timed out error for HTTP GET to '/my-bucket/?encoding-type=url&list-type=2&prefix=some-prefix

LINE 1: SELECT COUNT(1) FROM read_parquet("s3://my-bucket/some-prefix...

Has something changed on that front?

Perhaps it's a question more fit for https://github.com/duckdb/duckdb, but since it's a duckdb-java setup I figured it's better to start here.

[staticlibs]

Hi, thanks for the report! Yes, there were a number of HTTP fixes applied in 1.4.4, I will raise this internally.

If you can provide more details on proxy setup - this may be helpful!

[carlopi]

I can have a look at what might have been the difference, but to gather more informations (and potentially solve the issue), could you retry with both explcitly setting:

SET httpfs_client_implementation = curl;
--- do queries

or

SET httpfs_client_implementation = httplib;
--- do queries

They should take different codepaths, and might result in different behaviour, that might be interesting to understand what's going on.

There is also a:

SET http_timeout =  100; --- seconds

option, that might also help, you can find out what's the current setting (of both settings) via:

SELECT current_setting('http_timeout');

[staticlibs]

@eledhwen

I cannot reproduce this on 1.4.4.0 with a default Squid proxy server setup and with the following query to S3:

try (Connection conn = DriverManager.getConnection("jdbc:duckdb:");
     Statement stmt = conn.createStatement()) {
    stmt.execute("SET http_proxy = '127.0.0.1:3128'");
    try (ResultSet rs = stmt.executeQuery("SELECT * FROM 's3://alexkasko-test-bucket-1/iris.parquet'")) {
        while (rs.next()) {
            System.out.println(rs.getString(5));
        }
    }
}

Any more details about the problematic env may be helpful.

[eledhwen]

I had already tried setting an http_timeout to 60, without seeing a difference ; I suspect this has to do with proxy settings (which I don't control) that cap the transaction time anyway.

Here is what I use when I initialize the duckdb session:

SET autoinstall_known_extensions=false;
SET autoload_known_extensions=false;
SET custom_extension_repository='file:///dev/null';
SET extension_directory='/path/to/extensions';
INSTALL httpfs; LOAD httpfs;
CREATE OR REPLACE SECRET s3_config (
  TYPE s3,
  PROVIDER config,
  URL_STYLE 'path',
  KEY_ID 'my_key',
  SECRET 'my_secret',
  REGION 'my_region',
  ENDPOINT 'my_endpoint',
  USE_SSL false
);
CREATE OR REPLACE SECRET http_proxy (
  TYPE http,
  PROVIDER config,
  HTTP_PROXY 'my_host:my_port'
);

I won't have the time to test the suggested httpfs_client_implementation option today, but that sounds promising indeed.