What happened?
Reading an Arrow file, a TIMESTAMP column comes out different depending on the timezone of the machine the server runs on. The file is unchanged and the workflow is unchanged, so the same run on two deployments reports different data, with nothing to say so.
Arrow stores a timestamp as an instant. On the way in, ArrowUtils.getTexeraTuple takes the vector's value, which for a zoned field is epoch milliseconds, and passes it to AttributeTypeUtils.parseField as a TIMESTAMP, reaching new Timestamp(millis). That renders the instant in the JVM's default zone, and Texera's TIMESTAMP carries no zone of its own to record which one was used. The zone is therefore read off the server rather than off the file.
Expected: one file yields one set of values, on every server.
How to reproduce?
Write a one row .arrow file holding a timestamp column. Read it with the Arrow File Scan operator on servers in different zones. Below is one file, written once, then read by four JVMs started with different -Duser.timezone.
| Server timezone |
Value the operator produces |
| America/Los_Angeles |
2024-01-07 00:00:00.0 |
| UTC |
2024-01-07 08:00:00.0 |
| Asia/Tokyo |
2024-01-07 17:00:00.0 |
| Europe/Berlin |
2024-01-07 09:00:00.0 |
The file holds a single instant, 1704614400000 milliseconds, which is 2024-01-07 08:00 UTC. Reading the same file with pandas.read_feather reports 2024-01-07 08:00:00+00:00 in every case, so the file itself is unambiguous.
Version/Branch
1.3.0-incubating-SNAPSHOT (main)
Commit Hash (Optional)
557b84a
What happened?
Reading an Arrow file, a TIMESTAMP column comes out different depending on the timezone of the machine the server runs on. The file is unchanged and the workflow is unchanged, so the same run on two deployments reports different data, with nothing to say so.
Arrow stores a timestamp as an instant. On the way in,
ArrowUtils.getTexeraTupletakes the vector's value, which for a zoned field is epoch milliseconds, and passes it toAttributeTypeUtils.parseFieldas a TIMESTAMP, reachingnew Timestamp(millis). That renders the instant in the JVM's default zone, and Texera's TIMESTAMP carries no zone of its own to record which one was used. The zone is therefore read off the server rather than off the file.Expected: one file yields one set of values, on every server.
How to reproduce?
Write a one row
.arrowfile holding a timestamp column. Read it with the Arrow File Scan operator on servers in different zones. Below is one file, written once, then read by four JVMs started with different-Duser.timezone.The file holds a single instant, 1704614400000 milliseconds, which is 2024-01-07 08:00 UTC. Reading the same file with
pandas.read_featherreports 2024-01-07 08:00:00+00:00 in every case, so the file itself is unambiguous.Version/Branch
1.3.0-incubating-SNAPSHOT (main)
Commit Hash (Optional)
557b84a