Stream input from stdin, rather than reading the entire script into memory.

[rolandwalker]

Description

• Stream STDIN input, running queries a line at a time.
• Remove MemoryError check, and recommendation for the vendor client.
• Use CSV/TSV formats with headers for the first line only.
• Exit with an error code if we are unable to open /dev/tty.
• Add --noninteractive flag to suppress the destructive-warning prompt from the CLI.
• Commentary on edge cases and followups.

Fixes #1184.

Checklist

• I've added this contribution to the changelog.md.
• I've added my name to the AUTHORS file (or it's already there).
• I ran uv run ruff check && uv run ruff format && uv run mypy --install-types . to lint and format the code.

[rolandwalker]

Use CSV/TSV formats with headers for the first line only.

The one breaking change here is that previously, the header lines would have been printed for every query in the script, leading to every other line being a header. It seems improbable that anyone would be depending on that behavior.

Edit: this description of the previous behavior may not be correct, and needs checking.

Edit: the description of the previous behavior was correct.

[scottnemes]

What's the intended use case of only showing the header for the first query? That seems to be more confusing at first glance, as you lose the details of what is actually happening. I.e.:

test.sql queries:

select count(*) from dev.t1;
select * from dev.t2 limit 5;
select count(*) from dev.t3;

command

uv run mycli < test.sql

Current behavior (able to see the header for each, so you know what the values are for:

count(*)
19
id
2
2
1
2
1
count(*)
1

New behavior:

count(*)
19
2
2
1
2
1
1

Makes it look like all the values are count(*)s due to only the one header.

[rolandwalker]

What's the intended use case of only showing the header for the first query?

The intended use case is a uniform script which produces the same columns for each line. I think that's the most common usecase when scripting! But maybe we can make it more configurable.

[scottnemes]

The intended use case is a uniform script which produces the same columns for each line. I think that's the most common usecase when scripting! But maybe we can make it more configurable.

Hmm I'd have to see some examples of what you mean. Anything I can think of I'd want to preserve the headers for each query. I.e. for a migration script, there might be various selects to check different things, some updates/inserts, etc. where I'd expect each query to come back with whatever headers it needed. If you're actually doing something with the output, i.e. creating a CSV, it's going to be in one query, and only have one header then regarless.

[rolandwalker]

SELECT data FROM table WHERE id >= 1 AND id < 1000;
SELECT data FROM table WHERE id >= 1000 AND id < 2000;
SELECT data FROM table WHERE id >= 2000 AND id < 3000;
…

As a data practitioner, this pattern is common. But again, we can finesse it with the CLI args.

[scottnemes]

SELECT data FROM table WHERE id >= 1 AND id < 1000;
SELECT data FROM table WHERE id >= 1000 AND id < 2000;
SELECT data FROM table WHERE id >= 2000 AND id < 3000;
…

As a data practitioner, this pattern is common. But again, we can finesse it with the CLI args.

Huh interesting. I'd expect to do that in a loop; putting that all in a SQL file (and running as a combo of queries at once) seems like an anti-pattern.

[scottnemes]

SELECT data FROM table WHERE id >= 1 AND id < 1000;
SELECT data FROM table WHERE id >= 1000 AND id < 2000;
SELECT data FROM table WHERE id >= 2000 AND id < 3000;

Also the vendor client works as mycli currently does, with headers for each query. I know we are expanding on the vendor client, but in this case I prefer the current behavior that matches with headers per query.

Edited to add; if you are planning to add an option to control this, I would say have the current behavior be the default at least.

[rolandwalker]

My main issue is with the specification of --csv. Interspersed headers are not CSV. So the current (weird) behavior can be the default.