diff --git a/README.md b/README.md index 5c5226c..6a8d088 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ PG Raw Parse provides a low level wrapper around the PostgreSQL backend parser. These bindings, as well as some additional functionality are provided by -[libpg\_query](https://github.com/pganalyze/libpg_query). +[libpg\_query]. In addition to parsing, we provide mechanisms to [traverse an AST], [construct new ASTs], and [transform ASTs]. See the API docs for more details. @@ -12,12 +12,136 @@ new ASTs], and [transform ASTs]. See the API docs for more details. [construct new ASTs]: https://docs.rs/pg_raw_parse/latest/pg_raw_parse/make/index.html [transform ASTs]: https://docs.rs/pg_raw_parse/latest/pg_raw_parse/transform/index.html +## Design + +The primary goal of `pg_raw_parse` is to map to PostgreSQL's parser with as +little overhead as possible. This means mapping to the raw structures whenever +possible, using PostgreSQL's internal allocator, and avoiding any significant +copies of data. + +PostgreSQL does not publish any header files or libraries to expose its backend +functions. We use [libpg\_query], which embeds those files in a form that is +easy to compile without going through cmake, as well as makes a few changes +to enable multithreaded usage. We also use this library for its `deparse` +implementation, turning an AST back into a string. + +When possible, the structures in this library are cast directly from a pointer +to the C structure. The main exception to this is `Node *`, which is +semantically an unsized enum. There is no way to represent an enum with +different sizes for each variant in Rust, so we need our own wrapper enum. The +tag is identical to the tag of the C enum, so LLVM *should* be able to optimize +this away in many cases but it is not guaranteed. + +Everything in pg\_raw\_parse makes use of PostgreSQLs allocator, both for +manipulating the structures returned by `parse`, and for [constructors provided +by this library][construct new ASTs]. It is assumed that ASTs are retained at +the scope of a single query. Each call to `parse` will return an AST with its +own arena. Individual nodes do not implement `Drop`, and are not freed until the +entire arena is dropped. This can result in slightly higher memory usage when +mutating ASTs, as nodes that are replaced will still occupy memory. But the +result is much less overhead from `palloc`/`pfree` in the most common usage +patterns. + +To ensure that fields of an AST node are always allocated on the same arena as +its parent, we make use of [lifetime branding]. [`MemoryToken`] is a type that +is used for constructing node allocated on a specific arena. Constructors +require all fields to be [`Unique`], which represents a node allocated on that +same arena and is not assigned anywhere else. Once all construction/mutation is +complete, the result is wrapped in [`Owned`], which is responsible for freeing +the arena in its destructor. + +[lifetime branding]: https://plv.mpi-sws.org/rustbelt/ghostcell/ +[`MemoryToken`]: https://docs.rs/pg_raw_parse/latest/pg_raw_parse/make/struct.MemoryToken.html +[`Unique`]: https://docs.rs/pg_raw_parse/latest/pg_raw_parse/make/struct.Unique.html +[`Owned`]: https://docs.rs/pg_raw_parse/latest/pg_raw_parse/struct.Owned.html + +Because individual nodes are never freed on their own, once an arena is inside +of an `Owned`, it is frozen. It is only possible to get shared references to +fields within it, and its arena can never be used for allocations again. This +decision was made to make it impossible to cause a memory leak by holding a long +lived reference to an AST, and then mutating it repeatedly. Instead, to mutate +an `Owned` node, it must first be copied onto a new memory arena using +[`make_unique`]. + +[`make_unique`]: https://docs.rs/pg_raw_parse/latest/pg_raw_parse/make/struct.MemoryToken.html#method.make_unique + +The majority of the code in this library is generated from C header files, with +the exception of extremely generic code such as list manipulation. We first run +these header files through [bindgen], and then operate on the resulting code as +if it were a procedural macro. Although this code lives in +[build.rs](blob/main/build.rs), its patterns should be familiar to developers +familiar with writing procedural macros. + +[bindgen]: https://github.com/rust-lang/rust-bindgen + +We create our own layout compatible structs rather than directly exposing the +structs generated by bindgen. This is to give us control over the visibility of +fields, as we don't want raw pointer fields to be public. We generate accessor +methods that convert to our custom type, and check the tag so an invalid node +assignment results in a panic rather than undefined behavior. In particular, +this is required for `Node*`, which cannot be represented in Rust as a simple +pointer cast for the reasons mentioned above. + +C has no concept of generics, so all lists are untyped lists of nodes. However, +many of those fields have documentation stating that they are a list of a single +type of node. We look for those comments, and change the type of the field to a +typed list if we find one. + +[AST traversal][walk an AST] is done using PostgreSQL's internal +`raw_expression_tree_walker` function, with a thin wrapper to handle passing a +Rust closure to C and transform PostgreSQL's exceptions into Rust panics. [AST +transformation][transform ASTs] is done with generated code. + +As a result of relying on code generation for the majority of this library, +supporting new PostgreSQL versions requires very little work. It is usually +nothing more than a submodule update for [libpg\_query], pointing to a commit +which includes the PostgreSQL source for that version. + +## Comparison with pg_query.rs + +The other popular library in this space is [pg\_query.rs], which is maintained +by the same team who maintains [libpg\_query]. While both libraries depend on +[libpg\_query] to get access to PostgreSQL's internal parser, [pg\_query.rs] +uses [libpg\_query]'s protobuf serialization layer to somewhat decouple it from +PostgreSQL's internal details. This type of approach makes sense when you're +maintaining bindings for multiple languages. But Rust's strong C FFI means a +lower level binding allows us to avoid many of the drawbacks of that approach. + +We are able to avoid the overhead of protobuf de/serialization, as well as +memory cost of copying all those structures into a memory space controlled by +the global allocator. `protoc` also generates a fairly inefficient data +structure in this case, causing the `Node` enum to be 584 bytes large. + +In contrast, by binding directly to PostgreSQL's data structures, there is no +memory overhead beyond what would be used either way within PostgreSQL's parser. +And the cost of "constructing" the Rust structures is at most a pointer cast and +a tag check. These two factors result in pg\_raw\_parse performing significantly +better, with the gap increasing as the size of the AST increases. + +#### Parse time + +![Speed benchmark graph](raw/main/benchmark_time.png) + +#### Parse time (log scale) + +![Speed benchmark graph (log scale)](raw/main/benchmark_time_log.png) + +#### Memory Usage + +![Memory benchmark graph](raw/main/benchmark_mem.png) + +#### Memory Usage (log scale) + +![Memory benchmark graph (log scale)](raw/main/benchmark_mem_log.png) + +## Contributing + This library's API surface is primarily driven by the needs of [PgDog](https://github.com/pgdogdev/pgdog). It is not intended to be a complete, one-size-fits-all solution to PostgreSQL ASTs. Contributions are welcome, but pull requests adding large and complex features are unlikely to be accepted unless they align with PgDog's needs. For a more general purpose library, -consider [pg\_query.rs](https://github.com/pganalyze/pg_query.rs). +consider [pg\_query.rs]. ## License @@ -27,3 +151,19 @@ Licensed under either of these: https://www.apache.org/licenses/LICENSE-2.0) * MIT license ([LICENSE-MIT](LICENSE-MIT) or https://opensource.org/licenses/MIT) + +[libpg\_query]: https://github.com/pganalyze/libpg_query +[pg\_query.rs]: https://github.com/pganalyze/pg_query.rs + +## LLM Policy + +This library was primarily written by humans, without LLM assistance. While LLM +assisted pull requests are allowed, any code written by an LLM must be +disclosed. Regardless of whether they were written by a human or an LLM, pull +requests are expected to be of a reasonable size to be reviewed by a human. + +Your pull request description, and any communication with maintainers must be +written by a human. Do not copy/paste output from an LLM into a PR comment. Pull +requests that violate this policy will be closed and locked. + + diff --git a/benchmark_mem.png b/benchmark_mem.png new file mode 100755 index 0000000..c1de4ef Binary files /dev/null and b/benchmark_mem.png differ diff --git a/benchmark_mem_log.png b/benchmark_mem_log.png new file mode 100755 index 0000000..56aac2c Binary files /dev/null and b/benchmark_mem_log.png differ diff --git a/benchmark_time.png b/benchmark_time.png new file mode 100755 index 0000000..1a9d117 Binary files /dev/null and b/benchmark_time.png differ diff --git a/benchmark_time_log.png b/benchmark_time_log.png new file mode 100755 index 0000000..792203c Binary files /dev/null and b/benchmark_time_log.png differ