Add grouped-linemerge generalization

[leijurv]

Summary

This PR adds a new generalization strategy, grouped-linemerge, to osm2pgsql-gen. It maintains a table of merged linestrings, the equivalent of:

SELECT col1, col2, ..., (ST_Dump(ST_LineMerge(ST_Collect(geom)))).geom
FROM src
GROUP BY col1, col2, ...
WHERE condition

The key is that this table is global, rather than local to your tile. It is kept up to date incrementally as the source data changes (and it never re-scans the entire planet to do so).

It works similar to the existing generalizations rivers and vector-union. You configure it from your flex style like osm2pgsql.run_gen('grouped-linemerge', {...}), and you run osm2pgsql-gen the same way (after import, and after update).

Motivation

The motivation is merging ways in OSM-Carto with the same rendering. Here's the relevant issue: openstreetmap-carto/openstreetmap-carto#951

OSM ways are often extremely fragmented, only going a block or two at a time in urban areas, due to relations and tagging changes. This significantly impairs rendering (see images near the bottom of that issue). We could LineMerge these ways within each tile as it's rendered, but this makes it really challenging to have consistent geometries for long ways that span metatile boundaries. (see openstreetmap-carto/openstreetmap-carto#5229 and openstreetmap-carto/openstreetmap-carto@master...leijurv:openstreetmap-carto:recursive-cte)

This PR allows us to have a global LineMerge for all ways, grouped only by the colunms that actually affect rendering (like name, ref, highway, layer, etc). There are no tile-boundary artifacts, and it can be maintained incrementally quite efficiently.

How it works

Initial build

The initial build is very simple - we just run that exact query to do a grouped ST_LineMerge across the whole world. This takes a few minutes, because the number of ways in each group is often not so large. We also make an index on the start and end points of each way, this is not strictly required but it significantly helps performance of the incremental update.

Incremental updates

Generalizations are able to receive the expire list of tiles whose geometry has changed. The expire list includes tiles where something was deleted, changed, or added. One possible approach here would be to select all roads in the entire planet that match any grouping key that was touched, and re-LineMerge them all. I ruled this out because some road names are very frequent. Imagine if moving a node on "Main Street" in some town, resulted in re-scanning every road named "Main Street" in the entire world. I assumed this was not an option from a performance perspective. Instead, I used a recursive CTE to spider through the geometry from the expired tile. Each changed area becomes a seed from which it walks outwards. To bound this walk, it must match an exact endpoint, and it must match every element of the grouping columns. This finds the full connected component, which replaces the old connected component geometry.

Usage

-- An expire output on the source geometry feeds the incremental updates:
local exp = osm2pgsql.define_expire_output({ maxzoom = 18, table = 'expire_roads' })
-- ... attach `expire = {{ output = exp }}` to the source table's geometry column ...

-- A destination table holding the grouping columns + the merged geometry
-- (no id column; it's maintained by osm2pgsql-gen):
osm2pgsql.define_table({
  name = 'coalesced_roads',
  columns = {
      { column = 'highway', type = 'text' }, { column = 'name', type = 'text' },
      { column = 'ref', type = 'text' },     { column = 'layer', type = 'int4' },
      -- … any other grouping columns …
      { column = 'way', type = 'linestring', not_null = true },
  }
})

function osm2pgsql.process_gen()
  osm2pgsql.run_gen('grouped-linemerge', {
      src_table        = 'planet_osm_line',
      dest_table       = 'coalesced_roads',
      geom_column      = 'way',
      group_by_columns = 'highway, name, ref, layer',  -- comma-separated
      where            = 'name IS NOT NULL OR ref IS NOT NULL', -- optional pre-filter
      expire_list      = 'expire_roads',  -- } required in
      zoom             = 18,              -- } append mode
  })
end

Then:

osm2pgsql      -O flex -S style.lua --slim … data.osm.pbf   # import (creates the tables)
osm2pgsql-gen          -S style.lua                         # initial build
osm2pgsql   -a -O flex -S style.lua --slim … changes.osc.gz # update
osm2pgsql-gen -a       -S style.lua                         # incremental update

Details

I believe this works in general for incremental updates, but the recursive CTE was challenging to get correct and performant. The reason for the new index is so that we can look up ways by their exact endpoint match. Without the new index, we'd use the existing GiST to lookup the way, which means that even if we ask PostGIS for ST_Intersect or some such, it will actually mechanically do && on the way's entire bbox. This works, to be clear, but it means that if I want to find a way that ends on an exact point, GiST will actually give me all the ways whose entire bbox covers that point, which means less than one percent of the ways that the index gives me are actually eligible. See here openstreetmap-carto/openstreetmap-carto#951 (comment) for discussion about that.

Tests

I put in tests/test-gen-grouped-linemerge.cpp a differential fuzz test. It performs hundreds of random connect/disconnects in a small grid of points. I intentionally set it up so the degree of each node could vary from 0 to 4, which stresses how ST_LineMerge does not merge nodes with a degree above 2. At all times, the incrementally updated geometry must exactly match what ST_LineMerge would have done.

I have run this locally and it works beautifully. Here's the commit to run it leijurv/openstreetmap-carto@b79c0fa and it does add a bunch of labels (looks like this openstreetmap-carto/openstreetmap-carto#951 (comment))

[joto]

From a first quick glimpse this looks really interesting. I'll need a while to digest this though. Can you look at the build failures and fix them?

[leijurv]

That's great to hear that it's interesting!!!

I've fixed the clang-tidy, and I've fixed a performance mistake.

Here are some numbers for what this does, and how long it takes, over downtown Los Angeles.

Note that in practice, we would never expire an entire tile at Z13. More likely Z18 or Z19. I'm just showing it for clarity.

Note that the most relevant column is probably walk, which is how long the CTE takes. Note that even in a nasty torture test, a large region of los angeles, that made the CTE go to a recursion depth of 433, the time it took is very small, 0.08 seconds in the worst case. I think this is a really good sign - but - I don't actually know what the performance requirements are here. Roughly, fetching all the ways in the affected tile, took about as long as the recursion (at higher zooms), so hopefully that's fine?

As we get to higher zooms, the reason why region doesn't get faster, is the issue I mentioned earlier. The GiST index on a linestring indexes the entire bbox. Therefore, even at "zoom level infinity" (looking up a single point), the GiST will unavoidably give you every way whose entire bbox covers that point (way && point), and then Postgres will then filter it down to ST_Intersects or whatever the query actually was. So, by the time we get to Z17 or Z18, it's almost like we are querying a single point, but there are tons of ways in a dense urban area whose full min-to-max bbox covers that point. (and this is why the cte needs a new index - otherwise every one of those up to 433 lookups would take perhaps 30ms each)

zoom	seed ways	ways traversed	nodes	CTE recursion depth	groups	output geoms	region	walk	collect	delete	insert	total
z13	1,793	3,792	4,430	433	450	783	38ms	81ms	39ms	28ms	49ms	234 ms
z14	616	1,855	2,014	433	135	212	29ms	46ms	23ms	17ms	18ms	133 ms
z15	171	410	452	89	44	60	30ms	20ms	8ms	6ms	9ms	74 ms
z16	50	185	200	38	17	25	30ms	17ms	10ms	5ms	4ms	67 ms
z17	15	152	161	38	9	11	28ms	18ms	11ms	5ms	4ms	66 ms
z18	5	88	92	38	4	6	29ms	16ms	6ms	7ms	4ms	61 ms
z19	4	88	92	38	4	6	31ms	16ms	6ms	7ms	4ms	63 ms
z20	2	58	60	38	2	2	30ms	17ms	9ms	5ms	3ms	64 ms