Adding more layers for overture derived tiles

[michaelkirk]

We currently use planetiler to generate tiles for headway (a self hosted map stack) using osm.pbf as input.

For https://maps.earth (our headway deployment) that input data comes from https://daylightmap.org 's planet.pbf.

With the pending shutdown of https://daylightmap.org (https://daylightmap.org/2024/05/03/sunsetting-daylight.html), I've been looking at overture as a potential successor for QA'd source data.

I've successfully used https://github.com/onthegomap/planetiler-examples/blob/main/Overture.java as a starting point, but that example profile includes only a few layers - e.g. it doesn't have a POI layer or land use layer. Similar data exists in overture in some form or another (though I think there may be some significant differences in the data).

Is there an existing effort to implement more of the openmaptiles schema from overture? Or interest in incorporating such a thing?

[acalcutt]

Just an FYI, there is also https://github.com/onthegomap/planetiler-examples/blob/main/OvertureLandcover.java that has landcover layers from overture. Based on https://gist.github.com/dschep/9a4c875715e62c6b8e7d5697e33780d4 I used this code in my style.

    {
      "id": "land_cover",
      "type": "fill",
      "source": "overture_land_cover",
      "source-layer": "land_cover",
      "layout": {
        "visibility": "visible"
      },
      "paint": {
        "fill-outline-color": ["rgba", 0,0,0,0],
        "fill-color": [
          "case",
          [ "==", [ "get", "subtype" ], "urban" ],
          ["rgb",167, 162, 186],
          [ "==", [ "get", "subtype" ], "forest" ],
          ["rgb",134, 178, 137],
          [ "==", [ "get", "subtype" ], "barren" ],
          ["rgb",245, 237, 213],
          [ "==", [ "get", "subtype" ], "shrub" ],
          ["rgb",239, 218, 182],
          [ "==", [ "get", "subtype" ], "grass" ],
          ["rgb",254, 239, 173],
          [ "==", [ "get", "subtype" ], "crop" ],
          ["rgb",222, 223, 154],
          [ "==", [ "get", "subtype" ], "wetland" ],
          ["rgb",158, 207, 195],
          [ "==", [ "get", "subtype" ], "mangrove" ],
          ["rgb",83, 171, 128],
          [ "==", [ "get", "subtype" ], "moss" ],
          ["rgb",250, 230, 160],
          [ "==", [ "get", "subtype" ], "snow" ],
          ["rgb",255, 255, 255],
          ["rgba",255, 255, 255, 0]
        ]
      }
    },

You can see this landcover mixed with OSM data at
https://tiles.wifidb.net/styles/WDB_OSM_LAND/#6/42.378/-71.927

In the past alpha version of overture I had started to make a rough style that was a mix between osm basic and protomaps
https://github.com/acalcutt/wifidb-tileserver-gl/tree/master/tileserver-gl/styles/WDB_BASIC_OVERTURE
but you can see at https://tiles.wifidb.net/styles/WDB_BASIC_OVERTURE/#6/42.378/-71.927 it needs a bit of work with label clipping, though I am not sure if the newer overture profile improved things since that was made on the old alpha data and overture profile before it was merged.

[acalcutt]

It looks like my alpha version was based on https://github.com/onthegomap/planetiler/blob/overture-generic/planetiler-core/src/main/java/com/onthegomap/planetiler/overture/Overture.java which looks quite different from the newer example. It did have landuse, which I used in my style

It was from https://osmus.slack.com/archives/C031V9E9RMG/p1703037524824199

[msbarry]

Hello! There are a couple of different options here that I can think of:

1. Create a brand new overture profile from scratch: This would be continuing with what I started with Overture.java and might not be that bad since Overture already does a lot of the work that openmaptiles has to do to sort and filter features into thematic layers so it's mostly deciding on appropriate min zooms for features in each layer. This wouldn't be compatible with existing openmaptiles styles, but it would also let you ditch the extra copyright notice and cruft that's accrued in the schema over time.
2. Modify openmaptiles profile to add a flag that pulls data from overture: This would maintain compatibility with existing styles, and could be done one layer at a time, for example for onthegomap.com I made my openmaptiles fork use OSM for all layers, except pull buildings from Overture.
3. Add an Overture-to-OSM-compatibility layer inside planetiler: This would map Overture elements and attributes to what the canonical tags would look like if they came from OSM and pass that through so any OSM-based profile could switch over to using Overture. This would be quite a bit of work, but benefit all planetiler users on whatever schema they are using
4. Create an Overture-to-OSM.pbf translation tool: Similar to (3) except this would write out an osm.pbf from overture data using the same rules to generate canonical osm tags that (3) would have used on an in-memory transformation. (3) could easily be turned into (4) by just writing elements to disk instead of just passing them through a profile. This would be the most work (initial build-out and ongoing maintenance as Overture evolves), but have maximum benefit for all tools that currently process OSM data.

Thinking long-term I think (1) would be ideal to start to give people more options than just openmaptiles. You could also imagine an OSM.pbf-to-Overture translation tool that would let you plug live data into a pipeline instead of Overture snapshots, which would also be more straighforward to build as Overture publishes their translation rules in a machine-readable format.

For my own purposes, I'd probably continue on the path of (2) since it's the fastest way to start getting incremental benefit from incorporating bits of Overture data. I could contribute changes from my fork upstream.

But if someone else were willing to help then any of these options would be possible. What do you think?

[michaelkirk]

Thank you for the thorough response!

(1) ... would also be more straighforward to build as Overture publishes their translation rules in a machine-readable format.

I didn't know that. Can you link me to anything saying more about this?

To be honest, my initial instinct was (4.) — it'd be great to be able to build an .osm.pbf from overture data, and then continue to use the myriad of existing tools that accept osm.pbf.

But I have some hangups that might be more personal than technical here. For example, I'm still invested in supporting OSM as the ultimate source of truth. For Buildings, OSM entries are preferred, and ML buildings are only used if one doesn't exist in OSM, which works for me.

But for places:

The [Places] theme is derived from a conflation of Meta and Microsoft data

So I don't have a clear cut way to direct my users to "fix this data".

It still might be worthwhile to pursue, I'm just kind of thinking out loud.

As my first baby steps into planetiler, I think I'll start by trying to add some layers to https://github.com/onthegomap/planetiler-examples/blob/main/Overture.java (approach 1.)

And then if that goes well, maybe I'll try to see about extracting that into something that could be reusable for (approach 2.).

At the end of the day, I want to be able to use overture data for the tiles on https://maps.earth

[msbarry]

I didn't know that. Can you link me to anything saying more about this?

They publish these sql queries that map OSM data to overture: https://github.com/OvertureMaps/schema/blob/main/task-force-docs/base/land_use.sql - worst case you would have to port that, but ideally that would be a more language-independent format that other tools could read.

I'm also not sure how practical (4) is until more kinds of data from OSM are reflected in overture, or at least until OSM POIs are conflated with meta/microsoft like buildings. That's why I'm thinking a hybrid approach that replaces selective features from OSM with overture (ie. water, buildings) might be a practical compromise. I suppose you could also have a tool that takes OSM.pbf + overture and returns a new OSM.pbf with selective features added/replaced?

As my first baby steps into planetiler, I think I'll start by trying to add some layers to https://github.com/onthegomap/planetiler-examples/blob/main/Overture.java (approach 1.)

Sounds good, there are 2 versions of that one. The one you linked requires building with maven, whereas this one lets you modify/run the java profile without any build tools: https://github.com/onthegomap/planetiler-examples/blob/main/Overture.java. I'm thinking we might want to standardize on that one?

[davecraig]

Hi!

Obviously I'm a bit late to this thread, so playing catch up and looking for some advice. We've just gone into open testing with our app to help people with visual impairment navigate and explore the world - https://github.com/Scottish-Tech-Army/Soundscape-Android. We're been using planetiler to generate our maps from OSM data and it's been great, so a huge thank you! We have some extra metadata (e.g. crossing metadata, NAPTAN codes for UK busses) that we put in our maps, but no other changes.

One of the main areas of feedback we're getting comes down to OSM points of interest being thin on the ground in many areas and that's led me to looking at Overture Maps and this thread. One fix for us at least in the short term would be to have the Places layer from Overture Maps merged into our maps. The minimum for a place would be the name, primary category and location as a point. These then show up in our app as nearby places which users can easily select from a list to navigate to.

I think this type of map likely overlaps with some of the descriptions above, but I'm really not familiar enough with the details. Planetiler has meant I have been able to be mostly ignorant of the map building process! Having the Overture places appear in a wholly separate layer would be okay from our map parsing point of view, but would obviously need some additional maplibre styling work.

If you have any suggestions on how I can get from our current pure OSM maps to what I've described above, or whether you think that's a dead-end and there are other options I'd be very grateful to hear them.

Many thanks,

Dave

[msbarry]

This is doable! You could either update your planetiler-openmaptiles fork to remove the regular poi layer and add a new layer that reads from overture places. Or you could use https://github.com/onthegomap/planetiler-examples as a base to generate a profile that reads overture places and generates a separate pmtiles/mbtiles file and the client has to load tiles from both of them.

As far as data quality goes you can check out the overture explore site to see what's in its places layer https://explore.overturemaps.org/ - from what I've seen it's hit or miss.

There's also the foursquare poi dataset you could read with the same parquet input reader. It's also hit or miss but I think if you filter on date created/refreshed in last year or two it helps.

[davecraig]

This was remarkably easy, thanks again. The data quality is definitely a bit hit and miss though. There are both Meta and Microsoft places that are in the middle of a road and not even next to the business that they represent. The address information is correct, but there's been no effort to use that to improve the pin location.
It took 2 minutes in total to compile a layer with the places all just at maximum zoom.
I'll have a look at the foursquare data too now that I know what I'm doing,

Thanks

Dave

[davecraig]

This is doable!
Excellent, that's what I wanted to hear. I'll see what I can come up with!

Once I've figured out Overture support, then I guess FourSquare can be a 3rd option. The app is for worldwide use and I'm sure the different data sources have different regional strengths.

Thanks!

[michaelkirk]

I believe the overture places data also has a quality/confidence metric. Did you have a look at only including points above a certain confidence?

[davecraig]

I believe the overture places data also has a quality/confidence metric. Did you have a look at only including points above a certain confidence?

Yes, the confidence value seemed good for determining if businesses were still in operation, but didn't seem to have any correlation to the accuracy of the other data.
Closed businesses can sometimes be useful for navigation "Turn right where the bakery used to be", but you have to be clear that they are closed.