OSM Wrangler 2

Further development

The first version of the OsmWrangler is obviously quite limited in what it can do with the Osm file, and deliberately so - it was meant just to "scratch an itch" of removing unwanted amenity nodes from the maps being loaded into my GPS. But since then some other itches have appeared and therefore some work was started on a more comprehensive filtering tool.

The new version isn't in a state to be released yet, and it's not yet clear whether it will be. It's written in python rather than in java, mainly because python is fun to program in and its regular expression matching and list processing are quite convenient for this kind of thing. So it's starting again from scratch with a much wider set of aims:

Filtering out of nodes by tag values, but not just for amenity types, you could remove any kind of highway for example, or specific tourism elements, or indeed all amenities without listing each and every type
Filtering nodes by defining a latitude/longitude rectangle, and only keeping nodes within this rectangle
Filtering ways by way id, in this case only keeping ways which match the given id
Filtering relations by relation id, again only keeping elements in the given relation
"Squishing" the osm file by removing unwanted information like timestamps and who edited it last and so on
Perhaps converting into other formats such as GPX or CSV

Some of these functions are already implemented, such as the lat/long filtering and tag filtering for nodes and the squishing. But removing and/or modifying the ways based on which nodes have been removed is a bit of a trickier problem, and it's not clear the best way to handle that at the moment. Should the whole way be deleted, should the removed nodes be removed from the way, or should the already removed nodes be somehow retrieved and put back in to make the ways complete?

Reasons not to do it

A bigger problem threatening the completion of this python-based OsmWrangler is that it turns out that a lot of its intended applications can already be done by other tools, so it was more or less decided to shelve the project. For example:

Amenity and other node removal can be done by styles in programs such as mkgmap and the various tile renderers.
Filtering by lat/long can be done by bigger projects such as Osmosis, and several services (such as cloudmade) provide custom downloads for specific areas or countries, without you having to cut the data up yourself.
Way extraction can be done by loading a specific way id into JOSM and saving as OSM. Note that the OSM wiki makes it sounds like that's possible just by asking the XAPI, but that only delivers the node ids of the nodes in the way, not the details (and coordinates) of each node. But with JOSM that's quite easy.
Relation extraction can also be done using JOSM, or there are other services such as ra.osmsurround.org which even let you download GPX or KML versions of individual relations.
Conversion of an OSM file saved from JOSM or wherever into GPX or other formats can be done by GpsBabel.

That means I can already extract ways and relations and convert to GPX for use in Prune and other programs. And I don't feel the urge to split out my own sections of the planet.osm when the cloudmade downloads are so convenient. And all that together put the python version of OsmWrangler on hold, at least temporarily.

Reasons to do it after all

After deciding to leave it, and then thinking about the problems some more, there seemed to be good reasons to extend it after all. These included considerations such as:

It's not just about mkgmap and rendering, there are other uses for which a processed OSM file might be useful.
Maybe you don't always want to have to install a big program like Osmosis, or a database.
Web services such as ra.osmsurround.org only use OSM data, and have also shown themselves to be unreliable, for example the ability to export GPX files.
GpsBabel's conversion of GPX files is too dumb to be used for relations. It kind of works but what you get is the nodes in completely the wrong sequence, and a bunch of unwanted waypoints. So it's not at all suitable for what I want.

Current status

It's still an early prototype, but this command-line only, python-based tool can now not only filter on lat/long and node attributes (thereby already reproducing what version 1 can do and much more), but it also filters out the ways and relations in this function. If an included way contains nodes which didn't pass the node filter, then the extra nodes are included in the output, but ways containing 1 or fewer filtered nodes are removed. So the resulting OSM file is consistent.

Also any relation can be extracted to GPX format, and the nodes are given in the order in which they appear in the relation. Duplicate nodes are removed and ways are reversed as necessary to simplify the output. Relations containing relations are also handled.

Also the squishing is working, obviously only for the osm output, and it reduces the space quite nicely. On the other hand, some tools require the version attributes to be present for it to be considered valid osm format, so squish at your own risk.

There's still plenty to do before release and it seems that development has stalled due to scalability and performance issues. If you have any suggestions or feedback in the meantime then please get in touch by email.

OsmWrangler // OsmWrangler 2 // Timetabler // TextRactor // RockMySlocs // MobileJass

Page last updated September 2011. Feedback welcome at mail@activityworkshop.net.