A common library and set of test cases for transforming OSM tags to lane specifications

Overview

osm2lanes

See discussion for context. This repo is currently just for starting this experiment. No license chosen yet.

Structure

Example

Input JSON file with road OpenStreetMap tags:

{
    "lanes": "2",
    "oneway": "yes",
    "sidewalk": "both",
    "cycleway:left": "lane"
}

Output lane specifications from left to write:

[
    {"type": "sidewalk", "direction": "backward"},
    {"type": "cycleway", "direction": "forward"},
    {"type": "travel_lane", "direction": "forward"},
    {"type": "travel_lane", "direction": "forward"},
    {"type": "sidewalk", "direction": "forward"}
]

Kotlin

Run with Gradle

cd kotlin
gradle run --args "${INPUT_FILE} ${OUTPUT_FILE}"

Install and test

Create JAR file with gradle jar task and test with gradle test.

Run JAR

java -jar kotlin/build/libs/osm2lanes.jar ${INPUT_FILE} ${OUTPUT_FILE}

Python

Install and test

cd python
pip install .
cd ..
pytest

Run

osm2lanes ${INPUT_FILE} ${OUTPUT_FILE}

Rust

Install and test

After installing rust, run:

cd rust/osm2lanes
cargo test

Run

cargo run -- ${INPUT_FILE} ${OUTPUT_FILE}
Comments
  • Fix lane count calculation - use Scheme structs to avoid re-reading/re-interpreting tags

    Fix lane count calculation - use Scheme structs to avoid re-reading/re-interpreting tags

    I am submitting this draft PR to get input on the new approach to driving_lane_directions, rust style and tests.yml. This PR is aiming towards ~~#69~~ #72.

    Are we happy with this overall structure for driving_lane_directions?

    bf5df9d8abdad9b72b4a5b64d558784d6695d1e8 shows the concise version, and dc5de27a3ee539cf25735219d2fdd2c8ffc7e677 shows how useful tag validation warnings can be added (thinking about #101 and how an editor would like to use tags_to_lanes).

    Any feedback on my rust style?

    I want to fit into this project, more than anything else, but couldn't help trying out is_some_and and using lots of unwrap_or.

    How do the roundtrip tests work?

    I couldn't figure it out just yet. Some guidance in the docs about writing and running tests.yml tests would be super awesome. (I figured out cargo test --feature tests, which I'll add to the docs if I get there first.)

    opened by BudgieInWA 24
  • Rewrite Spec

    Rewrite Spec

    I think the current spec is a bit brittle if we want to be truly agnostic to our clients

    Adds: designated vehicle - to help renderers use - to inform on access restrictions by use case. width: to describe width of the lane.

    Open issue: Something I have been mulling over in my head is whether we shouldn't make all lane separators be just a variant of lane. Because what is the difference between a painted line, 2 painted lines, 2 painted lines with some more space in between, 2 painted lines with hashmarks inbetween, bollards, etc. IMO it will become important to describe in full detail what kind of vehicles can cross, e.g. for overtaking, and how wide the space is.

    opened by droogmic 14
  • Represent dividers semantically

    Represent dividers semantically

    I want to propose that dividers be represented internally by their semantics, instead of their visual/physical manifestation, and be translated into specific markings in a separate step. My initial feeling is that it would shift most or all locale dependent questions about dividers out of the tag parsing step, into its own divider to markings step. Providing the divider semantics in the output would also be useful for users.

    Semantic divider types

    Dividers would be broadly categorised something like:

    • curb
    • road edge line (inner / outer)
    • lane separating line
    • center line
    • median (see below)

    with additional semantic properties such as

    • stopping: default/no_standing/no_stopping/parking?/...
    • can_change: no/only_left/only_right/both (change= and overtaking=

    dividers_to_markings

    A separate function concerned with just the problem of converting semantic dividers to markings in the context of a given road would be a easier for new contributors to contribute to, which would be valuable for capturing global nuances. (I would be interested to see how static the mappings turn out to be.)

    Having the markings be a separate step would also be a useful to allow additional information to be added to the road after examing the semantics. E.g. I have two connected OSM ways: 2 lanes into 3 lanes. I get the semantic lane info, compare them and decide that first lane in the 3 lane way is an added lane. I annotate the appropriate divider as "merging" before getting markings, and receive a beautifully appropriate dotted line (here in Oz).

    Exposing the function as part of the API would also be a good way to query the lane marking database, because describing a situation from scratch in the output schema is much easier than describing it in the OSM schema!

    Divider widths

    Given that different locales might have different width markings for the same semantic divider, semantic dividers would have zero width: their markings fit within the the width(s) of the adjacent lane(s). (I'm not even sure this isn't how it already works). This is how I would prefer to receive osm2lanes output in semantic mode as a user.

    Medians

    This makes the "median" divider the odd one out, as the only divider with width of its own. Perhaps this suggest that the median itself should be a "lane" like a shoulder or a verge. One benefit is that the markings that divide the road from the median can be described too. In Australia, a median will often have a continuous solid line surrounding it, even as it transitions from a solid colour to a raised median to a diagonal hatching. The following output to describe the median would make sense to me:

    A continuous line (divider) surrounds the median (buffer)

        {"type": "travel_lane", "direction": "forward"},
        {"type": "separator", "markings": [{"style": "solid_line"}]},
        {"type": "median", "kind": "raised"}, // or "colored" or "striped" or whatever
        {"type": "separator", "markings": [{"style": "solid_line"}]},
        {"type": "travel_lane", "direction": "backward"},
    

    Medians are not well represented in OSM at the moment (divider= is the best I could find), but that is no reason not to represent then in this schema.

    opened by BudgieInWA 12
  • Generate more test cases to cover the current Rust implementation

    Generate more test cases to cover the current Rust implementation

    The current test case suite lacks bus lanes, cycleway separators, and probably other cases. I can generate a JSON test case for every single OSM way in a region using the Rust implementation. Then run the Python/Kotlin implementation against them, look for discrepancies, and boil down to the simplest test cases.

    opened by dabreegster 11
  • Start a web app to edit lanes and upstream tags to OSM

    Start a web app to edit lanes and upstream tags to OSM

    #240 The goal for this app is to make it easier to tag lanes correctly in OSM. Somebody would pick a way, see the current lanes in a cross-section view, drag the cards around and make things look correct, generate OSM tags to represent that, and upload the changeset.

    This PR is a minimal prototype of that flow, without uploading changesets yet. I'd like to make sure the basic structure is OK in code review, then send followups to flesh out its functionality more. I'm giving myself an aggressive goal of mid August to make this actually usable end-to-end in simple situations, for the SoTM conference. It's a non-goal right now to handle roads split into multiple ways.

    To run it, you should just be able to cd web_editor; ./serve_locally.sh. The only dependency is wasm-pack to compile and python3 to serve a dummy HTTP file server. The demo is not very impressive yet:

    https://user-images.githubusercontent.com/1664407/180864376-6f4b2984-8e62-4f5a-a8c4-f5bfa671b7a3.mp4

    opened by dabreegster 8
  • Process for starting a schema

    Process for starting a schema

    First there's the question of how we want to iterate on the schema.

    1. We could get test parity with the Rust implementation as it exists currently, then iterate on the schema and keep all 3 implementations in sync.
    2. We could iterate on the schema first with one implementation (Python, unless I get my Kotlin dev environment set up). Then once the dust has settled a bit, make the other 2 implementations match.

    I'd vote for the second, since it seems less work.

    How to express the schema

    Should we use https://json-schema.org or switch to protocol buffers or similar?

    Next steps with the schema

    First thing is the set of lane types, and how we want to add details to some of them. For instance, we could have parallel_parking and diagonal_parking... or type = parking, parking_type = parallel. If we used protos, we could be more precise and show that the parking_type field is only valid when type = parking by using oneof. Any preferences?

    The next major question on my mind is whether to represent a bus/transit-only lane, or instead list access/usage restrictions. We could just say "travel lane" with some subset of [motor vehicle, high-occupancy vehicle, public transit, taxi] allowed.

    I also don't want to get too bogged down in deciding things that're pretty quick to switch in the code, but I'm not convinced blindly following A/B Street's current schema is the right choice.

    opened by dabreegster 8
  • Flesh out the web editor, remove drag-and-drop

    Flesh out the web editor, remove drag-and-drop

    https://user-images.githubusercontent.com/1664407/199611654-8e9d39aa-a323-4a4e-b285-5abaf9fb0d82.mp4

    Remove drag-and-drop for now, since it complicates the code, we were using a library I'm not sure is the right choice, and the UX of dragging a trash can to a lane vs a lane to a trash can is a bit unclear to me. Instead, just have some buttons inline to each card, and add new lanes to the end always.

    opened by dabreegster 7
  • Choose license

    Choose license

    The readme states

    This repo is currently just for starting this experiment. No license chosen yet.

    https://github.com/a-b-street/osm2lanes#osm2lanes


    I just wanted to add this issue to not miss the point when the experiment becomes a project that should have a license …

    opened by tordans 7
  • Web interface to convert tags to lanes

    Web interface to convert tags to lanes

    We could have a simple webpage hosted on github pages where you copy in an OSM way's key/values, press a button, and get the JSON lanes output. Internally it can use Javascript and call the Rust implementation, compiled to WASM.

    Next steps after that could be a Leaflet or Mapbox map, where you click on a way, we query Overpass for the nearest way and its tags, and do the same thing.

    And then from there, maybe consider some kind of rendering/visualization, just for a single road.

    opened by dabreegster 7
  • Unexpected number of traffic lanes

    Unexpected number of traffic lanes

    The last test in A/B Street has the following specification:

                (
                    "https://www.openstreetmap.org/way/335668924",
                    vec!["lanes=1", "sidewalk=none"],
                    DrivingSide::Right,
                    "SddS",
                    "vv^^",
                ),
    

    Why do we assume two traffic lanes here?

    question 
    opened by enzet 7
  • Rust: Add Overpass and Async

    Rust: Add Overpass and Async

    move main.rs into its own crate...

    2 subcommands, convert and get or something

    and try to add a input form to the web view to also query overpass, just to confirm the reqwest library works in WASM.

    probably not included: driving_side and iso3166 retrieval from overpass

    opened by droogmic 6
  • busway on under-tagged roads

    busway on under-tagged roads

    https://www.openstreetmap.org/way/997453943 it'd be more reasonable to put 2 lanes in one direction or the other

    https://www.openstreetmap.org/way/228767989 no lanes tagged, but since it doesn't suggest general traffic is restricted, we should assume 4 lanes

    Also just mis-tagged cases like https://www.openstreetmap.org/way/46729604. lanes isn't 2

    opened by dabreegster 2
  • tags.subset can miss the relevant tags

    tags.subset can miss the relevant tags

    Way 893136312 produces the error Conversion Error: unsupported: 'lane count mismatch' - 'no tags' - osm2lanes/src/transform/tags_to_lanes/modes/bus/mod.rs:154:28

    The way in question has bus:lanes:forward, but not bus:lanes. We should consider including all tags with the prefix, since that's often how tag keys get used

    opened by dabreegster 0
  • Proposal for a rework of the `parking:lane` schema

    Proposal for a rework of the `parking:lane` schema

    Hey! For anyone who did not see it, yet: There is a proposal in the works with the goal of improving and reworking the parking:lane schema: https://wiki.openstreetmap.org/wiki/Proposed_features/street_parking_revision

    Since street parking is relevant for A/B Street, I wanted to inform you about this. We appreciate any comments and - when the time comes - help with the process of data migration/transferring existing data into the new schema. We are planning "translation" lists and maybe tools to support this. If anyone can help with this, it would be awesome!

    opened by SupaplexOSM 3
  • Local web development workflow broken

    Local web development workflow broken

    trunk serve can't be used for local development anymore. I'm getting CORS errors: Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://unpkg.com/[email protected]/dist/leaflet.js. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). Status code: 301.

    If I understand modern web dev best practices, the "right" answer is to instead grab local copies of our JS dependencies like leaflet and serve them. In osm2streets we're manually vendoring the dependencies here, but it's an unpleasant process. I manually grabbed files from unpkg, and watched browser network requests to find all of the files.

    Of the plethora of dependency management + bundler tools out there, what should we use? What'll play nice with trunk? Can https://trunkrs.dev/assets/#script-assets help?

    opened by dabreegster 8
  • osm2streets cutover

    osm2streets cutover

    #71, about cutting A/B Street over to using osm2lanes, kind of died off. I wanted to start fresh here based on new goals and problems. Since the last update there, we successfully detangled the overall OSM import and geometry logic out of A/B Street into https://github.com/a-b-street/osm2streets. It has a standalone web UI, its own unit tests, etc and is no longer tied to A/B Street. A/B Street directly depends on it. That's the same thing we want to achieve with osm2lanes, but now the integration is a bit more focused -- osm2streets will depend on osm2lanes. A/B Street will depend on osm2streets, and osm2lanes is an implementation detail one layer in.

    Goal

    osm2streets is still using the original lane_specs parsing code. In the short-term (next few weeks), I want to start adding a lane type for shared walking/cycling paths, fix bugs like this and many of the others filed in this repo, and handle lanes with explicitly tagged widths. I do not want to invest further in the monolithic lane_specs.rs; I want to cutover to osm2lanes.

    Problems

    When I've tried to jump in and work on osm2lanes for some of these things, it's felt like the codebase is very complex, even though I've been reviewing all the changes.

    Error handling / permissiveness

    cargo run way 260912073 fails outright because sidewalk=none is deprecated. I think we need to rethink how strict we are, maybe using something like taginfo popularity to guide decisions. Any pushback against treating this as a warning but otherwise proceeding?

    Alternatively, there can be an optional layer in osm2lanes that preprocesses tags and corrects common mistakes. Some examples in https://github.com/a-b-street/abstreet/blob/f0f13bc50aac28cf0f41d960602258ddfb61a566/raw_map/src/lane_specs.rs#L135 -- not the ones involving directions and such, or sidewalk inference.

    Adding a test

    osm2streets progress has happened quickly because of being able to iterate very quickly with tests. The table-driven tests in the original code were very quick to crank out:

                (
                    "https://www.openstreetmap.org/way/49207928",
                    vec!["cycleway:right=lane", "sidewalk=both"],
                    DrivingSide::Left,
                    "sddbs",
                    "^^vvv",
                ),
    

    The last time I attempted to add an osm2lanes test, I was a bit overwhelmed by having to figure out the locale and write out the full JSON (with separators or not?). Some ideas for making this experience easier:

    1. Add something to the web app to export as a copy-pasteable test case template
    2. Maybe make a tiny tool to transform the "sddb" + "^^vv" notation into full JSON, and also use that to copy into the yaml. It's easier to type.
    3. Maybe split the test yaml by how much we want to assert. Most test cases don't need to specify separators or width in the output. We can start a different file where we include those in the expected output and enforce that.

    If the drag-and-drop UI worked, we could also use that to quickly specify the expectation and copy it into the YAML.

    Matching up lanes

    osm2streets still uses the old LaneType enum. It'd be nice to use osm2lanes' representation directly at some point, but this is out of scope for now. One major change that has to happen in osm2streets and beyond is properly modelling Direction = Both or Direction = None. Until then, bidi lanes will get split into two directional lanes.

    Some of that logic is https://github.com/a-b-street/abstreet/blob/f0f13bc50aac28cf0f41d960602258ddfb61a566/raw_map/src/lane_specs.rs#L238. Since it's non-trivial, I think it makes sense for osm2streets unit tests to also include the translated lane config in the expectation. I will probably add that as a preliminary step.

    Rough plan

    Because of the way work is going right now, I get the most done when I quickly crank out code in a few days. What I would work on is:

    • fixing the above test issues, making it much easier to quickly iterate
    • getting parity with the current implementation. Maybe that means turning errors into warnings, or adding an optional layer to "fix" common tagging issues that happen in the real world.
    • a cutover
    • then fixing reported bugs and starting on things like lane widths

    Any feedback, @droogmic and @BudgieInWA? When I go rapid-fire mode and start this, do you want me to wait for reviews on PRs, or would y'all be OK leaving feedback whenever is convenient and I'll address it later? (If the former, I'll probably send bigger PRs to reduce communication latency)

    opened by dabreegster 2
Owner
A/B Street
Transportation planning and traffic simulation software for creating cities friendlier to walking, biking, and public transit
A/B Street
Core Fiberplane data models and methods for transforming them (templates, providers, markdown conversion)

fiberplane This repository is a monorepo for Rust code that is used throughout Fiberplane's product. Overview base64uuid - A utility for working with

Fiberplane 18 Feb 22, 2023
Shaping, Processing, and Transforming Data with the Power of Sulfur with Rust

Sulfur WIP https://www.youtube.com/watch?v=PAAvNmoqDq0 "Shaping, Processing, and Transforming Data with the Power of Sulfur" Welcome to the Sulfur pro

Emre 6 Aug 22, 2023
Given a set of kmers (fasta format) and a set of sequences (fasta format), this tool will extract the sequences containing the kmers.

Kmer2sequences Description Given a set of kmers (fasta / fastq [.gz] format) and a set of sequences (fasta / fastq [.gz] format), this tool will extra

Pierre Peterlongo 22 Sep 16, 2023
CLI Tool for tagging and organizing files by tags.

wutag ?? ??️ CLI tool for tagging and organizing files by tags. Install If you use arch Linux and have AUR repositories set up you can use your favour

Wojciech Kępka 32 Dec 6, 2022
An efficient pictures manager based on custom tags and file system organization.

PicturesManager An efficient pictures manager based on custom tags and file system organization. Developed with Tauri (web app) with a Rust backend an

Clément Grennerat 2 Dec 21, 2022
Coppers is a custom test harnass for Rust that measures the energy usage of your test suite.

Coppers Coppers is a test harness for Rust that can measure the evolution of power consumptions of a Rust program between different versions with the

Thijs Raymakers 175 Dec 4, 2022
A library to provide abstractions to access common utilities when developing Dioxus applications.

?? Dioxus Standard Library ?? A platform agnostic library for supercharging your productivity with Dioxus. dioxus-std is a Dioxus standard library tha

Miles Murgaw 5 Nov 9, 2022
Universal Windows library for discovering common render engines functions. Supports DirectX9 (D3D9), DirectX10 (D3D10), DirectX11 (D3D11), DirectX12 (D3D12).

Shroud Universal library for discovering common render engines functions. Supports DirectX9 (D3D9), DirectX10 (D3D10), DirectX11 (D3D11), DirectX12 (D

Chase 6 Dec 10, 2022
Rust-clippy - A bunch of lints to catch common mistakes and improve your Rust code

Clippy A collection of lints to catch common mistakes and improve your Rust code. There are over 450 lints included in this crate! Lints are divided i

The Rust Programming Language 8.7k Dec 31, 2022
This repo contains crates that are used to create the micro services and keep shared code in a common place.

MyEmma Helper Crates This repo contains crates that can are reused over different services. These crate are used in projects at MyEmma. But these crat

MyEmma 1 Jan 14, 2022
Work in progress NCBI's Common Tree alternative in the terminal

Lomanai Usage lomanai --species 'Mus musculus' --species 'Homo sapiens' #> Mammalia #> ++Rodentia #> | \-Mus musculus #> \+Primates #> \-Homo sapien

Jean Manguy 3 Dec 20, 2022
A simple common-line interface for ChatGPT API.

heygpt A simple common-line interface for ChatGPT API. ?? Streaming output! ?? One-shot mode to get a quick answer ?? Interactive mode to have a conve

Eric Fu 88 Apr 17, 2023
Rust TUI library - Clipping region is a set of min/max x/y values applied to the existing region

TinyBit Clipping region is a set of min/max x/y values applied to the existing region A TUI lib This is not yet production ready T O D O TODO: bugs: T

Togglebit 13 May 3, 2022
🍅 A command-line tool to get and set values in toml files while preserving comments and formatting

tomato Get, set, and delete values in TOML files while preserving comments and formatting. That's it. That's the feature set. I wrote tomato to satisf

C J Silverio 15 Dec 23, 2022
tmplt is a command-line interface tool that allows you to quickly and easily set up project templates for various programming languages and frameworks

tmplt A User Friendly CLI Tool For Creating New Projects With Templates About tmplt is a command-line tool that lets users quickly create new projects

Humble Penguin 35 Apr 8, 2023
create and test the style and formatting of text in your terminal applications

description: create and test the style and formatting of text in your terminal applications docs: https://docs.rs/termstyle termstyle is a library tha

Rett Berg 18 Jul 3, 2021
A CLI app to set and organize your favorite DNS servers.

rdns A CLI app to set and organize your favorite DNS servers. Introduction rdns is a CLI utility that can set your system DNS, either directly or by m

null 4 Feb 19, 2024
Black-box integration tests for your REST API using the Rust and its test framework

restest Black-box integration test for REST APIs in Rust. This crate provides the [assert_api] macro that allows to declaratively test, given a certai

IOmentum 10 Nov 23, 2022
trigger io::Error's in test, and annotate their source

fault-injection docs Similar to the try! macro or ? operator, but externally controllable to inject faults during testing. Unlike the try! macro or ?

Komora 18 Dec 16, 2022