Make Cabana Route

Utility that takes CSV formatted CAN log files and (optionally) accompanying videos, convert them to "routes" that can be opened in comma.ai's Cabana for CAN reverse engineering.

Motivation

The company comma.ai has an open source CAN log viewer and reverse engineering tool called Cabana. This is actually technically "New Cabana", a Qt desktop application distributed on GitHub in their openpilot repo.

Although Cabana is built to work with comma.ai's unique hardware and infrastructure, it has a very nice workflow for finding signals in CAN captures and creating DBC files to document them. One of its features is synchronising a video with the CAN log, which can make it easy to correlate events as they happen.

Normally, Cabana integrates directly with other comma.ai tools and their cloud "route" storage. The make_cabana_route tool is for when you have some random CAN log files and corresponding videos, and you want to load these into Cabana instead.

For background of how I personally came to want this, see this blog post.

IMPORTANT

This isn't a way to use comma.ai's driver assistance software without buying their hardware. It's a way to use the Cabana workflow for otherwise unrelated projects.

If you think you've found an issue with Cabana using a route generated by this tool, consider carefully whether it might be an issue with this tool before reporting it to comma.ai. This tool is entirely unofficial and unsupported by comma.ai.

Terminology

• In comma.ai's world, a "route" is a combination of a detailed vehicle log (including CAN messages) and one or more video streams that cover the same period in time.

Prerequisites

• CAN log files in the "gvret/SavvyCAN" CSV format, as supported by SavvyCAN. For examples, see here. As far as I know there's no official written specification for this format. PRs to add more CAN log input formats would be welcome.
• (Optional) video file that matches the CAN log.
• Currently this tool is only tested on Linux. It should work out of the box on macOS, but will require at least some small patch to work on Windows. PRs welcome!

Building

• You'll need to install Rust and its build tool Cargo.
• Clone the code from GitHub.
• Run git submodule update --init to get the "Cereal" submodule.
• cargo build --release to build an optimised version.
• If the build succeeds, you can find the binary at target/release/make_cabana_route (or use cargo run --release -- [...arguments...]).

You'll also need to build the Cabana program, I don't believe comma.ai distribute built versions of it. It is built as part of openpilot.

Usage

YAML file

Create a YAML file with the details of your CAN log inputs. Here's an example:

- car: Kona Electric 2021
  fingerprint: pcan-ccan-drive-modes-abs-traction
  logfile: 221217-3-2021-pcan-ccan-drive-modes-abs-traction.csv
  video: 221217-3-2021-pcan-ccan-drive-modes-abs-traction.mp4
  sync:
    video_s: 1.291
    log_us: 1469445700

• The YAML file can contain as many of these entries as you like.
• car and fingerprint are arbitrary strings, fingerprint is displayed above the video so you can put some identifying information here (the "route" itself is identified by its timestamp, so it's not very descriptive by itself.)
• logfile is the path to the CSV log file (relative to the YAML file).
• video is the path to the video file (relative to the YAML file). This is optional, if your log has no video then leave it and the sync key out.
• The sync values synchronise the video to the CAN log. Provide a timestamp for the video (in seconds) that corresponds to a particular microsecond timestamp in the log file. It's best to do this by recording a shot of the screen where the log is being captured at the start of each video, then you can step through frame by frame (mpv and VLC both let you do this) and find the exact timestamp of the frame when a particular CAN message timestamp first appears.

An additional optional key (not shown in the example) is route_timestamp that allows you to manually set the timestamp used to identify the route. If not found, make_cabana_route will use the video modification time (if there is a video file) or the CSV log file modification time otherwise.

Run make_cabana_route

Run the tool as follows:

make_cabana_route --yaml-path logs.yml --data-dir ./data_dir

The --data-dir argument provides the path to a "data directory" (created if missing) for Cabana. The route is created there as a series of sub-directories with names like 2022-12-17--09-35-30--0, 2022-12-17--09-35-30--1, 2022-12-17--09-35-30--2, etc.

The first part of each sub-directory name (before --) is the timestamp that uniquely identifies the "route" to Cabana. The final part (after the --) is the "segment" index, comma.ai splits each route into segments (presumably to save bandwidth when streaming them from their server).

Processing logs is pretty slow as it includes transcoding the video content.

You can also specify a filter on the command line in order to only process some logs:

make_cabana_route --yaml-path logs.yml --data-dir ./data_dir pcan-ccan-drive-modes

If a filter string like this is provided, only entries in the YAML file which contain this string in either the fingerprint, logfile, or video fields will be processed.

Run Cabana

To run Cabana and load a route, use a command line such as:

cabana --data_dir /path/to/data_dir --dbc optional/path/to/this.dbc "2022-12-17--09-35-30"

The --data_dir path is the same one as above. The --dbc argument is an optional one to open the Route in Cabana with a DBC file associated. Finally, the route name is the name of one of the directories inside the data directory, minus the final --0 for the segment number.

If the route doesn't contain any associated video, you need to add --no-vipc on the command line as well, or the route won't load.

Cabana should open immediately and start playing back the "route".

(Note, Cabana also has a UI for opening local routes like this, but I find it very difficult to get the right settings there compared to the command line.)

Shortcuts

To make loading Cabana easier, make_cabana_route also generates some shell script segments in the data_dir. These are named the same as the CSV log file, but with a .sh extension.

To run one, first ensure that cabana is on the PATH. Then you can run the script directly, like this:

path/to/data_dir/221217-2-2021-pcan-bcan-drive-modes.sh --dbc dbc/hyundai_kona.dbc

Additional arguments (like --dbc here) are passed to cabana as well. It's not necessary to manually pass --no-vipc here, the shell script will add it if/when it's needed.

Support

As mentioned above, please don't report issues to comma.ai until you're certain they're not issues introduced by this tool.

• PRs are welcome. If they seem well-written and don't make the project harder to maintain then I'm likely to merge them.
• Like every random thing published on GitHub, this project otherwise clearly comes without warranty!
• I'm not sure how much I intend to support this project into the future. There's a lot of "polish" that could be added, but at this stage it's useful enough for my purposes and I need to spend time on the project it was intended to assist.
• Nevertheless, feel free to open issues if you run into them. However, please don't be surprised if they don't get fixed (at least not by me).

Known Limitations

• Video transcoding is CPU only. This boils down to two things: the libavcodec API for hardware-assisted transcoding is relatively complex (as it's a complex concept), and rust-ffmpeg does not appear to fully support it.
• Error checking and propagation is pretty rubbish. In different places the program arbitrarily may panic without context on failure, or may return a useless Box<dyn Error> and then panic with even less context. There are a lot of TODOs in the code to improve this, but I haven't been back to tidy it up...