DWARF packaging utility, written in Rust, supporting GNU extension and DWARF 5 package formats.

Overview

thorin

thorin is an DWARF packaging utility for creating DWARF packages (*.dwp files) out of input DWARF objects (*.dwo files; or *.o files with .dwo sections), supporting both the pre-standard GNU extension format for DWARF packages and the standardized format introduced in DWARF 5.

thorin was written as part of the implementation of Split DWARF in rustc. A Rust implementation of a DWARF packaging utility is easier to integrate into the compiler and can support features like loading dwarf objects from archive files (or rustc's rlibs) which are helpful in supporting cross-crate Split DWARF packaging in rustc.

See the README documents of the thorin crate and the thorin-bin crate for usage details of the library and binary interfaces respectively.

Contributing to thorin

If you want help or mentorship, reach out to us in a GitHub issue, or ask davidtwco or in #t-compiler on the Rust Zulip instance.

thorin should always build on stable rustc. To build thorin:

$ cargo build

To run the tests, first install the relevant dependencies:

$ apt install --no-install-recommends --yes llvm-13 llvm-13-tools
$ pip install lit

Next, run the lit testsuite (replacing /path/to/llvm/bin with the correct path to your LLVM installation, if required):

$ cargo build # in debug mode..
$ lit -v --path "$PWD/target/debug/:/path/to/llvm/bin/" ./tests
$ cargo build --release # ..or in release mode
$ lit -v --path "$PWD/target/release/:/path/to/llvm/bin/" ./tests

We use rustfmt to automatically format and style all of our code. To install and use rustfmt:

$ rustup component add rustfmt
$ cargo fmt

Filing an issue

Think you've found a bug? File an issue! To help us understand and reproduce the issue, provide us with:

  • The (preferably minimal) test case
  • Steps to reproduce the issue using the test case
  • The expected result of following those steps
  • The actual result of following those steps

Definitely file an issue if you see an unexpected panic originating from within thorin! thorin should never panic unless it is explicitly documented to panic in the specific circumstances provided.


Name

thorin is named after Thorin Oakenshield from The Hobbit, as Thorin is a dwarf who leads other dwarves. thorin uses the gimli library (named after a dwarf from Lord of the Rings) to read DWARF format debug information, the name of which is a medieval fantasy complement to ELF, the file format for executables and object files.
You could also call this project rust-dwp, if you'd prefer that.

Author and acknowledgements

thorin is authored by David Wood of Huawei Technologies Research & Development (UK) Ltd. thorin is maintained by the Rust Compiler Team.
In addition, thanks to the authors of object and gimli, on which this utility depends heavily; and to Philip Craig for advice and reviews during initial implementation of thorin.

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Code of conduct

When contributing or interacting with this project, we ask abide the Rust Code of Conduct and ask that you do too.
Comments
  • release version 0.2.0

    release version 0.2.0

    Includes all changes since a59b8ee82:

    7e76f9d Use hashbrown cfc9c7d Reduce the number of reallocations for PackageStringTable::data 0ecb11d Avoid extra HashMap in PackageStringTable 12660a7 Change assert to debug_assert in PackageStringTable cf41f62 Handle entries with differing columns cf21bfb Update object dependency 025395b update repository links and readme

    opened by davidtwco 5
  • Is the relocation handling unnecessary?

    Is the relocation handling unnecessary?

    My understanding is that DWO files are generated such that relocations aren't needed, and they're definitely not needed for executable files.

    I tried changing DwpReader to be just EndianSlice, and the resulting DWP for my test case was unchanged.

    opened by philipc 4
  • Performance improvements

    Performance improvements

    This approximately doubles the speed for a large test case (thorin -e llvm-project/build/bin/opt -o opt.dwp). (Interestingly, both llvm-dwp and my Ubuntu's dwp fail to handle this test case.)

    Most of the time is still spend in PackageStringTable::remap_str_offsets_section. I think the main way we could further improve that is by using the strings in PackageStringTable::data as the hash keys instead of allocating a Vec for each one, but I don't know if that is possible with existing HashMap APIs. Maybe custom allocators will allow this.

    I also tried radix_trie, but it is much slower.

    opened by philipc 1
  • memmap2 on wasm

    memmap2 on wasm

    Wasm doesn't have mmap support. Rustc has a wrapper that uses Vec<u8> instead of memmap2::Mmap on wasm: https://github.com/rust-lang/rust/blob/2a9e0831d6603d87220cedd1b1293e2eb82ef55c/compiler/rustc_data_structures/src/memmap.rs#L7

    Originally posted by @bjorn3 in https://github.com/davidtwco/thorin/pull/5#discussion_r762118577

    opened by davidtwco 1
  • next_pow2

    next_pow2

    https://github.com/davidtwco/rust-dwp/blob/874db16e69573d5872f34b80372a5e71381c2601/src/main.rs#L1386 This can probably be replaced with u32::next_power_of_two.

    opened by philipc 1
  •  Does number_of_columns allow for differences in entries?

    Does number_of_columns allow for differences in entries?

    https://github.com/davidtwco/rust-dwp/blob/874db16e69573d5872f34b80372a5e71381c2601/src/main.rs#L1034-L1036

    The assert here checks that all entries have the same number of columns, but what if they are different while still having the same number?

    opened by philipc 1
  • Upgrade hashbrown, indexmap, and object

    Upgrade hashbrown, indexmap, and object

    I'm working on a PR to update rust-lang/rust to hashbrown 0.12 everywhere, and thorin is the last holdout. We only need new versions of indexmap and object here, as well as upgrading the direct hashbrown dependency.

    opened by cuviper 0
  • Update `object` dependency to 0.28

    Update `object` dependency to 0.28

    This PR simply updates the object dependency from 0.27 to 0.28.

    rust_codegen_ssa now depends on 0.28 since https://github.com/rust-lang/rust/pull/90001. Whenever thorin is updated in rustc in the future, removing the use of 0.27 would deduplicate them.

    opened by lqd 0
  • release version 0.1.1 and add per-crate readme

    release version 0.1.1 and add per-crate readme

    Instead of using the same README for both the library and binary crates, use a separate specific README for each. Upgrade the version to 0.1.1, including nagisa's patches (#10).

    opened by davidtwco 0
  • Review…

    Review…

    The code looks good, at least given my understanding of DWARF which is definitely not super extensive.

    I ended up with just a couple style nits and whatnot, best reviewed commit-by-commit.

    opened by nagisa 0
  • Testing, repeated sections and diagnostics

    Testing, repeated sections and diagnostics

    Fixes #1.

    • Adds test suite, based on llvm-dwp's test suite using lit
    • Various refactorings and tidying up
    • Support for repeated sections (e.g. multiple .debug_types.dwo)
    • Confirm that each index entry has same columns
    • Record loclist/rnglist contributions in type unit indices (tested by llvm-dwp but not in the DWARF 5 standard)
    • Add incompatible package version diagnostic
    • Add duplicate split compilation unit diagnostic
    • Probably some other things I've forgotten, check the commits.
    opened by davidtwco 0
Owner
The Rust Programming Language
The Rust Programming Language
Serde support for n-dimensional arrays from self-describing formats

serde-ndim Overview This crate provides a way to serialize and deserialize arrays of arbitrary dimensionality from self-described formats such as JSON

Ingvar Stepanyan 5 Apr 3, 2023
A typemap for a set of known types optionally without heap allocation, and supporting iterating by traits

fixed_typemap docs.rs GitHub Sponsors Implements typemaps that support a lot of extra funcctionality using procedural macros. docs.rs has a lot more t

Austin Hicks 2 Dec 27, 2021
Blazing fast, memory safe & modern Linux package manager written in Rust.

paket Blazing fast, memory safe & modern Linux package manager written in Rust. Roadmap Version: 0.1 Paket.toml file parsing. (#1, #2) CLI handling (p

null 4 Oct 19, 2023
An extension to Rust std for beginner exercises

note: this is for practicing rust only, do not use this in actual production programs! really, don't. [dependencies] simple-std = "0.1.1" simple-std i

nils 2 Jan 2, 2022
HP++: A Hazard Pointers Extension for Better Applicability

HP++: A Hazard Pointers Extension for Better Applicability This is an implementation of HP++, a safe memory reclamation scheme proposed in Jaehwang Ju

KAIST Concurrency & Parallelism Laboratory 3 May 10, 2023
Elton is a benchmark utility written in rust aimed to be used to benchmark HTTP calls.

Elton Elton is an HTTP Benchmark utility with options to be used within an HTTP interface. Installation Elton is currently available via Docker or by

Emil Priver 5 Sep 22, 2023
A Rust crate providing utility functions and macros.

介绍 此库提供四类功能:异常处理、http post收发对象、格式转换、语法糖。 在 Cargo.toml 里添加如下依赖项 [dependencies.xuanmi_base_support] git = "https://github.com/taiyi-research-institute/x

null 17 Mar 22, 2023
A query-building & utility crate for SurrealDB and its SQL querying language that aims to be simple

Surreal simple querybuilder A simple query-builder for the Surreal Query Language, for SurrealDB. Aims at being simple to use and not too verbose firs

Thibault H 11 Dec 30, 2022
The utility is designed to check the availability of peers and automatically update them in the Yggdrasil configuration file, as well as using the admin API - addPeer method.

Yggrasil network peers checker / updater The utility is designed to check the availability of peers and automatically update them in the Yggdrasil con

null 6 Dec 25, 2022
Modern Rust utility library delivering modularity, performance & extras; or simply Rust version of Lodash

Lorust - API Documentation Lorust is the Rust version of Lodash, which is a modern Javascript utilty library delivering modularity, performance & extr

Imamuzzaki Abu Salam 2 Jul 9, 2023
A small utility for modifying ELF shared library loading order.

elfpromote A small utility for modifying ELF shared library loading order. Usage $ cargo install elfpromote $ ldd blueboat_server linux-vdso.s

Heyang Zhou 5 Jul 21, 2022
An i386 operation system written in pure rust for fun and no profit.

OrustS An i386 operation system written in pure rust (for fun and no profit). This operation system is under active developing. Checklist implement a

M4tsuri 10 Aug 12, 2022
rusty-riscy is a performance testing and system resource monitoring tool written in Rust to benchmark RISC-V processors.

rusty-riscy rusty-riscy is a performance testing and system resource monitoring tool written in Rust to benchmark RISC-V processors. Objectives To cre

Suhas KV 4 May 3, 2022
A set of Zero Knowledge modules, written in Rust and designed to be used in other system programming environments.

Zerokit A set of Zero Knowledge modules, written in Rust and designed to be used in other system programming environments. Initial scope Focus on RLN

vac 44 Dec 27, 2022
An experimental, well-documented and expansion-ready virtual machine written in Rust.

Notice ivm is undergoing a complete refactor. ivm ivm is a rich, well-documented virtual machine written in Rust. Pros Documentation everywhere. Every

Imajin 4 Jul 9, 2022
Secure mTLS and gRPC backed runtime daemon. Alternative to systemd. Written in Rust.

Auraed A runtime daemon written in Rust. Designed to run as pid 1 mTLS backed gRPC API over unix domain socket Run executables Run containers Run virt

Aurae Runtime 57 Dec 22, 2022
A relatively simple puzzle generator application written in Rust and used via Javascript

Puzzlip Basic Overview This is a relatively simple puzzle generator application written in Rust and used via Javascript in https://puzzlip.com. If you

Nenad 5 Dec 7, 2022
🚀simple server that returns error codes with their respective messages and debug information, written in rust 🦀

ErrorServer ?? A simple & lightweight server that returns a HTML page of the error code with its respective message and debug information, written in

Jakob 2 Dec 15, 2022
Grimsby is an Erlang Port written in Rust that can close its standard input while retaining standard output (and error)

Grimsby An Erlang Port provides the basic mechanism for communication from Erlang with the external world. From the Ports and Port Drivers: Erlang Ref

Peter Morgan 5 May 29, 2023