Windows Linked Lists in idiomatic Rust (LIST_ENTRY, SINGLE_LIST_ENTRY)

Related tags

Cryptography nt-list
Overview

nt-list

crates.io docs.rs license: MIT OR Apache-2.0

by Colin Finck <[email protected]>

Provides compatible, type-safe, and idiomatic Rust implementations of the Windows NT Linked Lists, known as LIST_ENTRY and SINGLE_LIST_ENTRY.

In Action


Debugging a Rust application in WinDbg and using the !list extension to traverse a doubly linked list. A testament to the compatibility between nt-list and LIST_ENTRY.

Details

Singly and doubly linked lists of this format are fundamental data structures widely used in Windows itself and in drivers written for Windows. In the case of a doubly linked list, Windows defines a LIST_ENTRY structure with forward and backward pointers to other LIST_ENTRY structures. LIST_ENTRY is then embedded into your own element structure. Check the relevant Microsoft documentation for more details on linked lists in Windows.

This design exhibits several properties that differ from textbook linked list implementations:

  • A single element can be part of multiple lists (by having multiple LIST_ENTRY fields).
  • YOU are responsible for pushing only elements of the same type to a list. Without any type safety, the C/C++ compiler cannot prevent you from adding differently typed elements to the same list.
  • Links point to the LIST_ENTRY field of an element and not to the element itself. YOU need to retrieve the corresponding element structure using CONTAINING_RECORD, and YOU are responsible for all parameters passed to that macro.

The nt-list crate introduces type safety for these lists, taking away some responsibility from the user and moving it to the compiler. Additionally, it offers an idiomatic Rust interface similar to that of LinkedList and Vec.

Example

Creating a linked list with nt-list boils down to these three steps:

  1. You define an empty enum to identify your list (for type safety when pushing elements), and derive either NtList (doubly linked list) or NtSingleList (singly linked list).
  2. You define your element structure, declare an entry as #[boxed] if desired, and derive NtListElement.
  3. You call new of the respective list implementation with the element structure and empty enum as type parameters.

All of this taken together looks like:

#[derive(NtSingleList)]
enum MyList {}

#[derive(Default, NtListElement)]
#[repr(C)]
struct MyElement {
    #[boxed]
    entry: NtSingleListEntry<Self, MyList>,
    value: i32,
}

fn test() {
    let mut list = NtBoxingSingleListHead::<MyElement, MyList>::new();

    list.push_back(MyElement {
        value: 42,
        ..Default::default()
    });

    for element in list.iter() {
        println!("{}", element.value);
    }
}

Check the module-level documentation of list for a doubly linked list example.

no_std support

The crate is no_std-compatible and therefore usable from firmware-level code up to user-mode applications.

To support heap allocations in NtBoxingListHead and NtBoxingSingleListHead, the crate depends on the alloc library. If you want to use the crate in a pure no_std environment without heap allocations, include it with default-features = false to disable the default alloc feature.

License

This crate is licensed under either of

at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Comments
  • Calling `MaybeUninit::uninit().assume_init()` is UB

    Calling `MaybeUninit::uninit().assume_init()` is UB

    Hi,

    I saw your talk at EuroConf 2022 and was ~~horrified~~ surprised to see that you were using MaybeUninit::uninit().assume_init(), this is completely UB and is equivalent to using std::mem::uninitialized whish was deprecated for being UB.

    From the std docs:

    The compiler, in general, assumes that a variable is properly initialized according to the requirements of the variable’s type. For example, a variable of reference type must be aligned and non-null. This is an invariant that must always be upheld, even in unsafe code. And no matter whether that reference ever gets used to access memory.

    For more information's you can also read Ralf blog post: "What The Hardware Does" is not What Your Program Does: Uninitialized Memory.

    The offended code

    https://github.com/ColinFinck/nt-list/blob/4406c77cd00ea9114a6e5bfe5bf56fc026d6cd50/nt-list/src/list/base.rs#L49-L50 https://github.com/ColinFinck/nt-list/blob/4406c77cd00ea9114a6e5bfe5bf56fc026d6cd50/nt-list/src/single_list/base.rs#L263 https://github.com/ColinFinck/nt-list/blob/4406c77cd00ea9114a6e5bfe5bf56fc026d6cd50/nt-list/src/list/boxing.rs#L49-L50

    opened by Urgau 2
  • UB in safe code caused by unsafe code trusting `NtListElement::offset`

    UB in safe code caused by unsafe code trusting `NtListElement::offset`

    It's possible to cause UB in safe code, by implementing NtListElement by hand and providing a wrong offset. Here is a MRE:

    use nt_list::single_list::{NtSingleList, NtBoxingSingleListHead, NtSingleListEntry};
    use nt_list::{NtListElement, NtBoxedListElement};
    
    #[derive(NtSingleList)]
    enum MyList {}
    
    struct MyElement(NtSingleListEntry<Self, MyList>); // Not even `repr(C)` :)
    
    impl NtListElement<MyList> for MyElement {
        fn offset() -> usize { 1000000 }
    }
    
    impl NtBoxedListElement for MyElement { type L = MyList; }
    
    fn main() {
        let mut list = NtBoxingSingleListHead::new();
    
        list.push_front(MyElement(Default::default()));
    }
    
    ; cargo run
       Compiling nt-list-ub v0.1.0 (/home/waffle/projects/repos/nt-list-ub)
        Finished dev [unoptimized + debuginfo] target(s) in 1.06s
         Running `target/debug/nt-list-ub`
    zsh: segmentation fault (core dumped)  cargo run
    

    At the very least traits that unsafe code uses should be unsafe.

    opened by WaffleLapkin 1
  • Remove all int2ptr and ptr2int casts

    Remove all int2ptr and ptr2int casts

    int2ptr casts obscure the code and make it quite hard to analyze with tools like miri. It doesn't seem like you really need them (an artifact of C past?), so this PR removes them (replacing with pointer offsets).

    opened by WaffleLapkin 1
  • "alloc" feature is not mentioned in the generated documentation at docs.rs

    I had added an extra line to mention the "alloc" feature for all symbols of the boxing module: https://github.com/ColinFinck/nt-list/blob/c9f48d45dde2ce42ef5ea460265d14a1c2e6b023/nt-list/src/list/mod.rs#L69-L71

    However, this wasn't picked up: https://docs.rs/nt-list/0.1.0/nt_list/list/struct.NtBoxingListHead.html

    This should be fixed once I figure out why this didn't work and how to fix it.

    opened by ColinFinck 0
  • This library violates stacked borrows (TM) rules

    This library violates stacked borrows (TM) rules

    Even if you use the library as intended, with derives and fully safe code, it still causes UB, according to miri and SB:

    use nt_list::single_list::{NtSingleList, NtBoxingSingleListHead, NtSingleListEntry};
    use nt_list::NtListElement;
    
    #[derive(NtSingleList)]
    enum MyList {}
    
    #[derive(NtListElement)]
    #[repr(C)]
    struct MyElement { 
        #[boxed] 
        _entry: NtSingleListEntry<Self, MyList>
    }
    
    fn main() {
        let mut list = NtBoxingSingleListHead::new();
    
        list.push_front(MyElement { _entry: Default::default() });
    }
    
    ; cargo miri run
    Preparing a sysroot for Miri (target: x86_64-unknown-linux-gnu)... done
        Finished dev [unoptimized + debuginfo] target(s) in 0.02s
         Running `/home/waffle/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo-miri runner target/miri/x86_64-unknown-linux-gnu/debug/nt-list-ub`
    error: Undefined Behavior: trying to retag from <3005> for Unique permission at alloc1496[0x0], but that tag only grants SharedReadOnly permission for this location
       --> /home/waffle/projects/repos/nt-list/nt-list/src/single_list/base.rs:272:18
        |
    272 |         unsafe { &mut *(self.element_ptr() as *mut E) }
        |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |                  |
        |                  trying to retag from <3005> for Unique permission at alloc1496[0x0], but that tag only grants SharedReadOnly permission for this location
        |                  this error occurs as part of retag at alloc1496[0x0..0x8]
        |
        = help: this indicates a potential bug in the program: it performed an invalid operation, but the Stacked Borrows rules it violated are still experimental
        = help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md for further information
    help: <3005> was created by a SharedReadOnly retag at offsets [0x0..0x8]
       --> src/main.rs:18:1
        |
    18  | }
        | ^
        = note: BACKTRACE:
        = note: inside `nt_list::single_list::NtSingleListEntry::<MyElement, MyList>::containing_record_mut` at /home/waffle/projects/repos/nt-list/nt-list/src/single_list/base.rs:272:18
        = note: inside `<nt_list::single_list::IterMut<'_, MyElement, MyList> as std::iter::Iterator>::next` at /home/waffle/projects/repos/nt-list/nt-list/src/single_list/base.rs:231:31
        = note: inside `<nt_list::single_list::NtBoxingSingleListHead<MyElement, MyList> as std::ops::Drop>::drop` at /home/waffle/projects/repos/nt-list/nt-list/src/single_list/boxing.rs:179:24
        = note: inside `std::ptr::drop_in_place::<nt_list::single_list::NtBoxingSingleListHead<MyElement, MyList>> - shim(Some(nt_list::single_list::NtBoxingSingleListHead<MyElement, MyList>))` at /home/waffle/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:491:1
    note: inside `main` at src/main.rs:18:1
       --> src/main.rs:18:1
        |
    18  | }
        | ^
    
    note: some details are omitted, run with `MIRIFLAGS=-Zmiri-backtrace=full` for a verbose backtrace
    
    error: aborting due to previous error
    

    I'm not sure what exactly is the problem, but the way this library derives pointers is incorrect according to the rules (TM). Note that tests don't pass under miri either:

    ; cargo miri test   
    Preparing a sysroot for Miri (target: x86_64-unknown-linux-gnu)... done
        Finished test [unoptimized + debuginfo] target(s) in 0.02s
         Running unittests src/lib.rs (target/miri/x86_64-unknown-linux-gnu/debug/deps/nt_list-9ea035d0fbb79076)
    
    running 11 tests
    test list::boxing::tests::test_append ... error: Undefined Behavior: trying to retag from <198784> for SharedReadOnly permission at alloc80736[0x0], but that tag does not exist in the borrow stack for this location
       --> nt-list/src/list/base.rs:461:18
        |
    461 |         unsafe { &*self.element_ptr() }
        |                  ^^^^^^^^^^^^^^^^^^^^
        |                  |
        |                  trying to retag from <198784> for SharedReadOnly permission at alloc80736[0x0], but that tag does not exist in the borrow stack for this location
        |                  this error occurs as part of retag at alloc80736[0x0..0x18]
        |
        = help: this indicates a potential bug in the program: it performed an invalid operation, but the Stacked Borrows rules it violated are still experimental
        = help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md for further information
    help: <198784> was created by a SharedReadOnly retag at offsets [0x8..0x18]
       --> nt-list/src/list/base.rs:469:19
        |
    469 |         let ptr = self as *const Self;
        |                   ^^^^
        = note: BACKTRACE:
        = note: inside `list::base::NtListEntry::<list::boxing::tests::MyElement, list::boxing::tests::MyList>::containing_record` at nt-list/src/list/base.rs:461:18
    note: inside `<list::base::Iter<'_, list::boxing::tests::MyElement, list::boxing::tests::MyList> as core::iter::Iterator>::next` at nt-list/src/list/base.rs:299:31
       --> nt-list/src/list/base.rs:299:31
        |
    299 |                 let element = (*self.flink).containing_record();
        |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        = note: inside `<list::base::Iter<'_, list::boxing::tests::MyElement, list::boxing::tests::MyList> as core::iter::Iterator>::fold::<usize, [closure@<list::base::Iter<'_, list::boxing::tests::MyElement, list::boxing::tests::MyList> as core::iter::Iterator>::count::{closure#0}]>` at /home/waffle/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:2413:29
        = note: inside `<list::base::Iter<'_, list::boxing::tests::MyElement, list::boxing::tests::MyList> as core::iter::Iterator>::count` at /home/waffle/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:256:9
    note: inside `list::base::NtListHead::<list::boxing::tests::MyElement, list::boxing::tests::MyList>::len` at nt-list/src/list/base.rs:174:9
       --> nt-list/src/list/base.rs:174:9
        |
    174 |         self.iter().count()
        |         ^^^^^^^^^^^^^^^^^^^
    note: inside `list::boxing::NtBoxingListHead::<list::boxing::tests::MyElement, list::boxing::tests::MyList>::len` at nt-list/src/list/boxing.rs:162:18
       --> nt-list/src/list/boxing.rs:162:18
        |
    162 |         unsafe { self.inner().len() }
        |                  ^^^^^^^^^^^^^^^^^^
    note: inside `list::boxing::tests::test_append` at nt-list/src/list/boxing.rs:307:20
       --> nt-list/src/list/boxing.rs:307:20
        |
    307 |         assert_eq!(list1.as_ref().len(), 20);
        |                    ^^^^^^^^^^^^^^^^^^^^
    note: inside closure at nt-list/src/list/boxing.rs:293:5
       --> nt-list/src/list/boxing.rs:293:5
        |
    292 |       #[test]
        |       ------- in this procedural macro expansion
    293 | /     fn test_append() {
    294 | |         // Append two lists of equal size.
    295 | |         moveit! {
    296 | |             let mut list1 = NtBoxingListHead::<MyElement, MyList>::new();
    ...   |
    326 | |         verify_all_links(list3.as_ref().inner());
    327 | |     }
        | |_____^
        = note: this error originates in the attribute macro `test` (in Nightly builds, run with -Z macro-backtrace for more info)
    
    note: some details are omitted, run with `MIRIFLAGS=-Zmiri-backtrace=full` for a verbose backtrace
    
    error: aborting due to previous error
    
    error: test failed, to rerun pass `-p nt-list --lib`
    
    Caused by:
      process didn't exit successfully: `/home/waffle/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo-miri runner /home/waffle/projects/repos/nt-list/target/miri/x86_64-unknown-linux-gnu/debug/deps/nt_list-9ea035d0fbb79076` (exit status: 1)
    

    I would highly recommend running miri in CI and while developing, to avoid issues like this in the future :)

    opened by WaffleLapkin 0
Owner
Colin Finck
Long-time ReactOS developer and Rust enthusiast, bringing both worlds together
Colin Finck
A CLI application which allows you to archive Urbit channels and all linked content in them.

The Urbit Content Archiver is a small CLI application that exports channels from your Urbit ship and auto-downloads any directly linked content locall

Robert Kornacki 33 Sep 25, 2022
Doubly-linked list that stores key-node pairs.

key-node-list Doubly-linked list that stores key-node pairs. KeyNodeList is a doubly-linked list, it uses a hash map to maintain correspondence betwee

MaxXing 3 May 4, 2022
Experiments on blockchain technology (also known as Hashed & Zero-trust Verifiable Linked List)

AngeloChain Experiments on blockchain technology (also known as Hashed & Zero-trust Verifiable Linked List) ⚠️ Before We Get Started Before we get sta

Angelo 1 Jan 20, 2022
MimiRust - Hacking the Windows operating system to hand us the keys to the kingdom with Rust

MimiRust - Hacking the Windows operating system to hand us the keys to the kingdom with Rust

null 160 Dec 29, 2022
A Minimal Windows SDK.

Minimal Windows 10 SDK Installs only the necessary Windows 10 .lib files to save you having to download the full Visual Studio package. You can either

null 58 Dec 18, 2022
MyCitadel Wallet app for Linux, Windows & MacOS desktop made with GTK+

MyCitadel Desktop Bitcoin, Lightning and RGB wallet MyCitadel is a wallet for bitcoin, digital assets and bitcoin finance (#BiFi) smart contracts. It

My Citadel 88 Jan 2, 2023
Open-source tool to enforce privacy & security best-practices on Windows and macOS, because privacy is sexy 🍑🍆

privacy-sexy Open-source tool to enforce privacy & security best-practices on Windows and MacOs, because privacy is sexy ?? ?? privacy-sexy is a data-

Subconscious Compute 3 Oct 20, 2022
Nostr Vanity Address Generator (Windows, Linux and macOS)

Nostr Vanity Address Generator CLI tool to generate vanity addresses for Nostr Usage Download the latest release built by GitHub CI from the releases

Chawye Hsu 7 Mar 1, 2023
Package used by the cosmos-rust-interface. Makes direct use of cosmos-rust.

Package used by the cosmos-rust-interface. Makes direct use of cosmos-rust (cosmos‑sdk‑proto, osmosis-proto, cosmrs).

Philipp 4 Dec 26, 2022
Rust project for working with ETH - Ethereum transactions with Rust on Ganache and also deploy smart contracts :)

Just a test project to work with Ethereum but using Rust. I'm using plain Rust here, not Foundry. In future we will use Foundry. Hope you're already f

Akhil Sharma 2 Dec 20, 2022
An open source Rust high performance cryptocurrency trading API with support for multiple exchanges and language wrappers. written in rust(🦀) with ❤️

Les.rs - Rust Cryptocurrency Exchange Library An open source Rust high performance cryptocurrency trading API with support for multiple exchanges and

Crabby AI 4 Jan 9, 2023
Simple node and rust script to achieve an easy to use bridge between rust and node.js

Node-Rust Bridge Simple rust and node.js script to achieve a bridge between them. Only 1 bridge can be initialized per rust program. But node.js can h

Pure 5 Apr 30, 2023
A Rust library for working with Bitcoin SV

Rust-SV A library to build Bitcoin SV applications in Rust. Documentation Features P2P protocol messages (construction and serialization) Address enco

Brenton Gunning 51 Oct 13, 2022
Coinbase pro client for Rust

Coinbase pro client for Rust Supports SYNC/ASYNC/Websocket-feed data support Features private and public API sync and async support websocket-feed sup

null 126 Dec 30, 2022
Custom Ethereum vanity address generator made in Rust

ethaddrgen Custom Ethereum address generator Get a shiny ethereum address and stand out from the crowd! Disclaimer: Do not use the private key shown i

Jakub Hlusička 153 Dec 27, 2022
The new, performant, and simplified version of Holochain on Rust (sometimes called Holochain RSM for Refactored State Model)

Holochain License: This repository contains the core Holochain libraries and binaries. This is the most recent and well maintained version of Holochai

Holochain 741 Jan 5, 2023
IBC modules and relayer - Formal specifications and Rust implementation

ibc-rs Rust implementation of the Inter-Blockchain Communication (IBC) protocol. This project comprises primarily four crates: The ibc crate defines t

Informal Systems 296 Dec 31, 2022
A Rust implementation of BIP-0039

bip39-rs A Rust implementation of BIP0039 Changes See the changelog file, or the Github releases for specific tags. Documentation Add bip39 to your Ca

Infincia LLC 49 Dec 9, 2022
Rust Ethereum 2.0 Client

Lighthouse: Ethereum 2.0 An open-source Ethereum 2.0 client, written in Rust and maintained by Sigma Prime. Documentation Overview Lighthouse is: Read

Sigma Prime 2.1k Jan 6, 2023