Hey! I am trying to use XDP in driver mode but it doesn't work:
Caused by:
0: netlink error while attaching XDP program
1: Invalid argument (os error 22)
I've been testing on Amazon Linux with two different kernels and ENA drivers:
oligavin@dev] uname -r
5.4.209-129.367.amzn2.x86_64
oligavin@dev] modinfo ena | grep ^version
version: 2.7.4g
[oligavin@ip-10-0-115-159]~% uname -r
5.10.118-111.515.amzn2.x86_64
[oligavin@ip-10-0-115-159]~% modinfo ena | grep ^version
version: 2.7.1g
I have successfully been using AF_XDP (via another library) on 5.10 so I think this is an AYA issue.
I would be happy to attempt a fix if you can guide me in the right direction :)
Steps to reproduce
Running the generated XDP program:
oligavin@dev] cargo generate https://github.com/aya-rs/aya-template
โ ๏ธ Favorite `https://github.com/aya-rs/aya-template` not found in config, using it as a git repository: https://github.com/aya-rs/aya-template
๐คท Project Name: test_xdp_driver_mode
โ ๏ธ Renaming project called `test_xdp_driver_mode` to `test-xdp-driver-mode`...
๐ง Destination: /tmp/testing/test-xdp-driver-mode ...
๐ง project-name: test-xdp-driver-mode ...
๐ง Generating template ...
โ ๐คท Which type of eBPF program? ยท xdp
[ 1/34] Done: .cargo/config.toml
[ 2/34] Done: .cargo [ 3/34] Done: .gitignore [ 4/34] Done: .vim/coc-settings.json [ 5/34] Done: .vim
[ 6/34] Done: .vscode/settings.json
[ 7/34] Done: .vscode
[ 8/34] Done: Cargo.toml
[ 9/34] Done: README.md
[10/34] Done: xtask/Cargo.toml
[11/34] Done: xtask/src/build_ebpf.rs
[12/34] Done: xtask/src/main.rs
[13/34] Done: xtask/src/run.rs
[14/34] Done: xtask/src
[15/34] Done: xtask
[16/34] Done: test-xdp-driver-mode/Cargo.toml
[17/34] Done: test-xdp-driver-mode/src/main.rs
[18/34] Done: test-xdp-driver-mode/src
[19/34] Done: test-xdp-driver-mode
[20/34] Done: test-xdp-driver-mode-common/Cargo.toml
[21/34] Done: test-xdp-driver-mode-common/src/lib.rs
[22/34] Done: test-xdp-driver-mode-common/src
[23/34] Done: test-xdp-driver-mode-common
[24/34] Done: test-xdp-driver-mode-ebpf/.cargo/config.toml
[25/34] Done: test-xdp-driver-mode-ebpf/.cargo
[26/34] Done: test-xdp-driver-mode-ebpf/.vim/coc-settings.json
[27/34] Done: test-xdp-driver-mode-ebpf/.vim
[28/34] Done: test-xdp-driver-mode-ebpf/.vscode/settings.json
[29/34] Done: test-xdp-driver-mode-ebpf/.vscode
[30/34] Done: test-xdp-driver-mode-ebpf/Cargo.toml
[31/34] Done: test-xdp-driver-mode-ebpf/rust-toolchain.toml
[32/34] Done: test-xdp-driver-mode-ebpf/src/main.rs
[33/34] Done: test-xdp-driver-mode-ebpf/src
[34/34] Done: test-xdp-driver-mode-ebpf
๐ง Moving generated files into: `/tmp/testing/test-xdp-driver-mode`...
๐ก Initializing a fresh Git repository
โจ Done! New project created /tmp/testing/test-xdp-driver-mode
oligavin@] cargo xtask run
Finished dev [unoptimized + debuginfo] target(s) in 0.03s
Running `target/debug/xtask run`
Fresh unicode-ident v1.0.5
Fresh proc-macro2 v1.0.47
Fresh core v0.0.0 (/local/home/oligavin/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core)
Fresh rustc-std-workspace-core v1.99.0 (/local/home/oligavin/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/rustc-std-workspace-core)
Fresh quote v1.0.21
Fresh compiler_builtins v0.1.82
Fresh rustversion v1.0.9
Fresh syn v1.0.105
Fresh aya-bpf-cty v0.2.1 (https://github.com/aya-rs/aya?branch=main#16b029ed)
Fresh test-xdp-driver-mode-common v0.1.0 (/tmp/testing/test-xdp-driver-mode/test-xdp-driver-mode-common)
Fresh num_enum_derive v0.5.7
Fresh aya-bpf-macros v0.1.0 (https://github.com/aya-rs/aya?branch=main#16b029ed)
Fresh aya-bpf-bindings v0.1.0 (https://github.com/aya-rs/aya?branch=main#16b029ed)
Fresh num_enum v0.5.7
Fresh aya-bpf v0.1.0 (https://github.com/aya-rs/aya?branch=main#16b029ed)
Fresh aya-log-common v0.1.13 (https://github.com/aya-rs/aya?branch=main#16b029ed)
Fresh aya-log-parser v0.1.11-dev.0 (https://github.com/aya-rs/aya?branch=main#16b029ed)
Fresh aya-log-ebpf-macros v0.1.0 (https://github.com/aya-rs/aya?branch=main#16b029ed)
Fresh aya-log-ebpf v0.1.0 (https://github.com/aya-rs/aya?branch=main#16b029ed)
Fresh test-xdp-driver-mode-ebpf v0.1.0 (/tmp/testing/test-xdp-driver-mode/test-xdp-driver-mode-ebpf)
Finished dev [optimized] target(s) in 0.39s
Finished dev [unoptimized + debuginfo] target(s) in 0.27s
Error: failed to attach the XDP program with default flags - try changing XdpFlags::default() to XdpFlags::SKB_MODE
Caused by:
0: netlink error while attaching XDP program
1: Invalid argument (os error 22)
Modifying the XDP program to use SKB mode works
test-xdp-driver-mode/src/main.rs
:
- program.attach(&opt.iface, XdpFlags::default())
+ program.attach(&opt.iface, XdpFlags::SKB_MODE)
oligavin@dev] RUST_LOG=DEBUG cargo xtask run
Finished dev [unoptimized + debuginfo] target(s) in 0.03s
Running `target/debug/xtask run`
Fresh unicode-ident v1.0.5
Fresh proc-macro2 v1.0.47
Fresh core v0.0.0 (/local/home/oligavin/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core)
Fresh rustc-std-workspace-core v1.99.0 (/local/home/oligavin/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/rustc-std-workspace-core)
Fresh quote v1.0.21
Fresh compiler_builtins v0.1.82
Fresh rustversion v1.0.9
Fresh syn v1.0.105
Fresh aya-bpf-cty v0.2.1 (https://github.com/aya-rs/aya?branch=main#16b029ed)
Fresh test-xdp-driver-mode-common v0.1.0 (/tmp/testing/test-xdp-driver-mode/test-xdp-driver-mode-common)
Fresh num_enum_derive v0.5.7
Fresh aya-bpf-macros v0.1.0 (https://github.com/aya-rs/aya?branch=main#16b029ed)
Fresh aya-bpf-bindings v0.1.0 (https://github.com/aya-rs/aya?branch=main#16b029ed)
Fresh num_enum v0.5.7
Fresh aya-bpf v0.1.0 (https://github.com/aya-rs/aya?branch=main#16b029ed)
Fresh aya-log-common v0.1.13 (https://github.com/aya-rs/aya?branch=main#16b029ed)
Fresh aya-log-parser v0.1.11-dev.0 (https://github.com/aya-rs/aya?branch=main#16b029ed)
Fresh aya-log-ebpf-macros v0.1.0 (https://github.com/aya-rs/aya?branch=main#16b029ed)
Fresh aya-log-ebpf v0.1.0 (https://github.com/aya-rs/aya?branch=main#16b029ed)
Fresh test-xdp-driver-mode-ebpf v0.1.0 (/tmp/testing/test-xdp-driver-mode/test-xdp-driver-mode-ebpf)
Finished dev [optimized] target(s) in 0.38s
Finished dev [unoptimized + debuginfo] target(s) in 0.27s
[2022-12-06T20:34:41Z DEBUG aya::bpf] [FEAT PROBE] BPF program name support: true
[2022-12-06T20:34:41Z DEBUG aya::bpf] [FEAT PROBE] BTF support: true
[2022-12-06T20:34:41Z DEBUG aya::bpf] [FEAT PROBE] BTF func support: true
[2022-12-06T20:34:41Z DEBUG aya::bpf] [FEAT PROBE] BTF global func support: false
[2022-12-06T20:34:41Z DEBUG aya::bpf] [FEAT PROBE] BTF var and datasec support: true
[2022-12-06T20:34:41Z DEBUG aya::bpf] [FEAT PROBE] BTF float support: false
[2022-12-06T20:34:41Z DEBUG aya::bpf] [FEAT PROBE] BTF decl_tag support: false
[2022-12-06T20:34:41Z DEBUG aya::bpf] [FEAT PROBE] BTF type_tag support: false
[2022-12-06T20:34:41Z DEBUG aya::obj::relocation] relocating program test_xdp_driver_mode function test_xdp_driver_mode
[2022-12-06T20:34:41Z DEBUG aya::obj::relocation] finished relocating program test_xdp_driver_mode function test_xdp_driver_mode
[2022-12-06T20:34:41Z INFO test_xdp_driver_mode] Waiting for Ctrl-C...
[2022-12-06T20:34:41Z INFO test_xdp_driver_mode] received a packet
[2022-12-06T20:34:41Z INFO test_xdp_driver_mode] received a packet
[2022-12-06T20:34:41Z INFO test_xdp_driver_mode] received a packet
Modifying the XDP program to use DRV mode does not work
test-xdp-driver-mode/src/main.rs
:
- program.attach(&opt.iface, XdpFlags::default())
+ program.attach(&opt.iface, XdpFlags::DRV_MODE)
Same error as I started with:
Error: failed to attach the XDP program with default flags - try changing XdpFlags::default() to XdpFlags::SKB_MODE
Caused by:
0: netlink error while attaching XDP program
1: Invalid argument (os error 22)