This adds the IEEE 802.15.4 frame representation. Still a work in progress and interested to know what you think about this.

I really would like to add 6LowPAN as well, however I'm not sure where to put this in smoltcp, since 6LowPAN is kind of weird.

Hi

This is an attempt at creating a DHCP client for IPv4. Please give feedback.

Cc: @phil-opp, this PR is uses the work merged in #75.

I've taken a completely different approach compared to #63. UDP sockets don't allow us to send and receive on unspecified/broadcast IPv4 addresses. Therefore IPv4/UDP is parsed/serialized on top of a RawSocket.

~~There are still two problems with RawSocketBuffer:~~

• ~~How do I let the user provide storage? I have tried but failed to accept that as parameters in Client::new().~~ I've found the proper lifetime relation by now.
• ~~The RawSocket is getting stuck with Error::Exhausted when contig_window is too small for the DHCP egress packet. Somehow it appears to behave like sent packets don't get dequeued from the tx_buffer. Is that known? What is my code doing wrong regarding RawSocket?~~ See #187

Problem: smoltcp doesn't support raw sockets, low-level utilities like ping and custom IP protocols are impossible to implement.

Solution: implement raw sockets, provide an example of their usage.

Changes introduced by this pull request:

• socket::udp::SocketBuffer has been factored out as generic common::RingBuffer.
• socket::raw module has been implemented.
• iface::EthernetInterface has been changed to support raw sockets, giving them priority over the built-in handlers.
• The ping example has been implemented.

Status: the ping example works as expected.

TODOs:

• Currently, smoltcp uses iteration over the list of sockets for packet dispatch, which is highly inefficient on the platforms where standard containers like maps and hash tables are available. The existing SocketSet container should be complimented with lookup tables for raw and tcp/udp sockets.

Other:

• I'm using the current version of rustfmt for my code, sometimes it looks weird.
• I've used some recent stable Rust features like field shorthands and pub(crate), so I've bumped Rust version in Readme.md to 1.18.

Implementation of solution discussed in: https://github.com/m-labs/smoltcp/issues/83

Main things to look at:

1. I figured this only really makes sense for the ManagedMap::Owned case, so I conditionally compile for the alloc case. Let me know if this is wrong

2. There weren't any tests in the neighbor.rs file that use the Owned case, so I had to start using BTreeMap. I'm not totally sure what the downsides of this are, so let me know if there's anything I should be aware of.

3. The reason I needed to upgrade the version of managed is so that we could have len operator on ManagedMap.

Manual Testing

I did manually test this with the httpclient example (i changed around where run_gc gets run in order to make it do something) and checked that the sizes were correct.

thanks for taking a look at this!

Opening this PR so I can get feedback earlier in case there are problems with the approach I'm taking.

This is a first draft of #334. The goal is to split EthernetInterface into Ethernet and IP parts, so the IP parts can be reused in other non-ethernet interface types (linux tun, VPNs, PPP).

The IP module defines the following structs:

• Config: contains IP configuration (addresses, routes) that doesn't change when processing IP packets.
• State: contains state that can change when processing IP packets.
• Processor: contains a &Config and a &mut State. Most of the functions moved from ethernet are here. This is just a convenience wrapper, as most functions need the config and the state, so they don't have to be passed around as parameters all the time.

The motivation for separating Config and State is that all the ethernet code needs read-only access to Config. This way Config can be immutably borrowed many times and shared everywhere it's needed, at the same time State is mutably borrowed to process packets. Without this I was getting borrow issues with both the ethernet and ip code needing to access the ip config.

For packet egress, the Ethernet layer passes a struct implementing a Dispatcher traint to the IP layer, which is used to pass IP packets back to the Ethernet layer.

Missing items:

• Tests. I'll update them as soon as we've confirmed the code structure is OK
• IPv6 NDISC. I don't know what to do, I have 2 ideas:
  • Have ethernet::Interface inject a Socket that handles NDISC. The socket would have to mut borrow ethernet::InnerInterface so it would have to be created on the fly every time, not sure if it's even possible.
  • Pass a process_ndisc callback to process_ipv6... ugly but should work?
• Some random FIXMEs in the code.

My questions:

1. any general feedback on the taken approach, or possible improvements?
2. Most of the functions in the Ethernet code take as param the ip::Config, passing it around is somewhat annoying. This could benefit from an abstraction similar to ip::Processor: ethernet::InterfaceInner would become ethernet::State, and there would be an ethernet::Processor with ip_config, ethernet_state. Pros: no passing around ip all the time, design is consistent with IP. Cons: it's an extra struct. What do you think?
3. Should the IP structs be pub or pub(crate)? I guess the latter, unless we want people to implement their interfaces out-of-tree?
4. Is the route table an ethernet concept or an IP concept? To me it looks like it is an Ethernet concept. The only use in the IP code is for AnyIP. To me it feels a bit strange that AnyIP reads the route table instead of just using the CIDRs in ip_addrs. (Also AFAICT these CIDRs aren't used for anything?). If we change it to use just ip_addrs, then routes could be moved to ethernet.

So I was using google netstack gvisor tcp/ip in 'promiscuous mode' (thats their terminology) where it will listen and accept connections on 'any' port - which is extremely useful, and is what i need. I see that in smoltcp we need to specificy individual ports to listen/accept, on a quick scan through the source I did not find any way to listen on any port, can you please tell me if there is any way ?

Rgds, Gopa.

@whitequark

Changes:

1. Add cfg-if crate.
2. Update libc version to latest.
3. Make examples httpclient/ping/server/client/benchmark only work on linux platform.
4. Update example tcpdump.
5. Add phy::LinkLayer.
6. Rewrite phy::RawSocket and phy::sys::RawSocket.

Problems:

1. BPF buffer reader ( see code )

This code here https://github.com/m-labs/smoltcp/blob/master/src/socket/tcp.rs#L913 should either reset local_rx_dup_acks or let it count from 255 to 0 with a overflowing add or stay at 255 with a saturation. What is the desired behavior?

                        self.local_rx_dup_acks += 1;
                        if self.local_rx_dup_acks == 3 {
                        }

thread 'main' panicked at 'attempt to add with overflow', smoltcp/src/socket/tcp.rs:913:25

https://github.com/smoltcp-rs/smoltcp/blob/027f255f904b9b7c4226cfd8b2d31f272ffa5105/src/phy/raw_socket.rs#L54-L60 You can't simply limit the buffer size with MTU. Because MTU is the maximum packet size you can send in one eth frame, not how big a frame you can receive. In case that remote MTU is bigger than local MTU, it really hurts performance(very often in AWS). https://github.com/smoltcp-rs/smoltcp/issues/493#issuecomment-862211323

I recently ported a project over to using the wire types from smoltcp. The places in my code where I create Arp/Udp/Ipv4/Ethernet packets are now significantly more verbose and I also introduced several bugs that could have been prevented by a more typesafe API.

Basically the problem is this: Suppose I want to wrap a udp packet into an ipv4 packet. The way I'd currently do this is with something like.

let udp: UdpPacket<_> = ...;

let udp_raw_bytes = udp.into_inner();
let ipv4_repr = Ipv4Repr {
    src_addr: ...,
    dst_addr: ...,
    protocol: IpProtocol::Tcp,
    payload_len: udp_raw_bytes.len(),
};
let mut ipv4 = Ipv4Packet::new(allocate_some_bytes(ipv4_repr.buffer_len()));
ipv4_repr.emit(&mut ipv4);
ipv4.payload_mut().clone_from_slice(&udp_raw_bytes);

This is too long. Also it contains a couple of bugs that aren't caught by the typechecker or prevented by the API. Firstly, by converting the udp packet into raw bytes we lose information (at the type level) about what sort of data we're putting in the packet. It would be better if there was a packet-emitting function that took a UdpPacket directly so that I can't accidentally wrap one as a TCP packet. Also, I have to tell the API the lengths of data I'm dealing with in several places, and it's possible to get these wrong (by eg. calling ipv4_repr.buffer_len() thinking it will give me the total length of memory I need to allocate for the packet). The library already knows how much memory it needs, I shouldn't have to tell it. Also it's easy to forget to copy across the UDP packet data. This also shouldn't be possible to do.

I propose to add packet-creation APIs that look something like this:

Ipv4Packet::from_udp<T, B>(
    src_addr: Ipv4Address,
    dst_addr: Ipv4Address,
    ttl: u8,
    packet: UdpPacket<T>,
    alloc: impl FnOnce(usize) -> B,
) -> Ipv4Packet
where
    T: AsRef<[u8]>,
    B: AsMut<[u8]>,

That way, I can pass it the allocation function I want to use and trust it to create the packet correctly. Would something like this be desirable?

This should close #106 ~~once I put in a couple unit tests~~ (EDIT: they're done).

I would like an early review and a set of independent testing because I suspect this PR may cause future issues.

I did testing with the server example using buffers that were 10, 100, and 1000 times bigger, and did repeated 200 MB file transfers. The expanded windows did seem to behave correctly, and the file was correctly assembled at the other end.

However, my testing also suggested that this increased "speed limit" has made smoltcp quite timing sensitive, and is exposing unrelated edge cases and/or timing bugs.

The most annoying was when, on occasion, it would send a duplicate ACK every 20 microseconds between two received segments for reasons unknown. Another was more rare, but ominous. If segments arrived at a high enough speed with the CPU was starved, they would seem to get lost; the Assembler would read those that made it as "out of order". If the CPU remained starved for long enough, it was possible to exceed its tracking limit and hit a panic.

Both problems were intermittent, and I could not troubleshoot them because adding debug code made them go away. :unamused:

Also, changes were needed to allow rx_buffer > TCP window, even before any of the actual window sizing changes. It makes me wonder if there are more rx_buffer == TCP window baked-in assumptions that I missed somewhere. I hope not. :crossed_fingers:

And finally, it seems no one tried to run cargo bench lately, or they would have discovered my last patch broke it. :blush: That's fixed in this patch as well.

Some refactors trying to simplify the data structures and API for the assemblers. See individual commit messages for details.

@thvdveld since you wrote the original code: Could you take a look? Is there anything I might have missed?

One thing that's wrong for now is the offset correction. Previously, the PacketAssembler was initialized when receiving the first fragment, with the right offset correction, and non-first fragments were discarded if they arrived first. The new API has only .get() which creates the assembler if it doesn't exist, so now non-first fragments will still be processed, but with the wrong offset correction. This is a problem, because only the first packet has enough info to learn the offset correction, right? Perhaps what we're supposed to do is decompress the headers before writing them to the assembler?

Hi! This is the first time I am trying to use smoltcp, so it is entirely possible I am doing something wrong. I have been trying to add a DHCP socket to an Ethernet interface; as soon as I do that, the second call to poll() results in an assertion failure.

Here is a reduced source file that reproduces the issue (with smoltcp v0.8.2):

use std::collections::BTreeMap;

use log::info;
use smoltcp::{
    iface::{InterfaceBuilder, NeighborCache, Routes},
    phy::{Device, DeviceCapabilities, Medium, RxToken, TxToken},
    socket::Dhcpv4Socket,
    time::Instant,
    wire::HardwareAddress,
    Result,
};

struct FakeDevice {}

const PACKET_SIZE: usize = 1514;

struct FakeRxToken {
    packet: [u8; PACKET_SIZE],
    len: usize,
}

struct FakeTxToken {}

impl TxToken for FakeTxToken {
    fn consume<R, F>(self, _: Instant, len: usize, f: F) -> Result<R>
    where
        F: FnOnce(&mut [u8]) -> Result<R>,
    {
        let mut buf = [0u8; PACKET_SIZE];
        assert!(len <= PACKET_SIZE);
        let payload = &mut buf[..len];

        let ret = f(payload)?;

        info!("transmit: {:?}", payload);

        Ok(ret)
    }
}

impl RxToken for FakeRxToken {
    fn consume<R, F>(mut self, _: Instant, f: F) -> Result<R>
    where
        F: FnOnce(&mut [u8]) -> Result<R>,
    {
        let packet = &mut self.packet[..self.len];
        info!("receive: {:?}", packet);
        f(packet)
    }
}

impl<'d> Device<'d> for FakeDevice {
    type TxToken = FakeTxToken;
    type RxToken = FakeRxToken;

    fn receive(&'d mut self) -> Option<(Self::RxToken, Self::TxToken)> {
        None
    }

    fn transmit(&'d mut self) -> Option<Self::TxToken> {
        Some(FakeTxToken {})
    }

    fn capabilities(&self) -> DeviceCapabilities {
        let mut caps = DeviceCapabilities::default();
        caps.medium = Medium::Ethernet;
        caps.max_transmission_unit = PACKET_SIZE;
        caps.max_burst_size = Some(1);
        caps
    }
}

fn main() {
    env_logger::init();
    let device = FakeDevice {};

    let routes = Routes::new(BTreeMap::new());
    let neighbor_cache = NeighborCache::new(BTreeMap::new());
    let hw_addr = HardwareAddress::Ethernet(smoltcp::wire::EthernetAddress::from_bytes(&[
        0, 1, 2, 3, 4, 5,
    ]));

    let mut interface = InterfaceBuilder::new(device, Vec::new())
        .hardware_addr(hw_addr)
        .routes(routes)
        .neighbor_cache(neighbor_cache)
        .finalize();

    let dhcp_socket = Dhcpv4Socket::new();
    let dhcp_socket_handle = interface.add_socket(dhcp_socket);

    for i in 0.. {
        info!("IPs: {:?}", interface.ip_addrs());
        let status = interface.poll(Instant::from_micros(i));
        info!("Poll: {:?}", status);

        let dhcp_status = interface
            .get_socket::<Dhcpv4Socket>(dhcp_socket_handle)
            .poll();

        info!("{:?}", dhcp_status);
    }
}

and here are the corresponding logs:

luca@xps ~/smoltcp-repro  $ RUST_LOG=trace cargo run
   Compiling smoltcp-repro v0.1.0 (/home/luca/smoltcp-repro)
    Finished dev [unoptimized + debuginfo] target(s) in 0.49s
     Running `target/debug/smoltcp-repro`
[2022-12-11T01:03:38Z TRACE smoltcp::iface::socket_set] [0]: adding
[2022-12-11T01:03:38Z INFO  smoltcp_repro] IPs: []
[2022-12-11T01:03:38Z DEBUG smoltcp::socket::dhcpv4] DHCP send DISCOVER to 255.255.255.255: Repr { message_type: Discover, transaction_id: 4260165794, client_hardware_address: Address([0, 1, 2, 3, 4, 5]), client_ip: Address([0, 0, 0, 0]), your_ip: Address([0, 0, 0, 0]), server_ip: Address([0, 0, 0, 0]), router: None, subnet_mask: None, relay_agent_ip: Address([0, 0, 0, 0]), broadcast: false, requested_ip: None, client_identifier: Some(Address([0, 1, 2, 3, 4, 5])), server_identifier: None, parameter_request_list: Some([1, 3, 6]), dns_servers: None, max_size: Some(1432), lease_duration: None }
[2022-12-11T01:03:38Z TRACE smoltcp::iface::socket_meta] #0: neighbor 255.255.255.255 missing, silencing until t+3.000s
[2022-12-11T01:03:38Z INFO  smoltcp_repro] Poll: Ok(false)
[2022-12-11T01:03:38Z INFO  smoltcp_repro] Some(Deconfigured)
[2022-12-11T01:03:38Z INFO  smoltcp_repro] IPs: []
thread 'main' panicked at 'assertion failed: protocol_addr.is_unicast()', /home/luca/.cargo/registry/src/github.com-1ecc6299db9ec823/smoltcp-0.8.2/src/iface/neighbor.rs:199:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Any help would be appreciated!

Note that the problem seems to go away if using current GitHub main:

use std::collections::BTreeMap;

use log::info;
use smoltcp::{
    iface::{InterfaceBuilder, NeighborCache, Routes, SocketSet},
    phy::{Device, DeviceCapabilities, Medium, RxToken, TxToken},
    socket::dhcpv4::Socket as Dhcpv4Socket,
    time::Instant,
    wire::HardwareAddress,
    Result,
};

struct FakeDevice {}

const PACKET_SIZE: usize = 1514;

struct FakeRxToken {
    packet: [u8; PACKET_SIZE],
    len: usize,
}

struct FakeTxToken {}

impl TxToken for FakeTxToken {
    fn consume<R, F>(self, _: Instant, len: usize, f: F) -> Result<R>
    where
        F: FnOnce(&mut [u8]) -> Result<R>,
    {
        let mut buf = [0u8; PACKET_SIZE];
        assert!(len <= PACKET_SIZE);
        let payload = &mut buf[..len];

        let ret = f(payload)?;

        info!("transmit: {:?}", payload);

        Ok(ret)
    }
}

impl RxToken for FakeRxToken {
    fn consume<R, F>(mut self, _: Instant, f: F) -> Result<R>
    where
        F: FnOnce(&mut [u8]) -> Result<R>,
    {
        let packet = &mut self.packet[..self.len];
        info!("receive: {:?}", packet);
        f(packet)
    }
}

impl Device for FakeDevice {
    type TxToken<'d> = FakeTxToken;
    type RxToken<'d> = FakeRxToken;

    fn receive(&mut self) -> Option<(Self::RxToken<'_>, Self::TxToken<'_>)> {
        None
    }

    fn transmit(&mut self) -> Option<Self::TxToken<'_>> {
        Some(FakeTxToken {})
    }

    fn capabilities(&self) -> DeviceCapabilities {
        let mut caps = DeviceCapabilities::default();
        caps.medium = Medium::Ethernet;
        caps.max_transmission_unit = PACKET_SIZE;
        caps.max_burst_size = Some(1);
        caps
    }
}

fn main() {
    env_logger::init();
    let mut device = FakeDevice {};

    let routes = Routes::new(BTreeMap::new());
    let neighbor_cache = NeighborCache::new(BTreeMap::new());
    let hw_addr = HardwareAddress::Ethernet(smoltcp::wire::EthernetAddress::from_bytes(&[
        0, 1, 2, 3, 4, 5,
    ]));

    let mut interface = InterfaceBuilder::new()
        .hardware_addr(hw_addr)
        .routes(routes)
        .neighbor_cache(neighbor_cache)
        .finalize(&mut device);

    let dhcp_socket = Dhcpv4Socket::new();
    let mut socket_set = SocketSet::new(vec![]);
    let dhcp_socket_handle = socket_set.add(dhcp_socket);

    for i in 0.. {
        info!("IPs: {:?}", interface.ip_addrs());
        let status = interface.poll(Instant::from_micros(i), &mut device, &mut socket_set);
        info!("Poll: {:?}", status);

        let dhcp_status = socket_set
            .get_mut::<Dhcpv4Socket>(dhcp_socket_handle)
            .poll();

        info!("{:?}", dhcp_status);
    }
}

This error is pretty curious, I wonder if that is due to core changes since I remember Box should have had handled Format to passthrough transparently to the underlying type:

(base) PS C:\Users\steve\esp32c2-rust-hello-world> cargo build
   Compiling smoltcp v0.8.2
   Compiling esp32c3-hal v0.3.0
   Compiling esp-wifi v0.1.0 (https://github.com/esp-rs/esp-wifi.git?rev=4d8243f#4d8243f7)
error[E0277]: the trait bound `Box<[Contig; 32]>: Format` is not satisfied
   --> C:\Users\steve\.cargo\registry\src\github.com-1ecc6299db9ec823\smoltcp-0.8.2\src\storage\assembler.rs:104:5
    |
99  | #[cfg_attr(feature = "defmt", derive(defmt::Format))]
    |                                      ------------- required by a bound introduced by this call
...
104 |     contigs: Box<[Contig; CONTIG_COUNT]>,
    |     ^^^^^^^ the trait `Format` is not implemented for `Box<[Contig; 32]>`
    |
    = help: the following other types implement trait `Format`:
              &T
              &mut T
              ()
              (T0, T1)
              (T0, T1, T2)
              (T0, T1, T2, T3)
              (T0, T1, T2, T3, T4)
              (T0, T1, T2, T3, T4, T5)
            and 185 others
note: required by a bound in `defmt::export::fmt`
   --> C:\Users\steve\.cargo\registry\src\github.com-1ecc6299db9ec823\defmt-0.3.2\src\export\mod.rs:137:15
    |
137 | pub fn fmt<T: Format + ?Sized>(f: &T) {
    |               ^^^^^^ required by this bound in `defmt::export::fmt`

For more information about this error, try `rustc --explain E0277`.                                                                                                                                                                                                                                                    
error: could not compile `smoltcp` due to previous error
warning: build failed, waiting for other jobs to finish...

Cargo.toml entry:

smoltcp = { version = "0.8.2", default-features = false, features = [
     "medium-ethernet", "medium-ip", "medium-ieee802154",
     "proto-ipv4", "proto-igmp", "proto-dhcpv4", "proto-ipv6",
     "socket-raw", "socket-icmp", "socket-udp", "socket-tcp", "socket-dhcpv4",
     "async", "defmt", "alloc"
] }

If I disabled alloc or defmt it is absolutely working: https://github.com/smoltcp-rs/smoltcp/blob/0e614bc6cca8b97a1a11c27154f76221c5403739/src/storage/assembler.rs#L98-L99

Hi @whitequark / @Dirbaio - can you pls release a new version of smoltcp on crates.io so that the smoltcp GAT-device capability is available on crates.io

we are trying to use smoltcp in windows platform and transfer packet throgh wintun (we add windows platform support in rust-tun).

we found that some packet is not recognized, icmp packet will lost