A fully asynchronous, futures-based Kafka client library for Rust based on librdkafka

Overview

rust-rdkafka

crates.io docs.rs Build Status coverate Join the chat at https://gitter.im/rust-rdkafka/Lobby

A fully asynchronous, futures-enabled Apache Kafka client library for Rust based on librdkafka.

The library

rust-rdkafka provides a safe Rust interface to librdkafka. The master branch is currently based on librdkafka 1.6.0.

Documentation

Features

The main features provided at the moment are:

  • Support for all Kafka versions since 0.8.x. For more information about broker compatibility options, check the librdkafka documentation.
  • Consume from single or multiple topics.
  • Automatic consumer rebalancing.
  • Customizable rebalance, with pre and post rebalance callbacks.
  • Synchronous or asynchronous message production.
  • Customizable offset commit.
  • Create and delete topics and add and edit partitions.
  • Alter broker and topic configurations.
  • Access to cluster metadata (list of topic-partitions, replicas, active brokers etc).
  • Access to group metadata (list groups, list members of groups, hostnames, etc.).
  • Access to producer and consumer metrics, errors and callbacks.
  • Exactly-once semantics (EOS) via idempotent and transactional producers and read-committed consumers.

One million messages per second

rust-rdkafka is designed to be easy and safe to use thanks to the abstraction layer written in Rust, while at the same time being extremely fast thanks to the librdkafka C library.

Here are some benchmark results using the BaseProducer, sending data to a single Kafka 0.11 process running in localhost (default configuration, 3 partitions). Hardware: Dell laptop, with Intel Core i7-4712HQ @ 2.30GHz.

  • Scenario: produce 5 million messages, 10 bytes each, wait for all of them to be acked

    • 1045413 messages/s, 9.970 MB/s (average over 5 runs)
  • Scenario: produce 100000 messages, 10 KB each, wait for all of them to be acked

    • 24623 messages/s, 234.826 MB/s (average over 5 runs)

For more numbers, check out the kafka-benchmark project.

Client types

rust-rdkafka provides low level and high level consumers and producers.

Low level:

  • BaseConsumer: a simple wrapper around the librdkafka consumer. It must be periodically poll()ed in order to execute callbacks, rebalances and to receive messages.
  • BaseProducer: a simple wrapper around the librdkafka producer. As in the consumer case, the user must call poll() periodically to execute delivery callbacks.
  • ThreadedProducer: a BaseProducer with a separate thread dedicated to polling the producer.

High level:

For more information about consumers and producers, refer to their module-level documentation.

Warning: the library is under active development and the APIs are likely to change.

Asynchronous data processing with Tokio

Tokio is a platform for fast processing of asynchronous events in Rust. The interfaces exposed by the StreamConsumer and the FutureProducer allow rust-rdkafka users to easily integrate Kafka consumers and producers within the Tokio platform, and write asynchronous message processing code. Note that rust-rdkafka can be used without Tokio.

To see rust-rdkafka in action with Tokio, check out the asynchronous processing example in the examples folder.

At-least-once delivery

At-least-once delivery semantics are common in many streaming applications: every message is guaranteed to be processed at least once; in case of temporary failure, the message can be re-processed and/or re-delivered, but no message will be lost.

In order to implement at-least-once delivery the stream processing application has to carefully commit the offset only once the message has been processed. Committing the offset too early, instead, might cause message loss, since upon recovery the consumer will start from the next message, skipping the one where the failure occurred.

To see how to implement at-least-once delivery with rdkafka, check out the at-least-once delivery example in the examples folder. To know more about delivery semantics, check the message delivery semantics chapter in the Kafka documentation.

Exactly-once semantics

Exactly-once semantics (EOS) can be achieved using transactional producers, which allow produced records and consumer offsets to be committed or aborted atomically. Consumers that set their isolation.level to read_committed will only observe committed messages.

EOS is useful in read-process-write scenarios that require messages to be processed exactly once.

To learn more about using transactions in rust-rdkafka, see the Transactions section of the producer documentation.

Users

Here are some of the projects using rust-rdkafka:

If you are using rust-rdkafka, please let us know!

Installation

Add this to your Cargo.toml:

[dependencies]
rdkafka = { version = "0.25", features = ["cmake-build"] }

This crate will compile librdkafka from sources and link it statically to your executable. To compile librdkafka you'll need:

  • the GNU toolchain
  • GNU make
  • pthreads
  • zlib: optional, but included by default (feature: libz)
  • cmake: optional, not included by default (feature: cmake-build)
  • libssl-dev: optional, not included by default (feature: ssl)
  • libsasl2-dev: optional, not included by default (feature: gssapi)
  • libzstd-dev: optional, not included by default (feature: zstd-pkg-config)

Note that using the CMake build system, via the cmake-build feature, is encouraged if you can take the dependency on CMake.

By default a submodule with the librdkafka sources pinned to a specific commit will be used to compile and statically link the library. The dynamic-linking feature can be used to instead dynamically link rdkafka to the system's version of librdkafka. Example:

[dependencies]
rdkafka = { version = "0.25", features = ["dynamic-linking"] }

For a full listing of features, consult the rdkafka-sys crate's documentation. All of rdkafka-sys features are re-exported as rdkafka features.

Minimum supported Rust version (MSRV)

The current minimum supported Rust version (MSRV) is 1.45.0. Note that bumping the MSRV is not considered a breaking change. Any release of rust-rdkafka may bump the MSRV.

Asynchronous runtimes

Some features of the StreamConsumer and FutureProducer depend on Tokio, which can be a heavyweight dependency for users who only intend to use the low-level consumers and producers. The Tokio integration is enabled by default, but can be disabled by turning off default features:

[dependencies]
rdkafka = { version = "0.25", default-features = false }

If you would like to use an asynchronous runtime besides Tokio, you can integrate it with rust-rdkafka by providing a shim that implements the AsyncRuntime trait. See the smol runtime example for an example integration with smol.

Examples

You can find examples in the examples folder. To run them:

cargo run --example <example_name> -- <example_args>

Debugging

rust-rdkafka uses the log and env_logger crates to handle logging. Logging can be enabled using the RUST_LOG environment variable, for example:

RUST_LOG="librdkafka=trace,rdkafka::client=debug" cargo test

This will configure the logging level of librdkafka to trace, and the level of the client module of the Rust client to debug. To actually receive logs from librdkafka, you also have to set the debug option in the producer or consumer configuration (see librdkafka configuration).

To enable debugging in your project, make sure you initialize the logger with env_logger::init(), or the equivalent for any log-compatible logging framework.

rdkafka-sys

See rdkafka-sys.

Contributors

Thanks to:

Alternatives

  • kafka-rust: a pure Rust implementation of the Kafka client.
Comments
  • First time user

    First time user

    Hello! I've recently tried to use rust-rdkafka, and thought I would share some experiences. I think there may be some "issues" here, but there could also just be some helpful information about what people (person) stumble over.

    First off, thank you! This is in support of a timely dataflow Kafka adapter, which I wouldn't want to try and write without a crate. :)

    I wanted to try sending a message through kafka and back, so I wrote what seemed like a pretty simple program, following your examples but using the Base* variants of producer and consumer.

    fn round_trip(brokers: &str, topic_name: &str) -> Result<(), rdkafka::error::KafkaError> {
    
        let mut topic_config = TopicConfig::new();
        topic_config
            .set("produce.offset.report", "true")
            .finalize();
    
        let mut client_config = ClientConfig::new();
        client_config
            .set("group.id", "example")
            .set("bootstrap.servers", brokers)
            .set_default_topic_config(topic_config);
    
        let producer: BaseProducer<_> = try!(client_config.create());
        let consumer: BaseConsumer<_> = try!(client_config.create());
    
        try!(consumer.subscribe(&[topic_name]));
    
        let text = format!("hi there at {:?}", ::std::time::Instant::now());
        try!(producer.send_copy::<str,()>(topic_name, None, Some(text.as_str()), None,  None, None));
    
        for _ in 0 .. 100 {
    
            producer.poll(1);
    
            if let Some(result) = try!(consumer.poll(1)) {
                println!("{:?}:\t{:?}", ::std::time::Instant::now(), result.payload_view::<str>());
                try!(consumer.commit(None, rdkafka::consumer::CommitMode::Sync));
            }
            else {
                println!("{:?}:\tgot nothing", ::std::time::Instant::now());
            }
        }
    
        Ok(())
    }
    

    The behavior was a bit surprising, and it was tricky to find documentation to explain if this was expected.

    1. This seems to run to completion just fine, printing "got nothing" 100 times and returning Ok(()) when neither ZK nor Kafka are up and running. Where is the moment in the code it is supposed to complain if they are not installed or up and running? (when they are up and running, it does seem to connect and occasionally round-trip data; mysterious things happen).

    2. I'm not a Kafka expert, so I was surprised to see things like "nothing more to read now" surface as an error (PartitionEOF) rather than the absence of a message. I imagine that is just what rdkafka thinks is best and returns to you, so you want to surface it up?

    3. The BaseProducer documentation speaks of a BaseProducerTopic that I should use, but I couldn't find this. Is it just stale documentation?

    4. I was hoping to have a non-blocking poll, but I'm not sure if this is what an argument of 0 would mean. Sometimes 0 means "block". Is that something you could clarify in the documentation? It could be that rdkafka does what it does and won't let us fcntl the socket to non-blocking mode, or something like that, but would be good to know!

    Maybe as a meta comment, if BaseProducer and BaseConsumer are meant to be a very thin layer on top of rdkafka, with the caveat that you had best understand the details of rdkafka (which I don't yet), you could say that in the documentation (I'm still not sure). I'm sure it is still useful, but would keep me from scratching / banging my head.

    I'm going to try out the "higher-level" stuff next, but I'd have preferred to just have a non-blocking polling interface. If that should work out, and I just need more Kafka protocol expertise, I can push on it more and come back with comments on that.

    Thanks again!

    opened by frankmcsherry 38
  • Directly return a DeliveryFuture from FutureProducer::send_copy

    Directly return a DeliveryFuture from FutureProducer::send_copy

    Returning a Result made it very difficult to use send_copy() in a chain of futures, and, since a Future is just an asynchronous Result, this commit changes the DeliveryFuture to yield a KafkaError on failure. Previously, the DeliveryFuture would only yield Canceled, and, to continue supporting this, a new variant of KafkaError was added, FutureCanceled.

    This makes the FutureProducer API considerably more ergonomic. For example, in a function which returns a BoxFuture<(), ::std::io::Error>, this is what was necessary before:

    return match kafka_producer.send_copy::<_, ()>(
        "test", None, Some(&*format!("msg on {}", topic)), None, None) {
        Ok(report) => {
            report
                .map_err(|_| io::Error::new(io::ErrorKind::Other,
                                            "failed to send kafka message"))
                .and_then(|_| future::ok(()))
                .boxed()
        },
    
        Err(_) => {
            future::err(io::Error::new(
                    io::ErrorKind::InvalidData, "failed to send kafka message"))
                .boxed()
        }
    }
    

    This becomes even more complicated with the fact that match arms must return the same type and the types of chained Futures can easily get very complex.

    Compare that to the code one can write with this PR applied:

    return kafka_producer
        .send_copy::<_, ()>("test", None, Some(&*format!("msg on {}", topic)), None, None)
        .map_err(|_| io::Error::new(io::ErrorKind::Other,
                                    "failed to send kafka message"))
        .and_then(|_| future::ok(()))
        .boxed();
    

    Much less verbose, and, more importantly, it flows naturally as part of a chain of futures.

    Full disclosure: there is a compilation warning for the unused Result returned by tx.send() in two places, but this warning was present before the change and I'm also not sure what the correct way to handle an error arising in such a situation actually is.

    opened by wrl 25
  • Memory keeps growing

    Memory keeps growing

    Not sure if this is a problem or where it is but after creating a program with FutureProducer my memory skyrocketed to 10Gb usage by having 3 FutureProducers running at the same time (otherwise I get limited to 50k EPS).

    Even when running only one memory usage was at 4gb.

    After chasing this a little while, I have tried running the asynchronous_processing example, and memory also grows there at least until 1GB, any idea what can be causing this?

    ss-2020-02-23-130822

    opened by llitz 18
  • Running StreamConsumer with tokio's executor

    Running StreamConsumer with tokio's executor

    Currently how StreamConsumer stores the JoinHandle in a Cell prevents us of using the consumer in tokio's thread_pool executor, which expects the consumer to implement Send. In other hand using the current_thread executor doesn't comply with missing static lifetimes somewhere in the consumer code I haven't been able to locate yet.

    Is there plans to support the new executors, especially when the development of tokio-core and futures-cpupool seems to be stalled in favor of tokio?

    opened by pimeys 16
  • Compilation fails w/ rust-musl-builder

    Compilation fails w/ rust-musl-builder

    When compiling a rust application using rdkafka on rust-musl-builder the build fails when linking:

    My guess is that the rdkafka lib is build with glibc and then fails w/ musl.

    Is there a way to specify what to build it against the same way that dynamic linking can be specified?

    error: linking with `cc` failed: exit code: 1
      |
      = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-nostdlib" "-Wl,--eh-frame-hdr" "-Wl,-(" "-m64" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crt1.o" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crti.o" "-L" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.mshello0-66b6f0d31720f4bd44d6f265903cd36d.rs.rcgu.o" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.mshello1-66b6f0d31720f4bd44d6f265903cd36d.rs.rcgu.o" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.mshello2-66b6f0d31720f4bd44d6f265903cd36d.rs.rcgu.o" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.mshello3-66b6f0d31720f4bd44d6f265903cd36d.rs.rcgu.o" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.mshello4-66b6f0d31720f4bd44d6f265903cd36d.rs.rcgu.o" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.mshello5-66b6f0d31720f4bd44d6f265903cd36d.rs.rcgu.o" "-o" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.crate.allocator.rcgu.o" "-Wl,--gc-sections" "-no-pie" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-nodefaultlibs" "-L" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps" "-L" "/home/rust/src/target/release/deps" "-L" "/home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src" "-L" "/home/rust/src/target/x86_64-unknown-linux-musl/release/build/libz-sys-9e0dfdf6ce77c312/out/lib" "-L" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc_jemalloc-f68d8e8245e507c6.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka-a45674fad582d6de.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/liblibz_sys-d28970724bb40e9f.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/liblibc-eef0eb2737b72223.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/libfutures-f1a25787e6cfbe44.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/libserde_json-fded606be9e377b3.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/libitoa-539d7e09fab6c05d.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/libdtoa-19c34238b45a98bc.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/libserde-30812860af6099f8.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/liblog-4de18e7534a813d8.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/liblog-9a6200d5626a5e8a.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/libcfg_if-5f123802e50df72d.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-8d847bbe97fc9dc5.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-008c42aad548e7dd.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libunwind-0fe53371ec419e32.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc_system-9dec1cbd51097ce1.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/liblibc-6a4fb915dd86d140.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-7ebba6af2d3cc324.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd_unicode-7a26f8b3cf380464.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-b0c2d164a9741309.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-66d072e25a9acee3.rlib" "-static" "-Wl,-Bdynamic" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crtn.o" "-Wl,-)"
      = note: /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `vsnprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:77: undefined reference to `__vsnprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:77: undefined reference to `__vsnprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:64: more undefined references to `__snprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `vsnprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:77: undefined reference to `__vsnprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `rd_strndup':
              /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
              /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
              /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
              /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
              /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o):/home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: more undefined references to `__strndup' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `syslog':
              /usr/include/x86_64-linux-gnu/bits/syslog.h:31: undefined reference to `__syslog_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `vsnprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:77: undefined reference to `__vsnprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_msg.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_msg.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_conf.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_conf.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:64: more undefined references to `__snprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_conf.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_conf.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_conf.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_conf.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_offset.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_offset.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_offset.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_offset.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_offset.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_offset.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_transport.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_transport.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:64: more undefined references to `__snprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_transport.o): In function `poll':
              /usr/include/x86_64-linux-gnu/bits/poll2.h:41: undefined reference to `__poll_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_buf.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_buf.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_buf.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_queue.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_op.o): In function `vsnprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:77: undefined reference to `__vsnprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_op.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_op.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_request.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_request.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_request.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_request.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_request.o): In function `rd_strndup':
              /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_request.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_cgrp.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_cgrp.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_cgrp.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_cgrp.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_cgrp.o): In function `rd_strndup':
              /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
              /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_cgrp.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_pattern.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o): In function `rd_strndup':
              /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_assignor.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdstring.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:64: more undefined references to `__snprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_metadata.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_metadata.o): In function `rd_strndup':
              /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdbuf.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdbuf.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(snappy.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rddl.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rddl.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:64: more undefined references to `__snprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(crc32c.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(crc32c.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdaddr.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdaddr.o): In function `strncpy':
              /usr/include/x86_64-linux-gnu/bits/string3.h:126: undefined reference to `__strncpy_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdaddr.o): In function `strcpy':
              /usr/include/x86_64-linux-gnu/bits/string3.h:110: undefined reference to `__memcpy_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdlist.o): In function `printf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:104: undefined reference to `__printf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:104: undefined reference to `__printf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_metadata_cache.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_sasl.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_msgset_reader.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_msgset_reader.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_msgset_reader.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdvarint.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdvarint.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_plugin.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdlog.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdlog.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdlog.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdlog.o): In function `printf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:104: undefined reference to `__printf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:104: undefined reference to `__printf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:104: undefined reference to `__printf_chk'
    
    opened by Licenser 15
  • Batch Processing

    Batch Processing

    I understand the hesitation in adding to the publishing api an explicit batch processing fn (it already batches under the hood) - but it's unclear to me the purpose of not having this api in the consumer. We should remove complexity from the consumer. I note this after going through previous replies to the same feature request

    opened by lacasaprivata2 13
  • Is this function safe to use and support futures for async processing ? Need simple Producer usage

    Is this function safe to use and support futures for async processing ? Need simple Producer usage

    Hi,

    I need to use simple function to produce Kafka messages with a solid function, that process producing on demand when requested by users:

    //Send messages to Kafka
    fn produce(brokers: &str, topic_name: &str, messages: String) {
        let producer: FutureProducer = ClientConfig::new()
            .set("bootstrap.servers", brokers)
            .set("produce.offset.report", "true")
            .set("message.timeout.ms", "5000")
            .create()
            .expect("Producer creation error");
    
    
        producer.send(
            FutureRecord::to(topic_name)
                .payload(&format!("Message {}", messages))
                .key(&format!("Key {}", messages))
                .headers(OwnedHeaders::new()
                    .add("header_key", "header_value")),
            0
        );
    }
    

    Do this function is safe and performente ? support futures ?

    Thanks in advance.

    opened by KabDeveloper 13
  • 0.21.0 breaks build on Rust Docker image

    0.21.0 breaks build on Rust Docker image

    Problem

    Using the official Rust docker image, I am unable to build version 0.21.0, whereas 0.20.0 worked without issue. The error I see is:

    --- stderr
    Building and linking librdkafka statically
    Running command: "./configure --disable-sasl --enable-ssl --disable-lz4" in dir: librdkafka
    Running command: "make -j 4 libs" in dir: librdkafka
    thread 'main' panicked at 'Unable to find libclang: "couldn\'t find any valid shared libraries matching: [\'libclang.so\', \'libclang-*.so\', \'libclang.so.*\'], set the `LIBCLANG_PATH` environment variable to a path where one of these files can be found (invalid: [])"', src/libcore/result.rs:997:5
    note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
    

    Solution

    I am able to build successfully after installing some additional dependencies in the image (copied from the Dockerfile in this repo): apt-get install -y llvm-3.9-dev libclang-3.9-dev clang-3.9

    Based on the description in d9321f2b1f132d554d2c5854803fcb2869c6495c, it looks like the new dependencies are intended. Perhaps the new dependencies could be mentioned in the changelog to help anyone with broken builds?

    Let me know if there is additional information I can provide.

    Thanks!

    opened by austinhartzheim 12
  • Release 0.22 build from cargo fails after the release of 0.23

    Release 0.22 build from cargo fails after the release of 0.23

    I'm having problems building my project that is using release 0.22.

    It seems that problems started after the release of rust-rdkafka 0.23 and rdkafka-sys 1.3.1.

       Compiling rdkafka-sys v1.3.1
      ...
       Compiling rdkafka v0.22.0
    error[E0425]: cannot find function `primitive_to_rd_kafka_resp_err_t` in crate `rdsys`
       --> /Users/tkasu/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-0.22.0/src/client.rs:411:22
        |
    411 |     let err = rdsys::primitive_to_rd_kafka_resp_err_t(err)
        |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ not found in `rdsys`
    
    

    Release 0.22 is fetching rdkafka-sys 1.3.1 from which the function primitive_to_rd_kafka_resp_err_t has been removed.

    opened by tkasu 11
  • WIP - switch to std::future and async/await

    WIP - switch to std::future and async/await

    This PR moves the project over from futures 0.1 to the newly stabilized std::future, using tokio 0.2 (alpha) and futures-preview 0.3. It won't compile on stable until Rust 1.39 at the earliest since it uses async/await, so it'll obviously bump up the minimum Rust version too. It may be possible to remove the async/await which I think would allow it to compile on 1.36.0.

    Tests are passing and examples working on nightly.

    I guess this won't want to be merged until things stabilize a little more, but thought it might be useful when that time comes!

    opened by sd2k 11
  • Error linking libzstd.so on ubuntu 19.04

    Error linking libzstd.so on ubuntu 19.04

    building this repo or a project with it as a dependency on ubuntu 19.04 I get this error:

       Compiling rdkafka v0.21.0 (rust-rdkafka)
    error: linking with `cc` failed: exit code: 1
      |
      = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" [...] "-Wl,-Bdynamic" "-lz" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"
      = note: /usr/bin/ld: rust-rdkafka/target/debug/deps/librdkafka_sys-3d04a478eb2ea7f3.rlib(rdkafka_zstd.o): in function `rd_kafka_zstd_decompress':
              rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:44: undefined reference to `ZSTD_getFrameContentSize'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:91: undefined reference to `ZSTD_getErrorCode'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:80: undefined reference to `ZSTD_decompress'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:82: undefined reference to `ZSTD_isError'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:100: undefined reference to `ZSTD_getErrorName'
              /usr/bin/ld: rust-rdkafka/target/debug/deps/librdkafka_sys-3d04a478eb2ea7f3.rlib(rdkafka_zstd.o): in function `rd_kafka_zstd_compress':
              rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:130: undefined reference to `ZSTD_compressBound'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:141: undefined reference to `ZSTD_createCStream'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:155: undefined reference to `ZSTD_initCStream'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:157: undefined reference to `ZSTD_isError'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:213: undefined reference to `ZSTD_freeCStream'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:168: undefined reference to `ZSTD_compressStream'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:169: undefined reference to `ZSTD_isError'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:170: undefined reference to `ZSTD_getErrorName'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:213: undefined reference to `ZSTD_freeCStream'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:158: undefined reference to `ZSTD_getErrorName'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:213: undefined reference to `ZSTD_freeCStream'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:198: undefined reference to `ZSTD_endStream'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:199: undefined reference to `ZSTD_isError'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:213: undefined reference to `ZSTD_freeCStream'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:200: undefined reference to `ZSTD_getErrorName'
              collect2: error: ld returned 1 exit status
    
    
    error: aborting due to previous error
    
    error: Could not compile `rdkafka`.
    

    if I copy the command in note: and add "-lzstd" it seems to link

    from

    "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" [...] "-Wl,-Bdynamic" "-lz" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"
    

    to

    "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" [...] "-Wl,-Bdynamic" "-lz" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil" "-lzstd"
    

    here is a Dockerfile to reproduce:

    FROM ubuntu:19.04
    
    ARG UNAME=mariano
    ARG UID=1000
    ARG GID=1000
    RUN groupadd -g $GID -o $UNAME
    RUN useradd -m -u $UID -g $GID -o -s /bin/bash $UNAME
    
    RUN \
      sed -i 's/# \(.*multiverse$\)/\1/g' /etc/apt/sources.list && \
      apt-get update; \
      apt-get -y upgrade && \
      apt-get install -y --no-install-recommends wget build-essential clang libclang-dev libssl-dev zlib1g-dev libsasl2-dev libzstd-dev curl ca-certificates git
    
    USER $UNAME
    RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
    

    then

    # build it
    sudo docker build --build-arg UID=$$(id -u) --build-arg GID=$$(id -g) -t ubuntu-rust:1904-stable ubuntu-1904-rust
    
    sudo docker run -i ubuntu-rust:1904-stable sh << COMMANDS
        cd
        . .cargo/env
        git clone https://github.com/fede1024/rust-rdkafka.git
        cd rust-rdkafka
        cargo run --example simple_consumer
    COMMANDS
    

    it seems for some reason it doesn't find libzstd

    $ ldconfig -p | grep libzstd
    
            libzstd.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libzstd.so.1
    

    I tried creating symbolic links in /usr/local/lib just in case but it didn't work.

    opened by marianoguerra 11
  • Question - Accomulate N messages in T duration or timeout.

    Question - Accomulate N messages in T duration or timeout.

    I would like use poll with a specified timeout but i would like to get N number of messages per call. And poll other messages in the next call.

    consumer.iter().take(1000) // Acomulate messages 
    

    I would like to acomulate this messages but based in a specified Duration or Timeout.

    opened by ajosecueto 0
  • Help with clarifying dynamic-linking behavior

    Help with clarifying dynamic-linking behavior

    Hi there, had a quick question about the expectations of using dynamic linking. The readme in rdkafka-sys claims:

    The system version of librdkafka must exactly match the version of librdkafka bundled with this crate.

    However, in the build.rs appears to be searching for a version of librdkafka with *at least the version of the one bundled within itself. The commit on the readme appears to be newer, should the build.rs be updated to reflect what the readme says? Thanks.

    opened by hallfox 0
  • How to build for x86_64-unknown-linux-musl

    How to build for x86_64-unknown-linux-musl

    I have this in Cargo toml

    rdkafka = { version = "0.29.0", features = [ "ssl", "cmake-build"] }

    I tried to compile to x86_64-unknown-linux-musl using 2 options. Both failed.

    1. cargo build --target x86_64-unknown-linux-musl --release OUTPUT: linking with cc failed: exit status: 1 rdkafka_ssl.c:(.text.rd_kafka_ssl_ctx_term+0x2b): undefined reference to `ENGINE_free' collect2: error: ld returned 1 exit status

    2. cross build --target=x86_64-unknown-linux-musl --release OUTPUT: rdkafka_ssl.c:(.text.rd_kafka_ssl_ctx_term+0x2b): undefined reference to `ENGINE_free' collect2: error: ld returned 1 exit status

    They both give the same error.

    I am using Arch and it doesn't fail using just "cmake-build" feature

    Here is the github repo I made where the issue is recreated https://github.com/0xDjole/rust-rdkafka-musl

    Can rust-rdkafka be compiled to x86_64-unknown-linux-musl ?

    opened by 0xDjole 0
  • StreamConsumer commite message from tokio spawn.

    StreamConsumer commite message from tokio spawn.

    Hi, I have used the example provided for the asynchronous processing. and tried commited message. But I can't commited borrow message.

    tokio = { version = "1.22.0", features = ["full"] }

    rdkafka = { version = "0.29.0", features = ["cmake-build"] }

    
    
     pub async fn run_async_processor(brokers: String, group_id: String, topic: String, worker: i32) {
        let consumer: Arc<StreamConsumer> = Arc::new(ClientConfig::new()
            .set("group.id", &group_id)
            .set("bootstrap.servers", &brokers)
            .set("enable.auto.commit", "false")
            .create()
            .expect("Consumer creation failed")
        );
        consumer
            .subscribe(&[&topic])
            .expect("Can't subscribe to specified topic");
        let stream_processor = consumer.stream().try_for_each(|borrowed_message| {
            async move {
                let _ = match borrowed_message.payload_view::<str>() {
                    None => "",
                    Some(Ok(s)) => s,
                    Some(Err(e)) => {
                        println!("Error while deserializing message payload: {:?}", e);
                        ""
                    }
                };
                consumer.commit_message(&borrowed_message, CommitMode::Async).unwrap();
                Ok(())
            }
        });
        println!("Starting daemon consumer #{}", worker);
    
        stream_processor.await.expect("stream processing failed");
    }
    

    i get error

    error[E0507]: cannot move out of `consumer`, a captured variable in an `FnMut` closure
      --> src/broker/mod.rs:26:20
       |
    14 |       let consumer: Arc<StreamConsumer> = Arc::new(ClientConfig::new()
       |           -------- captured outer variable
    ...
    25 |       let stream_processor = consumer.stream().try_for_each(|borrowed_message| {
       |                                                             ------------------ captured by this `FnMut` closure
    26 |           async move {
       |  ____________________^
    27 | |             let _ = match borrowed_message.payload_view::<str>() {
    28 | |                 None => "",
    29 | |                 Some(Ok(s)) => s,
    ...  |
    35 | |             consumer.commit_message(&borrowed_message, CommitMode::Async).unwrap();
       | |             --------
       | |             |
       | |             variable moved due to use in generator
       | |             move occurs because `consumer` has type `Arc<StreamConsumer>`, which does not implement the `Copy` trait
    36 | |             Ok(())
    37 | |         }
       | |_________^ move out of `consumer` occurs here
    
    error[E0505]: cannot move out of `consumer` because it is borrowed
      --> src/broker/mod.rs:25:59
       |
    25 |     let stream_processor = consumer.stream().try_for_each(|borrowed_message| {
       |                            ----------------- ------------ ^^^^^^^^^^^^^^^^^^ move out of `consumer` occurs here
       |                            |                 |
       |                            |                 borrow later used by call
       |                            borrow of `consumer` occurs here
    ...
    35 |             consumer.commit_message(&borrowed_message, CommitMode::Async).unwrap();
       |             -------- move occurs due to use in closure
    

    Help me please.

    opened by BruAPAHE 0
  • Allow purging (Purge API) / don't leak on dropping producer

    Allow purging (Purge API) / don't leak on dropping producer

    Resolves #518

    What it does:

    • Purges producer on drop
    • Updates bindgen script to more recent bindgen version
    • Sets the parameter that makes macro constants signed (because that's how they are generally used in librdkafka)

    Todo:

    • [x] Check that it works
    • [x] Expose the full interface, possibly through bitflags
    opened by Ten0 2
  • Allow purging (purge API) / don't leak on dropping producer

    Allow purging (purge API) / don't leak on dropping producer

    I'm looking for a way to handle messages that have failed to send upon closing.

    Currently, dropping the producer when kafka server is not started gives the following message:

    [WARN] librdkafka: TERMINATE [thrd:app]: Producer terminating with 2 messages (13 bytes) still in queue or transit: use flush() to wait for outstanding message delivery
    

    Flushing beforehand (ThreadedProducer::flush) will obviously always timeout because server is not running, and doesn't change the previous message.

    Delivery callbacks from ProducerContext do not get called for these messages, leading to memory leaks (as Box/Arc/... are not cast back).

    librdkafka provides the purge API to solve this issue (https://github.com/edenhill/librdkafka/issues/990, https://github.com/edenhill/librdkafka/commit/107af2a5c9c564b71b06266227e76f1f8fad942c#diff-84ac6df9f4f6c13f1f4eecf3b9c0e2cee4d7c4c005feea512ff3f82c000fb84eR2211).

    It would be useful to have access to this API for scenarios such as mine or those described at https://github.com/edenhill/librdkafka/issues/990.

    As the Rust API also does some nice IntoOpaque casting where it may leak if callbacks are not called, I think it would also be best to actually force an instant purge/flush when dropping the BaseProducer, both ensuring that the DeliveryOpaques don't get somewhat-silently leaked, and making it more natural for users to handle failure to deliver messages. (The reason this wasn't done in librdkafka is that users may already be handling this separately, but that could just be released in 0.30.0 which would allow breaking changes - and this is unlikely to break much because such handling would be severely un-idiomatic as people would have to keep track of the set of raw pointers that were sent, removing them from the list as they get a callback, and cast them back using unsafe code to drop them in case they didn't receive the callback, instead of just opening the issue I'm opening right now)

    Thanks!

    opened by Ten0 2
Releases(v0.29.0)
Owner
Federico Giraud
Federico Giraud
A highly efficient daemon for streaming data from Kafka into Delta Lake

kafka-delta-ingest The kafka-delta-ingest project aims to build a highly efficient daemon for streaming data through Apache Kafka into Delta Lake. Thi

Delta Lake 173 Dec 28, 2022
An asynchronous, multi-producer, single-consumer (MPSC) bounded channel that operates at tachyonic speeds

tachyonix An asynchronous, multi-producer, single-consumer (MPSC) bounded channel that operates at tachyonic speeds. This library is an offshoot of As

Asynchronics 66 Jan 29, 2023
Easy-to-use beanstalkd client for Rust (IronMQ compatible)

rust-beanstalkd Easy-to-use beanstalkd client for Rust (IronMQ compatible) Install Add this dependency to your Cargo.toml beanstalkd = "*" Documentati

Johannes Schickling 44 Oct 4, 2022
The lightest distributed consensus library. Run your own replicated state machine! ❤️

Little Raft The lightest distributed consensus library. Run your own replicated state machine! ❤️ Installing Simply import the crate. In your Cargo.to

Ilya Andreev 359 Dec 26, 2022
Easy Hadoop Streaming and MapReduce interfaces in Rust

Efflux Efflux is a set of Rust interfaces for MapReduce and Hadoop Streaming. It enables Rust developers to run batch jobs on Hadoop infrastructure wh

Isaac Whitfield 31 Nov 22, 2022
libhdfs binding and wrapper APIs for Rust

hdfs-rs libhdfs binding library and rust APIs which safely wraps libhdfs binding APIs Current Status Alpha Status (Rust wrapping APIs can be changed)

Hyunsik Choi 32 Dec 1, 2022
Fluvio is a high-performance distributed streaming platform that's written in Rust

Fluvio is a high-performance distributed streaming platform that's written in Rust, built to make it easy to develop real-time applications.

InfinyOn 1.6k Dec 30, 2022
Magical Automatic Deterministic Simulator for distributed systems in Rust.

MadSim Magical Automatic Deterministic Simulator for distributed systems. Deterministic simulation MadSim is a Rust async runtime similar to tokio, bu

MadSys Research Group 249 Dec 28, 2022
The Raft algorithm implement by Rust.

Raft The Raft algorithm implement by Rust. This project refers to Eli Bendersky's website, the link as follows: https://eli.thegreenplace.net/2020/imp

Qiang Zhao 1 Oct 23, 2021
Raft distributed consensus for WebAssembly in Rust

WRaft: Raft in WebAssembly What is this? A toy implementation of the Raft Consensus Algorithm for WebAssembly, written in Rust. Basically, it synchron

Emanuel Evans 60 Oct 22, 2022
Paxakos is a pure Rust implementation of a distributed consensus algorithm

Paxakos is a pure Rust implementation of a distributed consensus algorithm based on Leslie Lamport's Paxos. It enables distributed systems to consistently modify shared state across their network, even in the presence of failures.

Pavan Ananth Sharma 2 Jul 5, 2022
librdkafka - the Apache Kafka C/C++ client library

librdkafka - the Apache Kafka C/C++ client library Copyright (c) 2012-2020, Magnus Edenhill. https://github.com/edenhill/librdkafka librdkafka is a C

Magnus Edenhill 6.4k Dec 31, 2022
Easy c̵̰͠r̵̛̠ö̴̪s̶̩̒s̵̭̀-t̶̲͝h̶̯̚r̵̺͐e̷̖̽ḁ̴̍d̶̖̔ ȓ̵͙ė̶͎ḟ̴͙e̸̖͛r̶̖͗ë̶̱́ṉ̵̒ĉ̷̥e̷͚̍ s̷̹͌h̷̲̉a̵̭͋r̷̫̊ḭ̵̊n̷̬͂g̵̦̃ f̶̻̊ơ̵̜ṟ̸̈́ R̵̞̋ù̵̺s̷̖̅ţ̸͗!̸̼͋

Rust S̵̓i̸̓n̵̉ I̴n̴f̶e̸r̵n̷a̴l mutability! Howdy, friendly Rust developer! Ever had a value get m̵̯̅ð̶͊v̴̮̾ê̴̼͘d away right under your nose just when

null 294 Dec 23, 2022
An API for getting questions from http://either.io implemented fully in Rust, using reqwest and some regex magic. Provides asynchronous and blocking clients respectively.

eithers_rust An API for getting questions from http://either.io implemented fully in Rust, using reqwest and some regex magic. Provides asynchronous a

null 2 Oct 24, 2021
A high level async Redis client for Rust built on Tokio and Futures.

A high level async Redis client for Rust built on Tokio and Futures.

Alec Embke 108 Jan 1, 2023
Ratchet is a fast, robust, lightweight and fully asynchronous implementation of RFC6455 (The WebSocket protocol).

Ratchet ?? Ratchet is a fast, robust, lightweight and fully asynchronous implementation of RFC6455 (The WebSocket protocol). Complete with an optional

SwimOS 7 Dec 16, 2022
The gRPC library for Rust built on C Core library and futures

gRPC-rs gRPC-rs is a Rust wrapper of gRPC Core. gRPC is a high performance, open source universal RPC framework that puts mobile and HTTP/2 first. Sta

TiKV Project 1.6k Jan 7, 2023
Rust client for Apache Kafka

Kafka Rust Client Project Status This project is starting to be maintained by John Ward, the current status is that I am bringing the project up to da

Yousuf Fauzan 902 Jan 2, 2023
Rust client for Apache Kafka

Kafka Rust Client Project Status This project is starting to be maintained by John Ward, the current status is that I am bringing the project up to da

Kafka Rust 900 Dec 26, 2022
Futures-based QUIC implementation in Rust

Pure-rust QUIC protocol implementation Quinn is a pure-rust, future-based implementation of the QUIC transport protocol undergoing standardization by

null 2.6k Jan 8, 2023