A fully asynchronous, futures-based Kafka client library for Rust based on librdkafka

Overview

rust-rdkafka

crates.io docs.rs Build Status coverate Join the chat at https://gitter.im/rust-rdkafka/Lobby

A fully asynchronous, futures-enabled Apache Kafka client library for Rust based on librdkafka.

The library

rust-rdkafka provides a safe Rust interface to librdkafka. The master branch is currently based on librdkafka 1.6.0.

Documentation

Features

The main features provided at the moment are:

  • Support for all Kafka versions since 0.8.x. For more information about broker compatibility options, check the librdkafka documentation.
  • Consume from single or multiple topics.
  • Automatic consumer rebalancing.
  • Customizable rebalance, with pre and post rebalance callbacks.
  • Synchronous or asynchronous message production.
  • Customizable offset commit.
  • Create and delete topics and add and edit partitions.
  • Alter broker and topic configurations.
  • Access to cluster metadata (list of topic-partitions, replicas, active brokers etc).
  • Access to group metadata (list groups, list members of groups, hostnames, etc.).
  • Access to producer and consumer metrics, errors and callbacks.
  • Exactly-once semantics (EOS) via idempotent and transactional producers and read-committed consumers.

One million messages per second

rust-rdkafka is designed to be easy and safe to use thanks to the abstraction layer written in Rust, while at the same time being extremely fast thanks to the librdkafka C library.

Here are some benchmark results using the BaseProducer, sending data to a single Kafka 0.11 process running in localhost (default configuration, 3 partitions). Hardware: Dell laptop, with Intel Core i7-4712HQ @ 2.30GHz.

  • Scenario: produce 5 million messages, 10 bytes each, wait for all of them to be acked

    • 1045413 messages/s, 9.970 MB/s (average over 5 runs)
  • Scenario: produce 100000 messages, 10 KB each, wait for all of them to be acked

    • 24623 messages/s, 234.826 MB/s (average over 5 runs)

For more numbers, check out the kafka-benchmark project.

Client types

rust-rdkafka provides low level and high level consumers and producers.

Low level:

  • BaseConsumer: a simple wrapper around the librdkafka consumer. It must be periodically poll()ed in order to execute callbacks, rebalances and to receive messages.
  • BaseProducer: a simple wrapper around the librdkafka producer. As in the consumer case, the user must call poll() periodically to execute delivery callbacks.
  • ThreadedProducer: a BaseProducer with a separate thread dedicated to polling the producer.

High level:

For more information about consumers and producers, refer to their module-level documentation.

Warning: the library is under active development and the APIs are likely to change.

Asynchronous data processing with Tokio

Tokio is a platform for fast processing of asynchronous events in Rust. The interfaces exposed by the StreamConsumer and the FutureProducer allow rust-rdkafka users to easily integrate Kafka consumers and producers within the Tokio platform, and write asynchronous message processing code. Note that rust-rdkafka can be used without Tokio.

To see rust-rdkafka in action with Tokio, check out the asynchronous processing example in the examples folder.

At-least-once delivery

At-least-once delivery semantics are common in many streaming applications: every message is guaranteed to be processed at least once; in case of temporary failure, the message can be re-processed and/or re-delivered, but no message will be lost.

In order to implement at-least-once delivery the stream processing application has to carefully commit the offset only once the message has been processed. Committing the offset too early, instead, might cause message loss, since upon recovery the consumer will start from the next message, skipping the one where the failure occurred.

To see how to implement at-least-once delivery with rdkafka, check out the at-least-once delivery example in the examples folder. To know more about delivery semantics, check the message delivery semantics chapter in the Kafka documentation.

Exactly-once semantics

Exactly-once semantics (EOS) can be achieved using transactional producers, which allow produced records and consumer offsets to be committed or aborted atomically. Consumers that set their isolation.level to read_committed will only observe committed messages.

EOS is useful in read-process-write scenarios that require messages to be processed exactly once.

To learn more about using transactions in rust-rdkafka, see the Transactions section of the producer documentation.

Users

Here are some of the projects using rust-rdkafka:

If you are using rust-rdkafka, please let us know!

Installation

Add this to your Cargo.toml:

[dependencies]
rdkafka = { version = "0.25", features = ["cmake-build"] }

This crate will compile librdkafka from sources and link it statically to your executable. To compile librdkafka you'll need:

  • the GNU toolchain
  • GNU make
  • pthreads
  • zlib: optional, but included by default (feature: libz)
  • cmake: optional, not included by default (feature: cmake-build)
  • libssl-dev: optional, not included by default (feature: ssl)
  • libsasl2-dev: optional, not included by default (feature: gssapi)
  • libzstd-dev: optional, not included by default (feature: zstd-pkg-config)

Note that using the CMake build system, via the cmake-build feature, is encouraged if you can take the dependency on CMake.

By default a submodule with the librdkafka sources pinned to a specific commit will be used to compile and statically link the library. The dynamic-linking feature can be used to instead dynamically link rdkafka to the system's version of librdkafka. Example:

[dependencies]
rdkafka = { version = "0.25", features = ["dynamic-linking"] }

For a full listing of features, consult the rdkafka-sys crate's documentation. All of rdkafka-sys features are re-exported as rdkafka features.

Minimum supported Rust version (MSRV)

The current minimum supported Rust version (MSRV) is 1.45.0. Note that bumping the MSRV is not considered a breaking change. Any release of rust-rdkafka may bump the MSRV.

Asynchronous runtimes

Some features of the StreamConsumer and FutureProducer depend on Tokio, which can be a heavyweight dependency for users who only intend to use the low-level consumers and producers. The Tokio integration is enabled by default, but can be disabled by turning off default features:

[dependencies]
rdkafka = { version = "0.25", default-features = false }

If you would like to use an asynchronous runtime besides Tokio, you can integrate it with rust-rdkafka by providing a shim that implements the AsyncRuntime trait. See the smol runtime example for an example integration with smol.

Examples

You can find examples in the examples folder. To run them:

cargo run --example <example_name> -- <example_args>

Debugging

rust-rdkafka uses the log and env_logger crates to handle logging. Logging can be enabled using the RUST_LOG environment variable, for example:

RUST_LOG="librdkafka=trace,rdkafka::client=debug" cargo test

This will configure the logging level of librdkafka to trace, and the level of the client module of the Rust client to debug. To actually receive logs from librdkafka, you also have to set the debug option in the producer or consumer configuration (see librdkafka configuration).

To enable debugging in your project, make sure you initialize the logger with env_logger::init(), or the equivalent for any log-compatible logging framework.

rdkafka-sys

See rdkafka-sys.

Contributors

Thanks to:

Alternatives

  • kafka-rust: a pure Rust implementation of the Kafka client.
Comments
  • First time user

    First time user

    Hello! I've recently tried to use rust-rdkafka, and thought I would share some experiences. I think there may be some "issues" here, but there could also just be some helpful information about what people (person) stumble over.

    First off, thank you! This is in support of a timely dataflow Kafka adapter, which I wouldn't want to try and write without a crate. :)

    I wanted to try sending a message through kafka and back, so I wrote what seemed like a pretty simple program, following your examples but using the Base* variants of producer and consumer.

    fn round_trip(brokers: &str, topic_name: &str) -> Result<(), rdkafka::error::KafkaError> {
    
        let mut topic_config = TopicConfig::new();
        topic_config
            .set("produce.offset.report", "true")
            .finalize();
    
        let mut client_config = ClientConfig::new();
        client_config
            .set("group.id", "example")
            .set("bootstrap.servers", brokers)
            .set_default_topic_config(topic_config);
    
        let producer: BaseProducer<_> = try!(client_config.create());
        let consumer: BaseConsumer<_> = try!(client_config.create());
    
        try!(consumer.subscribe(&[topic_name]));
    
        let text = format!("hi there at {:?}", ::std::time::Instant::now());
        try!(producer.send_copy::<str,()>(topic_name, None, Some(text.as_str()), None,  None, None));
    
        for _ in 0 .. 100 {
    
            producer.poll(1);
    
            if let Some(result) = try!(consumer.poll(1)) {
                println!("{:?}:\t{:?}", ::std::time::Instant::now(), result.payload_view::<str>());
                try!(consumer.commit(None, rdkafka::consumer::CommitMode::Sync));
            }
            else {
                println!("{:?}:\tgot nothing", ::std::time::Instant::now());
            }
        }
    
        Ok(())
    }
    

    The behavior was a bit surprising, and it was tricky to find documentation to explain if this was expected.

    1. This seems to run to completion just fine, printing "got nothing" 100 times and returning Ok(()) when neither ZK nor Kafka are up and running. Where is the moment in the code it is supposed to complain if they are not installed or up and running? (when they are up and running, it does seem to connect and occasionally round-trip data; mysterious things happen).

    2. I'm not a Kafka expert, so I was surprised to see things like "nothing more to read now" surface as an error (PartitionEOF) rather than the absence of a message. I imagine that is just what rdkafka thinks is best and returns to you, so you want to surface it up?

    3. The BaseProducer documentation speaks of a BaseProducerTopic that I should use, but I couldn't find this. Is it just stale documentation?

    4. I was hoping to have a non-blocking poll, but I'm not sure if this is what an argument of 0 would mean. Sometimes 0 means "block". Is that something you could clarify in the documentation? It could be that rdkafka does what it does and won't let us fcntl the socket to non-blocking mode, or something like that, but would be good to know!

    Maybe as a meta comment, if BaseProducer and BaseConsumer are meant to be a very thin layer on top of rdkafka, with the caveat that you had best understand the details of rdkafka (which I don't yet), you could say that in the documentation (I'm still not sure). I'm sure it is still useful, but would keep me from scratching / banging my head.

    I'm going to try out the "higher-level" stuff next, but I'd have preferred to just have a non-blocking polling interface. If that should work out, and I just need more Kafka protocol expertise, I can push on it more and come back with comments on that.

    Thanks again!

    opened by frankmcsherry 38
  • Directly return a DeliveryFuture from FutureProducer::send_copy

    Directly return a DeliveryFuture from FutureProducer::send_copy

    Returning a Result made it very difficult to use send_copy() in a chain of futures, and, since a Future is just an asynchronous Result, this commit changes the DeliveryFuture to yield a KafkaError on failure. Previously, the DeliveryFuture would only yield Canceled, and, to continue supporting this, a new variant of KafkaError was added, FutureCanceled.

    This makes the FutureProducer API considerably more ergonomic. For example, in a function which returns a BoxFuture<(), ::std::io::Error>, this is what was necessary before:

    return match kafka_producer.send_copy::<_, ()>(
        "test", None, Some(&*format!("msg on {}", topic)), None, None) {
        Ok(report) => {
            report
                .map_err(|_| io::Error::new(io::ErrorKind::Other,
                                            "failed to send kafka message"))
                .and_then(|_| future::ok(()))
                .boxed()
        },
    
        Err(_) => {
            future::err(io::Error::new(
                    io::ErrorKind::InvalidData, "failed to send kafka message"))
                .boxed()
        }
    }
    

    This becomes even more complicated with the fact that match arms must return the same type and the types of chained Futures can easily get very complex.

    Compare that to the code one can write with this PR applied:

    return kafka_producer
        .send_copy::<_, ()>("test", None, Some(&*format!("msg on {}", topic)), None, None)
        .map_err(|_| io::Error::new(io::ErrorKind::Other,
                                    "failed to send kafka message"))
        .and_then(|_| future::ok(()))
        .boxed();
    

    Much less verbose, and, more importantly, it flows naturally as part of a chain of futures.

    Full disclosure: there is a compilation warning for the unused Result returned by tx.send() in two places, but this warning was present before the change and I'm also not sure what the correct way to handle an error arising in such a situation actually is.

    opened by wrl 25
  • Memory keeps growing

    Memory keeps growing

    Not sure if this is a problem or where it is but after creating a program with FutureProducer my memory skyrocketed to 10Gb usage by having 3 FutureProducers running at the same time (otherwise I get limited to 50k EPS).

    Even when running only one memory usage was at 4gb.

    After chasing this a little while, I have tried running the asynchronous_processing example, and memory also grows there at least until 1GB, any idea what can be causing this?

    ss-2020-02-23-130822

    opened by llitz 18
  • Running StreamConsumer with tokio's executor

    Running StreamConsumer with tokio's executor

    Currently how StreamConsumer stores the JoinHandle in a Cell prevents us of using the consumer in tokio's thread_pool executor, which expects the consumer to implement Send. In other hand using the current_thread executor doesn't comply with missing static lifetimes somewhere in the consumer code I haven't been able to locate yet.

    Is there plans to support the new executors, especially when the development of tokio-core and futures-cpupool seems to be stalled in favor of tokio?

    opened by pimeys 16
  • Compilation fails w/ rust-musl-builder

    Compilation fails w/ rust-musl-builder

    When compiling a rust application using rdkafka on rust-musl-builder the build fails when linking:

    My guess is that the rdkafka lib is build with glibc and then fails w/ musl.

    Is there a way to specify what to build it against the same way that dynamic linking can be specified?

    error: linking with `cc` failed: exit code: 1
      |
      = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-nostdlib" "-Wl,--eh-frame-hdr" "-Wl,-(" "-m64" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crt1.o" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crti.o" "-L" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.mshello0-66b6f0d31720f4bd44d6f265903cd36d.rs.rcgu.o" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.mshello1-66b6f0d31720f4bd44d6f265903cd36d.rs.rcgu.o" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.mshello2-66b6f0d31720f4bd44d6f265903cd36d.rs.rcgu.o" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.mshello3-66b6f0d31720f4bd44d6f265903cd36d.rs.rcgu.o" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.mshello4-66b6f0d31720f4bd44d6f265903cd36d.rs.rcgu.o" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.mshello5-66b6f0d31720f4bd44d6f265903cd36d.rs.rcgu.o" "-o" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.crate.allocator.rcgu.o" "-Wl,--gc-sections" "-no-pie" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-nodefaultlibs" "-L" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps" "-L" "/home/rust/src/target/release/deps" "-L" "/home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src" "-L" "/home/rust/src/target/x86_64-unknown-linux-musl/release/build/libz-sys-9e0dfdf6ce77c312/out/lib" "-L" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc_jemalloc-f68d8e8245e507c6.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka-a45674fad582d6de.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/liblibz_sys-d28970724bb40e9f.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/liblibc-eef0eb2737b72223.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/libfutures-f1a25787e6cfbe44.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/libserde_json-fded606be9e377b3.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/libitoa-539d7e09fab6c05d.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/libdtoa-19c34238b45a98bc.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/libserde-30812860af6099f8.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/liblog-4de18e7534a813d8.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/liblog-9a6200d5626a5e8a.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/libcfg_if-5f123802e50df72d.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-8d847bbe97fc9dc5.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-008c42aad548e7dd.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libunwind-0fe53371ec419e32.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc_system-9dec1cbd51097ce1.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/liblibc-6a4fb915dd86d140.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-7ebba6af2d3cc324.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd_unicode-7a26f8b3cf380464.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-b0c2d164a9741309.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-66d072e25a9acee3.rlib" "-static" "-Wl,-Bdynamic" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crtn.o" "-Wl,-)"
      = note: /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `vsnprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:77: undefined reference to `__vsnprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:77: undefined reference to `__vsnprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:64: more undefined references to `__snprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `vsnprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:77: undefined reference to `__vsnprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `rd_strndup':
              /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
              /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
              /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
              /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
              /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o):/home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: more undefined references to `__strndup' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `syslog':
              /usr/include/x86_64-linux-gnu/bits/syslog.h:31: undefined reference to `__syslog_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `vsnprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:77: undefined reference to `__vsnprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_msg.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_msg.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_conf.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_conf.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:64: more undefined references to `__snprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_conf.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_conf.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_conf.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_conf.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_offset.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_offset.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_offset.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_offset.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_offset.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_offset.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_transport.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_transport.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:64: more undefined references to `__snprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_transport.o): In function `poll':
              /usr/include/x86_64-linux-gnu/bits/poll2.h:41: undefined reference to `__poll_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_buf.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_buf.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_buf.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_queue.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_op.o): In function `vsnprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:77: undefined reference to `__vsnprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_op.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_op.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_request.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_request.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_request.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_request.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_request.o): In function `rd_strndup':
              /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_request.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_cgrp.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_cgrp.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_cgrp.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_cgrp.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_cgrp.o): In function `rd_strndup':
              /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
              /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_cgrp.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_pattern.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o): In function `rd_strndup':
              /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_assignor.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdstring.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:64: more undefined references to `__snprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_metadata.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_metadata.o): In function `rd_strndup':
              /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdbuf.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdbuf.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(snappy.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rddl.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rddl.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:64: more undefined references to `__snprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(crc32c.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(crc32c.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdaddr.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdaddr.o): In function `strncpy':
              /usr/include/x86_64-linux-gnu/bits/string3.h:126: undefined reference to `__strncpy_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdaddr.o): In function `strcpy':
              /usr/include/x86_64-linux-gnu/bits/string3.h:110: undefined reference to `__memcpy_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdlist.o): In function `printf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:104: undefined reference to `__printf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:104: undefined reference to `__printf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_metadata_cache.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_sasl.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_msgset_reader.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_msgset_reader.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_msgset_reader.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdvarint.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdvarint.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_plugin.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdlog.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdlog.o): In function `snprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdlog.o): In function `fprintf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
              /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdlog.o): In function `printf':
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:104: undefined reference to `__printf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:104: undefined reference to `__printf_chk'
              /usr/include/x86_64-linux-gnu/bits/stdio2.h:104: undefined reference to `__printf_chk'
    
    opened by Licenser 15
  • Batch Processing

    Batch Processing

    I understand the hesitation in adding to the publishing api an explicit batch processing fn (it already batches under the hood) - but it's unclear to me the purpose of not having this api in the consumer. We should remove complexity from the consumer. I note this after going through previous replies to the same feature request

    opened by lacasaprivata2 13
  • Is this function safe to use and support futures for async processing ? Need simple Producer usage

    Is this function safe to use and support futures for async processing ? Need simple Producer usage

    Hi,

    I need to use simple function to produce Kafka messages with a solid function, that process producing on demand when requested by users:

    //Send messages to Kafka
    fn produce(brokers: &str, topic_name: &str, messages: String) {
        let producer: FutureProducer = ClientConfig::new()
            .set("bootstrap.servers", brokers)
            .set("produce.offset.report", "true")
            .set("message.timeout.ms", "5000")
            .create()
            .expect("Producer creation error");
    
    
        producer.send(
            FutureRecord::to(topic_name)
                .payload(&format!("Message {}", messages))
                .key(&format!("Key {}", messages))
                .headers(OwnedHeaders::new()
                    .add("header_key", "header_value")),
            0
        );
    }
    

    Do this function is safe and performente ? support futures ?

    Thanks in advance.

    opened by KabDeveloper 13
  • 0.21.0 breaks build on Rust Docker image

    0.21.0 breaks build on Rust Docker image

    Problem

    Using the official Rust docker image, I am unable to build version 0.21.0, whereas 0.20.0 worked without issue. The error I see is:

    --- stderr
    Building and linking librdkafka statically
    Running command: "./configure --disable-sasl --enable-ssl --disable-lz4" in dir: librdkafka
    Running command: "make -j 4 libs" in dir: librdkafka
    thread 'main' panicked at 'Unable to find libclang: "couldn\'t find any valid shared libraries matching: [\'libclang.so\', \'libclang-*.so\', \'libclang.so.*\'], set the `LIBCLANG_PATH` environment variable to a path where one of these files can be found (invalid: [])"', src/libcore/result.rs:997:5
    note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
    

    Solution

    I am able to build successfully after installing some additional dependencies in the image (copied from the Dockerfile in this repo): apt-get install -y llvm-3.9-dev libclang-3.9-dev clang-3.9

    Based on the description in d9321f2b1f132d554d2c5854803fcb2869c6495c, it looks like the new dependencies are intended. Perhaps the new dependencies could be mentioned in the changelog to help anyone with broken builds?

    Let me know if there is additional information I can provide.

    Thanks!

    opened by austinhartzheim 12
  • Release 0.22 build from cargo fails after the release of 0.23

    Release 0.22 build from cargo fails after the release of 0.23

    I'm having problems building my project that is using release 0.22.

    It seems that problems started after the release of rust-rdkafka 0.23 and rdkafka-sys 1.3.1.

       Compiling rdkafka-sys v1.3.1
      ...
       Compiling rdkafka v0.22.0
    error[E0425]: cannot find function `primitive_to_rd_kafka_resp_err_t` in crate `rdsys`
       --> /Users/tkasu/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-0.22.0/src/client.rs:411:22
        |
    411 |     let err = rdsys::primitive_to_rd_kafka_resp_err_t(err)
        |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ not found in `rdsys`
    
    

    Release 0.22 is fetching rdkafka-sys 1.3.1 from which the function primitive_to_rd_kafka_resp_err_t has been removed.

    opened by tkasu 11
  • WIP - switch to std::future and async/await

    WIP - switch to std::future and async/await

    This PR moves the project over from futures 0.1 to the newly stabilized std::future, using tokio 0.2 (alpha) and futures-preview 0.3. It won't compile on stable until Rust 1.39 at the earliest since it uses async/await, so it'll obviously bump up the minimum Rust version too. It may be possible to remove the async/await which I think would allow it to compile on 1.36.0.

    Tests are passing and examples working on nightly.

    I guess this won't want to be merged until things stabilize a little more, but thought it might be useful when that time comes!

    opened by sd2k 11
  • Error linking libzstd.so on ubuntu 19.04

    Error linking libzstd.so on ubuntu 19.04

    building this repo or a project with it as a dependency on ubuntu 19.04 I get this error:

       Compiling rdkafka v0.21.0 (rust-rdkafka)
    error: linking with `cc` failed: exit code: 1
      |
      = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" [...] "-Wl,-Bdynamic" "-lz" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"
      = note: /usr/bin/ld: rust-rdkafka/target/debug/deps/librdkafka_sys-3d04a478eb2ea7f3.rlib(rdkafka_zstd.o): in function `rd_kafka_zstd_decompress':
              rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:44: undefined reference to `ZSTD_getFrameContentSize'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:91: undefined reference to `ZSTD_getErrorCode'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:80: undefined reference to `ZSTD_decompress'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:82: undefined reference to `ZSTD_isError'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:100: undefined reference to `ZSTD_getErrorName'
              /usr/bin/ld: rust-rdkafka/target/debug/deps/librdkafka_sys-3d04a478eb2ea7f3.rlib(rdkafka_zstd.o): in function `rd_kafka_zstd_compress':
              rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:130: undefined reference to `ZSTD_compressBound'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:141: undefined reference to `ZSTD_createCStream'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:155: undefined reference to `ZSTD_initCStream'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:157: undefined reference to `ZSTD_isError'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:213: undefined reference to `ZSTD_freeCStream'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:168: undefined reference to `ZSTD_compressStream'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:169: undefined reference to `ZSTD_isError'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:170: undefined reference to `ZSTD_getErrorName'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:213: undefined reference to `ZSTD_freeCStream'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:158: undefined reference to `ZSTD_getErrorName'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:213: undefined reference to `ZSTD_freeCStream'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:198: undefined reference to `ZSTD_endStream'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:199: undefined reference to `ZSTD_isError'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:213: undefined reference to `ZSTD_freeCStream'
              /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:200: undefined reference to `ZSTD_getErrorName'
              collect2: error: ld returned 1 exit status
    
    
    error: aborting due to previous error
    
    error: Could not compile `rdkafka`.
    

    if I copy the command in note: and add "-lzstd" it seems to link

    from

    "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" [...] "-Wl,-Bdynamic" "-lz" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"
    

    to

    "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" [...] "-Wl,-Bdynamic" "-lz" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil" "-lzstd"
    

    here is a Dockerfile to reproduce:

    FROM ubuntu:19.04
    
    ARG UNAME=mariano
    ARG UID=1000
    ARG GID=1000
    RUN groupadd -g $GID -o $UNAME
    RUN useradd -m -u $UID -g $GID -o -s /bin/bash $UNAME
    
    RUN \
      sed -i 's/# \(.*multiverse$\)/\1/g' /etc/apt/sources.list && \
      apt-get update; \
      apt-get -y upgrade && \
      apt-get install -y --no-install-recommends wget build-essential clang libclang-dev libssl-dev zlib1g-dev libsasl2-dev libzstd-dev curl ca-certificates git
    
    USER $UNAME
    RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
    

    then

    # build it
    sudo docker build --build-arg UID=$$(id -u) --build-arg GID=$$(id -g) -t ubuntu-rust:1904-stable ubuntu-1904-rust
    
    sudo docker run -i ubuntu-rust:1904-stable sh << COMMANDS
        cd
        . .cargo/env
        git clone https://github.com/fede1024/rust-rdkafka.git
        cd rust-rdkafka
        cargo run --example simple_consumer
    COMMANDS
    

    it seems for some reason it doesn't find libzstd

    $ ldconfig -p | grep libzstd
    
            libzstd.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libzstd.so.1
    

    I tried creating symbolic links in /usr/local/lib just in case but it didn't work.

    opened by marianoguerra 11
  • rdkafka-sys: make CARGO_MAKEFLAGS optional

    rdkafka-sys: make CARGO_MAKEFLAGS optional

    Currently, the expectation that the CARGO_MAKEFLAGS env var be set causes the rdkafka-sys/build.rs script to fail when invoked in certain contexts -- notably, through https://github.com/bazelbuild/rules_rust rules.

    The CARGO_MAKEFLAGS env var is expected in order to populate the MAKEFLAGS env var. However, MAKEFLAGS is only required by GNU make. This commit refactors this section so that CARGO_MAKEFLAGS is only expected if GNU make is to be used.

    opened by mattbonnell 0
  • Bump lz4-sys from 1.8.3 to 1.9.4

    Bump lz4-sys from 1.8.3 to 1.9.4

    Bumps lz4-sys from 1.8.3 to 1.9.4.

    Changelog

    Sourced from lz4-sys's changelog.

    1.24.0:

    • Update to lz4 1.9.4 (lz4-sys 1.9.4) - this fixes CVE-2021-3520, which was a security vulnerability in the core lz4 library
    • export the include directory of lz4 from build.rs

    1.23.3 (March 5, 2022):

    • Update lz4 to 1.9.3
    • Add [de]compress_to_buffer to block API to allow reusing buffers (#16)
    • Windows static lib support
    • Support favor_dec_speed
    • Misc small fixes

    1.23.2:

    • Update lz4 to 1.9.2
    • Remove dependency on skeptic (replace with build-dependency docmatic for README testing)
    • Move to Rust 2018 edition

    1.23.0:

    • Update lz4 to v1.8.2
    • Add lz4 block mode api

    1.22.0:

    • Update lz4 to v1.8.0
    • Remove lz4 redundant dependency to gcc #22 (thanks to Xidorn Quan)

    1.21.1:

    • Fix always rebuild issue #21

    1.21.0:

    • Fix smallest 11-byte stream decoding (thanks to Niklas Hambüchen)
    • Update lz4 to v1.7.5

    1.20.0:

    • Split out separate sys package #16 (thanks to Thijs Cadier)

    1.19.173:

    • Update lz4 to v1.7.3

    1.19.131:

    • Update dependencies for correct work with change build environmet via rustup override

    1.18.131:

    • Implemented Send for Encoder/Decoder #15 (thanks to Maxime Lenoir)

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the Security Alerts page.
    dependencies 
    opened by dependabot[bot] 0
  • Problem consuming data always from beginning.

    Problem consuming data always from beginning.

    Hello there,

    I'm trying to get all the data from the Kafka topic from the beginning. I set the auto.offset.reset flag to earliest and group.id to a random string, this works almost fine, but the problem is when I read the data, because every time I run this consumer, it gives me a different count of data, the data on the topic is around 13k, The most I have been able to read is 12k.

    I also tried to set partitions offset to the beginning but the result was always the same. After that, I used the assign method instead of subscribing.

    let t: &str = &topic[..];
    partitions.add_partition_offset(t, 0, Offset::Beginning).unwrap();
    partitions.add_partition_offset(t, 1, Offset::Beginning).unwrap();
    partitions.add_partition_offset(t, 2, Offset::Beginning).unwrap();
    

    Here is my consuming function:

    type LoggingConsumer = StreamConsumer<CustomContext, AsyncStdRuntime>;
    
    async fn consume(consumer: &LoggingConsumer) -> Result<Vec<String>, Box<dyn Error>> {
        let mut result: Vec<String> = vec![];
        loop {
            info!("Consuming message ...");
            let mut stream = consumer.stream();
            let message = stream.next().await;
            match message {
                Some(Ok(message)) => println!(
                    "Received message: {}",
                    match message.payload_view::<str>() {
                        None => {
                            error!("Message payload is not a string");
                            ""
                        },
                        Some(Ok(mess)) => {
                            info!("Message received: {}", mess);
                            result.push(String::from(mess));
                            mess
                        },
                        Some(Err(e)) => {
                            error!("Message payload error: {}", e);
                            ""
                        },
                    }
                ),
                Some(Err(KafkaError::PartitionEOF(_))) => {
                    info!("Reached end of topic");
                    break;
                }
                Some(Err(e)) => {
                    eprintln!("Error receiving message: {}", e);
                    if let RDKafkaErrorCode::TopicAuthorizationFailed = e.rdkafka_error_code().unwrap() {
                        info!("Topic authorization failed: {}", e);
                        break;
                    }
                    break;
                }
                None => {
                    eprintln!("Consumer unexpectedly returned no messages");
                }
            }
        }
    
        Ok(result)
    }
    
    opened by sectasy0 3
  • Requesting a new published version of `rdkafka-sys`

    Requesting a new published version of `rdkafka-sys`

    Hello,

    This dependabot commit is a pretty important change for our use case and might warrant a new version of the crate with a new tag.

    Changing major version here, for a -sys crate that has the links manifest key means that currently there is no version of rdkafka-sys that can coexist with libraries that also want to link to zstd, such as async-compression for example.

    Would it be possible to have a new version of rdkafka-sys published with this change in the toml file please ? Currently rdkafka-sys still reports a 1.4 zstd-sys dependency

    Thanks for the package, it's been a huge time saver for us :)

    opened by gagbo 1
  • Build failed with ssl features on mac m1

    Build failed with ssl features on mac m1

    Build is failing with following message when we run rdkafka with ssl feature enabled on mac m1.

              Undefined symbols for architecture arm64:
                "_SSL_get1_peer_certificate", referenced from:
                    _rd_kafka_transport_ssl_handshake in librdkafka_sys-66bfa0eb454c4b25.rlib(rdkafka_ssl.c.o)
              ld: symbol(s) not found for architecture arm64
              clang: error: linker command failed with exit code 1 (use -v to see invocation)
    
    openssl version: OpenSSL 1.1.1q  5 Jul 2022
    uname -m: arm64
    
    opened by aiyajsupra 1
Releases(v0.29.0)
  • v0.29.0(Jun 17, 2022)

    What's Changed

    • Fix handling of NULL pointer in NativeClient::rebalance_protocol by @bruceg in https://github.com/fede1024/rust-rdkafka/pull/417
    • Setup dependabot by @duarten in https://github.com/fede1024/rust-rdkafka/pull/419
    • Bump env_logger from 0.7.1 to 0.9.0 by @dependabot in https://github.com/fede1024/rust-rdkafka/pull/428
    • ci: test minimal versions by @benesch in https://github.com/fede1024/rust-rdkafka/pull/430
    • Bump zstd-sys from 1.4.19+zstd.1.4.8 to 1.6.2+zstd.1.5.1 by @dependabot in https://github.com/fede1024/rust-rdkafka/pull/431
    • [producer] pass through error when calling flush by @cjubb39 in https://github.com/fede1024/rust-rdkafka/pull/434
    • Reduce futures dependency by @benesch in https://github.com/fede1024/rust-rdkafka/pull/435
    • Fix misleading error message on pkg_config failure by @qbx2 in https://github.com/fede1024/rust-rdkafka/pull/438
    • add tracing feature by @guswynn in https://github.com/fede1024/rust-rdkafka/pull/440
    • convert final log line to crate::log by @guswynn in https://github.com/fede1024/rust-rdkafka/pull/443
    • Improve documentation of CommitMode enum by @aarashy in https://github.com/fede1024/rust-rdkafka/pull/449
    • add support for OAuth by @jsurany-bloomberg in https://github.com/fede1024/rust-rdkafka/pull/442
    • Allow distinguishing between null and empty header values by @benesch in https://github.com/fede1024/rust-rdkafka/pull/454
    • Tweak improved commit mode docs by @benesch in https://github.com/fede1024/rust-rdkafka/pull/455
    • Bump zstd-sys from 1.6.2+zstd.1.5.1 to 2.0.1+zstd.1.5.2 by @dependabot in https://github.com/fede1024/rust-rdkafka/pull/452
    • Add missing changelog items for v0.29.0 by @benesch in https://github.com/fede1024/rust-rdkafka/pull/456
    • Add detach_headers for OwnedMessage by @scanterog in https://github.com/fede1024/rust-rdkafka/pull/459
    • Add Callysto to the README by @vertexclique in https://github.com/fede1024/rust-rdkafka/pull/463
    • Enable Atomic swap for statistics coming from librdkafka by enabling Clone on Statistics by @vertexclique in https://github.com/fede1024/rust-rdkafka/pull/462
    • Add a client method to fetch cluster_id by @dopuskh3 in https://github.com/fede1024/rust-rdkafka/pull/465
    • Bump crossbeam-channel from 0.4.2 to 0.4.4 by @dependabot in https://github.com/fede1024/rust-rdkafka/pull/475
    • Bump nb-connect from 1.0.2 to 1.2.0 by @dependabot in https://github.com/fede1024/rust-rdkafka/pull/474
    • Fix possible unsoundness in IntoOpaque to do with pointer provenance by @zRedShift in https://github.com/fede1024/rust-rdkafka/pull/472

    New Contributors

    • @bruceg made their first contribution in https://github.com/fede1024/rust-rdkafka/pull/417
    • @duarten made their first contribution in https://github.com/fede1024/rust-rdkafka/pull/419
    • @dependabot made their first contribution in https://github.com/fede1024/rust-rdkafka/pull/428
    • @cjubb39 made their first contribution in https://github.com/fede1024/rust-rdkafka/pull/434
    • @qbx2 made their first contribution in https://github.com/fede1024/rust-rdkafka/pull/438
    • @guswynn made their first contribution in https://github.com/fede1024/rust-rdkafka/pull/440
    • @aarashy made their first contribution in https://github.com/fede1024/rust-rdkafka/pull/449
    • @jsurany-bloomberg made their first contribution in https://github.com/fede1024/rust-rdkafka/pull/442
    • @scanterog made their first contribution in https://github.com/fede1024/rust-rdkafka/pull/459
    • @vertexclique made their first contribution in https://github.com/fede1024/rust-rdkafka/pull/463
    • @dopuskh3 made their first contribution in https://github.com/fede1024/rust-rdkafka/pull/465
    • @zRedShift made their first contribution in https://github.com/fede1024/rust-rdkafka/pull/472

    Full Changelog: https://github.com/fede1024/rust-rdkafka/compare/v0.28.0...v0.29.0

    Source code(tar.gz)
    Source code(zip)
Owner
Federico Giraud
Federico Giraud
A highly efficient daemon for streaming data from Kafka into Delta Lake

kafka-delta-ingest The kafka-delta-ingest project aims to build a highly efficient daemon for streaming data through Apache Kafka into Delta Lake. Thi

Delta Lake 156 Sep 18, 2022
Easy-to-use beanstalkd client for Rust (IronMQ compatible)

rust-beanstalkd Easy-to-use beanstalkd client for Rust (IronMQ compatible) Install Add this dependency to your Cargo.toml beanstalkd = "*" Documentati

Johannes Schickling 45 Sep 16, 2022
The lightest distributed consensus library. Run your own replicated state machine! ❤️

Little Raft The lightest distributed consensus library. Run your own replicated state machine! ❤️ Installing Simply import the crate. In your Cargo.to

Ilya Andreev 327 Sep 22, 2022
Easy Hadoop Streaming and MapReduce interfaces in Rust

Efflux Efflux is a set of Rust interfaces for MapReduce and Hadoop Streaming. It enables Rust developers to run batch jobs on Hadoop infrastructure wh

Isaac Whitfield 28 Aug 16, 2022
libhdfs binding and wrapper APIs for Rust

hdfs-rs libhdfs binding library and rust APIs which safely wraps libhdfs binding APIs Current Status Alpha Status (Rust wrapping APIs can be changed)

Hyunsik Choi 31 Apr 4, 2022
Fluvio is a high-performance distributed streaming platform that's written in Rust

Fluvio is a high-performance distributed streaming platform that's written in Rust, built to make it easy to develop real-time applications.

InfinyOn 1.4k Sep 22, 2022
Magical Automatic Deterministic Simulator for distributed systems in Rust.

MadSim Magical Automatic Deterministic Simulator for distributed systems. Deterministic simulation MadSim is a Rust async runtime similar to tokio, bu

MadSys Research Group 192 Sep 22, 2022
The Raft algorithm implement by Rust.

Raft The Raft algorithm implement by Rust. This project refers to Eli Bendersky's website, the link as follows: https://eli.thegreenplace.net/2020/imp

Qiang Zhao 1 Oct 23, 2021
Raft distributed consensus for WebAssembly in Rust

WRaft: Raft in WebAssembly What is this? A toy implementation of the Raft Consensus Algorithm for WebAssembly, written in Rust. Basically, it synchron

Emanuel Evans 57 Sep 22, 2022
Paxakos is a pure Rust implementation of a distributed consensus algorithm

Paxakos is a pure Rust implementation of a distributed consensus algorithm based on Leslie Lamport's Paxos. It enables distributed systems to consistently modify shared state across their network, even in the presence of failures.

Pavan Ananth Sharma 2 Jul 5, 2022
librdkafka - the Apache Kafka C/C++ client library

librdkafka - the Apache Kafka C/C++ client library Copyright (c) 2012-2020, Magnus Edenhill. https://github.com/edenhill/librdkafka librdkafka is a C

Magnus Edenhill 6.2k Sep 29, 2022
Easy c̵̰͠r̵̛̠ö̴̪s̶̩̒s̵̭̀-t̶̲͝h̶̯̚r̵̺͐e̷̖̽ḁ̴̍d̶̖̔ ȓ̵͙ė̶͎ḟ̴͙e̸̖͛r̶̖͗ë̶̱́ṉ̵̒ĉ̷̥e̷͚̍ s̷̹͌h̷̲̉a̵̭͋r̷̫̊ḭ̵̊n̷̬͂g̵̦̃ f̶̻̊ơ̵̜ṟ̸̈́ R̵̞̋ù̵̺s̷̖̅ţ̸͗!̸̼͋

Rust S̵̓i̸̓n̵̉ I̴n̴f̶e̸r̵n̷a̴l mutability! Howdy, friendly Rust developer! Ever had a value get m̵̯̅ð̶͊v̴̮̾ê̴̼͘d away right under your nose just when

null 293 Sep 22, 2022
An API for getting questions from http://either.io implemented fully in Rust, using reqwest and some regex magic. Provides asynchronous and blocking clients respectively.

eithers_rust An API for getting questions from http://either.io implemented fully in Rust, using reqwest and some regex magic. Provides asynchronous a

null 2 Oct 24, 2021
Ratchet is a fast, robust, lightweight and fully asynchronous implementation of RFC6455 (The WebSocket protocol).

Ratchet ?? Ratchet is a fast, robust, lightweight and fully asynchronous implementation of RFC6455 (The WebSocket protocol). Complete with an optional

SwimOS 6 Sep 2, 2022
A high level async Redis client for Rust built on Tokio and Futures.

A high level async Redis client for Rust built on Tokio and Futures.

Alec Embke 89 Sep 15, 2022
Rust client for Apache Kafka

Kafka Rust Client Project Status This project is starting to be maintained by John Ward, the current status is that I am bringing the project up to da

Yousuf Fauzan 844 Sep 19, 2022
Rust client for Apache Kafka

Kafka Rust Client Project Status This project is starting to be maintained by John Ward, the current status is that I am bringing the project up to da

Kafka Rust 846 Sep 24, 2022
The gRPC library for Rust built on C Core library and futures

gRPC-rs gRPC-rs is a Rust wrapper of gRPC Core. gRPC is a high performance, open source universal RPC framework that puts mobile and HTTP/2 first. Sta

TiKV Project 1.6k Sep 24, 2022
Futures-based QUIC implementation in Rust

Pure-rust QUIC protocol implementation Quinn is a pure-rust, future-based implementation of the QUIC transport protocol undergoing standardization by

null 2.3k Sep 25, 2022
An asynchronous Rust client library for the Hashicorp Vault API

vaultrs An asynchronous Rust client library for the Hashicorp Vault API The following features are currently supported: Auth AppRole JWT/OIDC Token Us

Joshua Gilman 46 Sep 29, 2022