A fully asynchronous, futures-based Kafka client library for Rust based on librdkafka

Federico Giraud

Last update: Jan 8, 2023

Related tags

Overview

rust-rdkafka

A fully asynchronous, futures-enabled Apache Kafka client library for Rust based on librdkafka.

The library

rust-rdkafka provides a safe Rust interface to librdkafka. The master branch is currently based on librdkafka 1.6.0.

Documentation

Features

The main features provided at the moment are:

Support for all Kafka versions since 0.8.x. For more information about broker compatibility options, check the librdkafka documentation.
Consume from single or multiple topics.
Automatic consumer rebalancing.
Customizable rebalance, with pre and post rebalance callbacks.
Synchronous or asynchronous message production.
Customizable offset commit.
Create and delete topics and add and edit partitions.
Alter broker and topic configurations.
Access to cluster metadata (list of topic-partitions, replicas, active brokers etc).
Access to group metadata (list groups, list members of groups, hostnames, etc.).
Access to producer and consumer metrics, errors and callbacks.
Exactly-once semantics (EOS) via idempotent and transactional producers and read-committed consumers.

One million messages per second

rust-rdkafka is designed to be easy and safe to use thanks to the abstraction layer written in Rust, while at the same time being extremely fast thanks to the librdkafka C library.

Here are some benchmark results using the BaseProducer, sending data to a single Kafka 0.11 process running in localhost (default configuration, 3 partitions). Hardware: Dell laptop, with Intel Core i7-4712HQ @ 2.30GHz.

Scenario: produce 5 million messages, 10 bytes each, wait for all of them to be acked
- 1045413 messages/s, 9.970 MB/s (average over 5 runs)
Scenario: produce 100000 messages, 10 KB each, wait for all of them to be acked
- 24623 messages/s, 234.826 MB/s (average over 5 runs)

For more numbers, check out the kafka-benchmark project.

Client types

rust-rdkafka provides low level and high level consumers and producers.

Low level:

BaseConsumer: a simple wrapper around the librdkafka consumer. It must be periodically poll()ed in order to execute callbacks, rebalances and to receive messages.
BaseProducer: a simple wrapper around the librdkafka producer. As in the consumer case, the user must call poll() periodically to execute delivery callbacks.
ThreadedProducer: a BaseProducer with a separate thread dedicated to polling the producer.

High level:

StreamConsumer: a Stream of messages that takes care of polling the consumer automatically.
FutureProducer: a Future that will be completed once the message is delivered to Kafka (or failed).

For more information about consumers and producers, refer to their module-level documentation.

Warning: the library is under active development and the APIs are likely to change.

Asynchronous data processing with Tokio

Tokio is a platform for fast processing of asynchronous events in Rust. The interfaces exposed by the StreamConsumer and the FutureProducer allow rust-rdkafka users to easily integrate Kafka consumers and producers within the Tokio platform, and write asynchronous message processing code. Note that rust-rdkafka can be used without Tokio.

To see rust-rdkafka in action with Tokio, check out the asynchronous processing example in the examples folder.

At-least-once delivery

At-least-once delivery semantics are common in many streaming applications: every message is guaranteed to be processed at least once; in case of temporary failure, the message can be re-processed and/or re-delivered, but no message will be lost.

In order to implement at-least-once delivery the stream processing application has to carefully commit the offset only once the message has been processed. Committing the offset too early, instead, might cause message loss, since upon recovery the consumer will start from the next message, skipping the one where the failure occurred.

To see how to implement at-least-once delivery with rdkafka, check out the at-least-once delivery example in the examples folder. To know more about delivery semantics, check the message delivery semantics chapter in the Kafka documentation.

Exactly-once semantics

Exactly-once semantics (EOS) can be achieved using transactional producers, which allow produced records and consumer offsets to be committed or aborted atomically. Consumers that set their isolation.level to read_committed will only observe committed messages.

EOS is useful in read-process-write scenarios that require messages to be processed exactly once.

To learn more about using transactions in rust-rdkafka, see the Transactions section of the producer documentation.

Users

Here are some of the projects using rust-rdkafka:

timely-dataflow: a distributed data-parallel compute engine. See also the blog post announcing its Kafka integration.
kafka-view: a web interface for Kafka clusters.
kafka-benchmark: a high performance benchmarking tool for Kafka.

If you are using rust-rdkafka, please let us know!

Installation

Add this to your Cargo.toml:

[dependencies]
rdkafka = { version = "0.25", features = ["cmake-build"] }

This crate will compile librdkafka from sources and link it statically to your executable. To compile librdkafka you'll need:

the GNU toolchain
GNU make
pthreads
zlib: optional, but included by default (feature: libz)
cmake: optional, not included by default (feature: cmake-build)
libssl-dev: optional, not included by default (feature: ssl)
libsasl2-dev: optional, not included by default (feature: gssapi)
libzstd-dev: optional, not included by default (feature: zstd-pkg-config)

Note that using the CMake build system, via the cmake-build feature, is encouraged if you can take the dependency on CMake.

By default a submodule with the librdkafka sources pinned to a specific commit will be used to compile and statically link the library. The dynamic-linking feature can be used to instead dynamically link rdkafka to the system's version of librdkafka. Example:

[dependencies]
rdkafka = { version = "0.25", features = ["dynamic-linking"] }

For a full listing of features, consult the rdkafka-sys crate's documentation. All of rdkafka-sys features are re-exported as rdkafka features.

Minimum supported Rust version (MSRV)

The current minimum supported Rust version (MSRV) is 1.45.0. Note that bumping the MSRV is not considered a breaking change. Any release of rust-rdkafka may bump the MSRV.

Asynchronous runtimes

Some features of the StreamConsumer and FutureProducer depend on Tokio, which can be a heavyweight dependency for users who only intend to use the low-level consumers and producers. The Tokio integration is enabled by default, but can be disabled by turning off default features:

[dependencies]
rdkafka = { version = "0.25", default-features = false }

If you would like to use an asynchronous runtime besides Tokio, you can integrate it with rust-rdkafka by providing a shim that implements the AsyncRuntime trait. See the smol runtime example for an example integration with smol.

Examples

You can find examples in the examples folder. To run them:

cargo run --example <example_name> -- <example_args>

Debugging

rust-rdkafka uses the log and env_logger crates to handle logging. Logging can be enabled using the RUST_LOG environment variable, for example:

RUST_LOG="librdkafka=trace,rdkafka::client=debug" cargo test

This will configure the logging level of librdkafka to trace, and the level of the client module of the Rust client to debug. To actually receive logs from librdkafka, you also have to set the debug option in the producer or consumer configuration (see librdkafka configuration).

To enable debugging in your project, make sure you initialize the logger with env_logger::init(), or the equivalent for any log-compatible logging framework.

rdkafka-sys

See rdkafka-sys.

Contributors

Thanks to:

Thijs Cadier - thijsc

Alternatives

kafka-rust: a pure Rust implementation of the Kafka client.

Comments

First time user
Hello! I've recently tried to use rust-rdkafka, and thought I would share some experiences. I think there may be some "issues" here, but there could also just be some helpful information about what people (person) stumble over.

First off, thank you! This is in support of a timely dataflow Kafka adapter, which I wouldn't want to try and write without a crate. :)

I wanted to try sending a message through kafka and back, so I wrote what seemed like a pretty simple program, following your examples but using the Base* variants of producer and consumer.

fn round_trip(brokers: &str, topic_name: &str) -> Result<(), rdkafka::error::KafkaError> { let mut topic_config = TopicConfig::new(); topic_config .set("produce.offset.report", "true") .finalize(); let mut client_config = ClientConfig::new(); client_config .set("group.id", "example") .set("bootstrap.servers", brokers) .set_default_topic_config(topic_config); let producer: BaseProducer<_> = try!(client_config.create()); let consumer: BaseConsumer<_> = try!(client_config.create()); try!(consumer.subscribe(&[topic_name])); let text = format!("hi there at {:?}", ::std::time::Instant::now()); try!(producer.send_copy::<str,()>(topic_name, None, Some(text.as_str()), None, None, None)); for _ in 0 .. 100 { producer.poll(1); if let Some(result) = try!(consumer.poll(1)) { println!("{:?}:\t{:?}", ::std::time::Instant::now(), result.payload_view::<str>()); try!(consumer.commit(None, rdkafka::consumer::CommitMode::Sync)); } else { println!("{:?}:\tgot nothing", ::std::time::Instant::now()); } } Ok(()) }

The behavior was a bit surprising, and it was tricky to find documentation to explain if this was expected.

This seems to run to completion just fine, printing "got nothing" 100 times and returning Ok(()) when neither ZK nor Kafka are up and running. Where is the moment in the code it is supposed to complain if they are not installed or up and running? (when they are up and running, it does seem to connect and occasionally round-trip data; mysterious things happen).

I'm not a Kafka expert, so I was surprised to see things like "nothing more to read now" surface as an error (PartitionEOF) rather than the absence of a message. I imagine that is just what rdkafka thinks is best and returns to you, so you want to surface it up?

The BaseProducer documentation speaks of a BaseProducerTopic that I should use, but I couldn't find this. Is it just stale documentation?

I was hoping to have a non-blocking poll, but I'm not sure if this is what an argument of 0 would mean. Sometimes 0 means "block". Is that something you could clarify in the documentation? It could be that rdkafka does what it does and won't let us fcntl the socket to non-blocking mode, or something like that, but would be good to know!

Maybe as a meta comment, if BaseProducer and BaseConsumer are meant to be a very thin layer on top of rdkafka, with the caveat that you had best understand the details of rdkafka (which I don't yet), you could say that in the documentation (I'm still not sure). I'm sure it is still useful, but would keep me from scratching / banging my head.

I'm going to try out the "higher-level" stuff next, but I'd have preferred to just have a non-blocking polling interface. If that should work out, and I just need more Kafka protocol expertise, I can push on it more and come back with comments on that.

Thanks again!
opened by frankmcsherry 38
Directly return a DeliveryFuture from FutureProducer::send_copy
Returning a Result made it very difficult to use send_copy() in a chain of futures, and, since a Future is just an asynchronous Result, this commit changes the DeliveryFuture to yield a KafkaError on failure. Previously, the DeliveryFuture would only yield Canceled, and, to continue supporting this, a new variant of KafkaError was added, FutureCanceled.

This makes the FutureProducer API considerably more ergonomic. For example, in a function which returns a BoxFuture<(), ::std::io::Error>, this is what was necessary before:

return match kafka_producer.send_copy::<_, ()>( "test", None, Some(&*format!("msg on {}", topic)), None, None) { Ok(report) => { report .map_err(|_| io::Error::new(io::ErrorKind::Other, "failed to send kafka message")) .and_then(|_| future::ok(())) .boxed() }, Err(_) => { future::err(io::Error::new( io::ErrorKind::InvalidData, "failed to send kafka message")) .boxed() } }

This becomes even more complicated with the fact that match arms must return the same type and the types of chained Futures can easily get very complex.

Compare that to the code one can write with this PR applied:

return kafka_producer .send_copy::<_, ()>("test", None, Some(&*format!("msg on {}", topic)), None, None) .map_err(|_| io::Error::new(io::ErrorKind::Other, "failed to send kafka message")) .and_then(|_| future::ok(())) .boxed();

Much less verbose, and, more importantly, it flows naturally as part of a chain of futures.

Full disclosure: there is a compilation warning for the unused Result returned by tx.send() in two places, but this warning was present before the change and I'm also not sure what the correct way to handle an error arising in such a situation actually is.
opened by wrl 25
Memory keeps growing

Not sure if this is a problem or where it is but after creating a program with FutureProducer my memory skyrocketed to 10Gb usage by having 3 FutureProducers running at the same time (otherwise I get limited to 50k EPS).

Even when running only one memory usage was at 4gb.

After chasing this a little while, I have tried running the asynchronous_processing example, and memory also grows there at least until 1GB, any idea what can be causing this?

opened by llitz 18
Running StreamConsumer with tokio's executor

Currently how StreamConsumer stores the JoinHandle in a Cell prevents us of using the consumer in tokio's thread_pool executor, which expects the consumer to implement Send. In other hand using the current_thread executor doesn't comply with missing static lifetimes somewhere in the consumer code I haven't been able to locate yet.

Is there plans to support the new executors, especially when the development of tokio-core and futures-cpupool seems to be stalled in favor of tokio?

opened by pimeys 16

Compilation fails w/ rust-musl-builder

When compiling a rust application using rdkafka on rust-musl-builder the build fails when linking:

My guess is that the rdkafka lib is build with glibc and then fails w/ musl.

Is there a way to specify what to build it against the same way that dynamic linking can be specified?

error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-nostdlib" "-Wl,--eh-frame-hdr" "-Wl,-(" "-m64" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crt1.o" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crti.o" "-L" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.mshello0-66b6f0d31720f4bd44d6f265903cd36d.rs.rcgu.o" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.mshello1-66b6f0d31720f4bd44d6f265903cd36d.rs.rcgu.o" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.mshello2-66b6f0d31720f4bd44d6f265903cd36d.rs.rcgu.o" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.mshello3-66b6f0d31720f4bd44d6f265903cd36d.rs.rcgu.o" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.mshello4-66b6f0d31720f4bd44d6f265903cd36d.rs.rcgu.o" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.mshello5-66b6f0d31720f4bd44d6f265903cd36d.rs.rcgu.o" "-o" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/mshello-2054ff820a64761c.crate.allocator.rcgu.o" "-Wl,--gc-sections" "-no-pie" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-nodefaultlibs" "-L" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps" "-L" "/home/rust/src/target/release/deps" "-L" "/home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src" "-L" "/home/rust/src/target/x86_64-unknown-linux-musl/release/build/libz-sys-9e0dfdf6ce77c312/out/lib" "-L" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc_jemalloc-f68d8e8245e507c6.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka-a45674fad582d6de.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/liblibz_sys-d28970724bb40e9f.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/liblibc-eef0eb2737b72223.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/libfutures-f1a25787e6cfbe44.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/libserde_json-fded606be9e377b3.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/libitoa-539d7e09fab6c05d.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/libdtoa-19c34238b45a98bc.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/libserde-30812860af6099f8.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/liblog-4de18e7534a813d8.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/liblog-9a6200d5626a5e8a.rlib" "/home/rust/src/target/x86_64-unknown-linux-musl/release/deps/libcfg_if-5f123802e50df72d.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-8d847bbe97fc9dc5.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-008c42aad548e7dd.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libunwind-0fe53371ec419e32.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc_system-9dec1cbd51097ce1.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/liblibc-6a4fb915dd86d140.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-7ebba6af2d3cc324.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd_unicode-7a26f8b3cf380464.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-b0c2d164a9741309.rlib" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-66d072e25a9acee3.rlib" "-static" "-Wl,-Bdynamic" "/home/rust/.rustup/toolchains/1.26.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crtn.o" "-Wl,-)"
  = note: /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `vsnprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:77: undefined reference to `__vsnprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:77: undefined reference to `__vsnprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:64: more undefined references to `__snprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `vsnprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:77: undefined reference to `__vsnprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `rd_strndup':
          /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
          /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
          /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
          /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
          /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o):/home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: more undefined references to `__strndup' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka.o): In function `syslog':
          /usr/include/x86_64-linux-gnu/bits/syslog.h:31: undefined reference to `__syslog_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `vsnprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:77: undefined reference to `__vsnprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_broker.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_msg.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_msg.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_conf.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_conf.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:64: more undefined references to `__snprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_conf.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_conf.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_conf.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_conf.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_offset.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_offset.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_offset.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_offset.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_offset.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_offset.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_transport.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_transport.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:64: more undefined references to `__snprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_transport.o): In function `poll':
          /usr/include/x86_64-linux-gnu/bits/poll2.h:41: undefined reference to `__poll_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_buf.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_buf.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_buf.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_queue.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_op.o): In function `vsnprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:77: undefined reference to `__vsnprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_op.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_op.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_request.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_request.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_request.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_request.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_request.o): In function `rd_strndup':
          /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_request.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_cgrp.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_cgrp.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_cgrp.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_cgrp.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_cgrp.o): In function `rd_strndup':
          /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
          /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_cgrp.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_pattern.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o): In function `rd_strndup':
          /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_partition.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_assignor.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdstring.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:64: more undefined references to `__snprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_metadata.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_metadata.o): In function `rd_strndup':
          /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.4-0/librdkafka/src/rd.h:127: undefined reference to `__strndup'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdbuf.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdbuf.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(snappy.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rddl.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rddl.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:64: more undefined references to `__snprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(crc32c.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(crc32c.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdaddr.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdaddr.o): In function `strncpy':
          /usr/include/x86_64-linux-gnu/bits/string3.h:126: undefined reference to `__strncpy_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdaddr.o): In function `strcpy':
          /usr/include/x86_64-linux-gnu/bits/string3.h:110: undefined reference to `__memcpy_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdlist.o): In function `printf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:104: undefined reference to `__printf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:104: undefined reference to `__printf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_metadata_cache.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_sasl.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_msgset_reader.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_msgset_reader.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_msgset_reader.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdvarint.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdvarint.o):/usr/include/x86_64-linux-gnu/bits/stdio2.h:97: more undefined references to `__fprintf_chk' follow
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdkafka_plugin.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdlog.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdlog.o): In function `snprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:64: undefined reference to `__snprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdlog.o): In function `fprintf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:97: undefined reference to `__fprintf_chk'
          /home/rust/src/target/x86_64-unknown-linux-musl/release/deps/librdkafka_sys-1dabf5d6227192b6.rlib(rdlog.o): In function `printf':
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:104: undefined reference to `__printf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:104: undefined reference to `__printf_chk'
          /usr/include/x86_64-linux-gnu/bits/stdio2.h:104: undefined reference to `__printf_chk'

opened by Licenser 15

Batch Processing

I understand the hesitation in adding to the publishing api an explicit batch processing fn (it already batches under the hood) - but it's unclear to me the purpose of not having this api in the consumer. We should remove complexity from the consumer. I note this after going through previous replies to the same feature request

opened by lacasaprivata2 13

Is this function safe to use and support futures for async processing ? Need simple Producer usage

Hi,

I need to use simple function to produce Kafka messages with a solid function, that process producing on demand when requested by users:

//Send messages to Kafka
fn produce(brokers: &str, topic_name: &str, messages: String) {
    let producer: FutureProducer = ClientConfig::new()
        .set("bootstrap.servers", brokers)
        .set("produce.offset.report", "true")
        .set("message.timeout.ms", "5000")
        .create()
        .expect("Producer creation error");


    producer.send(
        FutureRecord::to(topic_name)
            .payload(&format!("Message {}", messages))
            .key(&format!("Key {}", messages))
            .headers(OwnedHeaders::new()
                .add("header_key", "header_value")),
        0
    );
}

Do this function is safe and performente ? support futures ?

Thanks in advance.

opened by KabDeveloper 13

0.21.0 breaks build on Rust Docker image
Problem

Using the official Rust docker image, I am unable to build version 0.21.0, whereas 0.20.0 worked without issue. The error I see is:

--- stderr Building and linking librdkafka statically Running command: "./configure --disable-sasl --enable-ssl --disable-lz4" in dir: librdkafka Running command: "make -j 4 libs" in dir: librdkafka thread 'main' panicked at 'Unable to find libclang: "couldn\'t find any valid shared libraries matching: [\'libclang.so\', \'libclang-*.so\', \'libclang.so.*\'], set the `LIBCLANG_PATH` environment variable to a path where one of these files can be found (invalid: [])"', src/libcore/result.rs:997:5 note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

Solution

I am able to build successfully after installing some additional dependencies in the image (copied from the Dockerfile in this repo): apt-get install -y llvm-3.9-dev libclang-3.9-dev clang-3.9

Based on the description in d9321f2b1f132d554d2c5854803fcb2869c6495c, it looks like the new dependencies are intended. Perhaps the new dependencies could be mentioned in the changelog to help anyone with broken builds?

Let me know if there is additional information I can provide.

Thanks!
opened by austinhartzheim 12

Release 0.22 build from cargo fails after the release of 0.23

I'm having problems building my project that is using release 0.22.

It seems that problems started after the release of rust-rdkafka 0.23 and rdkafka-sys 1.3.1.

   Compiling rdkafka-sys v1.3.1
  ...
   Compiling rdkafka v0.22.0
error[E0425]: cannot find function `primitive_to_rd_kafka_resp_err_t` in crate `rdsys`
   --> /Users/tkasu/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-0.22.0/src/client.rs:411:22
    |
411 |     let err = rdsys::primitive_to_rd_kafka_resp_err_t(err)
    |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ not found in `rdsys`

Release 0.22 is fetching rdkafka-sys 1.3.1 from which the function primitive_to_rd_kafka_resp_err_t has been removed.

opened by tkasu 11

WIP - switch to std::future and async/await

This PR moves the project over from futures 0.1 to the newly stabilized std::future, using tokio 0.2 (alpha) and futures-preview 0.3. It won't compile on stable until Rust 1.39 at the earliest since it uses async/await, so it'll obviously bump up the minimum Rust version too. It may be possible to remove the async/await which I think would allow it to compile on 1.36.0.

Tests are passing and examples working on nightly.

I guess this won't want to be merged until things stabilize a little more, but thought it might be useful when that time comes!

opened by sd2k 11

Error linking libzstd.so on ubuntu 19.04

building this repo or a project with it as a dependency on ubuntu 19.04 I get this error:

   Compiling rdkafka v0.21.0 (rust-rdkafka)
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" [...] "-Wl,-Bdynamic" "-lz" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"
  = note: /usr/bin/ld: rust-rdkafka/target/debug/deps/librdkafka_sys-3d04a478eb2ea7f3.rlib(rdkafka_zstd.o): in function `rd_kafka_zstd_decompress':
          rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:44: undefined reference to `ZSTD_getFrameContentSize'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:91: undefined reference to `ZSTD_getErrorCode'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:80: undefined reference to `ZSTD_decompress'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:82: undefined reference to `ZSTD_isError'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:100: undefined reference to `ZSTD_getErrorName'
          /usr/bin/ld: rust-rdkafka/target/debug/deps/librdkafka_sys-3d04a478eb2ea7f3.rlib(rdkafka_zstd.o): in function `rd_kafka_zstd_compress':
          rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:130: undefined reference to `ZSTD_compressBound'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:141: undefined reference to `ZSTD_createCStream'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:155: undefined reference to `ZSTD_initCStream'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:157: undefined reference to `ZSTD_isError'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:213: undefined reference to `ZSTD_freeCStream'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:168: undefined reference to `ZSTD_compressStream'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:169: undefined reference to `ZSTD_isError'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:170: undefined reference to `ZSTD_getErrorName'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:213: undefined reference to `ZSTD_freeCStream'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:158: undefined reference to `ZSTD_getErrorName'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:213: undefined reference to `ZSTD_freeCStream'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:198: undefined reference to `ZSTD_endStream'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:199: undefined reference to `ZSTD_isError'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:213: undefined reference to `ZSTD_freeCStream'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:200: undefined reference to `ZSTD_getErrorName'
          collect2: error: ld returned 1 exit status


error: aborting due to previous error

error: Could not compile `rdkafka`.

if I copy the command in note: and add "-lzstd" it seems to link

from

"cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" [...] "-Wl,-Bdynamic" "-lz" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"

"cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" [...] "-Wl,-Bdynamic" "-lz" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil" "-lzstd"

here is a Dockerfile to reproduce:

FROM ubuntu:19.04

ARG UNAME=mariano
ARG UID=1000
ARG GID=1000
RUN groupadd -g $GID -o $UNAME
RUN useradd -m -u $UID -g $GID -o -s /bin/bash $UNAME

RUN \
  sed -i 's/# \(.*multiverse$\)/\1/g' /etc/apt/sources.list && \
  apt-get update; \
  apt-get -y upgrade && \
  apt-get install -y --no-install-recommends wget build-essential clang libclang-dev libssl-dev zlib1g-dev libsasl2-dev libzstd-dev curl ca-certificates git

USER $UNAME
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y

then

# build it
sudo docker build --build-arg UID=$$(id -u) --build-arg GID=$$(id -g) -t ubuntu-rust:1904-stable ubuntu-1904-rust

sudo docker run -i ubuntu-rust:1904-stable sh << COMMANDS
    cd
    . .cargo/env
    git clone https://github.com/fede1024/rust-rdkafka.git
    cd rust-rdkafka
    cargo run --example simple_consumer
COMMANDS

it seems for some reason it doesn't find libzstd

$ ldconfig -p | grep libzstd

        libzstd.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libzstd.so.1

I tried creating symbolic links in /usr/local/lib just in case but it didn't work.

opened by marianoguerra 11

Question - Accomulate N messages in T duration or timeout.
I would like use poll with a specified timeout but i would like to get N number of messages per call. And poll other messages in the next call.

consumer.iter().take(1000) // Acomulate messages

I would like to acomulate this messages but based in a specified Duration or Timeout.
opened by ajosecueto 0
Help with clarifying dynamic-linking behavior

Hi there, had a quick question about the expectations of using dynamic linking. The readme in rdkafka-sys claims:

The system version of librdkafka must exactly match the version of librdkafka bundled with this crate.

However, in the build.rs appears to be searching for a version of librdkafka with *at least the version of the one bundled within itself. The commit on the readme appears to be newer, should the build.rs be updated to reflect what the readme says? Thanks.

opened by hallfox 0
How to build for x86_64-unknown-linux-musl
I have this in Cargo toml

rdkafka = { version = "0.29.0", features = [ "ssl", "cmake-build"] }

I tried to compile to x86_64-unknown-linux-musl using 2 options. Both failed.

cargo build --target x86_64-unknown-linux-musl --release OUTPUT: linking with cc failed: exit status: 1 rdkafka_ssl.c:(.text.rd_kafka_ssl_ctx_term+0x2b): undefined reference to `ENGINE_free' collect2: error: ld returned 1 exit status

cross build --target=x86_64-unknown-linux-musl --release OUTPUT: rdkafka_ssl.c:(.text.rd_kafka_ssl_ctx_term+0x2b): undefined reference to `ENGINE_free' collect2: error: ld returned 1 exit status

They both give the same error.

I am using Arch and it doesn't fail using just "cmake-build" feature

Here is the github repo I made where the issue is recreated https://github.com/0xDjole/rust-rdkafka-musl

Can rust-rdkafka be compiled to x86_64-unknown-linux-musl ?
opened by 0xDjole 0

StreamConsumer commite message from tokio spawn.

Hi, I have used the example provided for the asynchronous processing. and tried commited message. But I can't commited borrow message.

tokio = { version = "1.22.0", features = ["full"] }

rdkafka = { version = "0.29.0", features = ["cmake-build"] }



 pub async fn run_async_processor(brokers: String, group_id: String, topic: String, worker: i32) {
    let consumer: Arc<StreamConsumer> = Arc::new(ClientConfig::new()
        .set("group.id", &group_id)
        .set("bootstrap.servers", &brokers)
        .set("enable.auto.commit", "false")
        .create()
        .expect("Consumer creation failed")
    );
    consumer
        .subscribe(&[&topic])
        .expect("Can't subscribe to specified topic");
    let stream_processor = consumer.stream().try_for_each(|borrowed_message| {
        async move {
            let _ = match borrowed_message.payload_view::<str>() {
                None => "",
                Some(Ok(s)) => s,
                Some(Err(e)) => {
                    println!("Error while deserializing message payload: {:?}", e);
                    ""
                }
            };
            consumer.commit_message(&borrowed_message, CommitMode::Async).unwrap();
            Ok(())
        }
    });
    println!("Starting daemon consumer #{}", worker);

    stream_processor.await.expect("stream processing failed");
}

i get error

error[E0507]: cannot move out of `consumer`, a captured variable in an `FnMut` closure
  --> src/broker/mod.rs:26:20
   |
14 |       let consumer: Arc<StreamConsumer> = Arc::new(ClientConfig::new()
   |           -------- captured outer variable
...
25 |       let stream_processor = consumer.stream().try_for_each(|borrowed_message| {
   |                                                             ------------------ captured by this `FnMut` closure
26 |           async move {
   |  ____________________^
27 | |             let _ = match borrowed_message.payload_view::<str>() {
28 | |                 None => "",
29 | |                 Some(Ok(s)) => s,
...  |
35 | |             consumer.commit_message(&borrowed_message, CommitMode::Async).unwrap();
   | |             --------
   | |             |
   | |             variable moved due to use in generator
   | |             move occurs because `consumer` has type `Arc<StreamConsumer>`, which does not implement the `Copy` trait
36 | |             Ok(())
37 | |         }
   | |_________^ move out of `consumer` occurs here

error[E0505]: cannot move out of `consumer` because it is borrowed
  --> src/broker/mod.rs:25:59
   |
25 |     let stream_processor = consumer.stream().try_for_each(|borrowed_message| {
   |                            ----------------- ------------ ^^^^^^^^^^^^^^^^^^ move out of `consumer` occurs here
   |                            |                 |
   |                            |                 borrow later used by call
   |                            borrow of `consumer` occurs here
...
35 |             consumer.commit_message(&borrowed_message, CommitMode::Async).unwrap();
   |             -------- move occurs due to use in closure

Help me please.

opened by BruAPAHE 0

Allow purging (Purge API) / don't leak on dropping producer
Resolves #518

What it does:

Purges producer on drop

Updates bindgen script to more recent bindgen version

Sets the parameter that makes macro constants signed (because that's how they are generally used in librdkafka)

Todo:

[x] Check that it works

[x] Expose the full interface, possibly through bitflags
opened by Ten0 2
Allow purging (purge API) / don't leak on dropping producer
I'm looking for a way to handle messages that have failed to send upon closing.

Currently, dropping the producer when kafka server is not started gives the following message:

[WARN] librdkafka: TERMINATE [thrd:app]: Producer terminating with 2 messages (13 bytes) still in queue or transit: use flush() to wait for outstanding message delivery

Flushing beforehand (ThreadedProducer::flush) will obviously always timeout because server is not running, and doesn't change the previous message.

Delivery callbacks from ProducerContext do not get called for these messages, leading to memory leaks (as Box/Arc/... are not cast back).

librdkafka provides the purge API to solve this issue (https://github.com/edenhill/librdkafka/issues/990, https://github.com/edenhill/librdkafka/commit/107af2a5c9c564b71b06266227e76f1f8fad942c#diff-84ac6df9f4f6c13f1f4eecf3b9c0e2cee4d7c4c005feea512ff3f82c000fb84eR2211).

It would be useful to have access to this API for scenarios such as mine or those described at https://github.com/edenhill/librdkafka/issues/990.

As the Rust API also does some nice IntoOpaque casting where it may leak if callbacks are not called, I think it would also be best to actually force an instant purge/flush when dropping the BaseProducer, both ensuring that the DeliveryOpaques don't get somewhat-silently leaked, and making it more natural for users to handle failure to deliver messages. (The reason this wasn't done in librdkafka is that users may already be handling this separately, but that could just be released in 0.30.0 which would allow breaking changes - and this is unlikely to break much because such handling would be severely un-idiomatic as people would have to keep track of the set of raw pointers that were sent, removing them from the list as they get a callback, and cast them back using unsafe code to drop them in case they didn't receive the callback, instead of just opening the issue I'm opening right now)

Thanks!
opened by Ten0 2