I tried to make a script that would allow reporting arbitrary metrics using a native source: https://gist.github.com/doubleyou/1e75fff02ff26b9a9f2fd5ca6c6d3417

It's a combination of the compiled .proto file and an actual script that implements CLI and network.

Payload length is calculated in accordance with https://docs.python.org/2/library/struct.html and https://github.com/postmates/cernan/blob/master/resources/protobufs/native.proto. The rest is pretty much packing a bunch of values into a protobuf packet.

Two issues arise when I try to test it:

1. Upon start, cernan reports the following in the logs:

cernan_1             | thread '<unnamed>' panicked at 'Unable to bind to TCP socket: Error { repr: Os { code: 98, message: "Address already in use" } }', /checkout/src/libcore/result.rs:906:4
cernan_1             | note: Run with `RUST_BACKTRACE=1` for a backtrace.
cernan_1             | thread '<unnamed>' panicked at 'Unable to bind to TCP socket: Error { repr: Os { code: 98, message: "Address already in use" } }', /checkout/src/libcore/result.rs:906:4

It's worth noting that changing the port doesn't affect that. Seems to be a race condition or something?

2. However, seems like later on cernan actually starts listening on the port, because when I run the script (my specific example: ./houston.py service.restarts 1.0 1 service=test 1.0), I see the following error in the logs:

cernan_1             | thread '<unnamed>' panicked at 'index out of bounds: the len is 0 but the index is 0', /checkout/src/liballoc/vec.rs:1561:14

Adding RUST_BACKTRACE=full didn't really help, I'll probably try to rebuild cernan with debug info, at least to see where the actual calls are happening.

Steps to reproduce:

1. Build cernan (https://github.com/postmates/cernan/commit/a4da52dc93eef606a3667dc6ad664e5d7248937f would work, but I have reproduced this on older versions too).

2. Start it with the following config:

data-directory = "/tmp/cernan"

flush-interval = 2

[sources]

  [sources.statsd.primary]
  enabled = true
  port = 8125
  forwards = ["sinks.console"]

    [sources.statsd.primary.mapping.default]
    mask = ".*"
    bounds = [0.0, 0.1, 10.0, 100.0]

[sinks]
  [sinks.console]
  bin_width = 1

3. pip install git+https://github.com/postmates/pystatsd (virtualenv recommended).

4. In Python repl, run the following:

import pystatsd
pystatsd.timing('test.timing', 50)

What happens

Output with RUST_STACKTRACE=full:

cernan_1             | thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: CannotSetError', /checkout/src/libcore/result.rs:906:4
cernan_1             | stack backtrace:
cernan_1             |    0:     0x563ec43a0063 - std::sys::imp::backtrace::tracing::imp::unwind_backtrace::h8ed7485deb8ab958
cernan_1             |                                at /checkout/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
cernan_1             |    1:     0x563ec439dd48 - std::panicking::default_hook::{{closure}}::h0088fe51b67c687c
cernan_1             |                                at /checkout/src/libstd/sys_common/backtrace.rs:69
cernan_1             |                                at /checkout/src/libstd/sys_common/backtrace.rs:58
cernan_1             |                                at /checkout/src/libstd/panicking.rs:381
cernan_1             |    2:     0x563ec439d1a5 - std::panicking::rust_panic_with_hook::h25b934bb4484e9e0
cernan_1             |                                at /checkout/src/libstd/panicking.rs:397
cernan_1             |                                at /checkout/src/libstd/panicking.rs:577
cernan_1             |    3:     0x563ec439cc28 - std::panicking::begin_panic::h59483e27e93d7bc6
cernan_1             |                                at /checkout/src/libstd/panicking.rs:538
cernan_1             |    4:     0x563ec439cba9 - std::panicking::begin_panic_fmt::h5f221297e8a3dbdb
cernan_1             |                                at /checkout/src/libstd/panicking.rs:522
cernan_1             |    5:     0x563ec43b489a - core::panicking::panic_fmt::h4d1ab9bae1f32475
cernan_1             |                                at /checkout/src/libstd/panicking.rs:498
cernan_1             |    6:     0x563ec4026b1b - core::result::unwrap_failed::h8824197d9878f263
cernan_1             |    7:     0x563ec410d124 - std::sys_common::backtrace::__rust_begin_short_backtrace::hb9f913159e607447
cernan_1             |    8:     0x563ec410a986 - <F as alloc::boxed::FnBox<A>>::call_box::hac99f2e9ac949407
cernan_1             |    9:     0x563ec43b0d07 - std::sys::imp::thread::Thread::new::thread_start::hbaf1b5aa1ca8e3ea
cernan_1             |                                at /checkout/src/liballoc/boxed.rs:736
cernan_1             |                                at /checkout/src/libstd/sys_common/thread.rs:24
cernan_1             |                                at /checkout/src/libstd/sys/unix/thread.rs:90
cernan_1             |   10:     0x7f5d9e3ab6b9 - start_thread
cernan_1             |   11:     0x7f5d9decb3dc - clone
cernan_1             |   12:                0x0 - <unknown>

Interestingly, when I use an InfluxDB sink, cernan stops reporting data there at all, once this error occurs. It keeps printing things out to the console though.

Also, interestingly, I tried to reproduce this on postal-console-stage and things didn't break there. Hard to tell how much the versions of pystatsd and cernan differ there. If anything, pystatsd way of serializing timings seems legit: https://github.com/postmates/pystatsd/blob/master/pystatsd/pystatsd.py#L179.

This is getting closer to working. There are (at least) the following tasks still to do (disregarding code review requests).

• [ ] Documentation,
• [ ] Matching DSL in config file (selecting which messages to ingest),
• [ ] Upstreaming rust-systemd pull-request (jmesmon/rust-systemd#35),
• [ ] cfg-config it out on platforms that don't support/run systemd,
• [ ] properly setting the timestamp,
• [ ] tests,
• [ ] fixing travis-ci, so it tests (at least building) this source.

Original message below.

This is an untested journald source.

It depends on an (still unreviewed) pull request to the systemd crate: jmesmon/rust-systemd#35

There are a few things missing. It cannot

• filter yet what messages to ingest (by unit, or anything else);
• ingest historical messages.

It also does not contain any documentation. Where should I put it?

I am still not sure what configuration API is best to describe matching the log records to get (systemd-journald can AND and OR matches. See sd_journal_add_match(3)). Or whether that is even desirable. What is easily possible and should be done before merging is simple filtering by one field.

ref #232

It would be convenient for some users if cernan had a default set of tags. @blakebarnett is the primary contact for this feature. For users that do not use tags we'd want to add a default-tags: Bool option to cernan to avoid tags, false by default.

Default tags we have so far:

• hostname

Open to suggestions.

After further discussion, we now want default-tags to be a list, not a boolean. See https://github.com/postmates/cernan/issues/279#issuecomment-312074665 for more details.

In order to preserve information about the distribution of request times or similar data, we can't rely on percentiles: there's no correct way to later aggregate two percentiles to get another percentile. A mean of a percentiles isn't usually what you want.

What we can do is calculate histograms for each reporting period, then report each bin of that histogram, for each flush interval, to Wavefront. From Wavefront's perspective it's just a bunch of separate metrics, but we can reassemble them and calculate a histogram for the distribution of some values for arbitrary intervals without loss of information.

The tricky part here is communicating from the application to Cernan what the bins should be. Statsd can be configured to calculate histograms in this way, but it involves defining the bins in statsd's config file which is awkward to realize.

So I propose an extension to the statsd protocol so applications can communicate to cernan at application start the desired histogram bins. Proposed format:

metric.name:-inf,1,10,100,inf|h

This configures four bins for the metric metric.name:

• less than 1
• [1, 10)
• [10, 100)
• greater than or equal to 100

Applications can send these each time they start, and Cernan will write them to the filesystem so they don't need to be sent again if Cernan restarts.

This introduces a new wire format version for the Avro source.

The existing version 1 Avro source doesn't provide any support for extensible metadata.

Introduced in version 2 is a set of key value pairs in the header that can be used to carry arbitrary metadata.

The protocol allows up to 255 key value pairs, each with a key of up to 255 UTF-8 encoded bytes and a value of up to 65535 bytes in length.

The new packet structure is described below in IETF RFC format:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                             Length                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            Version                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            Control                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                               ID                              +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                            ShardBy                            +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|   #KV Pairs   |   Key Length  |    Key (up to 255 bytes)      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          Value Length         |   Value (up to 65535 bytes)   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          Avro N Bytes                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The source remains backwards compatible with version 1 clients.

Without this, errors in lua cause the C lua code to abort().

I found it was easier to refactor the repeated process_metric/process_log/tick code than to make the change in three places.

It will also optionally try to parse the log line as JSON and include that at the top level.

This is needed for our structured logging so we can emit json to disk and have cernan forward it to kafka.

As of 2f08dbc0c11f0089e6e798241045a76730c3559a the only summarization type that cernan supports is the CKMS from our quantiles. Quantile structures are great but suffer from the need to aggregate the structures before their data makes sense, as in Wavefront or Prometheus. This is not a problem for binning histograms, that is histograms computed by using a client-supplied min/max range plus number of bins. Individual cernan instances are able to choose the same aggregating behaviour without coordination.

We should extend the native protocol to include a HISTOGRAM summary type. I believe we will need:

• a global configuration for default min/max
• a global configuration for default number of bins
• extension to the native protocol to include stream-specific min/max, bins

We may also choose to retarget statsd histogram/timers to this type. I'm unsure about this.

This is related to #66.

@lynchc @bitglue relevant to your interests

Kushal Pisavadia has reported on twitter that he's having trouble compiling cernan. Twitter's too wee to get much information so I'm creating this tracking issue to solicit detailed compilation failures.

Back last week @bitglue and I had a conversation about the QOS changes that landed in #126. In particular, he pointed out that figuring out how many points you'd elide in a flush interval was confusing and that the random sampling could tend to lose outlier points. With #131 it became possible to represent every bin as a quantile structure because Metrics were now quantile structures and each bin is a metric.

This commit removes the QOS concept entirely and provides a configurable bin-width per sink. That is, it's now possible to say that the console has a one-second bin width while the wavefront sink has a 10 second bin width. Each bin has full quantile introspection available to it though none of the sinks are able to consume that as yet.

Bumps net2 from 0.2.32 to 0.2.37.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps miow from 0.2.1 to 0.2.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps protobuf from 1.7.3 to 2.6.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps tiny_http from 0.6.0 to 0.8.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps libflate from 0.1.14 to 0.1.27.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

From the security review bug (https://github.com/postmates/cernan/issues/461):

None of the sources auth, the few sinks that have auth credentials are, iirc, not wired up to authenticate. IP whitelisting and/or presence in a blessed subnet was Good Enough. Obvious issues there.