Is there any plan to publish this package into crates.io to allow a simple cargo install and allow a better distribution model instead of cloning / building?

Fixes #20.

The file input:

• supports tail-like reading for existing files and dynamic recognition of new files (via inotify)
• handles multiple file inputs (src path can be glob compatible string with wildcards)

Example of working message on flowgger 0.1.X to gelf input with multiline: echo -e '{"version":"1.1", "host": "example.org", "short_message": "A short GELF message that helps you identify what is going on", "full_message": "Backtrace here\n\nmore stuff", "timestamp": 1470749969, "level": 1, "_user_id": 9001, "_some_info": "foo", "some_metric_num": 42.0}\0' | openssl s_client -quiet -no_ign_eof -connect localhost:12202

[input]
type = "tls"
format = "gelf"
listen = "0.0.0.0:12202"
framing = "nul"
...

Now with flowgger 0.2.0, I have the following error:

Invalid GELF input, unable to parse as a JSON object: [{"version":"1.1", "host": "example.org", "short_message": "A short GELF message that helps you identify what is going on", "full_message": "Backtrace here

more stuff", "timestamp": 1470749909, "level": 1, "_user_id": 9001, "_some_info": "foo", "some_metric_num": 42.0}]

to unlock https://github.com/awslabs/flowgger/issues/32 coio has to go

it's used in the TLS module as a scheduler and to control tcplistener and stream.

The package doesn't seem to have been touched for at least an year: https://github.com/zonyitoo/coio-rs

I may have missed it, but I didn't find in the docs how to get input from a log file.

It would need to:

• Not read the whole file from the start, but seek to the end and read from there (like tail -f)
• Keep working if a logrotate service renames the log file and creates a new file with the previous name (flowgger should start reading the new file).

Issue #, if available: part of #33

Description of changes: This PR updates the capnp library to 0.10. As a result the capnp schema was recompiled. I couldn't find a flag to tell the capnp compiler to put the sources in a specific package path (now it is crate::flowgger::capnp_record, in my PR it's under crate::capnp_record), so I moved the compiled schema to the crate root.

@jedisct1 , is there some way to change that, besides modifying the generated schema manually?

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

This little patch allows set CAfile to be set independantly of tls verify peer and add an option to specify one extra chain cert to the server chain cert.

Issue #, if available: none

Description of changes: This PR updates the Dockerfile, by changing LibreSSL to OpenSSL there and lowering the output image size do 76MB, by using a multi-stage Docker build and debian:buster-slim as a base image

--

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

I use nginx-1.12.0 version and flowgger-master on macOS 10.12.4

in the nginx configuration file nginx.conf, I instruct nginx to send error log to local syslog server

error_log   syslog:server=127.0.0.1 debug;

in flowgger.toml

[input]

### Syslog over UDP
type = "udp"
listen = "0.0.0.0:514"

[output]

### Debug output (stdout)
type = "stdout"

grep -rnw 'src' -e 'Unsupported version'
src/flowgger/decoder/rfc5424_decoder.rs:79:        return Err("Unsupported version");
nano +79 src/flowgger/decoder/rfc5424_decoder.rs

// I add a println! to this function to show line value
fn parse_pri_version(line: &str) -> Result<Pri, &'static str> {
    println!("line:{}", line); // <----show line value
    if !line.starts_with('<') {
        return Err("The priority should be inside brackets");
    }
    let mut parts = line[1..].splitn(2, '>');
    let pri_encoded: u8 =
        try!(try!(parts.next().ok_or("Empty priority")).parse().or(Err("Invalid priority")));
    let version = try!(parts.next().ok_or("Missing version"));
    if version != "1" {
        return Err("Unsupported version");
    }
    Ok(Pri {
           facility: pri_encoded >> 3,
           severity: pri_encoded & 7,
       })
}

after I recompile and restart flowgger, start nginx and make some error to generate a error log

target/release/flowgger
Flowgger 0.2.6
<184> Apr
Unsupported version

Do you have a plan to support input nginx log (both error_log and access_log)?

The decoder RFC 5424 does not correctly parse structured data. Currently, the parser parses only one structured data, which completely breaks the specification. As defined by the RFC 5424 ABF a Syslog message can contain no, one or more than one structured data.

Here is an example of a not properly parsed message:

<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource= "Application" eventID="1011"][examplePriority@32473 class="high"] hello world

In this example, the message contains two structured data (but we can have a message with more than two) but the flowgger decoder only parses the first one and consider the rest a message, which is not correct too as the structured data and the message must be separated with a space.

Dependencies are old and needs to be updated. There are some which are going to be tricky:

OpenSSL 0.9 -> 0.10: Blocked by Kafka using 0.9 and by API change in the library
CapNProto: Change of API in the library force implementation of a new trait
serde_json 0.8 -> 1.0: Change of API in the library

ported over from https://github.com/amazon-archives/flowgger-hold/issues/3

Hi. I'm getting errors while trying to build flowgger with --all-features (rust 1.62.0) as pointed out in https://gitlab.alpinelinux.org/alpine/aports/-/issues/13952#note_243393.

Changing the declaration to let mut ... (like the log suggests) makes the build pass but I honestly don't know whether things will break. This seems to be happen after redis-rs 0.11.0-beta1 release (ref: https://github.com/redis-rs/redis-rs/blob/main/CHANGELOG.md#0110-beta1---2019-05-30)

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tcp/tcpco_input.rs:52:18
   |
52 |     decoder: Box<Decoder>,
   |                  ^^^^^^^
   |
   = note: `#[warn(bare_trait_objects)]` on by default
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
52 -     decoder: Box<Decoder>,
52 +     decoder: Box<dyn Decoder>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tcp/tcpco_input.rs:53:18
   |
53 |     encoder: Box<Encoder>,
   |                  ^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
53 -     encoder: Box<Encoder>,
53 +     encoder: Box<dyn Encoder>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tcp/tcpco_input.rs:29:22
   |
29 |         decoder: Box<Decoder + Send>,
   |                      ^^^^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
29 -         decoder: Box<Decoder + Send>,
29 +         decoder: Box<dyn Decoder + Send>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tcp/tcpco_input.rs:30:22
   |
30 |         encoder: Box<Encoder + Send>,
   |                      ^^^^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
30 -         encoder: Box<Encoder + Send>,
30 +         encoder: Box<dyn Encoder + Send>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tls/tlsco_input.rs:52:18
   |
52 |     decoder: Box<Decoder>,
   |                  ^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
52 -     decoder: Box<Decoder>,
52 +     decoder: Box<dyn Decoder>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tls/tlsco_input.rs:53:18
   |
53 |     encoder: Box<Encoder>,
   |                  ^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
53 -     encoder: Box<Encoder>,
53 +     encoder: Box<dyn Encoder>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tls/tlsco_input.rs:29:22
   |
29 |         decoder: Box<Decoder + Send>,
   |                      ^^^^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
29 -         decoder: Box<Decoder + Send>,
29 +         decoder: Box<dyn Decoder + Send>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tls/tlsco_input.rs:30:22
   |
30 |         encoder: Box<Encoder + Send>,
   |                      ^^^^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
30 -         encoder: Box<Encoder + Send>,
30 +         encoder: Box<dyn Encoder + Send>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tcp/tcpco_input.rs:61:51
   |
61 |         "capnp" => Box::new(CapnpSplitter) as Box<Splitter<_>>,
   |                                                   ^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
61 -         "capnp" => Box::new(CapnpSplitter) as Box<Splitter<_>>,
61 +         "capnp" => Box::new(CapnpSplitter) as Box<dyn Splitter<_>>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tcp/tcpco_input.rs:62:49
   |
62 |         "line" => Box::new(LineSplitter) as Box<Splitter<_>>,
   |                                                 ^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
62 -         "line" => Box::new(LineSplitter) as Box<Splitter<_>>,
62 +         "line" => Box::new(LineSplitter) as Box<dyn Splitter<_>>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tcp/tcpco_input.rs:63:53
   |
63 |         "syslen" => Box::new(SyslenSplitter) as Box<Splitter<_>>,
   |                                                     ^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
63 -         "syslen" => Box::new(SyslenSplitter) as Box<Splitter<_>>,
63 +         "syslen" => Box::new(SyslenSplitter) as Box<dyn Splitter<_>>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tcp/tcpco_input.rs:64:47
   |
64 |         "nul" => Box::new(NulSplitter) as Box<Splitter<_>>,
   |                                               ^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
64 -         "nul" => Box::new(NulSplitter) as Box<Splitter<_>>,
64 +         "nul" => Box::new(NulSplitter) as Box<dyn Splitter<_>>,
   | 

warning: use of deprecated associated function `may::Config::set_io_workers`: use `set_workers` only
  --> src/flowgger/input/tls/tlsco_input.rs:33:23
   |
33 |         may::config().set_io_workers(tls_config.threads);
   |                       ^^^^^^^^^^^^^^
   |
   = note: `#[warn(deprecated)]` on by default

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tls/tlsco_input.rs:68:51
   |
68 |         "capnp" => Box::new(CapnpSplitter) as Box<Splitter<_>>,
   |                                                   ^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
68 -         "capnp" => Box::new(CapnpSplitter) as Box<Splitter<_>>,
68 +         "capnp" => Box::new(CapnpSplitter) as Box<dyn Splitter<_>>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tls/tlsco_input.rs:69:49
   |
69 |         "line" => Box::new(LineSplitter) as Box<Splitter<_>>,
   |                                                 ^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
69 -         "line" => Box::new(LineSplitter) as Box<Splitter<_>>,
69 +         "line" => Box::new(LineSplitter) as Box<dyn Splitter<_>>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tls/tlsco_input.rs:70:53
   |
70 |         "syslen" => Box::new(SyslenSplitter) as Box<Splitter<_>>,
   |                                                     ^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
70 -         "syslen" => Box::new(SyslenSplitter) as Box<Splitter<_>>,
70 +         "syslen" => Box::new(SyslenSplitter) as Box<dyn Splitter<_>>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tls/tlsco_input.rs:71:47
   |
71 |         "nul" => Box::new(NulSplitter) as Box<Splitter<_>>,
   |                                               ^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
71 -         "nul" => Box::new(NulSplitter) as Box<Splitter<_>>,
71 +         "nul" => Box::new(NulSplitter) as Box<dyn Splitter<_>>,
   | 

error[E0596]: cannot borrow `redis_cnx` as mutable, as it is not declared as mutable
   --> src/flowgger/input/redis_input.rs:105:46
    |
99  |         let redis_cnx = self.redis_cnx;
    |             --------- help: consider changing this to be mutable: `mut redis_cnx`
...
105 |             let dummy: RedisResult<String> = redis_cnx.rpoplpush(queue_key_tmp, queue_key);
    |                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cannot borrow as mutable

error[E0596]: cannot borrow `redis_cnx` as mutable, as it is not declared as mutable
   --> src/flowgger/input/redis_input.rs:110:38
    |
99  |         let redis_cnx = self.redis_cnx;
    |             --------- help: consider changing this to be mutable: `mut redis_cnx`
...
110 |             let line: String = match redis_cnx.brpoplpush(queue_key, queue_key_tmp, 0) {
    |                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cannot borrow as mutable

error[E0596]: cannot borrow `redis_cnx` as mutable, as it is not declared as mutable
   --> src/flowgger/input/redis_input.rs:117:40
    |
99  |         let redis_cnx = self.redis_cnx;
    |             --------- help: consider changing this to be mutable: `mut redis_cnx`
...
117 |             let res: RedisResult<u8> = redis_cnx.lrem(queue_key_tmp as &str, 1, line as String);
    |                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cannot borrow as mutable

For more information about this error, try `rustc --explain E0596`.
warning: `flowgger` (lib) generated 17 warnings
error: could not compile `flowgger` due to 3 previous errors; 17 warnings emitted

It would be great if Flowlogger supported output to NATS (https://nats.io/).

Kafka is quite complex and has JVM dependency, NATS is fast becoming a popular alternative for people who don't need all the bells and whistles of Kafka.

In log file name the separator with the datetime string and the extension strings are hardcoded.

So, using another char than "-" or no extension doesn't work.

file_path = "var/output/logs/fwsyslog.log"
  file_rotation_timeformat = ".%Y-%m-%d-%H-%M"
  file_rotation_time = 60

  # want fwsyslog.2021-12-17-15-06.log
  # get fwsyslog-.2021-12-17-15-39.log

 file_path = "var/output/logs/fwsyslog"
  file_rotation_timeformat = ".%Y-%m-%d-%H-%M"
  file_rotation_time = 60

  # want fwsyslog.2021-12-17-15-06
  # get fwsyslog-.2021-12-17-15-39.log

Update new_file.set_file_name(&format!("{}-{}.{}"

• The datetime separator must be removed and put in the config string.
• The ".{}" suffix should only be appended if the filename contains a suffix

Hello, We are trying flowgger for forwarding logs from remote site to our central Graylog server. Everything on our infrastructure is with timezone CEST. Flowgger seems to force the timestamp of the GELF to UTC Is there a way to configure and change this behavior ? Thanks in advance Flowgger seems to be a nice tool !!

Would you be open to accepting a PR to support a Lua parsing? This way some more custom log formats can be added without having to recompile, or add into source.

Found this while writing tests

As for https://github.com/jedisct1/flowgger/wiki/LTSV-Encoder the encoders should support the ability to add or replace, without duplicating tags and fields.

LTSV is not only not overriding it, but get duplicated as the encoding is done by appending data to a String:

https://github.com/awslabs/flowgger/blob/master/src/flowgger/encoder/ltsv_encoder.rs#L41

left: "some_info:foo\tsome_info:bar\thost:example.org\ttime:1385053862.3072\tmessage:A short message\tfull_message:Backtrace here with more\tlevel:1\tappname:appname\tprocid:44\tmsgid:msg_id", right: "some_info:bar\thost:example.org\ttime:1385053862.3072\tmessage:A short message\tfull_message:Backtrace here with more\tlevel:1\tappname:appname\tprocid:44\tmsgid:msg_id"', src/flowgger/encoder/ltsv_encoder.rs:175:9