I'm trying to compile sozu on debian testing but i have these errors :

   Compiling sozu-lib v0.1.1
error: no associated item named `get_2048_256` found for type `openssl::dh::Dh` in the current scope
   --> /home/mathieu/.cargo/registry/src/github.com-1ecc6299db9ec823/sozu-lib-0.1.1/src/network/tls.rs:433:11
    |
433 |     match Dh::get_2048_256() {
    |           ^^^^^^^^^^^^^^^^

error: no method named `set_ecdh_auto` found for type `openssl::ssl::SslContextBuilder` in the current scope
   --> /home/mathieu/.cargo/registry/src/github.com-1ecc6299db9ec823/sozu-lib-0.1.1/src/network/tls.rs:441:13
    |
441 |     context.set_ecdh_auto(true);
    |             ^^^^^^^^^^^^^

error: no associated item named `get_2048_256` found for type `openssl::dh::Dh` in the current scope
   --> /home/mathieu/.cargo/registry/src/github.com-1ecc6299db9ec823/sozu-lib-0.1.1/src/network/tls.rs:587:11
    |
587 |     match Dh::get_2048_256() {
    |           ^^^^^^^^^^^^^^^^

error: no method named `set_ecdh_auto` found for type `openssl::ssl::SslContextBuilder` in the current scope
   --> /home/mathieu/.cargo/registry/src/github.com-1ecc6299db9ec823/sozu-lib-0.1.1/src/network/tls.rs:594:9
    |
594 |     ctx.set_ecdh_auto(true);
    |         ^^^^^^^^^^^^^

error: aborting due to 4 previous errors

error: Could not compile `sozu-lib`.

Any idea how to solve this ?

As you can see on this picture, backend.connections increased at some point and never really went down. The unit value on the left is partially hidden, the actual value reported is: 36893488147419103000.

The setup has two workers. Each worker reports 18446744073709552000 backend connections, which gives the number above if we multiply it by the number of workers.

The value never decreased and is still that high a few hours after.

In gitlab by @Geal on Nov 15, 2016, 11:07

right now, we have a simple systemd service file, for a single process writing logs to stdout. In the near future, the proxy will use a main process with a few worker processes.

Questions:

• how to make a service file work correctly with a multiprocess arch?
• how to handle logging from the workers?

apparently, there's a KillMode=mixedoption to kill all the process in the cgroup. Ideally, in the case of a restart, the parent process would receive the signal and restart the workers itself, since it knows how to give them the relevant data. Right now, the workers will log on stdout, but they could send messages to syslog instead.

@keruspe @divarvel what do you think?

I open this issue with the current docker image. Maybe the problem is corrected outside docker.

Environment :

root@vertx:~/vertx-voting-app# docker version
Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:10:54 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:10:54 2017
 OS/Arch:      linux/amd64
 Experimental: false

The Linux is Ubuntu 16.04

root@vertx:~/vertx-voting-app# uname -a
Linux vertx 4.4.0-79-generic #100-Ubuntu SMP Wed May 17 19:58:14 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

The application comes from https://github.com/Treeptik/vertx-voting-app

And the stacktrace:

sozu_1          | 2017-06-19T19:29:39Z	6122965723685	9	DEBUG	HTTP-0	sozu_lib::network::protocol::http	2f5e0dfd-6460-492e-9cb7-a8b6ecec1c89	backend-vote		BACK [36->20036]: wrote 15 bytes of 15
sozu_1          | thread 'main' panicked at 'investigating single buffer usage: the front->back buffer should not be used while parsing and forwarding the response', /sozu-source/lib/src/network/protocol/http.rs:611
sozu_1          | stack backtrace:
sozu_1          |    1:     0x5605bbed94dc - std::sys::imp::backtrace::tracing::imp::write::hf33ae72d0baa11ed
sozu_1          |                         at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:42
sozu_1          |    2:     0x5605bbedf2ce - std::panicking::default_hook::{{closure}}::h59672b733cc6a455
sozu_1          |                         at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/panicking.rs:351
sozu_1          |    3:     0x5605bbedeed4 - std::panicking::default_hook::h1670459d2f3f8843
sozu_1          |                         at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/panicking.rs:367
sozu_1          |    4:     0x5605bbedf76b - std::panicking::rust_panic_with_hook::hcf0ddb069e7beee7
sozu_1          |                         at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/panicking.rs:555
sozu_1          |    5:     0x5605bbca7a13 - std::panicking::begin_panic::hcee73e0b293b8c83
sozu_1          |                         at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/panicking.rs:517
sozu_1          |    6:     0x5605bbda0a0b - <sozu_lib::network::protocol::http::Http<Front>>::back_readable::h4a1246cabf7ecd88
sozu_1          |                         at /sozu-source/lib/src/network/protocol/http.rs:611
sozu_1          |    7:     0x5605bbdbefb3 - <sozu_lib::network::http::Client as sozu_lib::network::proxy::ProxyClient>::back_readable::h876c202785470845
sozu_1          |                         at /sozu-source/lib/src/network/http.rs:272
sozu_1          |    8:     0x5605bbdf8ff5 - <sozu_lib::network::proxy::Server<ServerConfiguration, Client>>::ready::ha28e851867ca024d
sozu_1          |                         at /sozu-source/lib/src/network/proxy.rs:532
sozu_1          |    9:     0x5605bbdf4fc1 - <sozu_lib::network::proxy::Server<ServerConfiguration, Client>>::run::h033dcfd844da99f2
sozu_1          |                         at /sozu-source/lib/src/network/proxy.rs:415
sozu_1          |   10:     0x5605bbdcd748 - sozu_lib::network::http::start::h5253c70194772b51
sozu_1          |                         at /sozu-source/lib/src/network/http.rs:708
sozu_1          |   11:     0x5605bb929174 - sozu::worker::begin_worker_process::hfe3f858642ef9a0f
sozu_1          |                         at /sozu-source/bin/src/worker.rs:117
sozu_1          |   12:     0x5605bb930d4d - sozu::main::h3c24609eeed730d5
sozu_1          |                         at /sozu-source/bin/src/main.rs:74
sozu_1          |   13:     0x5605bbee67da - __rust_maybe_catch_panic
sozu_1          |                         at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libpanic_unwind/lib.rs:98
sozu_1          |   14:     0x5605bbedfed6 - std::rt::lang_start::hd7c880a37a646e81
sozu_1          |                         at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/panicking.rs:436
sozu_1          |                         at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/panic.rs:361
sozu_1          |                         at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/rt.rs:57
sozu_1          |   15:     0x5605bb946142 - main
sozu_1          |   16:     0x7fefa100182f - __libc_start_main
sozu_1          |   17:     0x5605bb602758 - _start
sozu_1          |   18:                0x0 - <unknown>
sozu_1          | 2017-06-19T19:46:04Z	7107772533523	1	INFO	MASTER	received client order AddInstance(Instance { app_id: "backend-sozu-vertxvotingapp", ip_address: "172.18.0.4", port: 80 }) with tag "HTTP"
sozu_1          | 2017-06-19T19:46:04Z	7107773357481	1	INFO	MASTER	received client order RemoveInstance(Instance { app_id: "backend-sozu-vertxvotingapp", ip_address: "172.19.0.2", port: 80 }) with tag "HTTP"

if there was an error handling the command and sozu doe not answer, sozuctl might just block waiting for the answer.

There's a solution I tested for the status command:

• run the code waiting for messages in a separate thread
• just before finishing, the thread sends a message. The calling code waits for this message with a timeout

Commands to update with this:

• [x] state save
• [x] state load
• [x] state dump
• [x] stop --hard
• [x] upgrade
• [x] metrics
• [x] add application
• [x] remove application
• [x] add frontend
• [x] remove frontend
• [x] add backend
• [x] remove backend
• [x] add certificate
• [x] remove certificate
• [x] query
• [x] logging filter

when using wrk2, the latency reports indicates a lot of timeouts, while sozu (current master at 6f4f67f ) does not indicate any issue:

$ ~/dev/clevercloud/wrk2/wrk -L -d 30s -c 200 -t 4 -R 10000 http://lolcatho.st:8080
Running 30s test @ http://lolcatho.st:8080
  4 threads and 200 connections
  Thread calibration: mean lat.: 2825.632ms, rate sampling interval: 10608ms
  Thread calibration: mean lat.: 2827.824ms, rate sampling interval: 10616ms
  Thread calibration: mean lat.: 3427.516ms, rate sampling interval: 11296ms
  Thread calibration: mean lat.: 2830.079ms, rate sampling interval: 10616ms
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency      nanus     nanus   0.00us    0.00%
    Req/Sec     0.00      0.00     0.00    100.00%
  Latency Distribution (HdrHistogram - Recorded Latency)
 50.000%    0.00us
 75.000%    0.00us
 90.000%    0.00us
 99.000%    0.00us
 99.900%    0.00us
 99.990%    0.00us
 99.999%    0.00us
100.000%    0.00us

  Detailed Percentile spectrum:
       Value   Percentile   TotalCount 1/(1-Percentile)

       0.000     1.000000            0          inf
#[Mean    =          nan, StdDeviation   =          nan]
#[Max     =        0.000, Total count    =            0]
#[Buckets =           27, SubBuckets     =         2048]
----------------------------------------------------------
  19800 requests in 30.25s, 623.75MB read
  Socket errors: connect 0, read 0, write 0, timeout 2114
Requests/sec:    654.45
Transfer/sec:     20.62MB

I put the access logs in a file, it reports 19775 successful HTTP requests (status 200, response times between 20ms and 120ms).

Here is my configuration:

#log_level = "info,sozu_lib::parser::http11=trace"
log_level = "info"
log_target = "stdout"
log_access_target = "file:///Users/geal/dev/rust/projects/yxorp/bin/sozu-access.log"

command_socket = "./command_folder/sock"

command_buffer_size = 16384
max_command_buffer_size = 163840

worker_count = 1

worker_automatic_restart = false
handle_process_affinity = false

max_connections = 500
max_buffers = 1200
buffer_size = 16384

[http]
address = "127.0.0.1"
port = 8080

[applications]
[applications.MyApp]
hostname = "lolcatho.st"
backends = [ "127.0.0.1:1026" ] # list of IP/port

I set the open files limit to 2048. The backend is a nginx server returning a static html page (32769 bytes).

From earlier runs, it appears that nginx returns Connection: keep-alive in each response, except the last ones.

I suspect there's an issue in the way sozu is handling the requests from wrk2 (maybe around the keep-alive/"Connection: close" behaviour), but I don't have the info yet.

This is annoying, because wrk2 is a really nice way to benchmark web servers, so it should work properly with sozu.

In gitlab by @Geal on Nov 3, 2016, 16:54

untested idea: the buffers are mainly used to parse headers, while the rest of the data can be transferred by splice() (on Linux). So keeping two buffers (one for front->back, one for back->front) could be a bit wasteful. The proxy could request one buffer from the pool to parse request headers, transfer the headers to the backend, then:

• if it is a headers-only request, reuse the buffer directly to parse back->front
• if it is a request with a full body, return the buffer to the pool, use splice to copy the body to the backend
• if it is a chunking request, return the buffer to the pool, copy and parse each chunk header in a fixed stack buffer (is there a limit to chunk size so we can infer a limit for the chunk header's size?), then use splice to copy the chunk's content

a few unknowns in this:

• how many changes to the current code it would require?
• how to keep compatibility with other platforms?
• can we really use the same buffer for back and front?

Hello, I wanted to try sozu on my server, unfortunately freeBSD does not seem to be supported because of this get_executable_path function.

I’ve never done this kind of unsafe / C call code in rust so please tell me if something is wrong with my code :)

I mostly open this issue to track it, I'll try to go deeper into why it behaves like this but right now, I don't have much clues.

I'm using sozu in front of a plex instance (plex being a self hosted netflix like). During video streaming, some requests hang and never complete. I have only 1 backend and 1 worker.

I don't really know from where this comes from. With haproxy in front of the plex instance, I can't reproduce the problem.

The last logs after a browser refresh, when the index.html page hangs:

2017-07-31T22:27:18Z    1219171966852505        8556    DEBUG   TAG-0   sozu_lib::network::protocol::http       b573d867-b590-48d8-82fa-1368ea8d5251    plex            FRONT [Some(Token(6148914691236517219))]: read 492 bytes
2017-07-31T22:27:18Z    1219171967105692        8556    TRACE   TAG-0   sozu_lib::parser::http11        PARSER  got Connection header: ""keep-alive""
2017-07-31T22:27:18Z    1219171967422359        8556    TRACE   TAG-0   sozu_lib::network::session      front readable  interpreting client order ConnectBackend
2017-07-31T22:27:18Z    1219171967500568        8556    DEBUG   TAG-0   sozu_lib::network::session      keepalive, reusing backend connection
2017-07-31T22:27:18Z    1219171967637816        8556    DEBUG   TAG-0   sozu_lib::network::protocol::http       b573d867-b590-48d8-82fa-1368ea8d5251    plex            BACK [6148914691236517219->6148914691236517718]: wrote 628 bytes of 628
2017-07-31T22:27:18Z    1219171967676536        8556    DEBUG   TAG-0   sozu_lib::network::protocol::http       b573d867-b590-48d8-82fa-1368ea8d5251    plex            BACK  [6148914691236517219<-6148914691236517718]: read 0 bytes
2017-07-31T22:27:18Z    1219171967731555        8556    TRACE   TAG-0   sozu_lib::network::session      front writable  interpreting client order Continue

This is not much context. It's just one request that hangs and some other still complete fine (not much though). Reloading the page doesn't help.

To elaborate on the "other requests still complete fine":

• I stream a movie
• The request downloading the video stream hangs at some point (it's one request only).
• I refresh the page, all assets do not load, some of them hang
• If I keep refreshing, more and more requests will hang until the index.html file hangs too (with the logs above)
• If I wait a few minutes (or even a few seconds) and refresh, everything is back to normal

Maybe it's the keep-alive between the proxy and the backend ? It's easy to reproduce but sometimes it takes 1 minute, some other times it takes 10 minutes.

I'll try to tcpdump this.

Continuation of #783

This pull request removes what @Geal started to work on but didn't finish. The cluster metrics are now stored without timestamps, which reduces the strain on the memory and thus is cleared only everyday at midgnight. They are stored in a simple binary tree map of the form cluster_or_backend_id -> (key, value)

The relation between is clusters and backend is tracked in another map of the form backend_id -> cluster_id

This pull request removes the option to enable the timed collection of metrics.

to do

• [x] queries for cluster and backend metrics, providing their ids and metric names.
• [x] error management on cluster & backend ids provided by the CLI
• [x] error management alltogether

solves #790 #789

As the SW is mainly developed for in-house use, I'd understand if you wanted me to leave the arrangement of your sitting room alone.

However, as a possible contributor, I'd appreciate being able to use rustfmt and not having to hand-indent code to conform to the existing style. Otherwise it would lead to lots of whitespace noise.

For now, Sōzu implements its own two layers of logging :

• a set of logging macros (info!, debug!, etc)
• a Logger instantiated in each worker to gather the log events and write them onto 5 possible targets:
  • a UDP address
  • a TCP address
  • a unix socket
  • a file
  • stdout

Since the birth of Sōzu, the tokio ecosystem has grown to become a defining standard in terms of asynchronous software architecture. It contains its own tracing crate, tracing, compatible with a number of loggers.

Switching Sōzu's home-made macros and logger will allow for a lot of enhancements down the way (such as structured logging: #437) and remove a great deal of code that is uneasy to maintain.

This PR aims to perform a drop-in replacement of Sōzu's logging logic, including notably:

• [x] a naive use of logging-subscriber to print logs to stdout throughout Sōzu
• [ ] the ability to change the log level during runtime
• [ ] send logs to all 5 targets (udp, tcp, unix socket, file, stdout)

In the spirit of the HTTP2 milestone, we want to split the functioning of current sessions (who pass the traffic from front socket to back socket) into:

• transport sessions (TCP)
• mux (for multiplexing) sessions, which entails:
  • an HTTP "server" (HTTP1 or H2) on the front side
  • one or several HTTP "clients" (HTTP1 or H2) on the backend side (this is multiplexing)
  • routing and multiplexing logic

Inspired by HAProxy's htx data format, we want to implement a generic HTTP data representation, in order to store and translate HTTP1 and H2 traffic.

HTX should be a middle layer for all HTTP traffic.

With the coming of Http2 and TCP over TLS, we may want to reuse and convert the HTTPS proxy as a generic SSL endpoint that redirects to Http2, Http1.1, or TCP clusters via ALPN.

Thoughts @FlorentinDUBOIS @Geal ?

As discovered in this Pull Request:

• #824

Errors are too often logged in the middle of low-level functions, when they should rather get trickled up and logged by the logic calling them.

A good example is in command/src/channel.rs:

https://github.com/sozu-proxy/sozu/blob/68e38591aaaa11ace37515c0781b8d14b8734ca1/command/src/channel.rs#L119-L132

This should rather return an anyhow::Result<()> with context, to be handled downstream.

Feel free to comment below on where this change should be applied (or not applied), with github permalinks. Also, quote this issue in ongoing pull requests.