An asynchronous dumb exporter proxy for prometheus. This aggregates all the metrics and exposes as a single scrape endpoint.

Last update: May 27, 2022

A dumb light weight asynchronous exporter proxy

This is a dumb lightweight asynchronous exporter proxy that will help to expose multiple application metrics over a single port. This will be helpful if it is difficult to open multiple ports because of firewall restrictions.

Exporter proxy is capable of receiving and polling.

Exporter proxy receives data either as capnp or as json. It will not accept both of them at the same time. Check the receiver.format section in the config.toml to choose one.

If multiple applications send metrics to the exporter proxy, it is the application's responsibility to add a prefix to the metric name to prevent collission. If there is a duplicate occurrence of same metric name from two different applications, the exporter proxy will expose both occurrences.

Both the TCP and the Unix socket receiver endpoints expects the data to be in a key-value form. The capnp schema is also a basic kv structure where the key is the name of the application that is sending the metrics and the value, a string representation of metrics. If the format is chosen as json, it could be either a single multiline string representation of the entire metrics, or a json array strings. This is just a way for the exporter proxy to distinguish between the metrices send by different application. Upon receiving a new set of metrics, the old ones exposed will be overwritten.

Eg: if at first, an application sends the following metrics

"http_requests_total_foo_bar_app1{method=\"post\",code=\"200\"} 1027 1395066363000",
"http_requests_total_foo_bar_app1{method=\"post\",code=\"400\"}    3 1395066363000"

and then it only sends

"http_requests_total_foo_bar_app1{method=\"post\",code=\"200\"} 1029 1395066363000",

the exporter proxy will only have

"http_requests_total_foo_bar_app1{method=\"post\",code=\"200\"} 1029 1395066363000",

and will not remember "http_requests_total_foo_bar_app1{method=\"post\",code=\"400\"} 3 1395066363000"

Endpoints

  • a scrape endpoint /metrics this exposes all the metrics combined. As mentioned earlier, this is a dumb proxy and will not validate the prometheus exposition format. It is the application's responsibility to validate it before sending it. Most prometheus libraries supports formatting the metrics in a valid exposition format.
  • a metadata endpoint /apps this exposes a list of apps that sends metrics, the source that received the metrics and the last received time in UTC
exporter-proxy/python_test_clients  🍣 master 🐍 v3.9.2 🐏 6GiB/8GiB | 1024MiB/1024MiB
✦3 🕙 23:51:49 ⬢ [Docker] ✖  curl localhost:6555/apps
+---------------------------------------------------------------------------------------------------------------+
| Current time: 2022-05-22 23:51:52.939898700 UTC                                                               |
+---------------------------------------------------------------------------------------------------------------+
| +-------------------------------+-----------------------------------+---------------------------------------+ |
| | App Name                      | Last Update Time (UTC)            | Source                                | |
| +-------------------------------+-----------------------------------+---------------------------------------+ |
| | external_poll: 127.0.0.1:1027 | 2022-05-22 23:51:47.971181300 UTC | external_poll: 127.0.0.1:1027         | |
| +-------------------------------+-----------------------------------+---------------------------------------+ |
| | external_poll: 127.0.0.1:1025 | 2022-05-22 23:51:47.970590700 UTC | external_poll: 127.0.0.1:1025         | |
| +-------------------------------+-----------------------------------+---------------------------------------+ |
| | foo_bar_app1_metrics          | 2022-05-22 23:51:07.406709200 UTC | /var/run/exporter-proxy-receiver.sock | |
| +-------------------------------+-----------------------------------+---------------------------------------+ |
| | external_poll: 127.0.0.1:1026 | 2022-05-22 23:51:47.970724 UTC    | external_poll: 127.0.0.1:1026         | |
| +-------------------------------+-----------------------------------+---------------------------------------+ |
| | foo_bar_app0_metrics          | 2022-05-22 23:51:42.737333400 UTC | 127.0.0.1:6554                        | |
| +-------------------------------+-----------------------------------+---------------------------------------+ |
+---------------------------------------------------------------------------------------------------------------+

Test sending metrics

These tests assume the default configuration as mentioned in the config.toml. If it is modified, change accordingly. A couple of json files and a python script is included in the python_test_clients directory. json_string.json has the metric data as a single string whereas the json_array.json has the metric data as an array of strings. These are the two schemas that are supported. If the metrics isn't in any of these json format, it will not be deserialized and proxied.

run the exporter proxy as

RUST_LOG="trace" cargo run

and cd python_test_clients from another shell.

To proxy the capnp serialized test metrics, first make sure that the receiver.format is capnp and not json. Restart the exporter proxy if it was already running.

  • install the capnp library
pip install -r requirements.txt
  • run the script test_send_tcp_socket_capnp.py and curl localhost:6555/metrics to see if it is exposing the metrics in prometheus exposition format. Run curl localhost:6555/apps to see the app names that sent the metrics.

To proxy the the json serialized message, make sure that the receiver.format is json and not capnp.

  • install netcat-openbsd

  • run the test with either of the json format as

# to send it over the unix socket
cat ./json_array.json  | nc -U /var/run/exporter-proxy-receiver.sock -q 0

or

# to send it over tcp socket
cat ./json_string.json | nc -s 127.0.0.1 127.0.0.1 6554 -q 0

To test the external poll, cd to python_test_clients directory and run

python3 -m http.server 1025&
python3 -m http.server 1026&
python3 -m http.server 1027&

to run 3 instances of webservers on port 1025, 1026 and 1027 respectively.

After this, the metrics can be checked by cURLing the endpoints mentioned in the config.toml's poll_external.other_scrape_endpoints section like

curl localhost:1026/1026.html

GitHub

https://github.com/darkstreams/exporter-proxy
You might also like...

A metrics collection application for Linux machines. Created for MSCS 710 Project at Marist College.

Linux-Metrics-Collector A metrics collection application for Linux machines. Created for MSCS 710 Project at Marist College. Development Environment S

May 2, 2022

A runtime for writing reliable asynchronous applications with Rust. Provides I/O, networking, scheduling, timers, ...

Tokio A runtime for writing reliable, asynchronous, and slim applications with the Rust programming language. It is: Fast: Tokio's zero-cost abstracti

Jun 20, 2022

BLEZ - Asynchronous Bluetooth Low Energy on Linux for Rust

BLEZ - Asynchronous Bluetooth Low Energy on Linux for Rust This library provides an asynchronous, fully featured interface to the Bluetooth Low Energy

Oct 21, 2021

General-purpose asynchronous socket stream.

async-socket This crate implements a general-purpose asynchronous socket. The Socket implements AsyncRead, AsyncWrite, Stream and Clone traits and thu

Oct 20, 2021

Asynchronous Linux SocketCAN - Broadcast Manager support (BCM) with tokio

tokio-socketcan-bcm The Broadcast Manager protocol provides a command based configuration interface to filter and send (e.g. cyclic) CAN messages in k

Mar 10, 2022

A fast, stable, efficient, and lightweight intranet penetration, port forwarding tool supports multiple connections, cascading proxy, and transmission encryption

A fast, stable, efficient, and lightweight intranet penetration, port forwarding tool supports multiple connections, cascading proxy, and transmission encryption

A fast, stable, efficient, and lightweight intranet penetration, port forwarding tool supports multiple connections, cascading proxy, and transmission encryption

Jun 20, 2022

A proxy implement with http / socks5 in-bound and vmess out-bound, written in Rust and tokio.rs

tokio-vmess an Asynchronous proxy implement with http / socks5 in-bound and vmess out-bound, written in Rust and tokio Run example first, Fill out the

Apr 20, 2022

A versatile and efficient proxy framework with nice features suitable for various use cases.

A versatile and efficient proxy framework with nice features suitable for various use cases.

Jun 16, 2022

Simple and fast layer 4 proxy in Rust

Fourth 这一波在第四层。 English Fourth是一个Rust实现的Layer 4代理,用于监听指定端口TCP流量,并根据规则转发到指定目标。 功能 监听指定端口代理到本地或远端指定端口 监听指定端口,通过TLS ClientHello消息中的SNI进行分流 安装方法 为了确保获得您架构

Jun 10, 2022
Proxy sentry request to a sentry server using a tunnel/proxy endpoint

Sentry Tunnel This is a proxy that forwards tunneled sentry requests to the real sentry server. The implementation is based on the explanation provide

Apr 20, 2022
A Prometheus exporter for WireGuard

wireguard_exporter An asynchronous Prometheus exporter for wireguard wireguard_exporter runs wg show [..] and scrapes the output to build Prometheus m

Jun 5, 2022
JSON-RPC endpoint proxy that dumps requests/responses for debugging
JSON-RPC endpoint proxy that dumps requests/responses for debugging

json_rpc_snoop How to build Ensure you have cargo installed and in your PATH (the easiest way is to visit https://rustup.rs/) make This will create t

Apr 6, 2022
A modern, simple TCP tunnel in Rust that exposes local ports to a remote server, bypassing standard NAT connection firewalls
A modern, simple TCP tunnel in Rust that exposes local ports to a remote server, bypassing standard NAT connection firewalls

bore A modern, simple TCP tunnel in Rust that exposes local ports to a remote server, bypassing standard NAT connection firewalls. That's all it does:

Jun 17, 2022
UDP proxy with Proxy Protocol and mmproxy support
UDP proxy with Proxy Protocol and mmproxy support

udppp UDP proxy with Proxy Protocol and mmproxy support. Features Async Support Proxy Protocol V2 SOCKET preserve client IP addresses in L7 proxies(mm

Jun 2, 2022
Web3-proxy: a fast caching and load balancing proxy for web3 (Ethereum or similar) JsonRPC servers.

web3-proxy Web3-proxy is a fast caching and load balancing proxy for web3 (Ethereum or similar) JsonRPC servers. Signed transactions (eth_sendRawTrans

Jun 20, 2022
Lightweight proxy that allows redirect HTTP(S) traffic through a proxy.

Proxyswarm Proxyswarm is a lightweight proxy that allows redirect HTTP(S) traffic through a proxy. WARNING: This app isn't recomended for download lar

Apr 16, 2022
A Prometheus Aggregation Gateway for FAAS applications

Gravel Gateway Gravel Gateway is a Prometheus Push Gateway for FAAS applications. In particular it allows aggregation to be controlled by the incoming

Jun 13, 2022
A Prometheus Aggregation Gateway for FAAS applications

Gravel Gateway Gravel Gateway is a Prometheus Push Gateway for FAAS applications. In particular it allows aggregation to be controlled by the incoming

Jun 13, 2022
Export statistics of Mosquitto MQTT broker (topic: $SYS) to Prometheus

Preface The Mosquitto MQTT broker provides a number of statistics on the special $SYS/# topic (see mosquitto(8)). Build requirements As a Rust program

Mar 9, 2022