Druid Exporter plays a fundamental role as a receiver of metrics events coming from Druid clusters, adopting the HTTP format as a means of communication

Overview

Druid Exporter


            ...................                                  ....................
           ~YYYYYYYYYYYYYYYYYYYYJ?~.                        .^7JYYYYYYYYYYYYYYYYYYYY7
                            ...:~7Y5J^                    :?5Y?~:...
                                   :?PJ.       .^:       7PJ^
?YYYY?. ^YYYYYYYYYYYYYYYYYYYJ?~.     ^P5.      !GJ      JG!      ~7JYYYYYYYYYYYYYYYYYYY!  7YYYYJ.
.:::::  .:::::::::::::::::::^!YP7     ~GJ      !GJ     !G?     ~557~:::::::::::::::::::.  .:::::
                               ?G7    .P5.     !GJ     JG^    ^GY.
              ~YYYYYYYYYYYYYYYYYGPYYYYYPPYYYYYY5GPYYYYYPG5YYYYPG5YYYYYYYYYYYYYYYYY7
              .::::::::::::::::YG7::::7G?::::::7GY:::::!GJ::::~G5^::::::::::::::::.
                             .?G?    .5P.      !GJ      YP:    !PY:
      :::::::::::::::::::^^!J5Y~    .YP^       !GJ      .5P^    :JPY7~^:::::::::::::::::::
     .?YYYYYYYYYYYYYYYYYYJJ?!:    .!P5:        :?~       .JP?.    .~7JJYYYYYYYYYYYYYYYYYYJ:
                                :755!                      ~YP?^.
             ....    .....::~!J55J~                          ^?55J7~^:.....    ....
            !YYYYJ. ^YYYYYYYJ?!^.                               :~7JYYYYYYY!  7YYYY?
             .....   .....                                            ......  ......
     ________________________________________________________________________________________
    /                                                                                        \
    |                                                                                        |
    |        ▗▄▄              █     ▗▖     ▗▄▄▄▖                                             |
    |        ▐▛▀█             ▀     ▐▌     ▐▛▀▀▘                     ▐▌                      |
    |        ▐▌ ▐▌ █▟█▌▐▌ ▐▌ ██   ▟█▟▌     ▐▌   ▝█ █▘▐▙█▙  ▟█▙  █▟█▌▐███  ▟█▙  █▟█▌          |
    |        ▐▌ ▐▌ █▘  ▐▌ ▐▌  █  ▐▛ ▜▌     ▐███  ▐█▌ ▐▛ ▜▌▐▛ ▜▌ █▘   ▐▌  ▐▙▄▟▌ █▘            |
    |        ▐▌ ▐▌ █   ▐▌ ▐▌  █  ▐▌ ▐▌     ▐▌    ▗█▖ ▐▌ ▐▌▐▌ ▐▌ █    ▐▌  ▐▛▀▀▘ █             |
    |        ▐▙▄█  █   ▐▙▄█▌▗▄█▄▖▝█▄█▌     ▐▙▄▄▖ ▟▀▙ ▐█▄█▘▝█▄█▘ █    ▐▙▄ ▝█▄▄▌ █             |
    |        ▝▀▀   ▀    ▀▀▝▘▝▀▀▀▘ ▝▀▝▘     ▝▀▀▀▘▝▀ ▀▘▐▌▀▘  ▝▀▘  ▀     ▀▀  ▝▀▀  ▀             |
    |                                                ▐▌                                      |
    |                                                                                        |
    \________________________________________________________________________________________/

Druid Exporter plays a fundamental role as a receiver of metrics events coming from Druid clusters, adopting the HTTP format as a means of communication. In addition to this capability, its primary function is to export these metrics to Prometheus, thus allowing the creation of meaningful graphs and visualizations.

The Project is entirely made in Rust with the Actix-web framework to provide maximum performance in receiving and delivering metrics for a large flow of requests.

Installation

Requires Docker

Please create .env file from .env.example and fill in all keys

In Apache Druid edit _common/common.runtime.properties change druid.emitter to http and add druid.emitter.http.recipientBaseUrl with Druid Exporter URL plus this path /druid and druid.emitter.logging.logLevel with value info, like this:

#
# Monitoring
#

druid.monitoring.monitors=["org.apache.druid.java.util.metrics.OshiSysMonitor", "org.apache.druid.java.util.metrics.JvmMonitor"]
druid.emitter=http
druid.emitter.http.recipientBaseUrl=http://localhost:7080/druid
druid.emitter.logging.logLevel=info

In Historical nodes we recommend to use this value in druid.monitoring.monitors

druid.monitoring.monitors=["org.apache.druid.server.metrics.HistoricalMetricsMonitor", "org.apache.druid.server.metrics.QueryCountStatsMonitor", "org.apache.druid.java.util.metrics.OshiSysMonitor", "org.apache.druid.java.util.metrics.JvmMonitor"]

And in brokers and router nodes we recommend to use this value in druid.monitoring.monitors

druid.monitoring.monitors=["org.apache.druid.server.metrics.QueryCountStatsMonitor", "org.apache.druid.java.util.metrics.OshiSysMonitor", "org.apache.druid.java.util.metrics.JvmMonitor"]

Feel free to add more metrics to monitor, we recommend the above ones as they facilitate cluster scalability and monitoring aspects

ℹ️ Please take a look at the Apache Druid emitter and metrics for more information and configurations https://druid.apache.org/docs/latest/configuration/#http-emitter-module

Run

In development mode you can run the following command on your terminal:

docker compose up

After the command is executed you can go to http://localhost:3000 and do login to Grafana with this credentials

user: admin

password: admin

Please in production mode change the user and password

Port Service
3000 Grafana
7080 Druid Exporter
9090 Prometheus

In production mode you can build the Dockerfile located in the docker/prod folder, to build this you may run this command:

docker build --platform=linux/amd64 --target prod -t druid-exporter -f docker/prod/Dockerfile .

⚠️ We recommend using Docker BuildKit for better build performance

After executing the command, a docker image called druid-exporter will be created which will only contain the Docker Exporter project which will be available on port 7080

Development

Want to contribute? Great!

The project using a simple code. Make a change in your file and be careful with your updates! Any new code will only be accepted with all viladations.

Kiwfy - Open your code, open your mind!

You might also like...
Converts Hikvision camera events to MQTT
Converts Hikvision camera events to MQTT

HikSink streams Hikvision camera and NVR events (motion, line crossing, tamper, illegal logins, etc.) to MQTT messages for consumption by home automat

A metrics collection application for Linux machines. Created for MSCS 710 Project at Marist College.

Linux-Metrics-Collector A metrics collection application for Linux machines. Created for MSCS 710 Project at Marist College. Development Environment S

`prometheus` backend for `metrics` crate
`prometheus` backend for `metrics` crate

metrics + prometheus = ❤️ API Docs | Changelog prometheus backend for metrics crate. Motivation Rust has at least two ecosystems regarding metrics col

Easily add metrics to your system -- and actually understand them using automatically customized Prometheus queries
Easily add metrics to your system -- and actually understand them using automatically customized Prometheus queries

Autometrics 📈 ✨ Autometrics is a macro that makes it trivial to add useful metrics to any function in your codebase. Easily understand and debug your

🚀 10x easier, 🚀 10x cheaper, 🚀 high performance,  🚀 petabyte scale - Elasticsearch/Splunk/Datadog alternative for 🚀 (logs, metrics, traces).
🚀 10x easier, 🚀 10x cheaper, 🚀 high performance, 🚀 petabyte scale - Elasticsearch/Splunk/Datadog alternative for 🚀 (logs, metrics, traces).

🚀 10x easier, 🚀 10x cheaper, 🚀 petabyte scale - Elasticsearch/Splunk/Datadog alternative for 🚀 (logs, metrics, traces). ZincObserve ZincObserve is

Easily add metrics to your system -- and actually understand them using automatically customized Prometheus queries
Easily add metrics to your system -- and actually understand them using automatically customized Prometheus queries

A Rust macro that makes it easy to understand the error rate, response time, and production usage of any function in your code. Jump from your IDE to

A minimal, allocation-free Prometheus/OpenMetrics metrics implementation for `no-std` and embedded Rust.

tinymetrics a minimal, allocation-free Prometheus/OpenMetrics metrics implementation for no-std and embedded projects. why should you use it? you may

🔍 Fully-featured metrics collection agent for First Tech Challenge competitions. Supports Prometheus.

Scout Scout is a fully-featured free and open source metrics collector for FTC competitions. The project is licensed under the GNU LGPLv3 license. Fea

🥧 Savoury implementation of the QUIC transport protocol and HTTP/3

quiche is an implementation of the QUIC transport protocol and HTTP/3 as specified by the IETF. It provides a low level API for processing QUIC packet

Releases(v1.0.1)
Owner
Kiwfy
Open code open mind
Kiwfy
An asynchronous dumb exporter proxy for prometheus. This aggregates all the metrics and exposes as a single scrape endpoint.

A dumb light weight asynchronous exporter proxy This is a dumb lightweight asynchronous exporter proxy that will help to expose multiple application m

Dark streams 3 Dec 4, 2022
An experimental HTTP server in Rust that supports HTTP/1.1, HTTP/2, and HTTP/3 over QUIC.

?? H123 An experimental HTTP server in Rust that supports HTTP/1.1, HTTP/2, and HTTP/3 over QUIC. Warning This is an experimental project and not inte

Naoki Ikeguchi 7 Dec 15, 2022
⏱ Cross-platform Prometheus style process metrics collector of metrics crate

⏱ metrics-process This crate provides Prometheus style process metrics collector of metrics crate for Linux, macOS, and Windows. Collector code is man

Alisue 12 Dec 16, 2022
Parse, edit and merge Prometheus metrics exposition format

promerge Promerge provides minimalistic and easy to use API to parse and manipulate Prometheus metrics. A simple usecase could be collecting metrics f

Mike Taghavi 4 Aug 12, 2023
An asynchronous Prometheus exporter for iptables

iptables_exporter An asynchronous Prometheus exporter for iptables iptables_exporter runs iptables-save --counter and scrapes the output to build Prom

Kevin K. 21 Dec 29, 2022
A Prometheus exporter for WireGuard

wireguard_exporter An asynchronous Prometheus exporter for wireguard wireguard_exporter runs wg show [..] and scrapes the output to build Prometheus m

Kevin K. 15 Dec 29, 2022
Rust wrapper for Eclipse iceoryx™ - true zero-copy inter-process-communication

iceoryx-rs Experimental rust wrapper for the iceoryx IPC middleware. clone and build The iceoryx repo is include as git submodule, therefore keep in m

null 43 Jan 4, 2023
Altruistic Angelshark is a project devoted to making Communication Manager (ACM) automation easier.

This project makes automating over one or more Communication Managers easier via OSSI over SSH.

ADP, LLC. 3 Feb 13, 2022
Elkodon - true zero-copy inter-process-communication in rust

elkodon - Zero-Copy Lock-Free IPC Purely Written In Rust Introduction Performance Getting Started Publish Subscribe Events Custom Configuration Suppor

null 12 Nov 27, 2023
Eclipse iceoryx2™ - true zero-copy inter-process-communication in pure Rust

iceoryx2 - Zero-Copy Lock-Free IPC Purely Written In Rust Introduction Performance Getting Started Publish Subscribe Events Custom Configuration Suppo

null 136 Jan 1, 2024