Druid Exporter plays a fundamental role as a receiver of metrics events coming from Druid clusters, adopting the HTTP format as a means of communication.

Related tags

Network programming rust grafana prometheus prometheus-exporter druid apache-druid
Overview

Druid Exporter 


            ...................                                  ....................
           ~YYYYYYYYYYYYYYYYYYYYJ?~.                        .^7JYYYYYYYYYYYYYYYYYYYY7
                            ...:~7Y5J^                    :?5Y?~:...
                                   :?PJ.       .^:       7PJ^
?YYYY?. ^YYYYYYYYYYYYYYYYYYYJ?~.     ^P5.      !GJ      JG!      ~7JYYYYYYYYYYYYYYYYYYY!  7YYYYJ.
.:::::  .:::::::::::::::::::^!YP7     ~GJ      !GJ     !G?     ~557~:::::::::::::::::::.  .:::::
                               ?G7    .P5.     !GJ     JG^    ^GY.
              ~YYYYYYYYYYYYYYYYYGPYYYYYPPYYYYYY5GPYYYYYPG5YYYYPG5YYYYYYYYYYYYYYYYY7
              .::::::::::::::::YG7::::7G?::::::7GY:::::!GJ::::~G5^::::::::::::::::.
                             .?G?    .5P.      !GJ      YP:    !PY:
      :::::::::::::::::::^^!J5Y~    .YP^       !GJ      .5P^    :JPY7~^:::::::::::::::::::
     .?YYYYYYYYYYYYYYYYYYJJ?!:    .!P5:        :?~       .JP?.    .~7JJYYYYYYYYYYYYYYYYYYJ:
                                :755!                      ~YP?^.
             ....    .....::~!J55J~                          ^?55J7~^:.....    ....
            !YYYYJ. ^YYYYYYYJ?!^.                               :~7JYYYYYYY!  7YYYY?
             .....   .....                                            ......  ......
     ________________________________________________________________________________________
    /                                                                                        \
    |                                                                                        |
    |        ▗▄▄              █     ▗▖     ▗▄▄▄▖                                             |
    |        ▐▛▀█             ▀     ▐▌     ▐▛▀▀▘                     ▐▌                      |
    |        ▐▌ ▐▌ █▟█▌▐▌ ▐▌ ██   ▟█▟▌     ▐▌   ▝█ █▘▐▙█▙  ▟█▙  █▟█▌▐███  ▟█▙  █▟█▌          |
    |        ▐▌ ▐▌ █▘  ▐▌ ▐▌  █  ▐▛ ▜▌     ▐███  ▐█▌ ▐▛ ▜▌▐▛ ▜▌ █▘   ▐▌  ▐▙▄▟▌ █▘            |
    |        ▐▌ ▐▌ █   ▐▌ ▐▌  █  ▐▌ ▐▌     ▐▌    ▗█▖ ▐▌ ▐▌▐▌ ▐▌ █    ▐▌  ▐▛▀▀▘ █             |
    |        ▐▙▄█  █   ▐▙▄█▌▗▄█▄▖▝█▄█▌     ▐▙▄▄▖ ▟▀▙ ▐█▄█▘▝█▄█▘ █    ▐▙▄ ▝█▄▄▌ █             |
    |        ▝▀▀   ▀    ▀▀▝▘▝▀▀▀▘ ▝▀▝▘     ▝▀▀▀▘▝▀ ▀▘▐▌▀▘  ▝▀▘  ▀     ▀▀  ▝▀▀  ▀             |
    |                                                ▐▌                                      |
    |                                                                                        |
    \________________________________________________________________________________________/

Druid Exporter plays a fundamental role as a receiver of metrics events coming from Druid clusters, adopting the HTTP format as a means of communication. In addition to this capability, its primary function is to export these metrics to Prometheus, thus allowing the creation of meaningful graphs and visualizations.

The Project is entirely made in Rust with the Actix-web framework to provide maximum performance in receiving and delivering metrics for a large flow of requests.

Installation

Requires Docker

Please create .env file from .env.example and fill in all keys

In Apache Druid edit _common/common.runtime.properties change druid.emitter to http and add druid.emitter.http.recipientBaseUrl with Druid Exporter URL plus this path /druid and druid.emitter.logging.logLevel with value info, like this:

#
# Monitoring
#

druid.monitoring.monitors=["org.apache.druid.java.util.metrics.OshiSysMonitor", "org.apache.druid.java.util.metrics.JvmMonitor"]
druid.emitter=http
druid.emitter.http.recipientBaseUrl=http://localhost:7080/druid
druid.emitter.logging.logLevel=info

In Historical nodes we recommend to use this value in druid.monitoring.monitors

druid.monitoring.monitors=["org.apache.druid.server.metrics.HistoricalMetricsMonitor", "org.apache.druid.server.metrics.QueryCountStatsMonitor", "org.apache.druid.java.util.metrics.OshiSysMonitor", "org.apache.druid.java.util.metrics.JvmMonitor"]

And in brokers and router nodes we recommend to use this value in druid.monitoring.monitors

druid.monitoring.monitors=["org.apache.druid.server.metrics.QueryCountStatsMonitor", "org.apache.druid.java.util.metrics.OshiSysMonitor", "org.apache.druid.java.util.metrics.JvmMonitor"]

Feel free to add more metrics to monitor, we recommend the above ones as they facilitate cluster scalability and monitoring aspects

ℹ️ Please take a look at the Apache Druid emitter and metrics for more information and configurations https://druid.apache.org/docs/latest/configuration/#http-emitter-module

Run

In development mode you can run the following command on your terminal:

docker compose up

After the command is executed you can go to http://localhost:3000 and do login to Grafana with this credentials

user: admin

password: admin

Please in production mode change the user and password

Port Service
3000 Grafana
7080 Druid Exporter
9090 Prometheus

In production mode you can build the Dockerfile located in the docker/prod folder, to build this you may run this command:

docker build --platform=linux/amd64 --target prod -t druid-exporter -f docker/prod/Dockerfile .

⚠️ We recommend using Docker BuildKit for better build performance

After executing the command, a docker image called druid-exporter will be created which will only contain the Docker Exporter project which will be available on port 7080

Development

Want to contribute? Great!

The project using a simple code. Make a change in your file and be careful with your updates! Any new code will only be accepted with all viladations.

Not Empty Foundation - Free codes, full minds

You might also like...
Converts Hikvision camera events to MQTT
Converts Hikvision camera events to MQTT

HikSink streams Hikvision camera and NVR events (motion, line crossing, tamper, illegal logins, etc.) to MQTT messages for consumption by home automat

Corner Bit 48 Dec 27, 2022
A metrics collection application for Linux machines. Created for MSCS 710 Project at Marist College.

Linux-Metrics-Collector A metrics collection application for Linux machines. Created for MSCS 710 Project at Marist College. Development Environment S

Christopher Ravosa 2 May 2, 2022
`prometheus` backend for `metrics` crate
`prometheus` backend for `metrics` crate

metrics + prometheus = ❤️ API Docs | Changelog prometheus backend for metrics crate. Motivation Rust has at least two ecosystems regarding metrics col

Instrumentisto Team 2 Dec 17, 2022
Easily add metrics to your system -- and actually understand them using automatically customized Prometheus queries
Easily add metrics to your system -- and actually understand them using automatically customized Prometheus queries

Autometrics 📈 ✨ Autometrics is a macro that makes it trivial to add useful metrics to any function in your codebase. Easily understand and debug your

Fiberplane 341 Feb 6, 2023
🚀 10x easier, 🚀 10x cheaper, 🚀 high performance, 🚀 petabyte scale - Elasticsearch/Splunk/Datadog alternative for 🚀 (logs, metrics, traces).
🚀 10x easier, 🚀 10x cheaper, 🚀 high performance, 🚀 petabyte scale - Elasticsearch/Splunk/Datadog alternative for 🚀 (logs, metrics, traces).

🚀 10x easier, 🚀 10x cheaper, 🚀 petabyte scale - Elasticsearch/Splunk/Datadog alternative for 🚀 (logs, metrics, traces). ZincObserve ZincObserve is

Zinc Labs Inc. 80 Feb 22, 2023
Easily add metrics to your system -- and actually understand them using automatically customized Prometheus queries
Easily add metrics to your system -- and actually understand them using automatically customized Prometheus queries

A Rust macro that makes it easy to understand the error rate, response time, and production usage of any function in your code. Jump from your IDE to

Autometrics 462 Mar 6, 2023
A minimal, allocation-free Prometheus/OpenMetrics metrics implementation for `no-std` and embedded Rust.

tinymetrics a minimal, allocation-free Prometheus/OpenMetrics metrics implementation for no-std and embedded projects. why should you use it? you may

Eliza Weisman 282 Apr 16, 2023
🔍 Fully-featured metrics collection agent for First Tech Challenge competitions. Supports Prometheus.

Scout Scout is a fully-featured free and open source metrics collector for FTC competitions. The project is licensed under the GNU LGPLv3 license. Fea

hivemind 3 Oct 24, 2023
🥧 Savoury implementation of the QUIC transport protocol and HTTP/3

quiche is an implementation of the QUIC transport protocol and HTTP/3 as specified by the IETF. It provides a low level API for processing QUIC packet

Cloudflare 7.1k Jan 8, 2023
Releases(v1.0.1)
Owner
Not Empty Free Software Foundation
Not Empty Free Software Foundation
GitHub
An asynchronous dumb exporter proxy for prometheus. This aggregates all the metrics and exposes as a single scrape endpoint.

A dumb light weight asynchronous exporter proxy This is a dumb lightweight asynchronous exporter proxy that will help to expose multiple application m

Dark streams 3 Dec 4, 2022
An experimental HTTP server in Rust that supports HTTP/1.1, HTTP/2, and HTTP/3 over QUIC.

?? H123 An experimental HTTP server in Rust that supports HTTP/1.1, HTTP/2, and HTTP/3 over QUIC. Warning This is an experimental project and not inte

Naoki Ikeguchi 7 Dec 15, 2022
⏱ Cross-platform Prometheus style process metrics collector of metrics crate

⏱ metrics-process This crate provides Prometheus style process metrics collector of metrics crate for Linux, macOS, and Windows. Collector code is man

Alisue 12 Dec 16, 2022
Parse, edit and merge Prometheus metrics exposition format

promerge Promerge provides minimalistic and easy to use API to parse and manipulate Prometheus metrics. A simple usecase could be collecting metrics f

Mike Taghavi 4 Aug 12, 2023
An asynchronous Prometheus exporter for iptables

iptables_exporter An asynchronous Prometheus exporter for iptables iptables_exporter runs iptables-save --counter and scrapes the output to build Prom

Kevin K. 21 Dec 29, 2022
A Prometheus exporter for WireGuard

wireguard_exporter An asynchronous Prometheus exporter for wireguard wireguard_exporter runs wg show [..] and scrapes the output to build Prometheus m

Kevin K. 15 Dec 29, 2022
Rust wrapper for Eclipse iceoryx™ - true zero-copy inter-process-communication

iceoryx-rs Experimental rust wrapper for the iceoryx IPC middleware. clone and build The iceoryx repo is include as git submodule, therefore keep in m

null 43 Jan 4, 2023
Altruistic Angelshark is a project devoted to making Communication Manager (ACM) automation easier.

This project makes automating over one or more Communication Managers easier via OSSI over SSH.

ADP, LLC. 3 Feb 13, 2022
Elkodon - true zero-copy inter-process-communication in rust

elkodon - Zero-Copy Lock-Free IPC Purely Written In Rust Introduction Performance Getting Started Publish Subscribe Events Custom Configuration Suppor

null 12 Nov 27, 2023
Eclipse iceoryx2™ - true zero-copy inter-process-communication in pure Rust

iceoryx2 - Zero-Copy Lock-Free IPC Purely Written In Rust Introduction Performance Getting Started Publish Subscribe Events Custom Configuration Suppo

null 136 Jan 1, 2024