Generate alerts for when metrics/recordings become absent

Stile Education

Last update: Sep 9, 2022

Related tags

Utilities prometheus-absent-data-alert-rule-generator

Overview

prometheus-absent-data-alert-rule-generator

prometheus-absent-data-alert-rule-generator is a tool to generate alerts for missing data in time-series that we use in our other alerts and recordings.

Usage

This is a normal Rust project and can be managed using Cargo. For the most up-to-date usage, run:

cargo run -- --help

For normal usage you just need to pass in the directory containing your Prometheus rules files in as the first (and only) positional argument, e.g.

cargo run -- ./rules

This will generate an absent.rules.yml file in the rules directory containing all your absent alerts.

Absent time-series alert generation

The high-level overview of how the alerts are generated is:

Parse all the *.rules.yml files in the given directory and pull out the expressions from the expr field
In each expression, pull out all the time-series selector used (e.g. stack:public_http_errors_5xx_non_L3:rate1m_sum or aws_firehose_delivery_to_redshift_success_minimum[1h])
Group the selectors into those that are all the same
For each group "merge" the selectors into a rule based on some rules

"for" field is chosen based on the minimum of all the selectors' origin rules with a floor of 1h

For each selector generate a rule of the form:

 )" alert: absent_
   annotations: description: "No data for '
  
   '. This alert rule was generated by prometheus-absent-data-alert-rule-generator." summary: "No data for '
   
    ' data" for: 
    
      labels: severity: business_hours_page "> 
     - expr: "absent(
       
        )
        "
       
  alert: absent_
       
  annotations:
    description: "No data for '
       
        '. This alert rule was generated by   prometheus-absent-data-alert-rule-generator.
        "
       
    summary: "No data for '
       
        ' data
        "
       
  for: 
       
  labels:
    severity: business_hours_page

NOTE: For range-vector selectors (e.g. aws_firehose_delivery_to_redshift_success_minimum[1h]) the absent_over_time function is used because it's the range-vector equivalent of absent.

Dump all the rules to absent.rules.yml in the input directory or to the specified output file.

Ignoring selectors

You can ignore selectors by listing their names, one line per name, in a text file. By default the text file is ignore_metrics.txt in this directory, or you can use the --ignore-file flag to pass in your own path.

Testing

Testing is done using the normal cargo test. The only external dependency that you need to have installed is promtool which you can get from the prometheus repository.

You might also like...

Rust library to generate word cloud images from text and images !

wordcloud-rs A Rust library to generate word-clouds from text and images! Example Code use std::collections::HashMap; use std::fs; use lazy_static::la

2 Dec 8, 2022

Generate commit messages using GPT3 based on your changes and commit history.

Commit Generate commit messages using GPT-3 based on your changes and commit history. Install You need Rust and Cargo installed on your machine. See t

40 Jan 3, 2023

"Philips Ambilight for desktops". A tool to generate color palettes from your desktop wallpaper and send them to Home Assistant.

Desktop Dye DesktopDye is an open source project written in Rust that allows users to have their lights paired with Home Assistant adjust to the most

7 Feb 22, 2023

Proc. macro to generate C-like `enum` tags.

Continuous Integration Documentation Crates.io #[derive(EnumTag)] This crate provides a proc. macro to derive the EnumTag trait for the given Rust enu

5 Mar 27, 2023

Generate an SPDX Software Bill of Materials for Rust crates.

cargo-spdx cargo-spdx is currently in development and not yet ready for use. cargo-spdx provides a cargo subcommand to generate an SPDX Software Bill

13 May 18, 2023

Easy c̵̰͠r̵̛̠ö̴̪s̶̩̒s̵̭̀-t̶̲͝h̶̯̚r̵̺͐e̷̖̽ḁ̴̍d̶̖̔ ȓ̵͙ė̶͎ḟ̴͙e̸̖͛r̶̖͗ë̶̱́ṉ̵̒ĉ̷̥e̷͚̍ s̷̹͌h̷̲̉a̵̭͋r̷̫̊ḭ̵̊n̷̬͂g̵̦̃ f̶̻̊ơ̵̜ṟ̸̈́ R̵̞̋ù̵̺s̷̖̅ţ̸͗!̸̼͋

Rust S̵̓i̸̓n̵̉ I̴n̴f̶e̸r̵n̷a̴l mutability! Howdy, friendly Rust developer! Ever had a value get m̵̯̅ð̶͊v̴̮̾ê̴̼͘d away right under your nose just when

294 Dec 23, 2022

With Dejavu, you can have a perfect memory by capturing and organizing your visual recordings efficiently.

Dejavu The content in README.md is assisted by ChatGPT. Overview Dejavu is an open-source, cross-platform tool designed to help you record and search

127 Jul 31, 2023

⏱ Cross-platform Prometheus style process metrics collector of metrics crate

⏱ metrics-process This crate provides Prometheus style process metrics collector of metrics crate for Linux, macOS, and Windows. Collector code is man

12 Dec 16, 2022

Posts CAP alerts to a chat room

CAP-CHAT Sends CAP (Weather, etc) Warnings to Chat. Options Option Description Default --cap URL for the Atom/RSS feed to CAP alerts (can have multipl

4 Jan 12, 2022

Display NOAA Weather Alerts For A Given Lat/Lon

wxwarn Display NOAA Weather Alerts For A Given Lat/Lon Grabs the NOAA weather alerts shapefile, checks to see if there are any alerts for the given co

4 Jul 25, 2022

Blaze is a Rust script that continuously monitors server resource usage and sends real-time summaries and alerts to a Discord channel via Webhook.

Blaze Blaze is a Rust script designed to run 24/7 on a server, providing a summary of resource usage every 30 minutes (configurable) to a Discord chan

18 Sep 23, 2024

A simple to use rust package to generate or parse Twitter snowflake IDs,generate time sortable 64 bits unique ids for distributed systems

A simple to use rust package to generate or parse Twitter snowflake IDs,generate time sortable 64 bits unique ids for distributed systems (inspired from twitter snowflake)

5 Oct 6, 2022

Scalable and fast data store optimised for time series data such as financial data, events, metrics for real time analysis

OnTimeDB Scalable and fast data store optimised for time series data such as financial data, events, metrics for real time analysis OnTimeDB is a time

2 Apr 5, 2022

Export validator metrics to Prometheus

3 Apr 8, 2022

cli application to visualize crate download metrics.

2 Dec 13, 2022

Comments

Handle errors parsing Prometheus expressions

Previously if we failed to parse an expression there would be a log at error level but the exit code would still be zero - not what we want! Now the errors are logged and failure is indicated so that once we've finished processing the other files we can bubble the error up.

opened by nick96 2
Allow specifying a playbook to add to absent alert rules

I think that a generic "how to deal with absent alerts" playbook is probably sufficient. When running this on our internal alerts we can provide a link to a Stile specific playbook.

opened by nick96 0