RusQTTbom takes weather data from the Bureau of Meteorology (BOM) and publishes that data via MQTT messages.

Overview

RusQTTbom

RusQTTbom collects weather data from the Bureau of Meterology (BOM) then publishes said data locally via MQTT messages. BOM weather data is only for Australian locations. This weather data is obtained via an undocumented API, which was seemingly discovered, or at least popularised, thanks to this Github repo - and this one. Because this is an undocumented API, it could be turned off at any time. BOM weather data is generally accepted as being the most accurate weather data in Australia.

The main idea with this program is to collect the BOM weather data via the API, parse the data, check for null or erroneous entries, then publish the wanted data in a local network via MQTT messages. This way, we can have many programs do things with the MQTT data in a de-coupled manner. For example, we can setup alerts and notifications. We can use various home automations. We can also write the data to a database. By using an MQTT message bus, we can loosely couple all of these services, which provides desired reliability in our system. In essense, RusQTTbom becomes a local weather sensor providing BOM weather data.

The BOM weather data API is used in this BOM weather site. Go to this website then enter your suburb / location in the search bar then click on the correct result. Take a note of the seven digit geohash in the URL in the address bar of your browser. This will be used to get data for your location and you need to add this to your config.toml file.

Weather data that is published by RusQTTbom via MQTT consists of the following values, by the MQTT topics:

  • outside/weather/current-temp
  • outside/weather/temp-feels
  • outside/weather/min-temp
  • outside/weather/max-temp
  • outside/weather/humidity
  • outside/weather/rain-today
  • outside/weather/wind-kms
  • outside/weather/wind-dir
  • outside/weather/gusts-kms
  • outside/weather/max-gust

All of the above MQTT topic names are user configurable (via the config.toml file). Other user configuration options consist of the following:

  • Location name
  • Location geohash
  • MQTT broker IP address
  • MQTT broker port
  • Minimum valid temperature
  • Maximum valid temperature
  • Minimum valid wind speed
  • Maximum valid wind speed
  • Minimum valid humidity
  • Maximum valid humidity
  • Minimum valid rainfall
  • Maximum valid rainfall

The data validation settings are to catch erroneous entries such as -99 or 999 and the like. RusQTTbom publishes the weather data asyncronously, which is totally unnecessary but it does. The RusQTTbom MQTT client name is 'rusqttbom'

RusQTTbom expects the config.toml file to be saved under the users home directory in /.config/rusqttbom/config.toml. The program will not create the directory or move the config file there. A user needs to do this, however it only needs to be done once. This works MacOS and it should work on other OSs too. Let me know if there are issues.

Issues can be viewed here, and the development plan can be found at the Project.

To use this program in its current state, the repo can be forked and cloned, configuration file edited and saved in the appropriate directory (as explained), then use cargo build --release. The binary itself, called rusqttbom, can be located at rusqttbom/target/release. The binary itself can be copied or moved anywhere else in your file system and can run by itself. Just remember the config.toml file. For best use of this program, be sure to set a cron job to run the binary every 10mins. Here is an example cron job that will run this program every 10mins.

crontab -e
*/10 * * * * /full/file/path/to/binary/rusqttbom

Once you get the data flowing into your environment via MQTT (via the cron jobs), you can setup automations, notifications, alerts, graph the data and / or save it to a database for longer term storage and analysis.

Important!: Do not hit this API more than once every ten minutes. Not abusing this API should help keep it from being shut down or modified. The API seems to only be updated every 10mins at a minimum so keeping your calls to a minimum makes sense.

Be sure to star this repo to watch the progress if this interests you!

Comments
  • Manage null values from the BOM API

    Manage null values from the BOM API

    In completing #1 and getting more familiar with the data from the API, I can see that null values appear to be relatively frequent. Because I am using structs and strict data types, the compiler really doesn't like it when there are null values. So I do need to sort that out and manage null values.

    bug 
    opened by adenoz 2
  • Setup configuration file

    Setup configuration file

    This is core functionality.

    A configuration file is necessary to allow a user to save sensitive MQTT broker details and optionally username and passwords.

    This file will also allow us to facilitate allowing a user to specify what data, from the data available in the API, a user wants. A user may want all of the data, a subset or even just one piece.

    core 
    opened by adenoz 2
  • Not parsing JSON from API due to null values in hierarchy

    Not parsing JSON from API due to null values in hierarchy

    This was due to the weather source used to develop the initial program. It had null values for wind gusts. However when a location that is providing the full data is available, it is not parsing properly due to the design of the structs. So need to design structs for full data then handle errors when null values are in the hierarchy.

    bug 
    opened by adenoz 1
  • Consider moving logic of sending MQTT message into standalone function

    Consider moving logic of sending MQTT message into standalone function

    This should tidy up the code in the get_weather() function and possibly allow for better concurrency. This may be too early of a time to work on this sort of optimisation. It is likely better to increase basic functioning and features first.

    core 
    opened by adenoz 1
  • Refactor logic of program into lib.rs file to facilitate testing

    Refactor logic of program into lib.rs file to facilitate testing

    Currently, all code is in the main.rs file. However it is currently in it's own function so moving should be trivial. This will be an important step to facilitate unit testing later as this cannot be done in the main.rs file.

    core 
    opened by adenoz 1
  • Publish weather data via MQTT messages

    Publish weather data via MQTT messages

    This is core functionality.

    ~~To do this, we'll also need a seperate configuration file to set broker details.~~ This is done, in #4

    We'll also need to determine topic naming conventions.

    Data should probably also be published with one data point to one topic rather than have one MQTT payload with all data in one json message. This will allow for easier later saving of MQTT messages into a database.

    core 
    opened by adenoz 1
  • Save the desired weather data into variables

    Save the desired weather data into variables

    This is core functionality.

    Saving the desired data into variables will allow for later error handling and prepare the data for publishing via MQTT messages.

    This could be individual variables or perhaps it will be best to use a Struct.

    core 
    opened by adenoz 1
  • GET and parse the BOM weather data

    GET and parse the BOM weather data

    This is core functionality.

    The program needs to be able to GET the weather data from the BOM API, parse the returned json and have it ready for subsequent processing.

    core 
    opened by adenoz 1
  • Add ability for user to change valid temperature bounds

    Add ability for user to change valid temperature bounds

    RusQTTbom currently has hard coded bounds for what a valid temperature value looks like. These are as follows:

    • min -20
    • max 50

    In colder areas of Australia (noting the BOM weather only covers Australia), there may be valid colder days than -20. Likewise, some areas may want to increase the lower bound as the temperate never even gets to freezing.

    This is a minor extra data validation step ensuring only as correct data as possible is being published.

    enhancement 
    opened by adenoz 0
  • Add ability for user to configure MQTT topics

    Add ability for user to configure MQTT topics

    Initially, the RusQTTbom program has hard coded MQTT topics. A user should be able to decide on the structure and naming of the MQTT topics.

    This functionality will likely be provided as options in a configuration file.

    This issue is related to the issue to provide user configuration of weather data (#7).

    enhancement 
    opened by adenoz 0
  • Change setup process from website to config file

    Change setup process from website to config file

    At the moment, users need to grab the geohash from the BOM website. However it would be better, and it is possible, do simply write in your location in the config.toml file that use an API call to grab the geohash. This would be much simpler and more streamlined from a user perspective.

    enhancement 
    opened by adenoz 0
  • Integrate daily weather forecasts

    Integrate daily weather forecasts

    So far, the program only gets the current weather observations. It makes sense to also want weather forecasts for at least the current day. This data is from a different endpoint though same overall API URL. This makes sense to start after the code refactoring is complete as at #10 as this will consist of a larger function as there are many more data points when considering the data is for multiple days.

    enhancement 
    opened by adenoz 1
  • Add ability of user to configure desired weather data

    Add ability of user to configure desired weather data

    Initially, there are no user configurable options to decide what weather data is wanted.

    A user should be able to select what weather data is collected from the BOM API.

    This feature will likely be implemented via a configuration file where options are provided.

    This will likely need to tie in with the MQTT topics configuration (#5).

    enhancement 
    opened by adenoz 0
Releases(v0.1.2)
  • v0.1.2(Jan 20, 2023)

    Backend, async publishing and user config updates

    • Tidied up codebase to prepare for unit testing #20
    • Added data validation and user configuration for data validation #23
    • MQTT publishing is now async, even though it doesn't need to be.

    Other general code refactoring splitting codebase up into modules for better expandability and upgradability.

    Full Changelog: https://github.com/athenars-io/rusqttbom/compare/v0.1.1...v0.1.2

    Source code(tar.gz)
    Source code(zip)
  • v0.1.1(Jan 18, 2023)

    What's Changed

    Incremental update focusing on error handling and minor refactoring.

    Overall, this means RusQTTbom can handle when some of the structure of the JSON from the API is missing. Previously, only the actual values were dealt with properly when missing or were the wrong type.

    Refactor #10 Error handling #6

    Full Changelog: https://github.com/athenars-io/rusqttbom/compare/v0.1...v0.1.1

    Source code(tar.gz)
    Source code(zip)
  • v0.1(Jan 11, 2023)

    Initial release

    This represents a minimally working program that takes weather data from the BOM API, conducts some type checking, then publishes only correct data types as MQTT messages to a local MQTT broker. The location and MQTT broker details (minus the port) are configurable via a config.toml file which is in the .gitignore file. No null values will be published.

    Full Changelog: https://github.com/athenars-io/rusqttbom/commits/v0.1

    Source code(tar.gz)
    Source code(zip)
Owner
Athenars
Athenars is building reliable, responsive, local and open source utilities for data driven smart home automation.
Athenars
Display NOAA Weather Alerts For A Given Lat/Lon

wxwarn Display NOAA Weather Alerts For A Given Lat/Lon Grabs the NOAA weather alerts shapefile, checks to see if there are any alerts for the given co

boB Rudis 4 Jul 25, 2022
Expand threaded messages without "Also sent to the channel"

slack-thread-expander Expand threaded messages without "Also sent to the channel" Usage Setup Slack App Create a Slack App for slack-thread-expander.

Kohei Suzuki 7 Feb 15, 2022
A simple Telegram bot that eats anonymous channel messages, written in Rust.

tgbot-nochanmsg By Asuna Right Control When creating your bot from botfather, it's not bad to disable privacy mode, because it needs to access the mes

h3a.moe 2 Aug 5, 2022
Automatic wallpaper downloader of posters of your favorite movies and TV shows via TMDb.

Wallpaperflix Automatic wallpaper downloader of posters of your favorite movies and TV shows via TMDb. Prerequisities https://tauri.app/v1/guides/gett

İsmail Karslı 2 Sep 8, 2022
A lock-free thread-owned queue whereby tasks are taken by stealers in entirety via buffer swapping

Swap Queue A lock-free thread-owned queue whereby tasks are taken by stealers in entirety via buffer swapping. This is meant to be used [thread_local]

Thomas Sieverding 20 Sep 9, 2022
cargo extension for flashing embedded rust programs via dfu based on jacobrosenthals cargo-hf2

cargo-dfu This crate provides a cargo subcommand to flash ELF binaries via dfu Most STM chips will probably work with this, although you might need to

Roman Kretschmer 0 Feb 6, 2022
A small monitoring process that checks if kstars is active, if not it sends a request to notify a user via telegram that it crashed

Astro monitor A small monitoring process that checks if kstars is active, if not it sends a request to notify a user via telegram that it crashed Inst

Mattia Procopio 2 Jan 10, 2022
Telegram bot help you to run Rust code in Telegram via Rust playground

RPG_BOT (Rust Playground Bot) Telegram bot help you to run Rust code in Telegram via Rust playground Bot interface The bot supports 3 straightforward

TheAwiteb 8 Dec 6, 2022
Canister for storing NNS principals via whitelist.

nns-registry Canister for storing NNS principals via whitelist. Development The canister can be built and deployed using dfx. Building Run dfx build T

DSCVR ONE 6 Nov 16, 2022
Write Cloudflare Workers in 100% Rust via WebAssembly. (A fork of workers-rs)

Note: This is a fork to workers-rs. Work-in-progress ergonomic Rust bindings to Cloudflare Workers environment. Write your entire worker in Rust! Read

Abid Omar 7 Jan 11, 2023
A library to access BGPKIT Broker API and enable searching for BGP data archive files over time from public available data sources.

BGPKIT Broker BGPKIT Broker is a online data API service that allows users to search for publicly available BGP archive files by time, collector, proj

BGPKIT 10 Nov 30, 2022
Uindex is a data store, for data that can be parsed as sentences in some context-free language.

Uindex - Universal index Uindex is a data store, for data that can be parsed as sentences in some context-free language.

Enrique Pérez Arnaud 3 Jul 20, 2021
NSE is a rust cli binary and library for extracting real-time data from National Stock Exchange (India)

NSE Check out the sister projects NsePython and SaveKiteEnctoken which are Python & Javascript libraries to use the NSE and Zerodha APIs respectively

Techfane Technologies 4 Nov 28, 2022
Showing how to deploy a Terra smart contract using Chainlink Data Feeds

Chainlink Terra Developing Requirements This demo requires the following components: Rust: rustup with cargo 1.44.1+. rustc and cargo 1.44.1+. Install

SmartContract 6 Aug 22, 2022
A tool using binrw to read FFXIV data files

last-legend-dob A tool using binrw to read FFXIV data files. Mainly made to harvest the music from the game files for personal consumption, since it t

Octavia Togami 2 Oct 10, 2022
An (unofficial) Rust library for querying db-ip.com data

db_ip An (unofficial) library for querying db-ip.com CSV databases in safe Rust. This library is not affiliated with or endorsed by db-ip.com. Be advi

Finn Bear 4 Dec 2, 2022
sblade or switchblade it's a multitool in one capable of doing simple analysis with any type of data, attempting to speed up ethical hacking activities

sblade or switchblade it's a multitool in one capable of doing simple analysis with any type of data, attempting to speed up ethical hacking activities

Gabriel Correia 1 Dec 27, 2022
A dead-simple tool for working with data in Kafka

ktool - a tool for Kafka ktool is a dead-simple tool for working with data in Kafka: Copy partitions / topics to disk Replay messages Inspect message

Dom 5 Nov 4, 2022
A Rust proc-macro crate which derives functions to compile and parse back enums and structs to and from a bytecode representation

Bytecode A simple way to derive bytecode for you Enums and Structs. What is this This is a crate that provides a proc macro which will derive bytecode

null 4 Sep 3, 2022