A CLI tool for CIs and build scripts, making file system based caching easy and correct (locking, eviction, etc.)

Overview

FS Dir Cache

A CLI tool for CIs and build scripts, making file system based caching easy and correct (locking, eviction, etc.)

When working on build systems / CIs it's often a requirement to utilize some best effort caching, with locking and eviction.

Not exactly rocket science, but non-trivial enough to not want to implement and maintain ad-hoc.

fs-dir-cache aims to be a simple to use utility from inside other scripts and programs taking care of the details.

Use case

CI cache

Imagine that you have a CI runner that can persist files between runs, and you'd like to utilize it to reuse and speed up some things:

set -euo pipefail
 
FS_DIR_CACHE_ROOT="$HOME/.cache/fs-dir-cache" # directory to hold all cache (sub)directories
FS_DIR_CACHE_LOCK_ID="pid-$$-rnd-$RANDOM"     # acquire lock based on the current pid and something random (just in case pid gets reused)
FS_DIR_CACHE_KEY_NAME="build-project-x"       # the base name of our key
FS_DIR_CACHE_LOCK_TIMEOUT_SECS="600"      # unlock after timeout in case our job fails misereably

fs-dir-cache gc unused --seconds "$((7 * 24 * 60 * 60))" # delete caches not used in more than a week

# create/reuse cache (sub-directory) and lock it (wait if already locked)
cache_dir=$(fs-dir-cache lock --key-file Cargo.toml)
# unlock it when the script finish
trap "fs-dir-cache unlock --dir ${cache_dir}" EXIT

# 'cache_dir' will now equal to something like '/home/user/.cache/fs-dir-cache/build-project-x-8jg9hsadjfkaj9jkfljdfsd'
# and script has up to 600s to use it exclusively

# build project
cargo build --target-dir="${cache_dir}/target"

Using just one tool, it's easy to get correct and practical caching including:

  • locking (including fallback timeouts)
  • evicition
  • timeouts
You might also like...
Shellharden is a syntax highlighter and a tool to semi-automate the rewriting of scripts to ShellCheck conformance, mainly focused on quoting
Shellharden is a syntax highlighter and a tool to semi-automate the rewriting of scripts to ShellCheck conformance, mainly focused on quoting

Shellharden is a syntax highlighter and a tool to semi-automate the rewriting of scripts to ShellCheck conformance, mainly focused on quoting

Shellcheck - a static analysis tool for shell scripts
Shellcheck - a static analysis tool for shell scripts

ShellCheck - A shell script static analysis tool ShellCheck is a GPLv3 tool that gives warnings and suggestions for bash/sh shell scripts: The goals o

A command-line tool for patching shell scripts inspired by resholve

patsh A command-line tool for patching shell scripts inspired by resholve nix run github:nix-community/patsh -- -f script.sh Usage Usage: patsh [OPTIO

The fastest memoizing and caching Python library written in Rust.

Cachebox Cachebox is a Python library (written in Rust) that provides memoizations and cache implementions with different cache replecement policies.

Implemented reverse-engineered signature algorithm to successfully register with Apple's caching server.

View as English 项目描述 本项目通过逆向得到苹果缓存服务器的签名算法,并可以成功注册缓存服务。算法分为两种运行模式。 运行模式 直接运行(x64): 效率较高,但只支持64位CPU。已测试可运行在Windows/Linux/macOS上。 模拟器运行: 兼容性极高,支持所有CPU架构

Non-interactive nREPL client for shell scripts and command-line

nreplops-tool (nr) nreplops-tool (nr) is a non-interactive nREPL client designed to be used in shell scripts and on the command-line. Early α warning:

Quickly save and retrieve values for shell scripts.

Quickly save and retrieve values for shell scripts.

solve scripts for all 3 of @0xhana's paradigm ctf challs!

hana solana ctf ok like all the best software engineers i got the technicals done on time and under budget but left documentation for future me. its n

FileSorterX is an automatic file sorting application that sorts your files into folders based on their file extension
FileSorterX is an automatic file sorting application that sorts your files into folders based on their file extension

FileSorterX is an automatic file sorting application that sorts your files into folders based on their file extension. With FileSorterX, you can easily keep your files organized and find what you need quickly.

Owner
Dawid Ciężarkiewicz
Born 1985, programming since ~1993, on Linux ~1998, professionally ~2003, on github ~2008, with Rust ~2013. Deep&wide generalist, FOSS enthusiast.
Dawid Ciężarkiewicz
SKYULL is a command-line interface (CLI) in development that creates REST API project structure templates with the aim of making it easy and fast to start a new project.

SKYULL is a command-line interface (CLI) in development that creates REST API project structure templates with the aim of making it easy and fast to start a new project. With just a few primary configurations, such as project name, you can get started quickly.

Gabriel Michaliszen 4 May 9, 2023
Punic is a remote caching CLI built for Apple's .xcframework

Punic is a remote caching CLI built for Carthage that exclusively supports Apple's .xcframeworks.

Shred Labs 26 Nov 22, 2022
🚀 Supercharge your development with easy to setup package scripts

Mist ?? Supercharge your development with easy to setup package scripts. Works with every codebase regardless of the language or framework used! Note:

Shiv 3 May 2, 2022
An easy file system based on eazy-fs of rcore.

fs-rs An easy mixed index file system based on eazy-fs of rcore. Usage if you don't have rust environment, you can download rust by: curl https://sh.r

Clstilmldy 9 Dec 21, 2022
A CLI tool you can pipe code and then ask for changes, add documentation, etc, using the OpenAI API.

AiBro This is your own little coding bro, immersed in the world of AI, crypto, and all other types of over hyped tech trends. You can pipe it code and

Josh Bainbridge 5 Sep 5, 2023
Self-contained template system with Handlebars and inline shell scripts

Handlematters Self-contained template system with Handlebars and inline shell scripts Introduction Handlematters is a template system that combines Ha

Keita Urashima 3 Sep 9, 2022
Easy, Simple, Clean. Making status bars reliable and up-to-date.

Simple Status Easy, Simple, Clean. Making status bars reliable and up-to-date. Installation Compiling simple_status yourself doesn't require much. Ins

James Butcher 5 Aug 1, 2022
try to find the correct word with only first letter and unknown letter count.

MOTUS Current dictionaries are provided in french and can contain some words not included in the official Motus dictionary. Additionally, dictionaries

Alexandre 6 Apr 11, 2022
languagetool-code-comments integrates the LanguageTool API to parse, spell check, and correct the grammar of your code comments!

languagetool-code-comments integrates the LanguageTool API to parse, spell check, and correct the grammar of your code comments! Overview Install MacO

Dustin Blackman 17 Dec 25, 2022
tidy-builder is a builder generator that is compile-time correct.

The Builder derive macro creates a compile-time correct builder which means that it only allows you to build the given struct if and only if you provi

M.Amin Rayej 7 Dec 18, 2022