Image optimization using Rust and Vips 🦀

Overview

Huffman

Image optimization using Rust and Libvips.

Requirements

You must have the following packages installed before getting started

  • Rust
  • Vips
  • pkg-config
  • cargo watch (Optional)
  • Docker (Optional. Required only for testing deploy phase)

Configuration

You must have the following env variables set before getting started. For local development create a .env file with these values:

  • AWS_ACCESS_KEY_ID: Access key id for AWS user
  • AWS_SECRET_ACCESS_KEY: Access key secret for AWS user
  • AWS_REGION: AWS region where buckets reside
  • SOURCE_BUCKET: The source bucket to read images from
  • CACHE_BUCKET: The bucket to store variants in
  • SQS_URL: URL for the SQS queue
  • SQS_POLL_INTERVAL= Polling interval for the SQS queue

Alternatively you can use doppler.io for the secrets

Running the server for development

$ RUSTFLAGS="$(pkg-config vips --libs)" cargo run

With cargo-watch for live reload during development

$ RUSTFLAGS="$(pkg-config vips --libs)" cargo watch -x run

Building & Publishing via Docker

Docker is used for building huffman into an image with all it's required dependencies. We use multistage builds for keeping the final container size low. Most of the Vips and Rust setup is borrowed from olxgroup-oss/dali.

  • Builder layer (Dockerfile.build): Uses the rust-alpine image and builds Vips from source.
  • Runtime layer (Dockerfile.runtime): Uses the alpine image and builds Vips from source. Install doppler-cli for pulling in the env variables.
  • App Layer (Dockerfile.app): Import builder and compile huffman into a binary read for execution. Import runtime and copies the built huffman binary into it. Copy the Rocket.toml config file from the build layer and start the huffman binary.

Since we copy this project into the docker image, any changes to the files that aren't included in .dockerignore will invalidate the cache for that layer. Make sure frequently changing files that aren't related to code are always added to the .dockerignore file to avoid this and keep builds fast.

During a push to master, the builder and runtime images are created if there is a change in the respective dockerfiles. This is because we don't include the project source in these images and so they won't change often. The app image is created for every commit and tagged with the commit version.

To build the docker images locally you can run

$ sh build.sh

Building docker images with a different target is incredibly slow on the M1 (>1 hour). If you are building the images to test locally, change the platform to arm64 in the build.sh file.

You might also like...
A Rust library that simplifies YAML serialization and deserialization using Serde.

Serde YML: Seamless YAML Serialization for Rust Serde YML is a Rust library that simplifies YAML serialization and deserialization using Serde. Effort

A simple workshop to learn how to write, test and deploy AWS Lambda functions using the Rust programming language

Rust Lambda Workshop Material to host a workshop on how to build and deploy Rust Lambda functions with AWS SAM and Cargo Lambda. Intro What is Serverl

Garden monitoring system using m328p and m2560 Arduino Uno boards
Garden monitoring system using m328p and m2560 Arduino Uno boards

Garden monitoring system using m328p and m2560 Arduino Uno boards. 100% Rust [no_std] using the avr hardware abstraction layer (avr-hal)

Generate commit messages using GPT3 based on your changes and commit history.
Generate commit messages using GPT3 based on your changes and commit history.

Commit Generate commit messages using GPT-3 based on your changes and commit history. Install You need Rust and Cargo installed on your machine. See t

🦀 Rust-based implementation of a Snowflake Generator which communicates using gRPC

Clawflake Clawflake is a Rust application which implements Twitter Snowflakes and communicates using gRPC. Snowflake ID numbers are 63 bits integers s

A pure Rust PLONK implementation using arkworks as a backend.

PLONK This is a pure Rust implementation of the PLONK zk proving system Usage use ark_plonk::prelude::*; use ark_ec::bls12::Bls12; use rand_core::OsRn

This crate allows you to safely initialize Dynamically Sized Types (DST) using only safe Rust.

This crate allows you to safely initialize Dynamically Sized Types (DST) using only safe Rust.

A working example of multi targets compilation for Rust using Github Actions.

A working example of multi targets compilation for Rust using Github Actions. Supports Windows, MacOSX, x86_64, ARM and Raspberry PI Linux.

Universal changelog generator using conventional commit+ with monorepo support. Written in Rust.
Universal changelog generator using conventional commit+ with monorepo support. Written in Rust.

chlog Universal changelog generator using conventional commit+ with monorepo support. chlog can generate the changelog from the conventional commits w

Owner
ChronicleHQ
ChronicleHQ
Prints the absolute path of all regular files in an unmounted btrfs filesystem image.

btrfs-walk-tut Prints the absolute path of all regular files in an unmounted btrfs filesystem image. Learning about btrfs: Btrfs Basics Series This re

Nishal Kulkarni 2 Feb 3, 2022
Ever wanted to torture your CPU by generating every image possible?

dumpsterfire Ever wanted to torture your CPU by generating every image possible? Well, now you can! This thing is worse than mining Bitcoin, since the

null 2 Jun 14, 2022
Boot tools: loader, image generator, etc as a library crate

ArcBoot v0 A uefi bootloader for riscv, arm and x86. Comes in the form of a single executable. the source code contains a single executable target and

Spectral Project 3 Oct 3, 2022
Small library for text to image steganography.

hips-lib Performs text to image steganography by hinding and retrieving secret text within images or pixel arrays. This is achieved by encoding the se

hewhocopypastes 4 Feb 21, 2023
A dynamic image generator.

dynimgen A dynamic image generator. How to use step 1: The designers export the design drawing as an svg file <svg> <rect /> <image src="img.png"

null 29 Jun 14, 2023
A tool that transforms an IMVU badge canvas into a single cohesive image.

IMVU Badge Canvas Image Generator This tool fetches an IMVU user's badge canvas and creates a single unified image representation of it. Instead of ha

Helba 3 Sep 20, 2023
Garden monitoring system using m328p Arduino Uno boards. 100% Rust [no_std] using the avr hardware abstraction layer (avr-hal)

uno-revive-rs References Arduino Garden Controller Roadmap uno-revive-rs: roadmap Components & Controllers 1-2 Uno R3 m328p Soil moisture sensor: m328

Ethan Gallucci 1 May 4, 2022
k-mer counter in Rust using the rust-bio and rayon crates

krust is a k-mer counter written in Rust and run from the command line that will output canonical k-mers and their frequency across the records in a f

null 14 Jan 7, 2023
Cookiecutter templates for Serverless applications using AWS SAM and the Rust programming language.

Cookiecutter SAM template for Lambda functions in Rust This is a Cookiecutter template to create a serverless application based on the Serverless Appl

AWS Samples 24 Nov 11, 2022
Awesome full-stack template using Yew and Rust

Docker + Actix + Yew Full Stack Template ??‍?? YouTube videos Full Stack Rust App Template using Yew + Actix! https://youtu.be/oCiGjrpGk4A Add Docker

Security Union 143 Jun 22, 2023