Image processing proxy and API, created to allow faster media delivery for the Holaplex storefront.

Overview

Description

imgopt is an image processing proxy with a very simple API, created to download and dynamically scaledown, convert, and cache different media formats on the fly to allow faster delivery for the Holaplex Storefront.

Supported actions

Getting started

Quickest way to start playing with imgopt is by launching the server via docker. By default, only proxying images from ipfs.io service is allowed (This can be changed by creating your own config. Use config-sample.toml as a guide to create your own config.toml file.

Quick start

docker run --network=host  mpwsh/imgopt:0.1.8

Open http://localhost:3030/health to validate your server is running. You should see 200 OK.

To get a scaled down version of an image or video, just make a GET request to your imgopt instance providing the desired width as a query parameter and the service in the URL path. The URL structure should look like this:

http://localhost:3030//?width=

In order to, for example, scale down this JPG image, you should point your browser to:

http://localhost:3030/ipfs/bafybeih26pot7dyvqkjabsx75fuypf6cy6derd6tojnfpctja75a2j7uk4?width=600

Click here to see it in action

Change the width to get the image size you want (Use a size from the allowed_sizes array in your config file. The following requests with same width will be served from cache directly and skip conversion entirely. To get the original image remove the ?width= parameter.

MP4 files work the same way, but those will be converted to GIF automatically and then scaled down to the desired width.

imgopt will create two folders inside the path specified in storage_path variable on the config.toml file to store the original and modified images and videos on start-up. If you will run imgopt from the container image, remember to mount a volume to persist the cached and original files in a folder on your control and send the modified config to the container as well.

mkdir imgopt-data
docker run -d --network=host -v $(pwd)/imgopt-data:/root/imgopt-data -v $(pwd)/config.toml:/root/config.toml mpwsh/imgopt:0.1.6

Customizing your configuration

The config file is pretty straightforward and all values are commented with a small description for ease of customization. To add more services to proxy and process through imgopt just add a new object as the one below, specifying name and uri endpoint. The endpoint should NOT contain a closing forward slash.

[[services]]
name = "ipfs"
endpoint = "https://ipfs.io/ipfs"
#max age header for media files (Optional, default 31536000 seconds)
cache.max_age = 31536000

[[services]]
name = "arweave"
endpoint = "https://arweave.net"

[[services]]
name = "yourservice"
endpoint = "https://servicewebsite.com"

Twitter request caching support

If you want to use imgopt to cache API calls to twitter, you need to set up the env var TWITTER_BEARER_TOKEN when executing.

Building from source

The code in this repository can be built using cargo without any further dependencies. Just clone the repo and execute cargo build --release. If you only need JPEG and PNG resizing you can stop installing things here and just just run the server located in ./target/release/. If MP4 and GIF are required on your setup then carry on.

Keep in mind that (as mentioned above) ffmpeg, gifsicle and gifski are required to trigger some conversions, which have their own dependencies.

Tooling dependencies

ffmpeg and gifsicle can be installed via apt in debian based systems.

apt install ffmpeg gifsicle libavformat-dev libavfilter-dev libavdevice-dev libclang-dev clang -y

gifski can be installed using brew, or you can download the binary directly from their github repo.

wget --quiet https://github.com/ImageOptim/gifski/releases/download/1.6.4/gifski_1.6.4_amd64.deb
dpkg -i gifski_1.6.4_amd64.deb

Last piece of the puzzle is copying a small wrapper script located in scripts/mp4-to-gif.sh, which will take care of calling ffmpeg and gifski to convert mp4 to gif.

Copy the script on the alongside imgopt (both should live in the same folder). You folder structure should look like this:

mp4-to-gif.sh
config.toml
imgopt

Once everything is setup, you should be able to just execute ./imgopt. The configuration being used is printed on startup when using log_level = "debug" to help troubleshooting.

License

See LICENSE

Comments
  • Assest Proxy Administrative API

    Assest Proxy Administrative API

    Goal

    Able to reset the asset cache through an admin API.

    Requirements

    • [x] Engineering team can pass a json list of broken urls to an API endpoint of asset proxy. This should purge the asset from the CDN, delete it from EFS, and reset retries for the asset.
    opened by kespinola 2
  • Track Requested URLs

    Track Requested URLs

    Goal

    Prevent continual requesting of "bad" URLs to prevent excessive NAT costs from reaching out to the Internet for content that does not exist.

    Requirements

    • imgopt tracks when an asset url is unresolvable (ie 4xx, 5xx)
    • imgopt updates attempts to fetch an asset url that is unresolvable
    • imgopt configured for max retries to asset urls
    • imgopt looks up the asset url in its internal asset store before requesting the asset from the Internet.
    • imgopt returns error to the caller instead of requesting from asset source when the asset has exceeded its maximum fetch attempts

    Tasks

    • [x] Document strategy for request URL tracking in Notion tech spec
    opened by kespinola 2
  • Include Cache-Control Headers

    Include Cache-Control Headers

    Background

    When the service being proxied by cloudflare sets Cache-Control header cloudflare will forward the header along to the browser which triggers it to lookup the context from its local cache instead of requesting from cloudflare. Screen Shot 2022-03-22 at 6 27 43 AM

    Goal

    Users that have viewed an image before from their browser serve images from local cache instead of requesting again from cloudflare.

    Requirements

    When image successfully resized set Cache-Control header max-age to trigger local cache lookups from a browser.

    https://github.com/holaplex/holaplex/blob/dev/pages/api/arweave/txt/%5Bid%5D.ts#L45

    opened by kespinola 1
  • Add templates

    Add templates

    Description

    Adds templates for pull requests and issues

    Type of change

    Please delete options that are not relevant.

    • [x] New feature (non-breaking change which adds functionality)

    How Has This Been Tested?

    • [x] Used in other repos successfully!
    opened by echohtp 0
  • implementing logging of asset download attempts

    implementing logging of asset download attempts

    imgopt will log failed attempts in a kv store to avoid requesting the same file over and over again. This will allow us to skip a lot of invalid or non-existent metadatas in devnet image

    opened by mpwsh 0
  • CORs Allow List

    CORs Allow List

    Goal

    Only approved domains can access the assets service.

    Requirements

    • CORs enforced by the CDN gateway before it reaches our internal network
    • Holaplex engineering can add or remove approved domains
    • CORs allow list should support wildcard selection like *.holaplex.com
    opened by kespinola 0
Owner
Holaplex
Holaplex - Build NFT stores on Solana
Holaplex
Image processing operations

imageproc An image processing library, based on the image library. There may initially be overlap between the functions in this library and those in i

image-rs 512 Jan 8, 2023
An advanced image processing library for Rust.

ImageProc Maintainers: @chyh1990 Note: this project is under active depvelopment, API may change! imageproc is a advanced image proccessing library fo

Chen Yuheng 97 Oct 18, 2022
ePaperify: Framebuffer/image pre-processing library for e-Paper displays

ePaperify: Framebuffer/image pre-processing library for e-Paper displays

Jackson Ming Hu 5 Mar 15, 2022
Takes a folder of images (as a palette), and an image, and figures out how to tile the palette to resemble the image!

Takes a folder of images (as a palette), and an image, and figures out how to tile the palette to resemble the image!

Jacob 258 Dec 30, 2022
Lust is a static image server designed to automatically convert uploaded image to several formats and preset sizes

What is Lust? Lust is a static image server designed to automatically convert uploaded image to several formats and preset sizes with scaling in mind.

Harrison Burt 242 Dec 22, 2022
Rust Lean Image Viewer - Fast and configurable image viewer inspired by JPEGView by David Kleiner

Rust Lean Image Viewer - Fast and configurable image viewer inspired by JPEGView by David Kleiner

3top1a 4 Apr 9, 2022
Pretty social media preview thumbnails on the edge

pretty-grauniad-thumbnails Created for the Guardian Fastly Hack day. I aimed to improve the rendering of our social media post sharing thumbnail image

Oliver Barnwell 3 Apr 27, 2022
Simple image resizer/cropper API

Simple image resizer/cropper API. This is a Rust ?? port of the following article on the Deno Blog: Build a Simple Image Resizing API in less than 100

Enrique Mejías 5 Mar 2, 2023
PixelWorldProxy - Simple Pixel World Proxy made in Rust.

PixelWorldProxy Simple Pixel World Proxy made in Rust. Feature Safe and fast Support subserver switching Usage Clone the repository: git clone https:/

null 3 Jan 5, 2022
Signed distance field font and image command line tool based on OpenCL.

SDFTool Signed distance field font and image command line tool based on OpenCL. Build Windows Run cargo build --release in Visual Studio developer x64

弦语蝶梦 7 Oct 16, 2022
A simple command-line utility (and Rust crate!) for converting from a conventional image file (e.g. a PNG file) into a pixel-art version constructed with emoji

EmojiPix This is a simple command-line utility (and Rust crate!) for converting from a conventional image file (e.g. a PNG file) into a pixel-art vers

Michael Milton 22 Dec 6, 2022
Rust library to get image size and format without loading/decoding

imageinfo-rs Rust library to get image size and format without loading/decoding. The imageinfo don't get image format by file ext name, but infer by f

xiaozhuai, Weihang Ding 47 Dec 30, 2022
Simple image metadata scrubber. Will remove EXIF, XMP and IPTC metadata.

Simple image metadata scrubber. Will remove EXIF, XMP and IPTC metadata.

Jae Lo Presti 12 Nov 29, 2022
Fastest image quadtree stylization implementation to date, capable of hundreds of fps and avoiding ugly non-squares.

Quadim Fastest image quadtree stylization implementation to date, capable of hundreds of fps and avoiding ugly non-squares. 简体中文 如果你是从B站来的…… “Listen t

K--A______ 13 May 11, 2023
Image Compression Algorithm

Image Compression Algorithm ?? A new lossless image compression algorithm. In the newest version the algorithm performs rather good, but manages to su

Hannes 31 May 10, 2022
tai (Terminal Ascii Image) tool to convert images to ascii written in Rust

TAI Terminal Ascii Image A tool to convert images to ascii art written in Rust ?? Notes This tool is still in development stage. Contributions All Con

Mustafa Salih 258 Dec 5, 2022
Artsy pixel image to vector graphics converter

inkdrop inkdrop is an artsy bitmap to vector converter. Command line interface The CLI binary is called inkdrop-cli and reads almost any image bitmap

Matthias Vogelgesang 62 Dec 26, 2022
Automated image compression for efficiently distributing images on the web.

Imager Apparently this project made it into the GitHub Archive Program. About Imager is a tool for automated image compression, and can competitively

Imager IO 487 Dec 25, 2022
Open Graphic Image Writer

Open Graphic Image Writer Documentation You can generate Open Graphic Image dynamically. A CSS-like API. You can generate image by using template imag

keiya sasaki 46 Dec 15, 2022