WIP: Rust implementation of packs for ruby

Alex Evanczuk

Last update: Jun 7, 2023

Related tags

Command-line packs

Overview

packs

WIP: Rust implementation of packs and packwerk for ruby

Features

It's entirely built in Rust, so it's really fast, and doesn't require any external dependencies since the binary contains everything that needed to run packs!
Currently all packs can do is generate a cache to be used by the ruby implementation.

Usage

One simple way to try out packs to generate your cache would be to create a bash function which wraps the call to bin/packwerk, like so:

# In your ~/.bash_profile or analogous file
packwerk() {
    if [ "$1" = "check" ] || [ "$1" = "update" ]; then
        echo "Calling packs generate_cache with args: ${@:2}"
        packs generate_cache "${@:2}"
    fi

    echo "Now calling packwerk with args: $@"
    bin/packwerk "$@"
}

You can also modify the bin/packwerk executable to call packs conditionally, e.g.

# In bin/packwerk
packs_executable = `which packs`.chomp
if !packs_executable.empty?
  if ARGV.first == 'check' || ARGV.first == 'update'
    puts "Calling packs generate_cache with args: #{ARGV[1..-1]}"
    system('packs', 'generate_cache', *ARGV[1..-1])
  end
end

Verification

As packs is still a work-in-progress, it's possible it will not produce the same results as the ruby implementation (see below). If not, please file an issue!

To verify:

Run rm -rf tmp/cache/packwerk to delete the existing cache.
Run packs generate_cache (see directions below to get binary)
Run bin/packwerk update to see how violations change using packs.

Downloading the Binary

Deployment ergonomics are still a WIP.

If you want to try it out:

Go to https://github.com/alexevanczuk/packs/releases
Download the packs asset and run chmod +x path/to/packs
Open the containing directory, right click on the binary, click open, and then accept the warning message that says its from an unknown developer (it's me!)
Execute path/to/packs to see the CLI help message.

You can add path/to/packs to your PATH so it's available in every terminal session.

Deployment Improvements

In the future, I hope to:

Somehow sign the binary so it does not get a warning message
Make it executable before download
Add directions to download via some other tool, or ship as a native ruby gem extension.

New to Rust?

Me too! This is my first Rust project, so I'd love to have feedback, advice, and contributions! If you're new to Rust, don't be intimidated! https://www.rust-lang.org has tons of great learning resources.

Comments

Parse packwerk.yml to allow generate_cache to read the correct set of files
write trivially passing test

remove panic

write failing test

make tests pass

make test pass

attempt with glob walker builder

add globset

make tests pass

add to test

light cleanup

add doc

make package paths tests pass

make cache enabled test pass

panic if cache is disabled but generate-cache is called

make cli commands snake_case

docs for greet

document list packs command -n

document remaining methods

use sets for package paths and included files

take the intersection of input files

make tests pass

move around readme contents

add failing text and fix it

fix faiing test

add test for dir case

add more info on how to use
opened by alexevanczuk 0
Organize code
move code into packwerk.rs

improve organization of pack tests

remove boiler plate test_ prefix of test functions

Rename Reference to UnresolvedReference

move cache implementation into cache module out of CLI

use simpler match arms

fill out todo
opened by alexevanczuk 0
Cover some reference extraction cases
write failing test

make test pass

write fialing test

fix other test

write failing test

address clippy warning

try to make test pass

clippy

make another tests pass

Fix typo

write failing test

make tests pass

fix some edge cases in association detection
opened by alexevanczuk 0
First pass at "pks generate_cache"
get md5 of filename to pass

add test for cache file

set cody directive

write failing file cache test

fix compiler errors

move test to be a unti test, get it to almost pass

start working on use serde...

set max width to 80 for my small personal laptop

start updating test...

delete dead code

implement proper shape serialization

getting closer

add to gitignore

delete tmp

getting closer

getting a lot closer...!

get tests to pass
opened by alexevanczuk 0
generate cache scaffolding
update readme

update fixture to simple_app

write failing test for generate_cache

fix bad reference to fixture

write comment

fix test

create instructions

Move CLI under packs namespace

extract string helpers into function
opened by alexevanczuk 0
Fill out some basic things with sample/example code
cargo new packs

add dependencies based on what is in codeowners-rs

add some failing tests (copied from codeowners-rs)

ignore existing tests for now, since they are for quick reference anyways

write failing test called "simple dependency test"

pull in some stuff from codeowners-rs and start changing, trying to get it to compile

try to get it to compile

cargo publish errors

add license

add dev.md

rename crate
opened by alexevanczuk 0
Performance Improvements
Although packs is intended to be fast, there are ways it can be made a lot faster!

Improve use of references

As I'm new to Rust, I don't take advantage of a lot of features in Rust that would improve the performance, such as making sure I minimize the use of deep clones and use references.

Using cache conditionally to avoid cost of a cache miss

Once packs check is implemented, it should potentially have different cache behaviors:

If a list of files is inputted (e.g. used with commit hook, vscode extension, etc), we should consider not attempting to read the cache, but instead parsing the file and then writing to the cache in a separate thread. The reason being that if someone is running packs check with few files, it's likely those files have changed and we'll get a cache miss, so we might be potentially faster by skipping the cache entirely.

If no files are inputted, the cache could be loaded for all files.

Faster cache load

We could potentially instead have a single, compressed cache file that can be loaded.

Using modified time instead of file contents digest

By using modified time, we can avoid opening the entire file and parsing it and calculating the md5 hash. It's possible this would not be a meaningful performance improvement.
opened by alexevanczuk 0
Design Considerations
Highly parallelized:

In parallel threads, should call two functions to generate constant resolver file map (i.e. definitions) and references

If possible, references should be pushed to a stack that can immediately be popped and passed through checkers in parallel

Validators should also run in parallel

Should iterate over files in parallel to build

Idea:

Cache could be a trie or even just an indexed database that loads into memory?

The current implementation does way too much and can be faster
opened by alexevanczuk 0
Various Enhancement Ideas
packs validate (could be part of pks check)

Could output connected components rather than "cycles", or just one example cycle.

Validate should spit out the count of connected subgraphs and look at modified time of pack YML files to determine which new dependency is causing cycle

packs init (could be always run if necessary)

packs create | move

CLI Could be interactive like use_packs, e.g. https://github.com/mikaelmello/inquire

New features turned on with environment flags, e.g.

Unnecessary dependency detection

Privacy violation inversion

Could potentially support "packs" file notation, e.g. package.yml => pack.yml. packwerk.yml => packs.yml. Disable with packwerk: true
opened by alexevanczuk 0
LSP Ideas
Make pack names in package.yml files a link

Highlight violations

Status bar packs (like Sorbet)

Run update

Jump to package.yml

Jump to pack public API

add violations for individual constant reference violations

One-click to solve violation should, for example, detect for cycles in advance before recommending adding a dependency. It can say “if you were to add this as a dependency to packs/foo/package.yml, you’ll create cycles in the dependency graph, resulting in X strongly connected components.
opened by alexevanczuk 0