A build system from the future 🖖

Maintaining these manifests in a separate repo has become troublesome in the last few weeks of iterating across the different tricorders. Putting them into the same repo allows us to also read them directly from disk during development.

Hello folks 👋🏽 opening this issue so we can have a conversation around this. Especially interested in your legal advise here @Enkronan!

I've tagged all of you to review this, so if you approve it you agree to having your contributions up to now and from now licensed as MPL 2.0.

When running the command below, we see all the right things are being handled BUT the counts of the different events are off.

We have built 5 things, and we have cached 5 things, but there is no mention of the 11 tests that were ran and cached.

; warp test --public-store-metadata-url http://localhost:9000 --force-redownload ./test/verl_SUITE.erl
     Started http://localhost:9000/tricorder/beam/manifest.json
    Readying http://localhost:9000/tricorder/beam/manifest.json
       Ready http://localhost:9000/tricorder/beam/manifest.json
   Cache-hit https://rules.warp.build/toolchains/openssl  
   Cache-hit https://rules.warp.build/toolchains/erlang  
   Cache-hit src/verl_parser.erl  
   Cache-hit src/verl.erl  
        PASS ./test/verl_SUITE.erl parse_test (CACHED)
        PASS ./test/verl_SUITE.erl compare_test (CACHED)
        PASS ./test/verl_SUITE.erl between_test (CACHED)
        PASS ./test/verl_SUITE.erl compile_requirement_test (CACHED)
        PASS ./test/verl_SUITE.erl ./test/verl_SUITE.erl (CACHED)
        PASS ./test/verl_SUITE.erl lte_test (CACHED)
        PASS ./test/verl_SUITE.erl lt_test (CACHED)
        PASS ./test/verl_SUITE.erl gt_test (CACHED)
        PASS ./test/verl_SUITE.erl is_match_test (CACHED)
        PASS ./test/verl_SUITE.erl parse_requirement_test (CACHED)
        PASS ./test/verl_SUITE.erl eq_test (CACHED)
        PASS ./test/verl_SUITE.erl gte_test (CACHED)
    Finished ./test/verl_SUITE.erl in 865ms (5 built, 5 cached)

What we'd expect to see would be:

    Finished ./test/verl_SUITE.erl in 865ms (11 passed, 0 errors, 5 built, 16 cached)

And also, one specific line where we built a library (the test/verl_SUITE.erl library) reads:

        PASS ./test/verl_SUITE.erl ./test/verl_SUITE.erl (CACHED)

where it should read:

   Cache-hit ./test/verl_SUITE.erl

Adds Warp pack <target> command to the CLI. Also adds packaging capability to the core. Given an ArtifactManifest, we're now able to turn its outputs and dependencies into N tarballs and generate a PackageManifest. This work will enable a future publisher that gets these packages and publish them to a remote warp store.

Still need to:

• [x] write tests
• [x] add docs

Since the Mark I we've had the problem of bootstrapping of tricorders. In the Mark I we dealt with this by vendoring every dependency to avoid the resolution process. In the Mark II the resolution process is very different, so we need to come up with a new approach.

In this PR we work to make any tricorder bootstrappable by building them in a way that allows the least amount of incrementality (and thus the least amount of signature generation), while still allowing us to add incremental build support to them once they are built.

; warp build ./Makefile

This skips the Makefile since we don't have a tricorder for it yet, but it never finishes since its not pulled out of the queue correctly, and so the WorkerPool still believes there's work to do.

Now that we're publishing binaries on every tag, we need a good way of smoke-testing if the binary will run on some sample repositories or not. This tests I'm adding will get us started on this path.

Since we know how often we are accessing certain artifacts in the local store, we can keep track of the last access to allow for routine cleanups of the store.

This will help us keep our store size down to rather stable numbers.

Some toolchains do not have a clear version that are readable from configuration files. One example of this is protoc, that usually is expected to be installed globally via brew or apt.

To do this, we should add support in the Warpfile to define toolchain versions:

{
  "toolchains": {
    "protobuf": "3.21.12"
  }
}

Currently we are missing a few things to get Warp to build with itself. We'll use this issue to keep track of those things.

• [ ] #26
• [x] #30
• [ ] #29
• [ ] build and release in multiple architectures
  • [x] #32
  • [ ] #35

Right now, you can only run warp and give a target from the root of the project. This needs to be extended to accept relative paths from the current directory.

We should probably use the invocation dir and the workspace root to figure out the difference and then construct a relative path to the root of the workspace.