Zero-Trust Decentralized Package Network

ci: avoid runs if rs files weren't changed

We don't need to run linting tasks if rust files are same with HEAD. GithHub Actions provide ability to skip task by conditions. Was added check that there are changed rust files. If check returns True we execute linting and others workflows, otherwise, GitHub Actions should skip them.

Close: pyrsia/pyrsia#479

Runs links with example

when we have to skip jobs: this when job should be triggered: that

PR Checklist

• [x] I've read the contributing guidelines.
• [x] I've read "What is a Good PR?"
• [x] I've included a good title and brief description along with how to review them.
• [x] I've linked any associated an issue.
• [X] I've requested a review from pyrsia/collaborators.

Code Contributions

I don't do next, because it is workflow changes

• [ ] I've built the code cargo build --all-targets successfully.
• [ ] I've run the unit tests cargo test --workspace and everything passes.

And I need some help to understand with what branches shoud we compare current. Is it good approach for your team? Any other points? Are these jobs chosen correctly?

Description

Fixes pyrsia/pyrsia#

This PR introduces relay protocol for central/authoritative node and also separation of network code from main node. This is the first setup for hole punching is to have relay node set.

Screenshots (optional)

PR Checklist

• [x] I've read the contributing guidelines.
• [x] I've read "What is a Good PR?"
• [x] I've included a good title and brief description along with how to review them.
• [x] I've linked any associated an issue.
• [x] I've requested a review from pyrsia/collaborators.

Code Contributions

• [ ] I've built the code cargo build --all-targets successfully.
• [ ] I've run the unit tests cargo test --workspace and everything passes.
• [ ] I've made sure my rust toolchain is current rustup update.

Description

Fixes pyrsia/pyrsia#501

This pull request adds a peer metrics so peers can find idle peers that are good candidates for downloading artifacts.

Screenshots (optional)

PR Checklist

• [x] I've read the contributing guidelines.
• [x] I've read "What is a Good PR?"
• [x] I've included a good title and brief description along with how to review them.
• [x] I've linked any associated an issue.
• [x] I've requested a review from pyrsia/collaborators.

Code Contributions

• [x] I've built the code cargo build --all-targets successfully.
• [x] I've run the unit tests cargo test --workspace and everything passes.

Signed-off-by: Abhijith Ganesh [email protected]

PR Checklist

• [x] I've built the code cargo build. For major changes, cargo build --workspace --release is recommended.
• [x] I've run the automated unit tests cargo test. This executes our automated unit tests.
• [x] I've not broken any existing tests or functionality. In addition to cargo test, I've run cargo clippy
• [x] I've not introduced any new known security vulnerabilities. I've run cargo audit

• [x] I've included a brief description and a good title of the proposed changes and how to test them.
• [x] I've read the contributing guidelines.
• [x] I've associated an issue with this Pull Request.
• [x] I've checked that the code is up-to-date with the pyrsia/main branch.
• [x] I've read "What is a Good PR?"
• [x] I've assigned this Pull Request to "pyrsia/collaborators"

Description

Fixes pyrsia/pyrsia#484 Closes #86

This PR deletes signed_struct, signed effectively removing the OpenSSL dependency for the Node.

Screenshots (optional)

• [x] I've built the code cargo build. For major changes, cargo build --workspace --release is recommended.
• [x] I've run the automated unit tests cargo test. This executes our automated unit tests.
• [x] I've not broken any existing tests or functionality. In addition to cargo test, I've run cargo clippy
• [x] I've not introduced any new known security vulnerabilities. I've run cargo audit

• [x] I've included a brief description and a good title of the proposed changes and how to test them.
• [x] I've read the contributing guidelines.
• [x] I've associated an issue with this Pull Request.
• [x] I've checked that the code is up-to-date with the pyrsia/main branch.
• [x] I've read "What is a Good PR?"
• [x] I've assigned this Pull Request to "pyrsia/collaborators"

Description

Fixes pyrsia/pyrsia#363 Fixes pyrsia/pyrsia#396 Closes pyrsia/pyrsia#425

When an docker blob is not available locally in the artifact manager, the blob is fetched from docker.io as a fallback. This PR introduces an intermediate step that allows to fetch the blob from another pyrsia node instead, before falling back to docker.io.

The flow changes to the following, where step 2 is the introduced change:

1. retrieve the artifact locally from the artifact manager
2. if an address to another pyrsia node is provided via the --peer option, retrieve the artifact from that pyrsia node
3. fallback to retrieving the artifact from docker.io

Testing

Make sure you configured the docker daemon with the registry-mirror set to localhost:7888 as described in step 5 in the getting started guide.

Testing can be done with the following steps:

Terminal one: setup pyrsia node 1

1. cargo build --workspace
2. mkdir /tmp/pyrsia
3. cp target/debug/pyrsia_node /tmp/pyrsia
4. cd /tmp/pyrsia
5. mkdir node1
6. cd node1
7. DEV_MODE=on RUST_LOG=debug,netlink_proto=info ../pyrsia_node -p 7889

Take not of the libp2p listener address, the info message to look for looks like:

INFO  pyrsia::network::p2p > Local node is listening on "/ip4/127.0.0.1/tcp/43975/p2p/12D3KooWBWMJK8rSrx313fkJGYjhpvGmekfYADCpsWLtMHKX1Nvb"

Copy the entire address that is printed between the " at the end of that line. This address is needed when starting the second pyrsia node later on.

Terminal two: populate pyrsia node 1 with the docker nginx image

1. docker pull localhost:7889/library/nginx
2. docker rmi localhost:7889/library/nginx

Terminal three: setup pyrsia node 2

1. cd /tmp/pyrsia
2. mkdir node2
3. cd node2
4. DEV_MODE=on RUST_LOG=debug,netlink_proto=info ../pyrsia_node -P ${node1_listen_address}

Terminal two: pull the docker nginx image from pyrsia node 2

1. docker pull nginx

The output of pyrsia node 2 should contain messages like the following:

DEBUG pyrsia::docker::v2::handlers::blobs     > Step 1: Does "sha256:1ae07ab881bd848493ad54c2ba32017f94d1d8dbfd0ba41b618f17e80f834a0f" exist in the artifact manager?
DEBUG pyrsia::docker::v2::handlers::blobs     > Step 1: NO, "sha256:1ae07ab881bd848493ad54c2ba32017f94d1d8dbfd0ba41b618f17e80f834a0f" does not exist in the artifact manager.
DEBUG pyrsia::docker::v2::handlers::blobs     > Step 2: Does "sha256:1ae07ab881bd848493ad54c2ba32017f94d1d8dbfd0ba41b618f17e80f834a0f" exist in the Pyrsia network?
DEBUG pyrsia::docker::v2::handlers::blobs     > Step 2: YES, "sha256:1ae07ab881bd848493ad54c2ba32017f94d1d8dbfd0ba41b618f17e80f834a0f" exists in the Pyrsia network.
DEBUG pyrsia::docker::v2::handlers::blobs     > Final Step: "sha256:1ae07ab881bd848493ad54c2ba32017f94d1d8dbfd0ba41b618f17e80f834a0f" successfully retrieved!

PR Checklist

• [x] I've built the code cargo build. For major changes, cargo build --workspace --release is recommended.
• [x] I've run the automated unit tests cargo test. This executes our automated unit tests.
• [x] I've not broken any existing tests or functionality. In addition to cargo test, I've run cargo clippy
• [x] I've not introduced any new known security vulnerabilities. I've run cargo audit

• [x] I've included a brief description and a good title of the proposed changes and how to test them.
• [x] I've read the contributing guidelines.
• [x] I've associated an issue with this Pull Request.
• [x] I've checked that the code is up-to-date with the pyrsia/main branch.
• [x] I've read "What is a Good PR?"
• [x] I've assigned this Pull Request to "pyrsia/collaborators"

Description

For #322

This adds the crate + a basic trait that required with minimal tests

Screenshots (optional)

The error I get:

error[E0515]: cannot return reference to temporary value
  --> pyrsia_node/src/block_chain/block_chain.rs:34:9
   |
34 |            &mut BlockChain {
   |  __________^____-
   | | _________|
   | ||
35 | ||             blocks: vec![]
36 | ||         }
   | ||         ^
   | ||_________|
   | |__________returns a reference to data owned by the current function
   |            temporary value created here

warning: variable does not need to be mutable
   --> pyrsia_node/src/main.rs:215:9
    |
215 |     let mut bc = BlockChain::new();
    |         ----^^
    |         |
    |         help: remove this `mut`
    |
    = note: `#[warn(unused_mut)]` on by default

resolves #56

See https://github.com/pyrsia/pyrsia/pull/102#discussion_r769751309 for conversation

This is extremely confusing and make it very error prone. We need to apply the suggestion from https://doc.rust-lang.org/cargo/guide/project-layout.html

https://doc.rust-lang.org/book/ch11-03-test-organization.html

https://github.com/pyrsia/pyrsia/pull/62#issuecomment-990199046

…enhancement #1254)

Description

This issues implements a new feature described in #1254, The idea is that users can check the status of the requested build. New feature added to track the requested build state, from the user perspective the following CLI commands were added: pyrsia build status --id <BUILD ID>

Few examples of returned messages: Build status for 'b024a136-9021-42a1-b8de-c665c94470f4' is 'RUNNING' Build status for 'b024a136-9021-42a1-b8de-c665c94470f4' is 'SUCCESS' Build status for 'f1548c9a-f2ee-46b9-8993-eed0a8e5c239' is 'FAILED'

When the build ID is not found or expired: Build status for 'b024a136-9021-42a1-b8de-c665c94470f4' was not found. Additional info related to the build might be available via 'pyrsia inspect-log' command.

The code follows patterns and style of other similar features including the "build request/start". It reuses the already existing API but also adds functionality to the CLI, build service, network and node API. New unit tests added also there is a pending integration test PR: pyrsia/pyrsia-integration-tests#18

Fixes pyrsia/pyrsia# The new feature is described here: #1254

This PR is a first step towards a more useful "build status" feature. Eventually, we should be able to add functionality which fetches the build logs and more. It should be relatively easy to add it based on this PR as long as the pipeline/build service supports it (provides API to get the build logs, etc).

PR Checklist

• [x] I've read the contributing guidelines.
• [x] I've read "What is a Good PR?"
• [x] I've included a good title and brief description along with how to review them.
• [x] I've linked any associated an issue.

Code Contributions

• [x] I've built the code cargo build --all-targets successfully.
• [x] I've run the unit tests cargo test --workspace and everything passes.

Steps taken/Steps to reproduce

According to the tutorial documentation quick-installation

• Download the latest Windows installer from release page.

• Double-click to run the installer

• Run

SET RUST_LOG=info
cd c:\Pyrsia\Pyrsia\service
pyrsia_node.exe

Then, Pyrsia_node stops and the output details are as follows:

C:\Pyrsia\Pyrsia\service>pyrsia_node
 2022-10-10T20:42:13.900Z INFO  pyrsia_node > Pyrsia Docker Node will start running on 127.0.0.1:7888
 2022-10-10T20:42:13.901Z INFO  pyrsia_node > Looking up bootstrap node
 2022-10-10T20:42:13.904Z INFO  pyrsia::network::event_loop > Local node is listening on "/ip4/127.0.0.1/tcp/55464/p2p/12D3KooWDhPM5KmLwHsnAYJ2mSdkiPCB4RAcT2RNhitYLJ5YDwME"
 2022-10-10T20:42:13.928Z INFO  pyrsia::network::event_loop > Local node is listening on "/ip4/10.6.252.115/tcp/55464/p2p/12D3KooWDhPM5KmLwHsnAYJ2mSdkiPCB4RAcT2RNhitYLJ5YDwME"
 2022-10-10T20:42:14.026Z INFO  pyrsia_node                 > Found bootstrap peer_addr "/ip4/104.198.178.107/tcp/44000/p2p/12D3KooWERQcvMrdmRc8a3oSuEDU2AmMZ974ADpisbbpFDrSZ8hx"
 2022-10-10T20:42:14.026Z INFO  pyrsia_node                 > Probing "/ip4/104.198.178.107/tcp/44000/p2p/12D3KooWERQcvMrdmRc8a3oSuEDU2AmMZ974ADpisbbpFDrSZ8hx"
Error: Anyhow Error

Caused by:
    Connection was closed before a response was received

Expected Behavior

Pyrsia_node works fine or reports CLI option errors and help messages.

Experienced Instance of the bug

OS: Windows 10 Pro Pyrsia version: 0.2.0-rc1

• refactors/moves the model (concern) of the blobs module into models folder

PR Checklist

• [x] I've built the code cargo build. For major changes, cargo build --workspace --release is recommended.
• [x] I've run the automated unit tests cargo test. This executes our automated unit tests.
• [x] I've not broken any existing tests or functionality. In addition to cargo test, I've run cargo clippy
• [x] I've not introduced any new known security vulnerabilities. I've run cargo audit

• [x] I've included a brief description and a good title of the proposed changes and how to test them.
• [x] I've read the contributing guidelines.
• [x] I've associated an issue with this Pull Request.
• [x] I've checked that the code is up-to-date with the pyrsia/main branch.
• [x] I've read "What is a Good PR?"
• [x] I've assigned this Pull Request to "pyrsia/collaborators"

Description

Starts to Fix pyrsia/pyrsia#307

This PR tests the append blob function with a simple string

• refactors/moves the storage (concern) of the blobs module into storage folder

Screenshots (optional)

Description

Fixes pyrsia/pyrsia#1464

This PR removes the hard exit on openssl, and removes the call to cargo remove (only available with nightly) and adds cargo tree when openssl is detected.

Screenshots (optional)

PR Checklist

• [x] I've read the contributing guidelines.
• [x] I've read "What is a Good PR?"
• [x] I've included a good title and brief description along with how to review them.
• [x] I've linked any associated an issue.

Details are as follows:

Run .github/workflows/build.sh
error: no such subcommand: `remove`

	Did you mean `report`?
Error: Process completed with exit code [10](https://github.com/pyrsia/pyrsia/actions/runs/3809065416/jobs/6480154412#step:7:11)1.

Link: https://github.com/pyrsia/pyrsia/actions/runs/3809065416/jobs/6480154412

Bumps byte-unit from 4.0.14 to 4.0.18.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Description

This PR fixes Local Code Coverage Run. Switching to test run specific user in docker container before tarpaulin installation to avoid rust registry cache write permission issue

To test run ./tests/test_local_code_coverage.sh from pyrsia project context root. On successful encounter you will be able to see something similar to below.

pyrsia_code_coverage  | || 
pyrsia_code_coverage  | 49.82% coverage, 1384/2778 lines covered
pyrsia_code_coverage exited with code 0
[+] Running 3/3
 ⠿ Container pyrsia_code_coverage                 Removed                                                                                                                                                                                                       3.5s
 ⠿ Image local_code_coverage-codecoverage:latest  Removed                                                                                                                                                                                                       0.0s
 ⠿ Network local_code_coverage_default            Removed 

Screenshots (optional)

PR Checklist

• [x] I've read the contributing guidelines.
• [x] I've read "What is a Good PR?"
• [x] I've included a good title and brief description along with how to review them.
• [x] I've linked any associated an issue.

Code Contributions

• [x] I've run pre-commit.sh (pre-commit.bat) successfully.
• [x] I've run the integrations tests on the PR branch and all tests passes.

Suggestion details

Adding test cases for CLIs.

As per my exploration here is the details.

[dev-dependencies]
assert_cmd = "2.0.5"
predicates = "2.1.1"

Sample test code. The test code runs the binary from the debug directory.

#[cfg(test)]
#[cfg(not(tarpaulin_include))]
mod tests {
    use crate::CONF_FILE_PATH_MSG_STARTER;
    use assert_cmd::Command;
    use predicates::prelude::*;

    #[tokio::test(flavor = "multi_thread")]
    async fn test_config_show_file_path_info_availability() {
        let mut cmd = Command::cargo_bin("pyrsia").unwrap();
        cmd.arg("config")
            .arg("--show")
            .assert()
            .success()
            .stdout(predicate::str::contains(CONF_FILE_PATH_MSG_STARTER));
    }
}

The code expects the binary will be present under target/debug directory. But tarpaulin is not going to build the binary under debug. Therefore a cargo build has to perform. In subsequent step while calling cargo tarpaulin, we need to preserve previously build debug dir. To achieve the same, add --skip-clean flag to cargo tarpaulin.

Please refer commits from

• https://github.com/pyrsia/pyrsia/pull/1245/commits/0c6903b6269d8133608ac930e4ddab31daebef7e
• https://github.com/pyrsia/pyrsia/pull/1245/commits/c870282ed43c0fbda7bbe04bd3fd51b4e54799a9

Disclaimer: Even if you will be able to run unit test but the coverage is not going to be included in code coverage report.

Additional details

No response

This test will only succeed after #1284 and #1446 have been resolved.

Consider this situation:

• nodeA (authorized)
• nodeB (authorized) bootstrapped to nodeA
• nodeC bootstrapped to nodeA
• nodeD bootstrapped to nodeB

Next:

• trigger a build from nodeA and wait for blocks to be published/broadcast
• verify the the logs for the new artifact are available on all nodes
• verify you can download an artifact on all nodes
• trigger a build from nodeB and wait for blocks to be published/broadcast
• verify the the logs for the new artifact are available on all nodes
• verify you can download an artifact on all nodes

Manual scenario to test this. Open 6 terminals,

Terminal X

go to the pyrsia build pipeline prototype and run

cargo run

Terminal A

build, set up and run node A

cargo build --workspace
mkdir -p /tmp/pyrsia-manual-tests/nodeA
mkdir -p /tmp/pyrsia-manual-tests/nodeB
mkdir -p /tmp/pyrsia-manual-tests/nodeC
mkdir -p /tmp/pyrsia-manual-tests/nodeD
cp target/debug/pyrsia_node /tmp/pyrsia-manual-tests/nodeA
cp target/debug/pyrsia_node /tmp/pyrsia-manual-tests/nodeB
cp target/debug/pyrsia_node /tmp/pyrsia-manual-tests/nodeC
cp target/debug/pyrsia_node /tmp/pyrsia-manual-tests/nodeD
cd /tmp/pyrsia-manual-tests/nodeA
RUST_LOG=pyrsia=debug DEV_MODE=on ./pyrsia_node --pipeline-service-endpoint http://localhost:8080  --listen-only -H 0.0.0.0 -p 7881 --init-blockchain

Terminal B

run node B:

cd /tmp/pyrsia-manual-tests/nodeB
RUST_LOG=debug ./pyrsia_node --bootstrap-url http://localhost:7881/status --pipeline-service-endpoint http://localhost:8080 -p 7882

Terminal C

run node C

cd /tmp/pyrsia-manual-tests/nodeC
RUST_LOG=debug ./pyrsia_node --bootstrap-url http://localhost:7881/status -p 7883

Terminal D

run node D, bootstrapping to node B

cd /tmp/pyrsia-manual-tests/nodeD
RUST_LOG=debug ./pyrsia_node --bootstrap-url http://localhost:7882/status -p 7884

Terminal Y

Back in the root of the pyrsia repo, let's use the Pyrsia CLI to run through our scenario. First set up the authorized nodes:

NODE_A_PEER_ID=`curl -s http://localhost:7881/status | jq -r .peer_id`
NODE_B_PEER_ID=`curl -s http://localhost:7882/status | jq -r .peer_id`
echo $NODE_A_PEER_ID
echo $NODE_B_PEER_ID
./target/debug/pyrsia config -e --port 7881
./target/debug/pyrsia authorize --peer $NODE_A_PEER_ID
./target/debug/pyrsia authorize --peer $NODE_B_PEER_ID

Trigger a build from node A:

./target/debug/pyrsia config -e --port 7881
./target/debug/pyrsia build docker --image alpine:3.16.0

Wait for the build to finish and check logs on all nodes:

./target/debug/pyrsia config -e --port 7881
./target/debug/pyrsia inspect-log docker --image alpine:3.16.0
./target/debug/pyrsia config -e --port 7882
./target/debug/pyrsia inspect-log docker --image alpine:3.16.0
./target/debug/pyrsia config -e --port 7883
./target/debug/pyrsia inspect-log docker --image alpine:3.16.0
./target/debug/pyrsia config -e --port 7884
./target/debug/pyrsia inspect-log docker --image alpine:3.16.0

All 4 inspect-log queries should return the transparency logs.

Next, check if the artifacts itself are accessible on all nodes:

curl http://0.0.0.0:7881/v2/library/alpine/manifests/3.16.0
curl http://0.0.0.0:7882/v2/library/alpine/manifests/3.16.0
curl http://0.0.0.0:7883/v2/library/alpine/manifests/3.16.0
curl http://0.0.0.0:7884/v2/library/alpine/manifests/3.16.0

All curl commands should output the manifest for the requested Docker image.

Then, check if a build by node B works in the same way:

./target/debug/pyrsia config -e --port 7882
./target/debug/pyrsia build docker --image alpine:3.16.1

Wait for the build to finish and check logs on all nodes:

./target/debug/pyrsia config -e --port 7881
./target/debug/pyrsia inspect-log docker --image alpine:3.16.1
./target/debug/pyrsia config -e --port 7882
./target/debug/pyrsia inspect-log docker --image alpine:3.16.1
./target/debug/pyrsia config -e --port 7883
./target/debug/pyrsia inspect-log docker --image alpine:3.16.1
./target/debug/pyrsia config -e --port 7884
./target/debug/pyrsia inspect-log docker --image alpine:3.16.1

All 4 inspect-log queries should return the transparency logs.

Next, check if the artifacts itself are accessible on all nodes:

curl http://0.0.0.0:7881/v2/library/alpine/manifests/3.16.1
curl http://0.0.0.0:7882/v2/library/alpine/manifests/3.16.1
curl http://0.0.0.0:7883/v2/library/alpine/manifests/3.16.1
curl http://0.0.0.0:7884/v2/library/alpine/manifests/3.16.1