I upgraded icx-proxy from e3866f0040d73cda4c5d4092b85af517b11b833f to 7624053a50e1683c48757ee5dacbb454921e39ef and now the initial call to /api/v2/status returns the HTML page of my asset canister, and a CBOR decoder tries to decode that and throws an error.

I downgraded back to e3866f0040d73cda4c5d4092b85af517b11b833f API calls were forwarded again.

Is there some new flag I need to pass to get the old behavior? I couldn't see anything obvious in the help text.

Hi there! I'm getting the below error when trying to run cargo install icx-proxy.

   Compiling ic-agent v0.9.0
   Compiling ic-utils v0.7.0
error[E0277]: the trait bound `ic_agent::export::Principal: CandidType` is not satisfied
   --> /Users/phuongvu/.cargo/registry/src/github.com-1ecc6299db9ec823/ic-utils-0.7.0/src/interfaces/management_canister/builders.rs:251:42
    |
251 | impl<'agent, 'canister: 'agent, T: Sync> AsyncCall<(Principal,)>
    |                                          ^^^^^^^^^^^^^^^^^^^^^^^ the trait `CandidType` is not implemented for `ic_agent::export::Principal`
    |
    = note: required because of the requirements on the impl of `for<'de> ArgumentDecoder<'de>` for `(ic_agent::export::Principal,)`
note: required by a bound in `AsyncCall`
   --> /Users/phuongvu/.cargo/registry/src/github.com-1ecc6299db9ec823/ic-utils-0.7.0/src/call.rs:38:10
    |
36  | pub trait AsyncCall<Out>
    |           --------- required by a bound in this
37  | where
38  |     Out: for<'de> ArgumentDecoder<'de> + Send,
    |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `AsyncCall`

For more information about this error, try `rustc --explain E0277`.
error: could not compile `ic-utils` due to previous error
warning: build failed, waiting for other jobs to finish...
error: failed to compile `icx-proxy v0.7.0`, intermediate artifacts can be found at `/var/folders/zj/f8bhtxm522vd_js96_y2w66c0000gn/T/cargo-installauI5dz`

This applies changes from https://github.com/dfinity/agent-rs/pull/195 that are specific to icx-proxy.

It necessarily depends on a newer version of ~agent-rs~ ic-agent and ic-utils. This will most likely be something like 10.X.0 but temporarily uses a Git URL for now.

This change creates a Validate trait and then uses it to wrap the existing validation logic. We then wrap our Validator struct using WithMetrics to expose whether requests passed/failed/skipped validation.

Some of the macos release tarballs contain a corrupt executable, in which the first 0x800000 bytes are all zeros.

For example, the macos release binary for commit https://github.com/dfinity/icx-proxy/commit/59674697309548b7021d457d95c32057e7681bf6 is corrupt:

$ wget https://github.com/dfinity/icx-proxy/releases/download/5967469/binaries-macos.tar.gz
$ tar -xzf binaries-macos.tar.gz 
$ ./icx-proxy --help
zsh: exec format error: ./icx-proxy
$ hexdump -C icx-proxy | head
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00800000  40 69 6e 00 01 00 00 00  62 00 00 00 00 00 00 00  |@in.....b.......|
00800010  74 01 00 00 09 00 00 00  c0 6a 6e 00 01 00 00 00  |t........jn.....|
00800020  0c 00 00 00 00 00 00 00  40 69 6e 00 01 00 00 00  |........@in.....|
00800030  62 00 00 00 00 00 00 00  ff 00 00 00 09 00 00 00  |b...............|
00800040  cc 6a 6e 00 01 00 00 00  0d 00 00 00 00 00 00 00  |.jn.............|
00800050  40 69 6e 00 01 00 00 00  62 00 00 00 00 00 00 00  |@in.....b.......|
00800060  0f 01 00 00 09 00 00 00  fe 6a 6e 00 01 00 00 00  |.........jn.....|
00800070  0e 00 00 00 00 00 00 00  40 69 6e 00 01 00 00 00  |........@in.....|

I have not found a way to ensure that the macos release tarball contains a valid binary. I tried:

• Adding retry logic to the workflow
• Splitting up the tar step from the gzip step

This PR makes two changes:

• Verify that the binary in the tarball can execute, or fail the CI run.
• Create all release artifacts before uploading any of them. An alternative would have been to upload whichever worked, which would then be overwritten on retry.

When we print log strings by taking slices, we must be mindful about utf8 code points. Otherwise it will crash the program:

thread 'tokio-runtime-worker' panicked at 'byte index 100 is not a char boundary; 

The fix is to take the slice before converting to utf8 (and use from_utf8_lossy).

That is missing part of body validation preprocessor. Without this, the verification of certificates for the assets-canister does not work (). Check the service-worker realization for details

Bumps axum-core from 0.2.7 to 0.2.8.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

This is a massive refactor which will enable us to collect better logs.

• Moves code out of main.rs
  • agent proxy code is now in proxy/agent.rs
  • forwarding proxy code is now in proxy/forward.rs
  • temporary src/http_transport.rs pending on dfinity/agent-rs#373
• Adds a new log format switch with json support. The exact format may need some fine tuning
• Completely gets rid of reqwest

Spawning a task has some slightly different async properties compared to the stream api, and in this case, we should prefer the stream api.

This allows us to buffer a preset amount and ties the cancellation of the downstream request more closely to the upstream request.

This change introduces the usage of http_body::Limited to ensure we don't allow incoming requests to exceed a certain size. We should consider using Tower going forward so that we can construct this using middlewares and not have all the logic live in one large function.

If some asset is uploaded to the asset canister under name api/xyz, then requesting it through icx-proxy will result in 400 error, which happens on both local dfx deployment and main net.

This is because icx-proxy hard coded path api to redirect to replica in these lines https://github.com/dfinity/icx-proxy/blob/4f732947846f3314e73db185e2932ad5a05c37b3/src/proxy/mod.rs#L157-L158

Given that /api is such a common prefix, shall we change it to something like /_/api ? Or at least give a better error message with some description of what could be going on?

That is a part of proposal about chunks certification (with backward compatibility).

These improvements are a proposal to improve the certification infrastructure around IC and might be considered as a recommendation for dfinity-team.

Goal

Make it possible to certify asset chunks. Validate chunk certificates on the service-worker and icx-proxy.

Why

At the moment, the service-worker and icx-proxy does not support the certification of chunkified files. Moreover, right now it is not possible to correctly stream chunkified and large audio and video files to the front-end. This problems could be solved independently if it would be possible to install an additional service-worker in the certified zone of the domain ic0.app (for 206 partial http-request handling). But is is impossible because there is unable to place custom worker on ic0.app domain. Making your own custom player for audio and video is extremely difficult due to the large number of formats and non-native implementation.

Details

To make this possible, support for HTTP-range requests for http_request query method has been added. This is done to support native html audio/video element (which uses 206 partial http-request) and to determine the index of the chunk throught 206 partial http-request. Using 206 partial http-requests allows you to focus only on certification in the worker and icx-proxy.

Steps

1. It all starts with PR for certified-assets-canister in cdk-rs
2. Did file was updated in PR for certified-assets-canister
3. Service-worker started supporting chunk_tree certificate verification in PR for ic
4. (Here) icx-proxy started supporting chunk_tree certificate verification here
5. Added support for new certified-assets-canister did in PR for agent-rs

When dfx starts a local replica, it automatically starts icx-proxy to reroute the incoming requests. It uses hyper for http logic, and hyper doesn't properly support http1/http2 dual stack. The main ic repo assumes http2 in a lot of places, one consequence of that is ic-rosetta-api being unable to connect to the local replica started by dfx. This patch fixes that problem.

See ticket https://dfinity.atlassian.net/browse/ROSETTA1-162 for more context.