Currently interledger-rs supports settling peer accounts whenever their balance goes over settle_threshold, settling for balance - settle_to and thus resetting the balance to settle_to. This PR adds a time-based method which works in addition to the existing threshold-based settlement. This feature allows driving the balance to settle_to later. The feature works with settle_threshold: if the balance goes over the threshold, a settlement is still fired off right away and any existing time-based settlement is cancelled.

The feature works by adding a delay on each prepare for every peering account id which did not already have a delay. The delay duration is set through the --settle_every configuration option. When the delay expires, a path similar to threshold settlement is taken to settle the balances, but now the balance's relation to the threshold is ignored.

The hot path of the prepare/fulfill processing in balance service is changed by addition of a non-blocking non-awaiting call to Sender::try_send, and on failing the send there is protection against flooding the logging by a non-awaited try to lock the mutex to log at most every minute. This guarding is because we don't anticipate running into issues with processing of the queue – the situtation should only arise when the node is badly overloaded, and in that case the logging should try not to make the bad situtation worse.

In balance_service there's additionally minor refactoring to share the settle_or_rollback for threshold and time based approaches which involved a few lines moved.

Note: In a cluster configuration where multiple nodes share a single database and database accounts, using this can result in many settlements.

Right now the ilp_over_btp_outgoing_token has to be in the form username:password while the ilp_over_btp_incoming_token has to be in the form password. We should change it so both values can be exactly the same because it's way too confusing otherwise.

We should also make sure the same applies to ilp_over_http

A part of #198

Resolves:

• #198 Make .mds testable
• #335 Make examples use CLI args vs env vars consistently
• Part of #230 (Docker images will be tagged)
• Part of #230 (Binaries are built and attached to a release)

• This PR enables run-md-test.sh to be run on CircleCI.
  • Also if the branch is master and all tests have passed, it updates docker images on DockerHub. Also, tagged commit would be pushed, being tagged with the git tag name.
• Because running docker command in containers requires docker-in-docker(dind), docker/run-md-test.dockerfile is based on circleci/buildpack-deps:latest-dind.
• Also, dind requires another job because it requires another image, so the job is defined as test-md.
• The setup_remote_docker command spins up a docker daemon in an outer machine. So we can't access it using curl localhost:xxxx which is used to test. In order to address this issue, ncat hack was used.
  • ncat -l -k -c "docker exec -i interledger-rs-node_a nc 127.0.0.1 7770" -p 7770 &
• interledgerrs/settlement-engine repo is deprecated, using interledgerrs/settlement-engines instead because the name looked wrong.

Questions:

• I'm wondering this is better or not. Using Cargo.toml doesn't seem to be efficient.
  • cargo-cache-2-{{ checksum "Cargo.lock" }}-{{ .Branch }} (changing Cargo.toml to Cargo.lock)
  • We have to remove Cargo.lock from .gitignore to do this.

The cli app will panic in ws_account_payments_incoming for several legitimate use cases:

• invalid URL
• invalid URL protocol/scheme
• ilp node not running

This extends the interpreter.rs module's Error enum, allowing the ? to gracefully report failures to the end user. Is this the style you prefer for error handling?

A few other notes:

• Refactored a nested match statement for readability, drawn from this clap example. The behavior should be the same.
• Is there a more descriptive name for ClientErr, say, RequestErr?
• Can socket.read_message().expect("Could not receive WebSocket message") occur in normal use?
• Is there is a reasonable way to spin up an ilp node for testing runtime failures? There doesn't seem to be anything that actually exercises NodeClient::ws_account_payments_incoming().

• Fixes
  • #171 Use gumdrop or structopt for CLI option parsing
  • #113 Fix --version flag in CLI
  • #206 Allow env vars to override config file values
  • #194 Make settlement-engine to take environmental values
  • #215 Add documentation for interledger node configuration
• Version is obtained from crate's version
• ~~Using structopt to reduce the complexity of the source.~~
• ~~Using #[allow(clippy::large_enum_variant)] because options of account add command are too many. It may be difficult to fix this unless we make a patch to structopt~~
  • ~~There is someone doing the same~~

BREAKING CHANGES

• Ethereum settlement engine now doesn't accept key argument. Use private_key instead.
• ~~Ethereum settlement engine now doesn't accept watch_incoming flag because it was a flag and default_value was true, which doesn't make sense and structopt considers it as an error. Instead I implemented without_incoming_watcher flag which disables incoming watcher.~~
  • ~~structopt: help: message: default_value is meaningless for bool~~
• ~~node now doesn't accept ilp_address argument. Instead address argument or ILP_ADDRESS env var will be accepted.~~

I noticed that we made .mds runnable so we could even test it. It has a pro that if the documents go outdated, we will be able to notice that.

That said, it will require more efforts, so... not now but sometime in the future.

• [x] Replace numeric IDs in interledger-store-redis / the node with unique ids
• [x] Add usernames to accounts as uid aliases
• [x] Make SPSP payments work with usernames
• [x] Add an auth method to the HTTP API that includes the username (HTTP Basic Auth, separate header, or JWT)
• [x] Remove need for auth tokens to be globally unique
• [x] Remove ilp_address from the Account (or make it optional, in which case if it is present it would override the automatically generated address)
• [x] Handle ILDCP queries by appending a dot (.) and the username to the Node's ILP address
• [x] Merge IldcpAccount trait into base Account trait and rename client_address method to ilp_address
• [ ] Don't broadcast node's child accounts if their addresses start with the node's ILP address (if node example.a has an account for example.a.b, it should broadcast a route to example.a but not to example.a.b)

Based on https://github.com/emschwartz/interledger-rs/issues/126, https://github.com/emschwartz/interledger-rs/issues/147, and https://github.com/emschwartz/interledger-rs/issues/161

Right now, the Store is responsible for managing secret values like API keys (for authenticating with peers via HTTP or BTP). The Redis store uses encryption and hmac keys derived from a configured seed.

This approach isn't great because:

• It's not very secure (all of they key material is on the same machine)
• The seed needs to be passed in as an environment variable (is that even a good approach?)
• Cannot easily change the seed
• Every store implementation needs to re-implement similar crypto functionality or import some shared code (this issue was inspired by @cjimison asking about the right way to approach this for https://github.com/emschwartz/interledger-rs/issues/52)

Also, the keys should be included in the Account type in such a way that they are never logged. For example, they could have special types that implement Debug and Display but only print <redacted> or something like that.

How should we improve the key storage? If necessary, we can also revisit the use of API keys (rather than more complex auth) if there is some decent solution that's more secure but not impossible to use.

@wliu @emschwartz

This is not actually working. Just playing with and figuring out what to do. connect-to-xpring.sh automatically connects our local node to the Xpring test node.

./scripts/testnet/connect-to-xpring.sh -f -x -c -s -a -l

(too many options lol)

Questions and suggestions:

• http://test.xpring.tech/api/accounts/xxx should accept some parameters so that we don't need to add our account to the Xpring side. For example, our ILP address and our endpoint URL.
• http://test.xpring.tech/api/accounts/xxx should return their ILP address.
• Can nodes handle SSL connections now? I'm not sure.

1. Should the asset scale conversion be done on the connector or on the engine? If we do it on the engine, it'd mean that each engine would have to implement that conversion properly, while we could do it on the operator once.

2. How do we handle settlement amounts that are bigger than u64 in the store? Can occur in high granularity EVM transactions (Ether / ERC20 / any other Ethereum fork)

cc @emschwartz @kincaidoneil @sappenin

The Simple and Ethereum Settlement examples have slightly different styles and we should try to make them more consistent.

Some of the notable differences:

• The Simple example puts all of the commands in shell scripts while the Ethereum Settlement example puts the code in the README. The advantage of shell scripts is that they can be run directly, while the disadvantage is that it's less nice to read through them than reading the markdown README file. Having both means we need to make sure the code and README stay in sync, which is a bit annoying.
• The Ethereum example gets the account balances directly from Redis while the other uses the HTTP API. We should definitely try to avoid calling the DB directly and instead encourage the use of the API, however it's slightly nicer to print the balance in line with the comment (as is done here) than to print the JSON {"balance":"100"} on the line after the comment.
• The Ethereum example suggests opening 7 terminal windows to run the different services, while the scripts in the other example run the services as forked processes and write the logs to different files. Opening 7 terminal windows is pretty excessive so I'd lean towards only requiring one, though that does make it more difficult to watch different processes happen simultaneously.
• The examples made in the settlement README.md's are hard to read due to how the POST parameters are added. We should move these to a file / improve formatting What do you think? cc @gakonst @bstrie @dora-gt @pgarg-ripple

Hello @koivunej @KristianWahlroos @matdehaast @emschwartz @frnsskk

Just wondering what the maintenance status of this project would be ?

I have open issue here to detail analysis which came after our discussion around unicase: https://github.com/rustsec/advisory-db/issues/1303

There is also concern around username comparison here to update unicase dependency: https://github.com/interledger-rs/interledger-rs/pull/744

Unicase dependency problem - recommend bumping to 2.6.0: https://github.com/rustsec/advisory-db/pull/1176

Would you like us to flag informational around that this crate is deprecated by unmaintained status and it may be also feasible to yank the old version of interledger-service that uses the old version of unicase ?

This informational would gentle advice any downstream to be vary of using it - since I am not sure whether these published crates are supposed / intended to be used - but the decision would be yours as we generally don't file unmaintained advisories where the maintainer is reachable and either does not agree as to unmaintained status - we may file advisories where the project has been officially deprecated or confirmed by the maintainer(s) where it is feasible to advice any downstream users along as to the status.

Thanks for the assistance !

Looking at the existing examples in serde/json for example, there probably is currently no supported way around installing cargo-fuzz on a nightly and just cargo fuzz build the targets under each crates/interledger-*/fuzz.

Implementing this might be difficult for the caching, because these fuzz targets are not part of the workspace and have their own target/ directory, which might not work out of the box with the rust-cache action introduced in #732.

Originally posted by @koivunej in https://github.com/interledger-rs/interledger-rs/issues/732#issuecomment-894163497

This was noticed while adding the benchmarks in #663 that the redis_helpers.rs is copypasted all over. I do remember attempting to split it, the latest one being https://github.com/KristianWahlroos/interledger-rs/pull/2. Since #729 will leave some FIXME and todo! related to these files, maybe they should be refactored into a single place before fixing.

Marking this as good-first-issue because I seem to have made quite small commits which should be easy to follow/replicate.

Deprecated Xpring API had an endpoint https://xpring.io/api/accounts/<assetCode> to create accounts on Xpring's side. ilp-cli testnet setup command was using this endpoint to create an account and peer with it to connect the testnet easily.

• https://github.com/omrtozd/interledger-rs/blob/0fcaf5d0a9b34d52bde08250a5f52f5398f6c7ab/crates/ilp-cli/src/interpreter.rs#L326-L335

But since Xpring rebranded as RippleX, API endpoints and responses changed with it. For now, I find out three things that need to change, in order to ilp-cli testnet setup command work.

1. API endpoint, which changed from https://xpring.io/api/accounts/<assetCode> to https://ripplex.io/portal/ilp/hermes/accounts/. Also, you should send a POST request instead of a GET request.

2. The method to change the account asset. You should send a POST request to https://ripplex.io/portal/ilp/hermes/accounts/ with a JSON payload, an object named assetCode in it to use an asset other than XRP. You can see the old method here: https://github.com/omrtozd/interledger-rs/blob/master/docs/manual-config.md#get-your-own-credentials

3. The XpringResponse struct, due to the change in the response format.

• XpringResponse Struct: https://github.com/omrtozd/interledger-rs/blob/0fcaf5d0a9b34d52bde08250a5f52f5398f6c7ab/crates/ilp-cli/src/interpreter.rs#L394-L403

• Example response:

"accountId": "user_w0vzjgqc",
"accountRelationship": "CHILD",
"assetCode": "XRP",
"assetScale": 9,
"maximumPacketAmount": null,
"linkType": {},
"connectionInitiator": true,
"isInternal": false,
"sendRoutes": true,
"receiveRoutes": false,
"balanceSettings": {
    "minBalance": null,
    "settleThreshold": null,
    "settleTo": 0
},
"rateLimitSettings": {
    "maxPacketsPerSecond": null
},
"settlementEngineDetails": null,
"customSettings": {
    "ilpOverHttp.outgoing.url": "https://rxprod.wc.wallet.ripplex.io./ilp",
    "ilpOverHttp.incoming.auth_type": "SIMPLE",
    "ilpOverHttp.incoming.simple.auth_token": "ygV47wq8mvI0b",
    "ilpOverHttp.outgoing.simple.auth_token": "enc:gcpkms:connector:secret0:1:gs:CiQAxw4QhykCtBPYDxd7n91yo3FGmiIGbDQE8bN1c6Ft_4tYvRUSWACjdIaO5ZBq-W9sN7OiSr3o9Hb8uQBv68QTmd61EMCXMHuEa6uxrJZNM2ZHwkvhMEPgBfTH13EGUGkeSQfLMVlp9ZXyluiP-m0t1OJbDfUAE4qv0LBIhRY=",
    "ilpOverHttp.outgoing.auth_type": "SIMPLE"
},
"parentAccount": false,
"childAccount": true,
"peerAccount": false,
"peerOrParentAccount": false,
"paymentPointer": "$rxprod.wc.wallet.ripplex.io./user_w0vzjgqc"

These should probably all be just error! outside tests and binary packages.

Originally posted by @koivunej in https://github.com/interledger-rs/interledger-rs/pull/723#discussion_r636770511