Sled seems not mature enough and development has slowed down. In the beginning I thought they might get to a stable release before MiniMint, but that's probably too optimistic.

I'd like to stick to dumb key-value stores with transactions, prefix-fetching and deterministic ordering because there seems to be less room for accidental consensus-incompatibility that way (e.g. complicated SQLite queries would be hard to analyze for being deterministic on all platforms). I'd also like the database to be embedded to reduce complexity (currently, if there's a DB error it's safe to assume something went horribly wrong, like the underlying drive unmounting, and we can shut down). That leaves a few options off the top of my head:

• LevelDB: my current favorite, used in Bitcoin Core, so already a dependency
• SQLite as a simple KV store: probably also a great choice, feels hacky though
• Berkeley DB: used in Bitcoin Core, but has caused problems historically afaik

I did not deeply evaluate any of these options and would be happy to hear opinions on these or other KV stores.

EDIT 1: I confused LMDB with LevelDB, fixed

Integration tests should be able to run concurrently.

This PR attempted a fix by using randomly generated ports, but I think it would be better to have an option to mock out network APIs so we don't need to worry about port conflicts at all.

Figured I'd make an issue because this is a recurring problem that needs to be fixed.

Starting the gateway CLN plugin runs the gateway. On run, the gateway attempts to register with a federation. If the federation is not ready, the gateway registration call fails, which in turn would fail the CLN plugin start.

The recent change to encrypt configs at rest (#808) marginally slows down the federation start, which then leads into the situation described above. As a temporary fix, this PR tweaks our sleep timers in the shell scripts we use to orchestrate federation and gateway start by:

• delaying CLN1 (with plugin) and CLN2 by 5s each to ensure the federation is started
• delaying channel opens by 10s to ensure the CLN instances are up and running

Follow Ups:

• [ ] We should look to complete work proposed by #699, such that the gateway does not need a federation configuration at boot, but can register federations later, at runtime. Alternatively
• [x] Alternative solution: We should use retries for the gateway registration action, instead of relying on timing to sequence the boot process [WIP] - #840

I am currently re-implementing #55 and had issues with tide not supporting http 1.0 (tides dependency respectively) which is needed for the json-rpcclient. Now I am wondering if we should migrate away from tide and use a different web-framework. Since I'm new to web-dev I really don't know how to choose one over another so a discussion about that would be great. My only criteria would be that it's lightweight (?) @elsirion said that somebody already suggested axum.

Also since I am re-implementing the API anyway this would be a chance to discuss general decisions and the API standard to choose. I wanted to go for JSON-2.0 RPC but feel free to discuss here if something else would be better (or multiple APIs make sense). Feel free to comment your thoughts (not only from a dev decision perspective but maybe also a API consumer (e.g app @justinmoon @Maan2003 )

cargo audit is very usefull (install it with cargo install cargo-audit) we should also add it to the CI on push (if .toml) changes but now fedimint is not regarded usable anyway so this would just slow down development

testing CI

• removed --all-features so we can use stable (avoiding the ''diagnostics` flag): should I now list all other flags manually ? we didn't run the tests with any flags but maybe we should ?
• can add a commit that removes udeps as well if wanted

analogous to the mint-client-cli

will be needed for testing the following endpoints (e.g peg_in) because clientd and mint-client-cli can never run simultaneously

It's great that we support setting up a dev environment in 2 steps: install nix and run tmuxinator.sh.

But the problem now it that it takes like 10+ minutes for tmuxinator.sh to run. Has to compile bitcoin core and lightningd. Is there any way we could just download binaries for these? Of course might want to build from source for deployments, but for development it's a huge friction point for prospective contributors.

This commit reformulates database key prefixes such that they are grouped into enums.

Currently, key prefixes are listed as separate const values. This formulation makes it difficult to manage key prefixes (import, iterate, test, etc), particularly when it comes to dbdump. This commit simplifies key prefix management using enums and a key prefix trait. It also makes it possible to test and ensure key prefixes are locally unique.

TODO:

• Fix the lifetimes for the raw prefix search
• Change the API and Database trait to support self referencing transactions
• Use native rocksdb column families instead of prefixing (e.g. collecting signature shares for a given token and deleting the column later)
• Perhaps create inherited traits for basic database -> transactional database -> transactional + column family database
• Crate generic trait for expected "transaction" behaviors and functionality
• Get rid of sled_impl and replace all references
• Remove Batch.rs
• Improve UT and add coverage for all expected trait functionality Note: transactions are soon to be merged into rust-rocksdb we are simply waiting on a couple changes for the next rocksdb you can track the issue here https://github.com/rust-rocksdb/rust-rocksdb/pull/565/files#. Obviously we will want to change to master as soon as possible.

The integration test should wait for the LN Gateway to actually receive the e-cash tokens for its LN payment. This probably requires creating a new (authenticated) API endpoint for LN Gateway operators to query its balance.

Two times in the last few days I've been confused about how to put an amount into our CLIs.

Yesterday:

$ gateway-cli withdraw rcklss 123456 <address>
error: Invalid value '123456' for '<AMOUNT>': invalid number format

The solution was to enter the amount field as "1203381 sats" in quotes.

Today:

$ fedimint-cli -- --workdir <workdir> info          
{
  "total_amount": 21000000,
  "total_num_notes": 10,
  "details": {
    ...
  }
}

$ fedimint-cli -- --workdir <workdir> spend 21000000
error: The following required arguments were not provided:
  <AMOUNT>

Usage: fedimint-cli --workdir <WORKDIR> spend <AMOUNT>

For more information try '--help'

The solution was to convert the 21000000 msat to 21000 sat value.

What to do?

Previously I had suggested that we try to use sats for all cli units because I thought it would be easier for users to work with. But it's a little problematic because that's not how we actually represent amounts internally.

What should be our plan for representing amounts in CLIs going forward?

The first warning should probably be logged via tracing and the second one should probably be a of level debug.

[root@hcpp:/var/lib/fedimint]# distributedgen run … --password password
WARNING: Passing in a password from the command line may be less secure!
2022-12-30T11:18:36.243536Z  WARN fedimintd::encrypt: READ "./tls-pk"

As inspired by https://github.com/jonasnick/bips/issues/32 and the fact that I need tke mint_pub_key in PublicKey but don't know the parity with which I can type convert it

Get rid of some hardcodings listing federation modules deep inside generic code, and move toward edges (final binaries), so it's easier to customize/inject new modules.

There's probably a lot more work to do around it, but it's a step towards the goal, I think.

See https://github.com/rust-bitcoin/rust-bitcoin/pull/1509#issue-1512287084 . We use actions-rs just in a handful of places, but still we should do the same.

We are going to need a way for the guardians to do things remotely via their app.

For social recovery the guardian needs to inform the fedimintd to create and publish decryption share, ideally from the conveniance of their app.

In the future we might want an option for the fedimintd to start in a locked mode and require remote config decryption (so we don't have to store any passphrase locally at all).

Other modules might require their own guardian authentication.