This issue tracks the discussion around creating an abstraction over the data that librustzcash needs to use. The goal is to define an API for this and then implement it.

As a wallet developer, I want to have a standard way to interact with the librustzcash library that gives me flexibility in how I interact with and store data so that librustzcash does not manage state.

Criteria:

1. Librustzcash data access API is documented
2. There exists at least one implementation of the data access API
3. The data access API is general enough to operate with iOS, Android, WASM, Javascript, and Desktop.
4. We manage as little data and state as possible
5. We account for reorgs and store enough state to allow for rollbacks up to 100 blocks

Data that probably needs to be managed:

• notes
• commitment trees
• sapling trees
• nullifiers

Diagram:

                     .====================================.             
                     |      Wallet SDK Implementation     |
                     .====================================.
                     |     +---------+--------+           |
                     |     | Networking Layer |<-.        |
                     |     +------------------+  |        |
+-------+------+     |    +----+----+            |        |
|              +--------> |  Data   |            |        |
| librustzcash |     |    |  Access |     +------+-----+  |  +------------+    
|              |<---------+  API    |<----| Wallet API |<--->|  Developer |
+--------------+     |    +----+----+     +------+-----+  |  +------------+    
                     +------------------------------------+

Diagram Legend:

Wallet SDK Implementation : Android SDK, iOS SDK, zecwallet-light, etc. that simply implement the glue that connects everything together in order to expose high level APIs for shielded wallet developers to leverage. These implementations should be as lightweight as possible, allowing everything complex to live within the robust core librustzcash code. networking layer : the component responsible for fetching compact blocks librustzcash : Rust crates like zcash_client_backend which contain the core logic needed for interacting with Zcash Data Access API : interface through which librustzcash and the Wallet API communicate Wallet API : high level functionality exposed to wallet developers such as getBalance and getAddress Developer : you (probably)

Background

One of the reasons we did not originally design it this way is that taking a functional approach across a language boundary (JNI) is expensive in terms of object allocation (memory), processing (cpu) and time. These are all the resources we want to conserve on mobile! This optimization may have been premature and now that we have a working product, we can create a more functional approach and benchmark its performance. Getting it right will provide a huge amount of benefits in usability and testability.

• Updates pairing to match its current master
• Pulls in the following PRs:
  • https://github.com/zkcrypto/pairing/pull/91
  • https://github.com/zkcrypto/bellman/pull/25
• Migrates remaining crates to use ff crate
• Pulls in the ff and group crates into the workspace.

The zcash_client_sqlite data DB needs to be updated to account for UAs. Specifically:

• We should have an addresses table for storing the derived diversified UAs within the known account(s).
• The accounts table should have its address-related columns removed (migrating the Sapling and transparent content into the addresses table), and a transparent_fvk column added.
  • The UFVK can then be derived from the component columns, and e.g. if a wallet doesn't want transparent support, they'd represent this by leaving the transparent_fvk column set to NULL.
  • The alternative would be to just have a ufvk column that stores the encoded UFVK, and parse it from there, but storing the individual components is probably more introspectable.
  • This migration requires the caller to provide access to the UFVKs.

Motivation

Zebra wants to use zcash_proofs to download both Sprout and Sapling parameters.

Changes

Add sprout:

• Add a download_sprout_parameters function.
• Stream sprout and sapling downloads to disk, rather than loading the whole file into RAM.

Functionality changes:

• If the parameter files already exist, check them instead of starting a download.
• Allow the caller to specify a download timeout.
• Check file sizes as well as file hashes (useful for debugging).
• Remove downloaded files on error (but leave existing files alone) .

Interface changes:

• Add download_sapling_parameters and deprecate download_parameters. This avoids confusion between sprout and sapling downloads, while maintaining backward compatibility.
• Make the Sprout and Sapling parameter file names public. This makes it easier for callers to supply the correct paths to load_parameters.

Bug fixes:

• Add a missing feature dependency: download_params -> directories. This makes sure download_parameters will work if it's the only feature specified.

Tweaks:

• Refactor file loads to use the same verifying function as downloads.

ZIP 212 alters the note plaintext to store a seed from which rcm is derived, rather than storing rcm directly. In the mobile SDKs we only need rcm, so for post-ZIP 212 notes, we derive rcm from the seed and store rcm in the data DB.

However, when selecting notes to spend, create_to_address was using the transaction's target height to determine if Canopy is active, and parsing the rcm value as the seed if so. This effectively applied a seed->rcm derivation to all selected notes' rcms once Canopy activated on the chain. As a result, the note commitments were incorrect, and thus the anchors derived from the witness paths were also incorrect. This caused two kinds of observed failures:

• If more than one note was selected, the builder would fail with "anchor mismatch", as the note commitments would be effectively randomised, causing the derived anchors to also randomise.
• If a single note was selected, the transaction would be built using the randomised anchor, and then rejected when sent to the network.

The fix is to "pretend" in create_to_address that all notes are pre-ZIP 212 notes. This works fine because we never need to serialize back to the note plaintext while spending a note.

As a mobile native app developer, I'd like to leverage the exposed functionality of librustzcash in order to support transparent transactions.

Criteria:

• [x] the SDK supports sending from t -> z
• [x] the SDK supports sending t->t
• [x] the SDK supports deriving t-addresses from the same seed or keys used for shielded transactions
• [x] the SDK supports processing transactions with transparent inputs or outputs from lightwalletd

Open Questions:

• How will SDK users create t-addresses?
  • Will key derivation happen inside librustzcash with something like secp256k1
  • Or will we have to use an external 3rd party implementation, similar to how zecwallet-cli does things?
  • This looks like it will also involve creating a public key from the secret key and then hashing it and doing a base58 check
• How will the SDK and librustzcash coordinate with UTXOs?
  • The SDK will probably download all UTXOs from lightwalletd
  • librustzcash will need access to the UTXOs that have been downloaded
  • this might be a great test case for the data access API
• How will the SDK construct transactions that have transparent inputs?
  • Option 1: create a different function
  • Option 2: add parameters to the existing function
    • 2a: add utxos parameter that require knowledge of specific UTXOs
    • 2b: add transparentValue parameter that does not require knowledge of specific UTXOs
• How will the SDK handle transparent transaction information downloaded from lightwalletd?
  • It seems that we'd need to parse the bytes using zcash_primitives, then process & store the results with the sqlite crate
  • Shielded outputs can mostly be processed using decrypt_and_store_transaction but there are some potential bugs like "is_change" and "nf" being set to null which violates the DB schema which defines them as "not null"
  • We don't have great support for processing transparent outputs, particularly in mixed transactions
  • For a given transaction we need to:
    • iterate over all inputs to check whether this transaction spends any of our UTXOs
    • iterate over all outputs to collect all that match any of our t-addrs
    • decrypt all shielded outputs and find and process those that match our wallet (including TXs that we've sent)

Note: This might be a good test case for integrating with the data access API. In fact, I'd even suggest that this feature not be implemented without it in place.

Previously we were retrieving nullifiers for notes that were not marked spent, but not checking against whether the spending transaction had been included in a block.

My note-spending-v7 branch is pretty stable at this point, and now being depended on more widely (after the ECC wallet was recently open-sourced). I rebased my branch on current master to fix merge conflicts (due to recent refactors), but it's otherwise unaltered.

Closes #71.

according to emerging spec here: https://github.com/zcash/zips/pull/220

Not sure it belongs here or to the separate lib!

The main design goals of this mmr implementation were

1. Avoid database callbacks. As it is implemented, calling side must just smartly pre-load MMR nodes from the database (about log2(tree length) for append, twice as much for deletion).

2. Reuse as much code/logic between rust and c++ clients.

3. Close to zero memory consumption.

This PR was originally submitted here: https://github.com/zcash-hackworks/sapling-crypto/pull/101

This pulls in a bunch of tests QED-it wrote during their audit that were spread across many branches in their sapling-crypto-internal repository. I've left their history intact so the best way to review it is to just look at the final diff, except for...

• This commit, which includes changes outside of tests, and
• This commit, which is my attempt to fix the broken linear relation testing in that commit (the check in that commit effectively just checks that p1 != p2 again, not what it was intended to do).
• I deleted a find_group_hash test since it didn't look like it was doing anything useful, and I don't want merging to block on me figuring out how to implement that test well. Raise a separate issue and assign me if you would like some find_group_hash unit tests!

I did not bring in this change to the circuit code, because I don't understand it. I also did not bring in a bunch of changes to comments and variable renamings in the aurel_comments branch and the sum_bug branch, but I did bring in the new tests from those branches.

The implementation of io:Read for ResponseLazyReader used to return only one byte regardless of the size of the provided buf, which degraded the performance of loading the response.

This change makes use of the provided buf. On my machine with Rust 1.64, the downloading speed went up from ~100 KiB/s to ~2 MiB/s. However, as before this change, the process still uses 100% of a single CPU thread when downloading the response. The most likely reason is that we still manually read each byte of the response. I can download the same data with ~ 50 MiB/s on my machine using a different HTTP client.

This fix seems to be sufficient for now.

Failed to write block metadata to FsBlockDb: Db(SqliteFailure(Error { code: Unknown, extended_code: 1 }, Some(\"no such table: compactblocks_meta\")))"

when writing a block to the FsBlockDb the database file exists at the cache directory but its size is zero bytes. It's not being initialized with the corresponding table even though the documentation implies so

impl FsBlockDb {
    /// Creates a filesystem-backed block store at the given path.
    ///
    /// This will construct or open a SQLite database at the path
    /// `<fsblockdb_root>/blockmeta.sqlite` and will ensure that a directory exists at
    /// `<fsblockdb_root>/blocks` where this block store will expect to find serialized block
    /// files as described for [`FsBlockDb`].
    pub fn for_path<P: AsRef<Path>>(fsblockdb_root: P) -> Result<Self, FsBlockDbError> {

In order to support note encryption for zsa, we suggest extending the current zcahsh_note_encryption implementation. Currently, the COMPACT_NOTE_SIZE is a constant, however we need to support variable note sizes to include the AssetId field for zsa notes.

Currently, in zcash_note_encryption:

/// The size of a compact note.
pub const COMPACT_NOTE_SIZE: usize = 1 + // version
    11 + // diversifier
    8  + // value
    32; // rseed (or rcm prior to ZIP 212)
/// The size of [`NotePlaintextBytes`].
pub const NOTE_PLAINTEXT_SIZE: usize = COMPACT_NOTE_SIZE + 512;

and

pub const ENC_CIPHERTEXT_SIZE: usize = NOTE_PLAINTEXT_SIZE + AEAD_TAG_SIZE;

We suggest moving the constants into the specific implementation (impl Domain for OrchardDomain and Spling) of the Domain trait by adding abstract types to NotePlaintextBytes, NoteCiphertextBytes, CompactNotePlaintextBytes, CompactNoteCiphertextBytes.

We get

pub trait Domain {
    type EphemeralSecretKey: ConstantTimeEq;
    type EphemeralPublicKey;
    type PreparedEphemeralPublicKey;
    type SharedSecret;
    type SymmetricKey: AsRef<[u8]>;
    type Note;
    type Recipient;
    type DiversifiedTransmissionKey;
    type IncomingViewingKey;
    type OutgoingViewingKey;
    type ValueCommitment;
    type ExtractedCommitment;
    type ExtractedCommitmentBytes: Eq + for<'a> From<&'a Self::ExtractedCommitment>;
    type Memo;

    // Types for variable note size handling:
    type NotePlaintextBytes: AsMut<[u8]> + for<'a> From<&'a [u8]>;
    type NoteCiphertextBytes: AsRef<[u8]> + for<'a> From<&'a [u8]>;
    type CompactNotePlaintextBytes: AsMut<[u8]> + for<'a> From<&'a [u8]>;
    type CompactNoteCiphertextBytes: AsRef<[u8]>;

Also, the constant will be removed from functions' signatures since they are unknown during the compilation time. For example:

pub fn try_note_decryption<D: Domain, Output: ShieldedOutput<D, ENC_CIPHERTEXT_SIZE>>(

Will be replaced with simply

pub fn try_note_decryption<D: Domain, Output: ShieldedOutput<D>>(

We provided our initial implementation to be complemented by the appropriate changes in Orchard::note_encryption.rs. Currently can be seen here for v2 notes https://github.com/QED-it/orchard/blob/8f02a5fdad2c1750ec5f2d372d6f9bb56346114e/src/note_encryption.rs#L197.

The changes will allow us to implement an Orchard::Domain for V3 notes while keeping compatibility with the existing Orchard Domain ( for V2 notes ) and Sapling.

Bumps actions/cache from 3.0.11 to 3.2.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

We need to implement a comprehensive set of errors for the FFI layer.

  Err(e) => Err(format_err!("Error while scanning blocks: {}", e)),
     |                                                                          ^ `FsBlockDbError` cannot be formatted with the default formatter
     |
     = help: the trait `std::fmt::Display` is not implemented for `FsBlockDbError`