I also bumped to the latest version of cryptoxide. There's been a lot of performance improvements that have been made since then and this would be a shame to not leverage this (though it's mostly for Curve25519).

This PR remove the temporary (useless?) serialisation of the Encode-able object. That way we now have a memory overhead allocation to do the hash.

• [ ] there is no way this can fail minicbor::Encoder have only the Write error. In the case of Blake2b being the writeable object, there is not error possible (none we can/should handle in the code. Shall we remove the Result<(), Box<dyn Error>> ?
• [ ] shall we have a pallas-crypto with the different crypto utils we will be using in our code?

In order to start planning a roadmap for the development of the storage layer, we need some due diligence on the state of existing Rust libraries to avoid overlap with other open-source development efforts.

In particular, we need to research:

• is there an existing Rust implementation of the ChainDB component?
• should Pallas be binary-compatible a the file system level with the Haskell implementation?

cc @mark-stopka

Opening this issue to collectively discuss alternatives to CBOR-handling libraries. I'll append my own findings from when I first evaluated minicbor and similar libraries. If a more suitable candidate is selected, we can work on the appropriate changes.

Fixes #211.

I've seen you already decided to fix the problem of ambiguous CBOR encodings with KeepRaw and keeping the initial bytes to be used for hashing etc.

But it is only a plus to make encodings here follow the ones in the haskell implementation, so I made the change.

I've looked on PlutusData encoding only. The differences to haskell implementation were:

• empty lists and lists of arguments to a constructor are encoded as a special case as 0-length definite length arrays
• maps are encoded as definite arrays (!)
• bytestrings over 64 bytes are encoded as indefinite bytestrings

I modified those 3 to match haskell. There is a new type PlutusBytes, same as Bytes but the encoding/decoding different. There is a test case plutus_data_isomorphic_decoding_encoding.

I'm not sure what is the policy for flattening fields for Pallas. Presumably based on the way blocks and txs are handled, the policy is to flatten stuff when it's shared between eras? Based on this I added some getters for the witness types

Cardano doesn't use a deterministic mechanism for CBOR encoding. The same block / tx data can be encoded in practically infinite variations, each one resulting in a different hash.

The process of decoding CBOR into a Rust struct involves loosing the information about these slight variations in encoding (eg: indefinite / definite arrays, int size, map orders, etc). So far, Pallas has been dealing with this problem by enriching the structures with extra data that allows to replicate the same variation when re-encoding a struct.

After dealing with several edge cases I thought I had won the battle. We actually managed to process the whole mainnet history without a mismatch. I felt a nice sense of accomplishment, I fought against the odds and survived...

... but I was wrong, CBOR had more tricks under the sleeve. A testnet Tx was found that had a mismatch in the resulting hash: https://github.com/txpipe/oura/issues/307

This time, the problem was that a Tx input tuple (hash, index) was encoded as a CBOR indefinite array. Why on earth would someone encode a fixed tuple as an indefinite array? I don't know, but it was clear that this is not a war that can be won.

It is now 100% clear to me that the only way to maintain consistency on the hash generation process is to retain the original CBOR data, something that @NicolasDP has been telling me since quite a while now.

The question is how do we accomplish this without an impact on memory.

In this PR I introduce a generic structure called KeepRaw<T> that wraps an inner CBOR-encodable structure while tracking the start / end positions that represent the segment of the original bytestream relevant to that particular inner struct:

use pallas_codec::utils::KeepRaw;

let a = (123u16, (456u16, 789u16), 123u16);
let data = minicbor::to_vec(a).unwrap();

let (_, keeper, _): (u16, KeepRaw<(u16, u16)>, u16) = minicbor::decode(&data).unwrap();

// this returns a &[u8] slice of the total bytestream
keeper.raw_cbor()

In this way, we can start wrapping ledger primitives, allowing us to retain the original CBOR and hash accordingly:

impl ToHash<32> for KeepRaw<'_, TransactionBody> {
    fn to_hash(&self) -> pallas_crypto::hash::Hash<32> {
        Hasher::<256>::hash(self.raw_cbor())
    }
}

I'm not ashamed to say that CBOR has won, it was an honourable fight.

The line here uses the type transaction_metadata: &KeyValuePairs<Metadatum, Metadatum> which doesn't match the Cardano binary spec. While Metadatum uses int, the spec has a special type for top-level keys called transaction_metadatum_label which uses uint instead.

Concretely, you can see in this transaction that the metadata representation used by Oura ends up being 82a21b12127f810d7dcee28264594554491907e53b08ef38d198e051b4a080 which gives the following cbor.me representation:

82                        # array(2)
   A2                     # map(2)
      1B 12127F810D7DCEE2 # unsigned(1302243434517352162)
      82                  # array(2)
         64               # text(4)
            59455449      # "YETI"
         19 07E5          # unsigned(2021)
      3B 08EF38D198E051B4 # negative(643795744601428404)
      A0                  # map(0)
   80                     # array(0)

As you can see, instead of the expected 17802948329108123211 key, we instead get -643795744601428404

You can see, for example, Cardano Multiplatform Lib uses the special label type here

closes #29

The goal is to have the following network-related crates:

• pallas-multiplexer
• pallas-miniprotocols

The argument against merging the two crates into a potential "pallas-network" is that it would be an artificial coupling not really required at the code level. Multiplexing is agnostic of the mini-protocols (even agnostic of the cbor encoding). Having it as different crates would allow easier repurposing of either crate.

Updates the requirements on cryptoxide to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

CBOR lists can be encoded in two ways: definite and indefinite length.

As seen for example here the encoding of PlutusData - here Constr concretely - uses indefinite length lists. I've seen you changed the implementation all over the place to use indefinite length lists in the commit 4f9fc1e40d94bfcbee532010738f2e50ad74970e. The comment says "we use indef array by default to match the approach used by the cardano-cli". So my question is

• In what scenario does cardano-cli do this encoding? Is it when we provide plutus data for redeemers/datums in the json format?

And my problem is

• haskell's Codec.Serialise.serialise (which I'd call "canonical" implementation right?) serialises plutus-data Constructor using definite length list. When PlutusData value gets serialized as a part of plutus script it results in a different script serialization and then the hash.

So it looks like either

• the haskell implementations already encode cbor inconsequentialy
• there's a default that luckily is used by every "official" (iog haskell) implementation

And my hope is that you were wrong with how cardano-cli encodes plutus data and we can revert the implementation to use definite length lists (which suits me).

While being a nice to have feature in general, this will also allow us to progress filtering by address payment or delegate part in Scrolls in a cleaner way.

Prefixes from CIP 5.

Updates the requirements on minicbor to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Updates the requirements on env_logger to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

function epoch_no() -> u64

function time() -> std::time::Instant

This code available in scrolls could be useful to write at least one of this method:

// TODO this is temporary, we should actually use this code from Pallas as this
// is very generic code

use pallas::ledger::traverse::MultiEraBlock;

fn post_byron_epoch_for_slot(shelley_known_slot: u64, shelley_epoch_length: u32, slot: u64) -> u64 {
    let last_byron_epoch_no = 208;

    let shelley_known_slot = shelley_known_slot as u64;
    let shelley_epoch_length = shelley_epoch_length as u64;

    let shelley_epoch_no = (slot - shelley_known_slot) / shelley_epoch_length;

    return last_byron_epoch_no + shelley_epoch_no;
}

fn byron_epoch_for_slot(byron_epoch_length: u32, byron_slot_length: u32, slot: u64) -> u64 {
    let byron_epoch_length = byron_epoch_length as u64;
    let byron_slot_length = byron_slot_length as u64;

    return slot / (byron_epoch_length / byron_slot_length);
}

pub fn block_epoch(chain: &super::ChainWellKnownInfo, block: &MultiEraBlock) -> u64 {
    let slot = block.slot();

    match block.era() {
        pallas::ledger::traverse::Era::Byron => {
            byron_epoch_for_slot(chain.byron_epoch_length, chain.byron_slot_length, slot)
        }   
        _ => post_byron_epoch_for_slot(chain.shelley_known_slot, chain.shelley_epoch_length, slot),
    }
}

Pallas aims to provide an implementation of each of the mini-protocols described in the Ouroboros network spec. We rely on the CDDL definitions to implement the CBOR wire-format required by each protocol.

Most of the wire-format is fully defined by the spec, except for certain messages which are specific to the ledger implementation. In particular, the "Local Tx-Submission" mini-protocol specification skips the definitions for the rejection reason message.

We've tried reverse-engineering the Haskell code to infer the CBOR structures, but it's a hard and error-prone process. On top of that, each cardano-node version might potentially change / augment these definitions.

We need to build / acquire a CDDL document that describes the missing structures of the "Local Tx-Submission" mini-protocol to facilitate the development of the corresponding Pallas implementation.

Pallas aims to provide an implementation of each of the mini-protocols described in the Ouroboros network spec. We rely on the CDDL definitions to implement the CBOR wire-format required by each protocol.

Most of the wire-format is fully defined by the spec, except for certain messages which are specific to the ledger implementation. In particular, the "Local State Query" mini-protocol specification skips the definitions for the query and result messages.

We've tried reverse-engineering the Haskell code to infer the CBOR structures, but it's a hard and error-prone process. On top of that, each cardano-node version might potentially change / augment these definitions.

We need to build / acquire a CDDL document that describes the missing structures of the "Local State Query" mini-protocol to facilitate the development of the corresponding Pallas implementation.