Neptune-core is the reference implementation for the Neptune protocol

Last update: Jul 10, 2023

Related tags

Overview

Neptune Core

Neptune-core is the reference implementation for the Neptune protocol. The implementation is not complete yet, but already supports many integral components. In particular, alpha-net is live.

Compiling from Source

Linux Debian/Ubuntu

Install curl: apt install curl
Install the rust compiler and accessories: curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
Source the rust environment: source $HOME/.cargo/env
Install LebelDB: apt install libleveldb-dev libsnappy-dev cmake
Download the repository: git clone https://github.com/Neptune-Crypto/neptune-core.git
Enter the repository: cd neptune-core
Build for release and put the binaries in your local path (~/.cargo/bin/): cargo install --path . (needs at least 3 GB of RAM)

Running & Connecting

Generate a wallet file: wallet_gen
Run neptune-core daemon: neptune-core with flags
- --peers [ip_address:port] to connect to a given peer, for instance --peers 51.15.139.238:9798 or --peers 139.162.193.206:9798 or both
- --mine to mine, and with --throttled-mining to avoid assigning too much CPU power to mining
- --help to get a list of available command-line arguments

Dashboard

This software comes with a dashboard that communicates with the daemon. The dashboard is a console-based user interface to generate addresses, receive and send money, and monitor the behavior of the client. The daemon must be running before the dashboard is started. To start the dashboard, run: neptune-dashboard. (If you set daemon's RPC port to a custom value specify that value with the flag --port [port].)

Command-Line Interface

In addition to a dashboard, the software comes with a CLI client to invoke procedures in the daemon. This can be invoked from another terminal window when the daemon is running. To get all available commands, execute

neptune-cli --help

To get e.g. the block height of a running daemon, execute

neptune-cli --server-addr block-height

If you set up neptune-core on a different address or port from the default (127.0.0.1:9799), then the flag --server-addr [ip_address:port] is your friend.

Setup for Development (Ubuntu)

build-essential (for make) -- apt install build-essential
install vscode
in vscode install the plugin rust-analyzer
in vscode activate format-on-save via File > Preferences > Settings then check the box for "Format on Save"
install cpulimit for nicer, and more quiet integration tests: apt install cpulimit

Logging

All logging is output to standard out.

The log level can be set through the environment variable RUST_LOG. Valid values are: trace, debug, info, warn, and error. The default value is info. E.g.: RUST_LOG=trace cargo run.

For development purposes it can sometimes be nice to get a more succint logging output by piping stdout through sed with the below command. This will only print the namespace of the logging event and the log text. The log output can also be stored to file by piping it to tee, like this: cargo run 2>&1 | tee -a integration_test.log.

sed 's/.*neptune_core:\+\(.*\)/\1/g'

If the RPC server is spamming your log too much, set the logging-level environment to RUST_LOG='info,tarpc=warn'.

Local Integration Test Strategy

This repository contains unit tests, but multi-threaded programs are notoriously hard to test. And the unit tests usually only cover narrow parts of the code within a single thread. When you are making changes to the code, you can run through the following checks

cargo b to verify that it builds without warnings
cargo t to verify that all unit tests work
run-multiple-instances.sh to spin up three nodes that are connected through localhost. Instance I0 and I2 should be mining and all three clients should be converging on the same blocks. You can read the hashes of the blocks in the log output and verify that they all store the same blocks.
Run make restart followed by run-multiple-instances.sh to verify that the nodes can start from the genesis block, create a database and store subsequent blocks in this database. This test is important to verify that the client software doesn't need an existing database to function.
If you encounter an error in some of the stages later then (2), i.e. an error that wasn't caught by the compiler or the tests, consider if you could add a unit test that would have caught this error. If that's not possible consider if you can add a manual test (for example a shell script) where the tests would have been visible. Also consider if you can add anything to this list that would have caught this error (assuming you didn't write a unit test that caught it).

Crash Procedures

If any cryptographic data ends up in an invalid state, and the note crashes as a result, please back up your entire data directory (except wallet.dat) and share it publicly. If you're not on main net, which hasn't been released yet, it should be OK to share wallet.dat, which contains your secret key, as well.

Comments

Reconnect to addresses in `peers` CLI list, if connection to any of those peers is lost

If a connection to a peer in the peers CLI list is lost, reconnect through a scheduled task from main_loop. You can reuse the scheduled task peer_discovery_handler for this. But maybe it should be renamed?

opened by Sword-Smith 0

Node crashes on synchronization, in `resync_membership_proofs`

Node crashes when attempting to sync while having UTXOs it needs to maintain.

This happened during block syncing/catch where blocks are batch-downloaded. I think the problem is a missing lock in the resync_membership_proofs method in main_loop. We need to hold a lock over the wallet database while fetching the current tip digest from light_state.

thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `MutatorSetAccumulator { kernel: MutatorSetKernel { aocl: MmrAccumulator { leaf_count: 133, peaks: [Digest([BFieldElement(9929542416540497177), BFieldElement(3191210005660334294), BFieldElement(887889175830241397), BFieldElement(2279789915348933990), BFieldElement(14568546562018005927)]), Digest([BFieldElement(1204433268204926216), BFieldElement(10990260493988805888), BFieldElement(729594735194224085), BFieldElement(14181650499388803979), BFieldElement(7371473686563791308)]), Digest([BFieldElement(17286232771861285200), BFieldElement(13950877626221494711), BFieldElement(2134864756203853409), BFieldElement(11787128406142950228), BFieldElement(6637212853588538053)])], _hasher: PhantomData<twenty_first::shared_math::tip5::Tip5> }, swbf_inactive: MmrAccumulator { leaf_count: 16, peaks: [Digest([BFieldElement(8470128677405051069), BFieldElement(9778619303736034449), BFieldElement(7808618770791401409), BFieldElement(15160525649937678128), BFieldElement(1402338975408590376)])], _hasher: PhantomData<twenty_first::shared_math::tip5::Tip5> }, swbf_active: ActiveWindow { sbf: [1970, 2659, 3827, 4597, 5599, 8822, 9114, 9872, 9875, 9900, 10606, 11461, 12026, 12678, 16829, 18474, 18916, 19753, 19996, 21396, 21645, 23858, 24365, 26129, 27031, 31326, 34263, 36434, 36461, 37111, 39287, 43575, 44047, 44680, 45958, 48173, 51565, 52575, 53129, 54545, 56051, 56404, 56669, 63458, 63463, 63639, 64476, 65714, 65849, 66241, 66264, 67431, 68460, 68935, 71135, 71384, 76143, 77531, 78835, 79264, 79414, 80527, 83691, 85133, 86948, 91695, 94433, 95657, 96227, 97693, 98028, 100428, 101499, 102744, 103702, 106350, 106810, 107217, 107830, 108220, 109650, 112042, 112467, 113889, 114621, 115598, 115778, 116115, 116640, 119392, 120997, 123156, 124357, 125564, 130515, 135303, 135398, 137635, 140344, 140807, 144194, 145635, 149337, 149872, 150052, 150203, 151044, 151455, 151463, 155419, 157436, 160878, 163987, 164725, 166164, 167141, 171566, 173515, 175230, 177208, 177319, 177639, 177801, 178374, 178390, 180073, 180697, 181313, 186706, 186808, 189836, 192010, 192376, 194231, 194532, 195644, 197129, 198470, 201198, 203484, 204106, 205495, 206168, 206328, 207735, 207834, 211249, 213347, 213848, 215978, 216873, 217478, 218271, 218502, 219204, 219243, 224146, 225022, 225421, 226264, 227711, 229034, 229087, 232800, 234047, 237852, 241479, 241935, 242451, 246272, 246483, 247799, 248633, 249149, 250198, 250223, 252096, 255811, 256131, 256730, 257810, 260005, 261148, 261755, 261760, 262299, 262762, 264615, 265610, 266057, 267143, 267567, 268386, 269097, 270562, 270707, 271445, 272187, 275752, 276118, 276414, 277012, 277812, 280358, 283327, 284712, 288339, 288388, 288711, 289660, 290452, 295377, 295477, 297735, 300023, 302766, 303909, 305063, 305229, 305670, 305756, 305844, 306204, 306900, 308182, 308383, 309328, 309667, 311179, 311289, 311886, 311997, 312298, 313880, 315167, 315872, 318283, 321026, 323234, 325584, 325770, 325980, 328428, 328760, 332350, 332655, 333517, 334531, 335097, 335574, 336781, 339715, 340868, 340872, 341196, 343911, 345710, 348229, 350929, 355687, 357207, 359073, 359635, 360848, 362523, 363802, 365692, 366644, 367907, 368253, 368805, 368884, 374821, 374959, 377687, 381838, 381998, 383378, 384665, 384925, 386831, 388535, 388989, 390810, 394314, 397888, 398556, 401641, 403890, 403974, 404075, 404970, 405354, 411220, 411588, 412782, 413163, 416946, 418009, 418226, 418335, 420184, 428250, 428604, 429531, 433877, 435327, 435495, 435926, 436096, 437198, 437768, 441068, 442805, 442961, 444255, 444324, 444596, 445350, 445469, 446499, 446900, 448036, 449725, 450120, 451612, 455569, 456263, 456687, 456975, 457210, 459964, 463836, 464394, 468831, 470923, 471535, 471654, 472193, 472765, 474361, 474909, 479123, 480170, 481556, 484730, 485200, 487133, 488261, 488840, 495046, 495236, 495584, 496111, 496272, 496457, 497719, 498067, 499405, 500731, 503028, 503429, 504032, 506394, 507067, 508040, 509362, 509374, 510302, 510657, 511107, 517366, 518340, 522097, 522448, 522545, 524073, 524702, 525902, 527379, 528132, 528145, 532473, 536698, 540712, 542372, 544079, 544230, 544623, 548631, 548880, 550534, 552994, 553494, 554136, 554143, 555254, 555623, 557989, 558810, 558971, 560895, 562099, 562771, 564875, 565716, 566174, 566758, 568604, 568655, 570715, 571112, 571397, 572024, 574076, 575056, 575873, 578176, 582568, 583069, 589859, 591387, 591864, 594099, 594723, 597250, 598439, 599985, 601228, 603023, 603258, 603375, 605087, 605855, 605900, 606529, 607424, 609074, 609206, 610103, 613213, 615733, 616706, 618865, 619333, 619745, 620148, 622714, 623659, 626734, 627497, 628736, 631050, 631655, 633004, 633498, 634670, 635900, 637647, 646472, 647501, 647513, 649811, 650798, 652833, 652872, 654212, 655782, 655955, 659269, 661752, 663905, 663960, 664483, 667642, 668052, 671931, 672014, 673672, 675131, 675330, 677822, 678757, 682933, 683582, 684001, 687503, 688094, 688216, 692742, 698650, 698916, 700003, 700032, 700407, 702682, 703289, 704552, 705443, 706583, 709127, 713231, 713739, 717025, 717791, 718212, 721264, 722257, 722549, 723822, 723941, 724131, 729942, 729996, 732184, 732827, 735177, 737098, 737830, 740438, 742337, 745051, 747567, 751173, 753082, 753142, 754392, 755270, 755487, 756817, 757536, 762940, 763277, 763324, 763737, 767464, 768180, 768764, 771125, 772067, 772243, 772747, 773001, 773229, 773584, 774345, 774958, 776388, 777129, 778687, 780270, 782487, 783451, 786184, 786877, 792426, 793273, 794419, 794726, 796625, 797048, 800228, 802792, 804979, 808373, 808615, 809796, 811211, 812994, 813925, 816074, 818008, 819127, 820677, 821431, 825269, 828046, 828420, 830680, 832251, 832370, 832650, 833340, 833463, 833481, 834700, 838323, 838382, 838505, 839914, 841469, 842002, 848628, 850792, 853698, 854647, 856657, 858402, 858972, 863191, 863606, 864619, 864980, 867908, 870937, 873944, 877348, 877379, 877975, 879572, 879699, 882381, 882492, 882777, 885096, 885148, 885916, 886963, 887393, 887930, 889196, 889990, 890186, 890686, 892407, 892622, 895153, 896730, 898053, 900073, 900368, 901202, 901765, 905768, 906118, 907255, 909816, 910329, 912844, 916036, 920022, 921354, 921908, 923363, 924780, 929133, 929457, 933847, 935178, 937216, 937284, 939667, 941570, 941804, 942395, 947458, 951671, 953811, 954891, 956926, 958783, 961434, 963498, 964400, 966712, 968162, 968807, 969933, 971480, 972557, 972657, 973446, 974827, 977952, 982875, 983290, 983990, 984725, 984747, 986569, 987081, 987976, 988958, 989716, 993766, 994703, 998892, 999921, 1005394, 1009194, 1012058, 1013328], _hasher: PhantomData<twenty_first::shared_math::tip5::Tip5> } } }`,
 right: `MutatorSetAccumulator { kernel: MutatorSetKernel { aocl: MmrAccumulator { leaf_count: 133, peaks: [Digest([BFieldElement(9929542416540497177), BFieldElement(3191210005660334294), BFieldElement(887889175830241397), BFieldElement(2279789915348933990), BFieldElement(14568546562018005927)]), Digest([BFieldElement(1204433268204926216), BFieldElement(10990260493988805888), BFieldElement(729594735194224085), BFieldElement(14181650499388803979), BFieldElement(7371473686563791308)]), Digest([BFieldElement(17286232771861285200), BFieldElement(13950877626221494711), BFieldElement(2134864756203853409), BFieldElement(11787128406142950228), BFieldElement(6637212853588538053)])], _hasher: PhantomData<twenty_first::shared_math::tip5::Tip5> }, swbf_inactive: MmrAccumulator { leaf_count: 16, peaks: [Digest([BFieldElement(6653023960902911483), BFieldElement(6390006220384842927), BFieldElement(4912385678695940590), BFieldElement(10367813628548529532), BFieldElement(7177573908953688142)])], _hasher: PhantomData<twenty_first::shared_math::tip5::Tip5> }, swbf_active: ActiveWindow { sbf: [1970, 2659, 3827, 4597, 5599, 8822, 9114, 9872, 9875, 9900, 10606, 11461, 12026, 12678, 16829, 18474, 18916, 19753, 19996, 21396, 21645, 23858, 24365, 26129, 27031, 31326, 34263, 36434, 36461, 37111, 39287, 43575, 44047, 44680, 45958, 48173, 51565, 52575, 53129, 54545, 56051, 56404, 56669, 63458, 63463, 63639, 64476, 65714, 65849, 66241, 66264, 67431, 68460, 68935, 71135, 71384, 76143, 77531, 78835, 79264, 79414, 80527, 83691, 85133, 86948, 91695, 94433, 95657, 96227, 97693, 98028, 100428, 101499, 102744, 103702, 106350, 106810, 107217, 107830, 108220, 109650, 112042, 112467, 113889, 114621, 115598, 115778, 116115, 116640, 119392, 120997, 123156, 124357, 125564, 130515, 135303, 135398, 137635, 140344, 140807, 144194, 145635, 149337, 149872, 150052, 150203, 151044, 151455, 151463, 155419, 157436, 160878, 163987, 164725, 166164, 167141, 171566, 173515, 175230, 177208, 177319, 177639, 177801, 178374, 178390, 180073, 180697, 181313, 186706, 186808, 189836, 192010, 192376, 194231, 194532, 195644, 197129, 198470, 201198, 203484, 204106, 205495, 206168, 206328, 207735, 207834, 211249, 213347, 213848, 215978, 216873, 217478, 218271, 218502, 219204, 219243, 224146, 225022, 225421, 226264, 227711, 229034, 229087, 232800, 234047, 237852, 241479, 241935, 242451, 246272, 246483, 247799, 248633, 249149, 250198, 250223, 252096, 255811, 256131, 256730, 257810, 260005, 261148, 261755, 261760, 262299, 262762, 264615, 265610, 266057, 267143, 267567, 268386, 269097, 270562, 270707, 271445, 272187, 275752, 276118, 276414, 277012, 277812, 280358, 283327, 284712, 288339, 288388, 288711, 289660, 290452, 295377, 295477, 297735, 300023, 302766, 303909, 305063, 305229, 305670, 305756, 305844, 306204, 306900, 308182, 308383, 309328, 309667, 311179, 311289, 311886, 311997, 312298, 313880, 315167, 315872, 318283, 321026, 323234, 325584, 325770, 325980, 328428, 328760, 332350, 332655, 333517, 334531, 335097, 335574, 336781, 339715, 340868, 340872, 341196, 343911, 345710, 348229, 350929, 355687, 357207, 359073, 359635, 360848, 362523, 363802, 365692, 366644, 367907, 368253, 368805, 368884, 374821, 374959, 377687, 381838, 381998, 383378, 384665, 384925, 386831, 388535, 388989, 390810, 394314, 397888, 398556, 401641, 403890, 403974, 404075, 404970, 405354, 411220, 411588, 412782, 413163, 416946, 418009, 418226, 418335, 420184, 428250, 428604, 429531, 433877, 435327, 435495, 435926, 436096, 437198, 437768, 441068, 442805, 442961, 444255, 444324, 444596, 445350, 445469, 446499, 446900, 448036, 449725, 450120, 451612, 455569, 456263, 456687, 456975, 457210, 459964, 463836, 464394, 468831, 470923, 471535, 471654, 472193, 472765, 474361, 474909, 479123, 480170, 481556, 484730, 485200, 487133, 488261, 488840, 495046, 495236, 495584, 496111, 496272, 496457, 497719, 498067, 499405, 500731, 503028, 503429, 504032, 506394, 507067, 508040, 509362, 509374, 510302, 510657, 511107, 517366, 518340, 522097, 522448, 522545, 524073, 524702, 525902, 527379, 528132, 528145, 532473, 536698, 540712, 542372, 544079, 544230, 544623, 548631, 548880, 550534, 552994, 553494, 554136, 554143, 555254, 555623, 557989, 558810, 558971, 560895, 562099, 562771, 564875, 565716, 566174, 566758, 568604, 568655, 570715, 571112, 571397, 572024, 574076, 575056, 575873, 578176, 582568, 583069, 589859, 591387, 591864, 594099, 594723, 597250, 598439, 599985, 601228, 603023, 603258, 603375, 605087, 605855, 605900, 606529, 607424, 609074, 609206, 610103, 613213, 615733, 616706, 618865, 619333, 619745, 620148, 622714, 623659, 626734, 627497, 628736, 631050, 631655, 633004, 633498, 634670, 635900, 637647, 646472, 647501, 647513, 649811, 650798, 652833, 652872, 654212, 655782, 655955, 659269, 661752, 663905, 663960, 664483, 667642, 668052, 671931, 672014, 673672, 675131, 675330, 677822, 678757, 682933, 683582, 684001, 687503, 688094, 688216, 692742, 698650, 698916, 700003, 700032, 700407, 702682, 703289, 704552, 705443, 706583, 709127, 713231, 713739, 717025, 717791, 718212, 721264, 722257, 722549, 723822, 723941, 724131, 729942, 729996, 732184, 732827, 735177, 737098, 737830, 740438, 742337, 745051, 747567, 751173, 753082, 753142, 754392, 755270, 755487, 756817, 757536, 762940, 763277, 763324, 763737, 767464, 768180, 768764, 771125, 772067, 772243, 772747, 773001, 773229, 773584, 774345, 774958, 776388, 777129, 778687, 780270, 782487, 783451, 786184, 786877, 792426, 793273, 794419, 794726, 796625, 797048, 800228, 802792, 804979, 808373, 808615, 809796, 811211, 812994, 813925, 816074, 818008, 819127, 820677, 821431, 825269, 828046, 828420, 830680, 832251, 832370, 832650, 833340, 833463, 833481, 834700, 838323, 838382, 838505, 839914, 841469, 842002, 848628, 850792, 853698, 854647, 856657, 858402, 858972, 863191, 863606, 864619, 864980, 867908, 870937, 873944, 877348, 877379, 877975, 879572, 879699, 882381, 882492, 882777, 885096, 885148, 885916, 886963, 887393, 887930, 889196, 889990, 890186, 890686, 892407, 892622, 895153, 896730, 898053, 900073, 900368, 901202, 901765, 905768, 906118, 907255, 909816, 910329, 912844, 916036, 920022, 921354, 921908, 923363, 924780, 929133, 929457, 933847, 935178, 937216, 937284, 939667, 941570, 941804, 942395, 947458, 951671, 953811, 954891, 956926, 958783, 961434, 963498, 964400, 966712, 968162, 968807, 969933, 971480, 972557, 972657, 973446, 974827, 977952, 982875, 983290, 983990, 984725, 984747, 986569, 987081, 987976, 988958, 989716, 993766, 994703, 998892, 999921, 1005394, 1009194, 1012058, 1013328], _hasher: PhantomData<twenty_first::shared_math::tip5::Tip5> } } }`', /home/thv/repos/neptune-core/src/models/state/mod.rs:503:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tokio-runtime-worker' panicked at 'Failed to read from main loop: channel closed', /home/thv/repos/neptune-core/src/peer_loop.rs:974:35
thread 'tokio-runtime-worker' panicked at 'Failed to read from main loop: channel closed', /home/thv/repos/neptune-core/src/peer_loop.rs:974:35

opened by Sword-Smith 1

Excessive disk space being used by `wallet_block_db`

After ~1900 blocks my node ran out of disk space. I only had 1.8 GB available, but all that was used by the ~/.local/share/neptune/alpha/databases/wallet_block_db directory.

The archival mutator set was only 1.7MB, and all blocks stored on disk was only 33MB.

Investigate why wallet_block_db is using so much disk space, and fix the issue.

opened by Sword-Smith 5
Archive Size
The dashboard has a field on the overview screen informing the user as to how much space is being used by archiving blocks, membership proofs, and mutator set data. But currently it only holds a placeholder because we don't have a way to calculate that yet.

Tasks:

[ ] Introduce and implement RPC calls to compute archive size.

[ ] Introduce and implement RPC calls to compute fraction of blockchain covered by archive (since we only have archival nodes at the moment, for the time being this should be different from 100% only while syncing.)

[ ] Use RPC calls in dashboard.

good first issue
opened by aszepieniec 0
Roadmap
[x] Peer-to-peer Networking

[x] peer discovery

[ ] peer authentication

[ ] end-to-end encryption

[ ] onion routing

[x] Daemon Application Architecture

[x] main thread

[x] peer thread

[x] mine thread

[x] RPC thread

[x] RPC interface

[ ] windows & mac support

[ ] binaries

[x] Synchronization

[x] archival block synchronization

[x] UTXO synchronization from blocks

[x] Blockchain Logic

[x] Transaction

[x] kernel definition and MAST hash

[x] validity program

[x] lock scripts halt

[ ] type scripts halt

[x] removal records integrity

[ ] kernel to lock scripts

[ ] kernel to type scripts

[ ] public scripts halt

[ ] single proof

[ ] merging

[x] on faith

[ ] with recursion

[x] UTXO

[x] privacy: mutator sets

[x] lockable UTXOs

[ ] lock-free UTXOs

[ ] currency logic

[ ] arbitrary state evolution

[x] addresses

[x] generation

[ ] change (untraced)

[ ] change (traced)

[ ] online

[x] encoding

[x] bech32m

[ ] bech32m error location

[ ] alternative

[ ] Block

[ ] parameter adjustment (spec)

[ ] parameter adjustment (logic)

[ ] validity program

[ ] transaction validity (rust / tasm)

[ ] parameter update (rust / tasm)

[ ] previous block validity (rust / tasm)

[ ] single proof

[x] Wallet

[x] generate tool

[x] addresses

[x] generation

[ ] change (untraced)

[ ] change (traced)

[ ] online

[x] encoding

[x] bech32m

[ ] bech32m error location

[ ] alternative

[x] Documentation

[x] User Interface

[x] console

[x] command-line

[x] overview

[x] peers

[x] transaction history

[x] send

[x] receive

[ ] blocks

[ ] graphical
opened by aszepieniec 0
Redefine or remove all references to listen address

It doesn't represent what I thought it did, at least not in its current form. It is the listen address of the internal network interface, not the public socket address.

opened by Sword-Smith 1