It would be useful to observe which pools (don't) include fee bumped transactions that are compatible with the new -mempoolfullrbf setting. I suppose this detection would be more useful on top of a node with this setting enabled, because otherwise you can't see how long such a fee bump has been in the mempool.

If we can sort the list by amount-bumped we can also see if @petertodd's bounty program is working.

ref: https://github.com/0xB10C/ofac-sanctioned-digital-currency-addresses/commit/96f89d770659edad674fa2ea101a0d79a7f4c768 ofac: https://home.treasury.gov/policy-issues/financial-sanctions/recent-actions/20220915

This prevents "ThreadRPCServer incorrect password attempt from 127.0.0.1:..." errors.

I'm using cookie based authentication, and the cookie changes every time bitcoind is restarted.

There might be a more elegant way to fix this, but this works. I haven't had this problem with c-lightning, electrs and nbexplorer, so their implementation might provide some hints.

cargo 1.63.0 on Ubuntu 22.04.1 on master @ 39c48032b39d6df565723610c5aaa37fb689ca50

cargo build --release --bin miningpool-observer-daemon

...

   Compiling miningpool_observer_shared v0.1.0 (/home/miningobs/miningpool-observer/shared)
warning: derive helper attribute is used before it is introduced
  --> shared/src/model.rs:16:3
   |
16 | #[primary_key(hash)]
   |   ^^^^^^^^^^^
17 | #[derive(Queryable, Serialize, Identifiable)]
   |                                ------------ the attribute is introduced here
   |
   = note: `#[warn(legacy_derive_helpers)]` on by default
   = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
   = note: for more information, see issue #79202 <https://github.com/rust-lang/rust/issues/79202>

   Compiling migrations_macros v1.4.2
   Compiling diesel_migrations v1.4.0
warning: `miningpool_observer_shared` (lib) generated 1 warning
warning: unused import: `std::convert::TryFrom`
 --> daemon/src/main.rs:2:5
  |
2 | use std::convert::TryFrom;
  |     ^^^^^^^^^^^^^^^^^^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

warning: unused imports: `PackedLockTime`, `Sequence`, `Witness`
  --> daemon/src/processing.rs:20:74
   |
20 | use bitcoin::{network::constants::Network, Address, Amount, Transaction, Sequence, PackedLockTime, Witness};
   |                                                                          ^^^^^^^^  ^^^^^^^^^^^^^^  ^^^^^^^

warning: unused `Result` that must be used
   --> daemon/src/main.rs:905:21
    |
905 |                     db::update_transaction_tags(&new_tags, &tx_in_db.txid.clone(), &conn);
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: `#[warn(unused_must_use)]` on by default
    = note: this `Result` may be an `Err` variant, which should be handled

warning: `miningpool-observer-daemon` (bin "miningpool-observer-daemon") generated 3 warnings
    Finished release [optimized] target(s) in 1m 03s

The feerate graph would previously not render for blocks with zero-fee transactions. Zero values don't work with the y-axis log scale.

closes https://github.com/0xB10C/miningpool-observer/issues/27

AJ noted this on twitter for the blocks 684266 and 684267.

Hmm, https://miningpool.observer/template-and-block/ is missing blocks at height 684266 and 684267 (267 was only 3s after 266, so fair enough, but not seeing why 266 would be missing).

The daemon skipped the block 267 as it couldn't request a template before the block was found. The daemon had templates for the the block 266, but currently does not process it. This should be improved.

Logs

INFO  [stats] New Template based on 684265 | 620 tx | coinbase 7.36356666 BTC
INFO  [stats] New Template based on 684265 | 627 tx | coinbase 7.36383182 BTC
INFO  [stats] New Template based on 684267 | 777 tx | coinbase 7.26011534 BTC
WARN  [processing] Can't compare the previous_template to the new block. Was there a reorg or multiple blocks found in rapid succession?
INFO  [stats] New Template based on 684267 | 777 tx | coinbase 7.26011534 BTC
INFO  [stats] New Template based on 684267 | 775 tx | coinbase 7.26158040 BTC
INFO  [stats] New Template based on 684267 | 765 tx | coinbase 7.26284898 BTC
INFO  [stats] New Template based on 684267 | 749 tx | coinbase 7.26845930 BTC

(I've replaced the block header hashes with block heights here)

Over the last 7 days, we saw 10 "Blocks skipped"-events, that means at-least 20 unprocessed blocks, 10 of which we could have processed.

Steps:

• [x] refactor: split the processing_loop() function into a loop+check and a processing function. The processing function takes the templates and block as arguments. This enables us to process the block we have templates for even if it's not the newest.
• [x] for new templates, log the height as well (easier to reason about than the block hash only)
• [x] get block we have templates for and pass it into the new processing function
• [x] improve the wording in this comment

One thing I couldn't find on the pages was the software version you used to generate the templates, since that can also be a source of differences. (E.g., imagine taproot activates; Bitcoin Core 0.21.0 won't mine taproot transactions; 0.21.1 will. Although not related to your primary goal of detecting censorship, comparing different version's templates against blocks may tell you what software miners are running.)

via https://x0f.org/web/statuses/106189627623104063

This allows using a relative path for the "www" directory.

Discussed in https://github.com/0xB10C/miningpool-observer/pull/40#discussion_r971808209

The first commit (only) sorts the list of addresses as in https://github.com/0xB10C/ofac-sanctioned-digital-currency-addresses/pull/3 and the second commit updates the list with the newly sanctioned addresses from https://home.treasury.gov/policy-issues/financial-sanctions/recent-actions/20220914.

This avoids the need to apply a patch and self compile.

https://github.com/bitcoin/bitcoin/pull/18772 was merged and made it into ~v23.0~ v22.0

(haven't tested this yet)

When adding new tags, the tags for the transactions in the database can now be updated. This can be controlled via the configuration parameter retag_transactions (true/false).

Closes https://github.com/0xB10C/miningpool-observer/issues/31

Detecting censorship is one of the main purposes of miningpool-observer. The code detecting a sanctioned transaction not making it into a block is however currently not covered by unit tests. It should probably be covered by unit tests better sooner than later.

Currently, the list of sanctioned transactions is baked into the binary during compilation. While this is nice for performance, it also means that I need to pull the list from GitHub, commit it to the repo, do a minor release, notify users, and users need to recompile and redeploy every time the list changes. This works when I'm the only one running miningpool-observer, but doesn't really scale.

An auto-update list feature could help here. One option could be to just download the list from time to time in the daemon. However, web needs the list too (to show e.g. https://miningpool.observer/faq#sanctioned-tx-addresses).

Loading the list from GitHub (or any other configurable URL), and then saving it to the database should work. Both, web and daemon can load and reload it when they need it.

As GETing things from URLs requires new dependencies, an alternative solution could be to have something that does this externally, e.g. a cron script. This comes with extra setup-work for users and might not be worth it.

Currently, if cookie-authentication is used for RPC and Bitcoin Core is restarted, the old cookie fails to authorize (reported in #48). If this happens, it might be worth trying to reload the cookie file once to see if that helps. I'm not sure if that's doable within the current rust-bitcoincore-rpc setup, but it can't hurt to try.

RPC users and password auth is not affected and should work across restarts.

• run only once after new tags have been added, probably mentioned in a release note
• requires txindex, non-pruned
• might take a while
• uses current sanctioned address list, but only updates TO SANCTIONED and not FROM SANCTIONED
• ...