Watching a bunch of addresses using -zmqpubrawtx=<address>

TheArchaeologist

sr. member

Activity: 310

Merit: 727

---------> 1231006505

Quote from: NotATether on November 04, 2021, 12:36:18 PM

But what if the Hash160, by chance, happens to match another part of the raw transaction (that is not in the designated vout or vin areas)? It's something that I thought about just now.

In practice that won't happen. But it doesn't even matter. Just like with a bloom filter you could see it as a false positive if it would. If the hash160 isn't found in the raw transaction you can be sure the address you are looking for is not funded in the transaction. If however it is found you move on to the next step, decode the transaction and look for the address, if it doesn't happen to be there the address wasn't funded in there after all. But like I said this won't happen.

And in case you wondering what if the hash160 is mentioned in the vin-part (so it is used for funding): it won't. The vin mentions the txid of an unspent transaction + the index. So if the address you are watching is used for spending the hash160 won't be found in the raw transaction you are evaluating.

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

But what if the Hash160, by chance, happens to match another part of the raw transaction (that is not in the designated vout or vin areas)? It's something that I thought about just now.

TheArchaeologist

sr. member

Activity: 310

Merit: 727

---------> 1231006505

Quote from: NotATether on November 04, 2021, 09:03:36 AM

That's actually a great idea. The zeromq message queue that bitcoind-rpc exposes emits Buffer objects as the raw transaction which are just a bunch of bytes. If I can serialize the addresses down to bytes of hash160s as well, I can avoid all those expensive calls to decoderawtransaction and then for transactions which *do* match, call decoderawtransaction on them later to retrieve the value sent to those addresses.

Yes, exactly the strategy I meant. To elobarate:

Say you want to monitor if address 3ChwpX32vBgXAZdheYGiFskMAZqQpKUZiT is getting any transactions:

Code:

3ChwpX32vBgXAZdheYGiFskMAZqQpKUZiT -> hash160= 78d65fb3b8fa013867a396d610a542a7213c1d99
Can be found by doing a b58 decode (won't work for bech32 addresses ofc)

Now let's say you will get the following transaction from zmq:

Code:

txid:e8e2a211e269d8105669367ce336eb4e2145f52816d1c1f528c3eb4795018c84

Raw-tx:
010000000154a8e81dd0fb53f0f82f2d6b90d2972a3dd1718ee4c1e9113a1add12f6c5e334000000006a47304402203f96e1b966ac9a8eaf7c6ee011f280ec9ba0f4a31eff9e678ffcbacd2d9ef3a10220292d03b0694ddf8e4773ceea0a85341574febc0243a76cbdbaaae9cc7c998d390121037873692731f2bd925132c2c05852deb44718dee093d8ce1567bf174e8440f03dffffffff0166aa2d000000000017a91478d65fb3b8fa013867a396d610a542a7213c1d998700000000

Now check to see if the hash 160 (78d65fb3b8fa013867a396d610a542a7213c1d99) is contained in the raw transaction (so without any decoding):

010000000154a8e81dd0fb53f0f82f2d6b90d2972a3dd1718ee4c1e9113a1add12f6c5e33400000 0006a47304402203f96e1b966ac9a8eaf7c6ee011f280ec9ba0f4a31eff9e678ffcbacd2d9ef3a1 0220292d03b0694ddf8e4773ceea0a85341574febc0243a76cbdbaaae9cc7c998d3901210378736 92731f2bd925132c2c05852deb44718dee093d8ce1567bf174e8440f03dffffffff0166aa2d0000 00000017a91478d65fb3b8fa013867a396d610a542a7213c1d998700000000

Yes it does. In that case do whatever extra steps you want including decoding the transaction, etc.

PrimeNumber7

copper member

Activity: 1652

Merit: 1901

Amazon Prime Member #7

If you are going to accept transactions with n confirmations, you can improve your workflow by doing the following:

Code:

sudo code
latest_block = 0
time.sleep(CHECKFREQUENCY) # CHECKFREQUENCY = the amount of time you will wait to check for a new block
current_block = RPC command getblockcount
if current_block > latest_block:
    block_to_check = RPC command getblockhash(current_block - REQUIREDCONFIRMATIONS) #REQUIREDCONFIRMATIONS = confirmations needed to accept transaction
    block_data = RPC command getblock(block_to_check, verbosity=2)
    block_transactions = block_data['tx']
    for transaction in block_transactions:
        decoded_tx = RPC command decoderawtransaction(transaction)
        for vout in decoded_tx['vout']:
            #get output address and amount
    latest_block = current_block

The above assumes that any orphans will not result in your transactions going from confirmed to unconfirmed (while this is rare, this is something you will want to address).

It will also make it so you do not have to keep all unconfirmed transactions in RAM. Today, most transactions that make it into the mempool will get quickly confirmed, but the small number of never-confirmed transactions will pile up over time, and when the mempool is more full, many more transactions that make it to your mempool will never confirm.

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

Quote from: TheArchaeologist on November 04, 2021, 08:59:33 AM

Quote from: NotATether on November 04, 2021, 08:45:51 AM

I've been getting errors about exceeding it on testnet by calling decoderawtransaction on every single incoming transaction my node detects.

If all you are doing is monitoring for incoming transactions for specific addresses you could do without the decoding and just check if the raw transaction contains the ripemd-160 of the address instead.

That's actually a great idea. The zeromq message queue that bitcoind-rpc exposes emits Buffer objects as the raw transaction which are just a bunch of bytes. If I can serialize the addresses down to bytes of hash160s as well, I can avoid all those expensive calls to decoderawtransaction and then for transactions which *do* match, call decoderawtransaction on them later to retrieve the value sent to those addresses.

TheArchaeologist

sr. member

Activity: 310

Merit: 727

---------> 1231006505

Quote from: NotATether on November 04, 2021, 08:45:51 AM

I've been getting errors about exceeding it on testnet by calling decoderawtransaction on every single incoming transaction my node detects.

If all you are doing is monitoring for incoming transactions for specific addresses you could do without the decoding and just check if the raw transaction contains the ripemd-160 of the address instead.

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

Quote from: TheArchaeologist on November 04, 2021, 07:30:38 AM

~

Genius. Exactly what I needed. Now I can avoid muddling with Blockcypher.

I still have to deal with performance issues though, as with the default work queue depth (16), I've been getting errors about exceeding it on testnet by calling decoderawtransaction on every single incoming transaction my node detects. I suspect the rate the transaction messages will come are going to be even higher on mainnet. How much, I don't know.

I've increased the work queue depth to 32 (and introduced a sleep delay of 10s when the depth is exceeded) and it seems to be stable for now.

TheArchaeologist

sr. member

Activity: 310

Merit: 727

---------> 1231006505

Quote from: NotATether on November 03, 2021, 02:26:03 PM

Meanwhile if someone here knows how to track tx's in a particular block, and also get the current block height...

Getting the blockheight (latest block) from bitcoin-core:

Code:

bitcoin-cli getblockcount

Tracking transactions in a particular block:

Code:

bitcoin-cli getblockhash 

where  is the block of which you want to see the included transactions, use the recveived blockhash for the next command:

bitcoin-cli getblock

Working example:

Code:

bitcoin-cli getblockcount
returns -> 708171

bitcoin-cli getblockhash 708171
returns -> 00000000000000000008677cb8837ed65aae105d7c69a8c59929d419acd82eae

bitcoin-cli getblock 00000000000000000008677cb8837ed65aae105d7c69a8c59929d419acd82eae
returns:

{
  "hash": "00000000000000000008677cb8837ed65aae105d7c69a8c59929d419acd82eae",
  "confirmations": 2,
  "height": 708171,
  "version": 536870916,
  "versionHex": "20000004",
  "merkleroot": "c97736339fbc33d74f37b567752e44a21fcf16681cd8a20cc9616f3263d362>
  "time": 1636028408,
  "mediantime": 1636024965,
  "nonce": 794427675,
  "bits": "170cfecf",
  "difficulty": 21659344833264.85,
  "chainwork": "000000000000000000000000000000000000000023a58a560224f1764bbd2b1>
  "nTx": 1021,
  "previousblockhash": "0000000000000000000a5b363eb16cef32f96d426ab0e12d788f440>
  "nextblockhash": "0000000000000000000ba2dc289a1c88fcc2038fb6c29e456a20843cb87>
  "strippedsize": 766974,
  "size": 1697345,
  "weight": 3998267,
  "tx": [
    "903340bf415d3e8b60ee4b05ec86fc74a4e4a4b3e6d4a00c781e9bfe91843aee",
    "ee2567364be24addd23ea77edfebc0bb66d8d9b182dadb91cd0a39908d2c6fa4",
    "e8beceac0cf99d303975036b89d1c9e3e4f96e596fcac70d2451644eb62639d1",
    "6b1bce4995656bd6c547e32f0719eb649838e7d34bdb423f3316b2093c9e9a8b",
    "f8bf080de438d1a1b71364ba9203e704b8ef9767964fdf3698bbd82c327a37b0",
    "b8a5ccbb753a394cb3f5d6e4fd4cc5f9649a67eee878f389acf23f16becdefbd",
    "812291d79c5b347ba801da97f353f58bf1e51a4cc17006ab5e7ebedb571eff5d",
    
    "0cadeaa28319ee1074a57ceebc2da373d3a929cc427d82183e7be16e26c4d419",
    "e4a2d32e42a5a31df12217ae561da6ea5032fe27c86d7c957610c1963da59df1",
    "51327517a743d787b8f5186210f5bf835e61e0dfbd8eca3a47632cddab6362e6"
  ]
}

Note: Json "tx"-field consists of a list of all the transaction-hashes in the block.

Examples are from cli, ofcourse you can interact with rpc as well to do this. Let me/us know if this was of any help or if you are looking for something different / extra. I can probably help

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

So, fiddling around making the function for searching the address list an async function, and putting all the addresses in a red-black tree, has really helped with performance. Right now, I am limited to testing with addresses on testnet and managed to track tBTC that I sent between Electrum wallets.

Bitcoin Core nodes won't tell you easily if a given transaction is in a block yet (as far as I know), so I'm going to use Blockcypher's API for that. Meanwhile if someone here knows how to track tx's in a particular block, and also get the current block height, I might be able to code up an alternative and save hundreds of dollars per month [blockcypher prices are outrageous].

HCP

legendary

Activity: 2086

Merit: 4361

Quote from: NotATether on November 01, 2021, 05:13:23 AM

So from here I can inspect the vouts, and each "value" and "address". The question now is, although address is an array, it appears that each vout only corresponds to one address.

Can I safely take the first address in each vout and assume it has the value listed? Or will it ever emit multiple addresses in one out? (That doesn't make any sense though).

This answer by Peter Wuille sums it up nice and succintly: https://bitcoin.stackexchange.com/a/11311

Essentially, it is theoretically possible, but very unlikely, that you could get multiple addresses here.

You can either code it expecting to only ever get 1 address, and then pray it never happens (Bad Plan™ refer: Murphy's Law)... or implement some kind of exception handling to handle the multiple address case so your program doesn't crash... even it is just logs it and then ignores the transaction and continues on it's merry way. ("Better Plan"™).

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

Thanks for the replies guys. @HCP I had already seen that page and was working on a program derived from that

I managed to output transactions as they are passed to the Bitcoin Core client via tx messages in real-time:

Code:

var zmq = require('zeromq')
  , sock = zmq.socket('sub')
  , RpcClient = require('bitcoind-rpc');
sock.connect('tcp://127.0.0.1:29000');

var config = {
    protocol: 'http',
    user: 'bitcoin',
    pass: 'local321',
    host: '127.0.0.1',
    port: '8332',
};

var rpc = new RpcClient(config);

sock.subscribe('rawtx')
sock.on('message', function(topic, message) {
  rpc.decodeRawTransaction(message.toString('hex'), function(err, resp) {
        console.log(JSON.stringify(resp, null, 4)) // Change code here
  })
})

Note: Should only be ran after Core's done verifying to prevent Core from crashing.

This is one of the thousands of transactions that are emitted by this:

Code:

{
    "result": {
        "txid": "d0c7061f2932aab93b4dfb1e7061ca06e7bceb2f46db2d8d31ae4af881079e14",
        "hash": "e25d73f479b07feec55115afd670ac86ef3b8565eeff040a1e5b573f122da5f0",
        "version": 2,
        "size": 258,
        "vsize": 176,
        "weight": 702,
        "locktime": 0,
        "vin": [
            {
                "txid": "793c35452a0afe536b29ac282fd34bfa10a9b11e622c50a968e96828b4987980",
                "vout": 2,
                "scriptSig": {
                    "asm": "",
                    "hex": ""
                },
                "txinwitness": [
                    "3045022100f99d5710d33f49cc2dd8d697af8e84ffc3618ae1c8726c769d257d30e6c31dff0220149bb4f31fbda2078a91fba08c7060a135ecaf8c52b042cdf9a90dbd6ad4373701",
                    "026e5628506ecd33242e5ceb5fdafe4d3066b5c0f159b3c05a621ef65f177ea286"
                ],
                "sequence": 4294967293
            }
        ],
        "vout": [
            {
                "value": 0.01963328,
                "n": 0,
                "scriptPubKey": {
                    "asm": "OP_DUP OP_HASH160 df8baa7ce04baa355dfdff9553315ba8a5092e73 OP_EQUALVERIFY OP_CHECKSIG",
                    "hex": "76a914df8baa7ce04baa355dfdff9553315ba8a5092e7388ac",
                    "reqSigs": 1,
                    "type": "pubkeyhash",
                    "addresses": [
                        "1MNzzwWMzL8fkBCeTZ5dZU3HL4o6WTX8qm"
                    ]
                }
            },
            {
                "value": 0.0026072,
                "n": 1,
                "scriptPubKey": {
                    "asm": "OP_HASH160 908504c01b5b27b90087cbf945b9b846382ddd2c OP_EQUAL",
                    "hex": "a914908504c01b5b27b90087cbf945b9b846382ddd2c87",
                    "reqSigs": 1,
                    "type": "scripthash",
                    "addresses": [
                        "3EsAaTZAqqdPf8gVtxXnkMCh2dLCVk5bKE"
                    ]
                }
            },
            {
                "value": 11.26734088,
                "n": 2,
                "scriptPubKey": {
                    "asm": "0 f60834ef165253c571b11ce9fa74e46692fc5ec1",
                    "hex": "0014f60834ef165253c571b11ce9fa74e46692fc5ec1",
                    "reqSigs": 1,
                    "type": "witness_v0_keyhash",
                    "addresses": [
                        "bc1q7cyrfmck2ffu2ud3rn5l5a8yv6f0chkp0zpemf"
                    ]
                }
            }
        ]
    },
    "error": null,
    "id": 9989
}

So from here I can inspect the vouts, and each "value" and "address". The question now is, although address is an array, it appears that each vout only corresponds to one address.

Can I safely take the first address in each vout and assume it has the value listed? Or will it ever emit multiple addresses in one out? (That doesn't make any sense though).

HCP

legendary

Activity: 2086

Merit: 4361

This tutorial regarding the ZeroMQ Interface might help: https://bitcoindev.network/accessing-bitcoins-zeromq-interface/

Granted it uses Javascript for the handler logic, but it illustrates the fundamentals of how to configure your node to use the ZeroMQ interface, and what sort of data your handler will be receiving from it.

domob

legendary

Activity: 1135

Merit: 1166

The

in this case is not a Bitcoin address but a TCP address on which the ZMQ notifications will be published. You will get notifications about all transactions, not filtered by Bitcoin address. But you can then of course just implement your logic (checking for your set of addresses) in the handler.

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

It has come to my attention that the -zmqpubrawtx=

parameter to Bitcoind can be used to listen for raw transactions published on the network and send the messages to the 127.0.0.1:29000 address and port.

Is it possible to pass this parameter multiple times with different Bitcoin addresses?

I need to monitor up to 200k addresses for incoming transactions so I'm interested to know if the software can handle that many event listeners that well (I'm only expecting a tiny percentage of these addresses firing events at any given time).

Topic: Watching a bunch of addresses using -zmqpubrawtx=<address> (Read 251 times)