Topic: Checking Address Balance with Bitcoin Core (Read 671 times)

Had the exact same problem.  Decided not to check the balances on the addresses, but the other way around - to scan the blockchain real-time and see if anything has arrived.  I wrote a simple bash script which is able to scan with a speed of 30 transactions per second.

https://bitcointalksearch.org/topic/m.52352018

I am running bitcoind in pruned mode.  Everything seems to work nice so far.

My SQL statement I forgot to include VALUES on INSERT. Fixed that. Now I'm getting amount: 0 for all transactions but it is saving them to my DB. When I use the CLI I get the same problem. When I use sendtoaddress it gives me a txid then when I use gettransaction on the transaction it says amount: 0 every time. This is what I get:


I'm seeing both a receiving and sending address and its using the same address for both. I thought [details][address] was only there for the receiving address? I'm assuming my regtest config isnt set up right or something and since it says its both sending and receiving it evens out to 0. Why would it have this behavior? The transaction is just to an address I generated using getnewaddress which should automatically have a zero balance so it shouldnt even be able to send to itself?

Alright, so we at least found the issue.
Since core and the config is working fine, we can focus on your php/sql issue.


The way to debug this, would be to run your php script from the command line and give it a (non-existing, randomly chosen or real) transaction ID.
Then see what it does or doesn't do.

I don't think there is an error log being written when these scripts are called from bitcoind.
But you can always create your own log. Either use the logger command or redirect the output from your commands into your own logfile (using >>) (sorry i don't know the php equivalent of these commands).

I think the best way would be to get them running using the command line.
Since you already found some syntax errors in your php script, is it running through completely now ? If so, it could be safe to assume that there is an issue with SQL.
If they are then running through fine via the command line, the next step would be to test them via bitcoind / walletnotify.

Ok, so I reset my bitcoin.conf to run your test script


Except that it writes to the same directory the script is in. I did this to test if permissions is the issue. The script ran fine and generated a file with 6 entries of the date after starting daemon, a list transactions call, generating a new address and then sending some coin to it. So permissions arent the issue and the shell script worked. There was also nothing in the regtest debug.log so it's either PHP isnt running or theres an error in my PHP script. Where could I find error logs for my PHP script since bitcoind called it instead of my server? I figured out syntax errors by testing scripts from command line but there were no transactions added to my database. I'm trying to figure out if I have an error in my SQL statements or logic doesnt work or what. I have custom errors that would normally be thrown to my server error logs but as I've said I don't know how to check PHP or MySQL error logs when it was called from bitcoind.

What bothers me is the return code 65280.

Unfortunately i am not sure what causes this, but it might be a faulty script ? Are you able to call it from the command line properly using /usr/bin/php /home/bitcoin/script.php ?

You might try it with a simple shell script to see whether your php script is the reason or some configuration. Something like this should be enough:

Place it anywhere and change walletnotify to:

See whether it can be called properly. If so, it should create a file in /tmp and the current time/date should be inside of the file.

If this dummy script is working, your script seems to be faulty. In this case post it somewhere so we can take a look at it. If you don't want to paste it publicly, feel free to PM it to me. I'll try to take a look at it.
If this dummy script isn't working either, some configuration is faulty, but currently not sure what exactly. Permissions maybe ?

So if I have errors thrown in my script where will I find the error logs? It should be somewhere different because bitcoind is running it and not my web server right?

I added the php executable to my config file and still getting an error


My table is still empty in my database so the script didnt run (or work). What behavior will it do if the script throws an error?

Your config looks right to me.

You have basically two options to call your php script.

1)
Like you are trying it to do:

This requires your php file to be executable (chmod +x) and a shebang pointing to the php executable in the first line of your script (e.g.  #!/usr/bin/php)


2)

This does not require the file to be executable or to contain a shebang in the first line.


Note that you have to replace /usr/bin/php with your actual path to the php executable, which can be found using this command:

Ok, so I wrote PHP scripts for blocknotify and wallet notify along with the needed tables. So when I generate new blocks and send bitcoin to a generated address I can see in the regtest error log that the scripts arent working:


I see that for every new block generated and also the same for the script for walletnotify. My table is empty so I dont think the script is even running. I set walletnotify and blocknotify in bitcoin.conf and just start the daemon using ./bitcoind -regtest -daemon

Is there a way that I'm supposed to type it differently because I'm using PHP? I'm assuming its because .php files are not executables. I could write an executable but right now I'm basically just making a proof of concept in PHP because its faster to do so. Also, how would I check for error logs for my PHP scripts if theres syntax errors and such? I'm assuming they wont be in the same place as PHP error logs for site code because its being run from bitcoind rather than my server software. I'm using Debian Linux.

This is my bitcoin.conf


I've tried adding php -f before the location of the file but still same error.

You could associate one order to one address and then keep track of all transactions to this order. You could have tables like this:
order <-> address (1:1)
address <-> transactions (1:n)
transaction <-> confirmation (1:n)

I think this should work (not absolutely sure about any edge cases).




You should use blocknotify only to count confirmations.

Better use walletnotify for all incoming transactions -> assigning in your database etc. and for the first confirmation.
Then increase the count of the confirmations the next time blocknotify is called. I don't think you need blocknotify for anything else.

Or am i missing something ?

With walletnotify I'd just get a new transaction and I can use the tx info to determine which address it goes towards and update in mysql the number of transactions for that order, however I want to handle cases where users need to send multiple transactions to the address for example if the first transaction wasnt enough to pay the whole tab. II was thinking use blocknotify then search my database for all orders with less than required confirmations and have orders associated with the address for the order. With multiple transactions, using one confirmations field in the database doesnt work so whats a strategy I could use to keep track of all the different transactions for each address/order? I'd have to keep track of all the transactions confirmations in a separate table and associate them with the order ID in the table keeping track of orders wouldnt I?

I'm kind of having trouble coming up with how to handle block notify since it sends block hash instead of tx hash.

Those scripts would be run whenever your node receive a block / transaction.

If your node is down, it doesn't receive anything.
When you start it up again, and it starts catching up those missing blocks, all received blocks will call blocknotify and all transactions to your wallet received will call walletnotify.

It does not skip anything. It resumes operation after the node is up again.




listtransactions gives you the last X transaction sent to your wallet.

So I just thought about it, what would I do in the case that my node went down and wasnt running for all the new transactions or blocks? Would it just skip over those notifications or handle them when the node restarts?

When I use listtransactions does it list all transactions in the block or just transactions in that block for my addresses associated with my xpub?

I wish there were more people like you (not using bitpay) to protect the privacy of your customers.  You are doing similar to How I take VPN monthly payments (see a recent jump in number of user using this type of payment option), and just to share my experience - (correct me please if any of those steps were deemed sub-optimal):

1) with offline and isolated PHYSICAL, non virtualized (Zen, vmware, etc) computer (with much older hardware, say Pentium III no IPMI/other build in BMI chips to scrape your memory during run-time) pre-generate pool of cold private keys address, using dumpprivkey(getnewaddress()). I keep a changing pool of address on live (or more exposed) system and keep no logs for privacy, and those address are only good for receive 1 payment for 1 instance only.
    - example:  getnewaddress "" bech32 to create bc1 prefixed native-segwit receive address to let your customer save fees.. assume you have core running.
    - Also create like 15% of those payment address in 3xx p2sh-segwit format simple getnewaddress() "" p2sh-segwit, then dumpprivkey() those said address and guard it / safe keeping those off-line.
    - Note current block count with getblockcount() for faster load for preparing your bitcoin core wallet for manual coinjoin transaction in step 5
    - After generated btc payment address/private key in safe place (see step 5)    
2) call API to generate QR bitmaps (zxing, etc.. a little off topic so I can't say more)
3) check those pre-generated balance using getreceivedbyaddress bc1xxxxxxxxxx etc.
4) No need to send to cold storage.  you pre-generated those private keys they are in floppy disks (or what ever else you choose.. automated Type 316 stainless steel plate tele-type would be my choice)
5) when you need to pay hosting or in my case co-location fees and need fiat, send it to chipmixer if its large (not feasible in my case because payments are normally 8 - 100 usd and chipmixer only do 0.001 btc). Or, if you want to do it yourself,
   a) on a different computer, with freshly install OS importprivkey() those address where in step 3 have getreceivedbyaddress(addr)>0. make sure to importprivkey "privatekey" false to keep your system away from rescanning private key.
   b) then rescanblockchain (Chain height from Step 1) to ready your wallet for manual coin-join / chipmix hybrid mixing.
6) create larger coinjoin transactions (I'm still writing software to automate RPC call with createrawtransaction() to a pool of chipmixer inputs.  You can tor and than hit chipmixer and get a pool say 32 session keys,
    a) now you have 32 output address - Note: Don't wait long between this step and complete your coin join transaction, because chipmixer only good for limited time.. I usually drag it out to 24 to save on tx fees (core is good at estimate this or to be sure visit bitcoinfees.earn.com i.e.,)
    b) each one with base 2 power of kbtc and have say around 64 inputs.  So you have output to each of those 32 as 0.001, 0.002 ... 0.00x; where x^2 represents the chip size you want
    c) enjoy lower fees with bc1 / p2sh-segwit enabled receive addresses
I manually create and select mgmt enabled).  I found p2sh-segwit address receives payment confirmation than legacy. bc1 comes in at about the same speed as legacy but with much reduced size and higher satoshi/byte it generally arrives as quick as legacy payment input address.
7) face2face and cash in those chips... I usually keep like 100 or so chipmixer private keys so I can cash out exact amount during LBC face-2-face meetups to cash out NTD fiats.  I usually PGP a reputable bitcoin dealer from HK/Korea/ in LBC and they will stop regularly to help give me cash at spot price.

Note: Help on 6 on createrawtransaction() bc1 segwit-enabled coin-join transaction is much appreciated. Anyone could provide me with example / sample script would save me about 2 hrs each week to make this transaction bulletproof manually.

He could.

But the address is only used for one single transaction.

Note that a public key is not an address. The address is being 'derived' from the public key.
And with a master public key (xpub) you are able to derive multiple public keys using a counter (and therefore multiple addresses).

While the master private key (xpriv) would be stored in cold storage, you could derive a new address for each transaction on the online webserver using the xpub.


A customer would not be able to determine the full balance, simply because he only knows that one address which is used for this single transaction.




From a privacy point of view it doesn't matter where your private keys are stored. There is absolutely no difference.
But from a security point of view, it definitely is favorable to store them in cold storage.

I don't see a single disadvantage in directly receiving coins to the cold storage. Except maybe for the fact that it takes a bit longer to access them in case you want to spend them.

No, I get it that the public keys can be known publicly without giving away anything. But can't the sender figure out the address from the tx id? That would expose the cold wallet address itself and make it prone to being monitored.

I can't remember when exactly but I have come across multiple wallet watching websites where anyone can track a wallet's transactions. Hell, if someone's that desperate they can do with any blockchain explorer.

All I am saying is perhaps accepting payments directly into the cold wallet could not be the wisest thing to do.

The xpub would be stored on the web server handling the payment requests.
It wouldn't be publicly available to customer / visitors of the site.

You'd be generating a new receiving address for each customer using this xpub. While the private keys are in cold storage.

This does not have any implications on the privacy compared to receiving coins to a hot wallet and afterwards sending it to the cold storage.


Or what exactly was your concern ? Maybe i misunderstood it ?




The xpub is not an address, but is used to derive public keys (and therefore addresses).
However, i doubt the admint of a shop which accepts crypto could even fall for such a phishing attempt. I mean.. who would be the sender ?
There is no 'official authority' who could request anything. If you are the admin, you are the admin. No one else should ask you for anything.

From a privacy standpoint, is that the best thing to do? If it were my business I'd keep my cold storage out of the public eye and perhaps only reveal it to authorities for taxation requirements.

Otherwise, there can be phishing attempts on the admin email with the cold storage address included in an official-looking email. People have fallen victim to much simpler fraud attempts than this.

It is being handed over to the script as an argument when calling it (%s is the TXID):



Then it depends on the scripting language how to access it.

For a shell script, you would use:

For a php script:


walletnotify will run each time:

• you receive a transaction associated with your wallet
• you send a transaction
• an incoming transaction receives its first confirmation

For X incoming transactions, it will be run 2*X times in total.

Note that you can set up core to be run on the testnet, to see how it exactly behaves and how to make use of it most efficiently.



P.s.
I appreciate that you don't want to use some 3rd party services and want to be in full control over the funds - basically how it is supposed to be.
If you have any concerns or questions, even going further than just setting this up, feel free to ask.
There are a lot of people here who appreciate that engagement and would help you with your questions / concerns.

So when walletnotify runs the script where do I get the transaction ID from to use $TX in gettransaction? What happens if theres multiple transactions at once? Will it just run multiple instances or will $tx be returned as an array?

Well, you'd call the script with the transaction id as a parameter:


And inside of your script (shell, php, whatever..), you could call

to get all necessary data from the transaction.

Afterwards process the data as you need and/or fill your database.


You wouldn'd directly check whether address X received a transaction, but you'd call that script upon receiving any transaction.
Then using gettransaction, you'd get all necessary information (receiving address, amount, etc.. ) for further processing.

So how do I check if that specific address had a transaction? After walletnotify tells the bash script to go, what methods do I use to check specific addresses vs the whole wallets balance?

While i can't speak out of experience, i don't think you'll have performance issues.

walletnotify and blocknotify execute a shell script upon receiving a transaction / block.

You probably won't be able to process thousands of transactions each minute with a raspberry pi, but since the network can't handle so much itself you shouldn't get any problem with a medicore VPS / whatever you are using.

The current amount of transactions the bitcoin network can handle is at roughly 200 per minute.
Besides the fact that i doubt you alone will fill 50% of all blocks, it definitely should be possible without performance issues.

Was most likely going to do this in deployment environment. Was just going to add extra functionality to send to single address if I wanted it stored in one address for whatever reason.


If I have thousands of transactions couldnt that cause performance issues if theres payments in many addresses in very short periods of time? I'm trying to build it to scale from the beginning.

Why don't you use the master public key to derive public keys and addresses while the private keys are in cold storage ?
That way, you would directly accept payments into your cold wallet and would save on transaction fees.




You could use walletnotify to get notified each time the state of your wallet changes (i.e. transaction received, transaction sent, first confirmation on a TX).
That'd be more efficient than having a cronjob checking the addresses each X seconds/minutes.

If you require more than 1 confirmation, you can combine this with blocknotify, to get notified each time core receives a block.
If walletnotify says that you received 1 confirmation on transaction X, you can be sure the next time blocknotify is called, this TX has one more confirmation.

I have an online store and already accept CC payments and I want to start accepting crypto. I'm using the regtest to test the application and so I can't rely on third party APIs plus I feel doing that will hurt at scaling. I'm trying to set it up where each new order will have its own address and I'll have a cronjob periodically check if the address received the payment then send money to cold storage wallet as follows:

Generate address -> receive payment -> wait for confirmations -> send coins to cold storage and show confirmed payment

Currently I save the address in a database associated with the order and generate it using getnewaddress(). I'm stuck on checking the balance of said address and how many confirmations the payment has. I saw accounts looked kind of like what I was looking for but its deprecated so I cant use that. I looked into using blockchain.info JSON API but then with scaling I have to worry about request limits and if I'm trying to request information on too many addresses at once I think it will result in an error. For example, say I have like 4000 orders and the cronjob needs to check all 4000 addresses during the cronjob to get it done efficiently then the addresses that received enough confirmation switch order to paid in database and send coins to cold storage wallet off of server. Plus I'm trying to test it in regtest mode so obviously I cant use a 4rd party API. I saw listunspent() after manually adding addresses and rescanning the chain but I feel if you have thousands of orders and addresses it will be very inefficient every time the job is run and slow the server down.

What is a solution to what I'm trying to accomplish?

I'm very tech savvy and am interested in the be my own bank aspect so I'm not trying to use third party processors or use blockchain.info's wallet API.