Reading the block chain with a library? - page 2.

Pieter Wuille

legendary

Activity: 1072

Merit: 1189

Quote from: Sukrim on April 23, 2012, 08:11:23 PM

Quote from: etotheipi on April 23, 2012, 09:30:11 AM

If you tell me what you're trying to do with it, I can write more directed code samples for you.

I personally want to get the timestamp of blocks with a certain block height once per day, but didn't get around to looking at either libbitcoin or Armory.
Ideally it would be something like "TheBDM.getHeaderByHeight(12345).getTimestamp()"...

libbitcoin or armory's library will surely give you a much more flexible interface, but you *can* just do this using bitcoind as well. See the getblockhash and getblock RPC commands.

etotheipi

legendary

Activity: 1428

Merit: 1093

Core Armory Developer

Quote from: Xenland on April 24, 2012, 07:02:20 PM

Oh yeah I noticed that part of your code, I just assumed that there was a quick way of going about this since it takes a few seconds to scan the block chain for one address and I'm looking for a way to scan and update the database for multiple addresses. Does that sound possible or am I stuck with this route you presented me? Thanks for your help btw I appreciate it.

You can use as many addAddress_1_() calls as you want before a rescan, and the rescan will still take the same amount of time. Load 20 addresses into your wallet before loading the blockchain, and it will take the same time as if you loaded in 10,000.

After the blockchain is loaded, if you add any new addresses to the wallet, the chain will be re-scanned on the "scanBlockchainForTx()" call, and again, it won't matter how many addresses that is, it'll take the same amount of time.

Xenland

legendary

Activity: 980

Merit: 1003

I'm not just any shaman, I'm a Sha256man

Quote from: etotheipi on April 24, 2012, 10:53:52 AM

Quote from: Xenland on April 24, 2012, 12:55:41 AM

How does one just query the balance of a Bitcoin address with armory? is there any calculations involved(like looping over tx history and sum up a balance) or can I just call it and print out a simple balance?

Im' sure a lot of programmers could benefit the solution to this question to write their own software to use armory for like making their own Blockchain.info website or w/e.

The code was in my example, but I guess my example was too long. So here's an extraction of it.

Code:

from armoryengine import *

cppWallet = Cpp.BtcWallet()
cppWallet.addAddress_1_( addrStr_to_hash160('1EbAUHsitefy3rSECh8eK2fdAWTUbpVUDN') ) # addrStr
TheBDM.registerWallet(cppWallet)
BDM_LoadBlockchainFile()
TheBDM.scanBlockchainForTx(cppWallet)

fullBalance = cppWallet.getFullBalance()

print '\n\nBalance of this wallet:', coin2str(fullBalance)

Oh yeah I noticed that part of your code, I just assumed that there was a quick way of going about this since it takes a few seconds to scan the block chain for one address and I'm looking for a way to scan and update the database for multiple addresses. Does that sound possible or am I stuck with this route you presented me? Thanks for your help btw I appreciate it.

etotheipi

legendary

Activity: 1428

Merit: 1093

Core Armory Developer

Quote from: Xenland on April 24, 2012, 12:55:41 AM

How does one just query the balance of a Bitcoin address with armory? is there any calculations involved(like looping over tx history and sum up a balance) or can I just call it and print out a simple balance?

Im' sure a lot of programmers could benefit the solution to this question to write their own software to use armory for like making their own Blockchain.info website or w/e.

The code was in my example, but I guess my example was too long. So here's an extraction of it.

Code:

from armoryengine import *

cppWallet = Cpp.BtcWallet()
cppWallet.addAddress_1_( addrStr_to_hash160('1EbAUHsitefy3rSECh8eK2fdAWTUbpVUDN') ) # addrStr
TheBDM.registerWallet(cppWallet)
BDM_LoadBlockchainFile()
TheBDM.scanBlockchainForTx(cppWallet)

fullBalance = cppWallet.getFullBalance()

print '\n\nBalance of this wallet:', coin2str(fullBalance)

Xenland

legendary

Activity: 980

Merit: 1003

I'm not just any shaman, I'm a Sha256man

How does one just query the balance of a Bitcoin address with armory? is there any calculations involved(like looping over tx history and sum up a balance) or can I just call it and print out a simple balance?

Im' sure a lot of programmers could benefit the solution to this question to write their own software to use armory for like making their own Blockchain.info website or w/e.

etotheipi

legendary

Activity: 1428

Merit: 1093

Core Armory Developer

Quote from: Sukrim on April 23, 2012, 08:11:23 PM

Quote from: etotheipi on April 23, 2012, 09:30:11 AM

If you tell me what you're trying to do with it, I can write more directed code samples for you.

I personally want to get the timestamp of blocks with a certain block height once per day, but didn't get around to looking at either libbitcoin or Armory.
Ideally it would be something like "TheBDM.getHeaderByHeight(12345).getTimestamp()"...

Also for armory I still have to run bitcoind to get the blockchain updated, right?

To use Armory, you need bitcoind running to receive blockchain updates and send&receive transactions. For the example code above, you only need to have the blk0001.dat file produced by the bitcoind, but it doesn't need to be running when you run the script.

Edit: Wow, I must have telepathy. Glad I could answer your question... before you asked it!

Sukrim

legendary

Activity: 2618

Merit: 1007

Quote from: etotheipi on April 23, 2012, 09:30:11 AM

If you tell me what you're trying to do with it, I can write more directed code samples for you.

I personally want to get the timestamp of blocks with a certain block height once per day, but didn't get around to looking at either libbitcoin or Armory.
Ideally it would be something like "TheBDM.getHeaderByHeight(12345).getTimestamp()"...

Also for armory I still have to run bitcoind to get the blockchain updated, right?

Edit:
Wow - this is awesome, you manage to answer my posts before I even post them! Shocked

Edit2:
And I even guessed both function names correctly to the letter! This is getting creepy...

etotheipi

legendary

Activity: 1428

Merit: 1093

Core Armory Developer

Here you go! I just put together a script that not only scans the blockchain, it does something useful! (it's at the bottom of the post)

It took 18 seconds for me to scan the whole blockchain from a cold start
It took 0.2 seconds to collect a list of every difficulty change since the genesis block.
It took 24 seconds on my system to count 3,532,497 unique addresses in the blockchain! (as of block 176953)

Timings will vary depending on RAM. If you have a lot of RAM, just about the whole blockchain will be cached from the first scan, and rescans will be nearly instantaneous. If you want to avoid rescans, make sure everything is added to your wallet and registered with TheBDM.registerWallet before BDM_LoadBlockChain().

The sample python script at the bottom of this post produced the following output (the difficulty changes):

Code:

Collect all difficulty changes...
   Block Diff Date
   0 1.0 2009-Jan-03 01:15pm
   32256 1.2 2009-Dec-30 01:11am
   34272 1.3 2010-Jan-11 05:48pm
   36288 1.3 2010-Jan-25 08:07am
   38304 1.8 2010-Feb-04 04:43pm
   40320 2.5 2010-Feb-14 06:52pm
   42336 3.8 2010-Feb-24 03:41am
   44352 4.5 2010-Mar-07 08:14pm
   46368 4.6 2010-Mar-21 06:54pm
   48384 6.1 2010-Apr-01 07:07am
   50400 7.8 2010-Apr-12 04:39am
   52416 11.5 2010-Apr-21 05:52pm
   54432 12.8 2010-May-04 05:46am
   56448 11.8 2010-May-19 10:13am
   58464 16.6 2010-May-29 09:57am
   60480 17.4 2010-Jun-11 07:26pm
   62496 19.4 2010-Jun-24 08:27am
   64512 23.5 2010-Jul-05 09:57pm
   66528 45.4 2010-Jul-13 04:03am
   68544 181.5 2010-Jul-16 12:29pm
   70560 244.2 2010-Jul-26 10:42pm
   72576 352.2 2010-Aug-05 03:46pm
   74592 511.8 2010-Aug-15 07:11am
   76608 623.4 2010-Aug-26 07:13pm
   78624 712.9 2010-Sep-08 01:04am
   80640 917.8 2010-Sep-18 10:04pm
   82656 1318.7 2010-Sep-28 03:58pm
   84672 1378.0 2010-Oct-12 01:35am
   86688 2149.0 2010-Oct-21 01:13am
   88704 3091.7 2010-Oct-30 06:58pm
   90720 4536.4 2010-Nov-09 07:29am
   92736 6866.9 2010-Nov-18 01:44pm
   94752 8078.2 2010-Nov-30 11:37am
   96768 12252.0 2010-Dec-09 05:20pm
   98784 14484.2 2010-Dec-21 01:34pm
   100800 16307.4 2011-Jan-03 12:10am
   102816 18437.6 2011-Jan-15 09:26am
   104832 22012.4 2011-Jan-27 03:16am
   106848 25997.9 2011-Feb-07 11:53pm
   108864 36459.9 2011-Feb-18 12:15am
   110880 55589.5 2011-Feb-27 04:59am
   112896 76192.6 2011-Mar-09 10:25am
   114912 68977.8 2011-Mar-24 10:39pm
   116928 82345.6 2011-Apr-05 04:09pm
   118944 92347.6 2011-Apr-18 03:49am
   120960 109670.1 2011-Apr-29 10:53pm
   122976 157416.4 2011-May-09 05:17pm
   124992 244112.5 2011-May-18 06:04pm
   127008 434877.0 2011-May-26 02:41pm
   129024 567269.5 2011-Jun-06 08:25am
   131040 876954.5 2011-Jun-15 09:49am
   133056 1379192.3 2011-Jun-24 07:45am
   135072 1563028.0 2011-Jul-06 04:35pm
   137088 1690895.8 2011-Jul-19 03:23pm
   139104 1888786.7 2011-Aug-01 04:11am
   141120 1805700.8 2011-Aug-15 07:44pm
   143136 1777774.5 2011-Aug-30 01:15am
   145152 1755425.3 2011-Sep-13 05:31am
   147168 1689334.4 2011-Sep-27 06:47pm
   149184 1468195.4 2011-Oct-13 09:44pm
   151200 1203461.9 2011-Oct-30 11:42pm
   153216 1192497.8 2011-Nov-14 01:56am
   155232 1090715.7 2011-Nov-29 09:20am
   157248 1155038.3 2011-Dec-12 02:42pm
   159264 1159929.5 2011-Dec-26 01:43pm
   161280 1250757.7 2012-Jan-08 01:26pm
   163296 1307728.4 2012-Jan-21 10:55pm
   165312 1379647.4 2012-Feb-04 05:32am
   167328 1376302.3 2012-Feb-18 06:24am
   169344 1496978.6 2012-Mar-02 03:25am
   171360 1498294.4 2012-Mar-16 04:09am
   173376 1626553.5 2012-Mar-29 01:41am
   175392 1577913.5 2012-Apr-12 12:04pm
Took 0.2 seconds to collect difficulty list

And here is the script itself. It illustrates a variety of ways you can access block data. Mainly, scanning for addresses in the blockchain with balances and unspent outputs, and walking through every TxOut of every Tx of every Block -- we grab the address from ever standard TxOut and add it to a set() object which only allows unique addresses.

Code:

from armoryengine import *

# NOTE:
# ALL ADDRESSES THROUGHOUT EVERYTHING ARE IN 20-BYTE BINARY FORM (hash160/addr20)
# Use hash160_to_addrStr() and addrStr_to_hash160() to convert...

print '\n\nCreating a new C++ wallet, add a few addresses...'
cppWallet = Cpp.BtcWallet()
cppWallet.addAddress_1_( hex_to_binary('11b366edfc0a8b66feebae5c2e25a7b6a5d1cf31') ) # hash160
cppWallet.addAddress_1_( addrStr_to_hash160('1EbAUHsitefy3rSECh8eK2fdAWTUbpVUDN') ) # addrStr
cppWallet.addAddress_1_('\x1b~\xa7*\x85\t\x12\xb7=\xd4G\xf3\xbd\xc1\x00\xf1\x00\x8b\xde\xb0') # binary

print 'Addresses in this wallet:'
for i in range(cppWallet.getNumAddr()):
   print '\t', hash160_to_addrStr(cppWallet.getAddrByIndex(i).getAddrStr20())

print '\n\nRegistering the wallet with the BlockDataManager & loading...'
start = RightNow()
TheBDM.registerWallet(cppWallet)
BDM_LoadBlockchainFile() # optional argument to specify blk0001.dat location
print 'Loading blockchain took %0.1f sec' % (RightNow() - start)

topBlock = TheBDM.getTopBlockHeight()
print '\n\nCurrent Top Block is:', topBlock
TheBDM.getTopBlockHeader().pprint()

# Add new addresses -- will rescan (which will be super fast if you ahve a lot of RAM)
cppWallet.addAddress_1_( hex_to_binary('0cdcd0f388a31b11ff11b1d8d7a9f978b37bc7af') )
TheBDM.scanBlockchainForTx(cppWallet)

print '\n\nBalance of this wallet:', coin2str(cppWallet.getSpendableBalance())
print 'Unspent outputs:'
unspentTxOuts = cppWallet.getSpendableTxOutList(topBlock)
for utxo in unspentTxOuts:
   utxo.pprintOneLine()

print '\n\nTransaction history of this wallet:'
ledger = cppWallet.getTxLedger()
for le in ledger:
   le.pprintOneLine()

print '\n\n'
print '-'*80
print 'Now for something completely different...'
start = RightNow()
print '\n\nCollect all difficulty changes...'
prevDiff = 0
for h in xrange(0,topBlock+1):
   header = TheBDM.getHeaderByHeight(h)
   currDiff = header.getDifficulty()
   if not prevDiff==currDiff:
   print str(h).rjust(10),
   print ('%0.1f'%currDiff).rjust(10),
   print '\t',unixTimeToFormatStr(header.getTimestamp())
   prevDiff = currDiff

print 'Took %0.1f seconds to collect difficulty list' % (RightNow()-start)


print '\n\nCount the number of unique addresses in the blockchain'
start = RightNow()
allAddr = set()
for h in xrange(0,topBlock+1):
   if h%10000 == 0:
   print '\tScanned %d blocks' % h

   header = TheBDM.getHeaderByHeight(h)
   txList = header.getTxRefPtrList()
   for tx in txList:
   for nout in range(tx.getNumTxOut()):
   txout = tx.getTxOutRef(nout)
   if txout.isStandard():
   allAddr.add(txout.getRecipientAddr())

print 'Took %0.1f seconds to count all addresses' % (RightNow()-start)
print 'There are %d unique addresses in the blockchain!' % len(allAddr)

This scanning is lower-level than what happens in Armory, so you are mainly using C++ objects. You can find a full list of all the methods in BlockObjRef.h and BlockUtils.h. Please ask questions if something isn't clear, or if you want to use some functionality not exemplified here.

Xenland

legendary

Activity: 980

Merit: 1003

I'm not just any shaman, I'm a Sha256man

Quote from: etotheipi on April 23, 2012, 09:30:11 AM

Armory is perfect this. It is C++ accessed in Python (via SWIG). You only ever touch python.

I'm not at my development computer now but there is example code on my github page (though many of the examples were created before RAM reduction, they will mostly still work)

Code:

from armoryengine import *

   print '\n\nLoading Blockchain from:', BLK0001_PATH
   BDM_LoadBlockchainFile() # optional argument to specify blk0001.dat location
   print 'Done!'

   print '\n\nCurrent Top Block is:', TheBDM.getTopBlockHeader().getBlockHeight()
   TheBDM.getTopBlockHeader().pprint()

   tx = TheBDM.getTxByHash( hex_to_binary('9c633b5689e462ddf3d52a6edc64226cedd1e1749d0b8e2f70cd9550bfa74c72') )
   tx.pprint()

   # Create and register the wallet before the LoadBlockchainFile call, to merge the scans
   # For now, this will rescan the blockchain
   cppWallet = Cpp.BtcWallet()
   cppWallet.addAddress_1_( hex_to_binary('9c633b56899c633b56899c633b56899c633b5689') ) # hash160
   TheBDM.registerWallet(cppWallet)
   TheBDM.scanBlockchainForTx(cppWallet)
   print cppWallet.getSpendableBalance()

   unspentTxOuts = cppWallet.getSpendableTxOutList()
   for utxo in unspentTxOuts:
   utxo.pprintOneLine()

(note, this code is notional since I can't test it where I am, but the syntax is nearly correct)

When I get home later, I can post more example code for how to access what you want. If you tell me what you're trying to do with it, I can write more directed code samples for you.

Nice I heard about armory but I thought it was just a GUI client with advanced features. I'll try it out with python see how it is.

etotheipi

legendary

Activity: 1428

Merit: 1093

Core Armory Developer

Armory is perfect this. It is C++ accessed in Python (via SWIG). You only ever touch python.

I'm not at my development computer now but there is example code on my github page (though many of the examples were created before RAM reduction, they will mostly still work)

Code:

from armoryengine import *

   print '\n\nLoading Blockchain from:', BLK0001_PATH
   BDM_LoadBlockchainFile() # optional argument to specify blk0001.dat location
   print 'Done!'

   print '\n\nCurrent Top Block is:', TheBDM.getTopBlockHeader().getBlockHeight()
   TheBDM.getTopBlockHeader().pprint()

   tx = TheBDM.getTxByHash( hex_to_binary('9c633b5689e462ddf3d52a6edc64226cedd1e1749d0b8e2f70cd9550bfa74c72') )
   tx.pprint()

   # Create and register the wallet before the LoadBlockchainFile call, to merge the scans
   # For now, this will rescan the blockchain
   cppWallet = Cpp.BtcWallet()
   cppWallet.addAddress_1_( hex_to_binary('9c633b56899c633b56899c633b56899c633b5689') ) # hash160
   TheBDM.registerWallet(cppWallet)
   TheBDM.scanBlockchainForTx(cppWallet)
   print cppWallet.getSpendableBalance()

   unspentTxOuts = cppWallet.getSpendableTxOutList()
   for utxo in unspentTxOuts:
   utxo.pprintOneLine()

(note, this code is notional since I can't test it where I am, but the syntax is nearly correct)

When I get home later, I can post more example code for how to access what you want. If you tell me what you're trying to do with it, I can write more directed code samples for you.

Xenland

legendary

Activity: 980

Merit: 1003

I'm not just any shaman, I'm a Sha256man

Libbitcoin is quite confusing even looking through the source code, are there any alternatives to libbitcoin that could help me query the blockchain?

kokjo

legendary

Activity: 1050

Merit: 1000

You are WRONG!

Quote from: Xenland on April 23, 2012, 12:54:10 AM

So how do I use this libbitcoin thing, for example something simple like a balance of a particular address, or a list of details of a particular address? (with python)

don't know, try read the api... im not a libbitcoin guru.

Xenland

legendary

Activity: 980

Merit: 1003

I'm not just any shaman, I'm a Sha256man

So how do I use this libbitcoin thing, for example something simple like a balance of a particular address, or a list of details of a particular address? (with python)

Xenland

legendary

Activity: 980

Merit: 1003

I'm not just any shaman, I'm a Sha256man

Nice responses mates, libbitcoin should work fine for what i need

ThomasV

legendary

Activity: 1896

Merit: 1353

Quote from: arby on April 22, 2012, 02:07:40 AM

libbitoin as i see is in cpp, not php or java, python as the OP mentioned.

yes, it has a python frontend

arby

donator

Activity: 112

Merit: 10

keybase.io/arblarg

Quote from: kokjo on April 22, 2012, 02:09:42 AM

Quote from: arby on April 22, 2012, 02:07:40 AM

libbitoin as i see is in cpp, not php or java, python as the OP mentioned.

I thin he is interested in using this on linux, and not using any executables.

swig!

Ok I will give that a try also, thanks for the info

kokjo

legendary

Activity: 1050

Merit: 1000

You are WRONG!

Quote from: arby on April 22, 2012, 02:07:40 AM

libbitoin as i see is in cpp, not php or java, python as the OP mentioned.

I thin he is interested in using this on linux, and not using any executables.

swig!

arby

donator

Activity: 112

Merit: 10

keybase.io/arblarg

libbitoin as i see is in cpp, not php or java, python as the OP mentioned.

I think he is interested in using this on linux, and not using any executables.

I am searching for this also, but have not found anything yet.

What I found that you can reliably use with PHP is:

https://blockchain.info/api

https://blockchain.info/q

To explore the blockchain.

kokjo

legendary

Activity: 1050

Merit: 1000

You are WRONG!

libbitcoin: https://github.com/libbitcoin/libbitcoin

ThomasV

legendary

Activity: 1896

Merit: 1353

Quote from: Xenland on April 22, 2012, 01:54:57 AM

How do I read the Bitcoin blockchain with an api or library with a programming language like php, python or java?

use libbitcoin

Topic: Reading the block chain with a library? - page 2. (Read 9753 times)