[ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff - page 41.

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: tiker on February 13, 2012, 10:26:12 AM

Does Abe re-organize blocks when it re-scans a block chain?

[...]

Will adding blocks in the middle of a chain cause any problems with the block order for Abe?

I know of one, possibly two bugs affecting this situation. First, Abe sometimes crashes by exceeding Python's recursion limit when rescanning. This does not produce wrong data, and I work around it by deleting rows from chain_candidate, updating chain.chain_last_block_id, and retrying.

I have noticed slight deviations in statistical values like Coin Days Destroyed from one Abe instance to another. (The values on the homepage depend on server system time, but on a given block's page they should be identical.) I suspect rescanning may create statistical discrepancies. The ones I've seen were slight (under 1%).

I do not know of any bugs affecting address balances, transaction links, etc. But I can offer no guarantee.

eurekafag

full member

Activity: 185

Merit: 100

John Tobey, well, the type was "int". I did as you said and now it passed that block. Thanks! I left a comment on the config line, seems like a mistype occured.

John Tobey

hero member

Activity: 481

Merit: 529

eurekafag:

Yes, you are right about not needing sqlite2.

Please run "sqlite3 abe.sqlite" from the command line and issue

Code:

select configvar_value from configvar where configvar_name = 'int_type';

The output should be "str" indicating that the workaround for this SQLite limitation is in effect. (A result of integer arithmetic is too big for 64 bits. The workaround forces the use of floating point.)

If int_type is not "str", please update via git where I've added support for --int-type=str. Delete abe.sqlite and abe.sqlite-journal and retry with int-type=str in your config.

tiker

sr. member

Activity: 459

Merit: 250

Does Abe re-organize blocks when it re-scans a block chain?

I had a problem importing the LiquidCoin blockchain because the linux liquidcoind couldn't download the chain properly. It would get to block 1000 then complain about invalid blocks. Abe was showing 1500 at this point though.

I grabbed the entire data directory from a windows download which was up-to-date, moved it over to the linux box and restarted the linux client using the windows data. After I cleared the offset value in Abe's DB and relaunched the import it filled in missing blocks. Here's an example:

Code:

commit
block 462660 already in chain 12
commit
block 462661 already in chain 12
commit
block 462662 already in chain 12
commit
block 462663 already in chain 12
commit
block 462664 already in chain 12
commit
block_tx 1464778 4137316
commit
block 462665 already in chain 12
commit
block 462666 already in chain 12

Will adding blocks in the middle of a chain cause any problems with the block order for Abe?

The site for this install is http://blockexplorer.funkymonkey.org if you want to look. (It is slow but that's due to older hardware.)

eurekafag

full member

Activity: 185

Merit: 100

Quote from: John Tobey on February 13, 2012, 09:54:34 AM

Please list your versions of Abe, pysqlite, and SQLite. I could not produce the error using Abe latest, pysqlite-2.6.0, and libsqlite3 3.7.4-2ubuntu5.

I'm using Debian wheezy and here are versions of my libs:
Abe (from git): commit dd07123fb1b5cb6c633c08d90366b879b3cb8ce9
pysqlite - ?
libsqlite3-0 - 3.7.9-2
I didn't find pysqlite and python-sqlite isn't installed either (and it shouldn't be because it's for sqlite2). But it works anyway because of the file /usr/lib/python2.6/lib-dynload/_sqlite3.so which belongs to python2.6 package.

I've tried your command but it resulted in the same error. Installing python-pysqlite2 or python-pysqlite1.1 doesn't help either. The version of my python bindings is 1.1.8:

Code:

Python 2.7.2+ (default, Nov 30 2011, 19:22:03) 
[GCC 4.6.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite
>>> sqlite.version
'1.1.8'
>>>

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: eurekafag on February 13, 2012, 09:25:00 AM

John Tobey, what about my case? Why am I getting this exception and what can I do to break through that block?

This error takes more thought. I got this error with SQLite and thought I'd worked around it in the latest code. Please list your versions of Abe, pysqlite, and SQLite. I could not produce the error using Abe latest, pysqlite-2.6.0, and libsqlite3 3.7.4-2ubuntu5. I tried this command, and it succeeds past block_tx 2261 2291:

Code:

python -m Abe.abe --dbtype sqlite3 --commit-bytes 10000 --connect-args test.sqlite

Note that SQLite will slow down horribly if you make it to block 70000+. Another database engine such as PostgreSQL or MySQL will not have the same error, which is specific to SQLite.

eurekafag

full member

Activity: 185

Merit: 100

John Tobey, what about my case? Why am I getting this exception and what can I do to break through that block?

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: andrehorta on February 12, 2012, 07:04:14 PM

Sorry Jonh,

But for me it´s not clear. I´m trying to make a query, to know how much some addess sent to another address, but i´m not understanding the database. For exemple, i would like to know how much the address 12cbQLTFMXRnSzktFkuoG3eHoMeFtpTu3S sent to 1Q2TWHE3GMdB6BZKafqwxXtWAWgFt5Jvm3.

I think this will do it. Note that the question is a little ambiguous, since a transaction can have more than one sending address. I'll count every transaction where the given address is among the senders. This will unfortunately double-count transactions where it appears in two inputs. But at least it should illustrate the table relationships.

Note the hashes of the two addresses:
sender = 12cbQLTFMXRnSzktFkuoG3eHoMeFtpTu3S = 11B366EDFC0A8B66FEEBAE5C2E25A7B6A5D1CF31
receiver = 1Q2TWHE3GMdB6BZKafqwxXtWAWgFt5Jvm3 = FC916F213A3D7F1369313D5FA30F6168F9446A2D

Note that Abe stores hashes as lowercase hex strings when it does not figure out how to use a binary type. Only SQLite has a binary type that Abe can use, so for SQLite instead of LOWER('hex string') you would put X'hex string', IIRC. (To force use of hex, pass --binary-type=hex when first creating the tables.)

Code:

SELECT SUM(txout.txout_value) / 100000000 AS btc
FROM txout
JOIN txin ON (txout.tx_id = txin.tx_id)
JOIN txout prevout ON (txin.txout_id = prevout.txout_id)
JOIN pubkey addr_out ON (addr_out.pubkey_id = txout.pubkey_id)
JOIN pubkey addr_in ON (addr_in.pubkey_id = prevout.pubkey_id)
WHERE addr_in.pubkey_hash = LOWER('11B366EDFC0A8B66FEEBAE5C2E25A7B6A5D1CF31')
AND addr_out.pubkey_hash = LOWER('FC916F213A3D7F1369313D5FA30F6168F9446A2D');

andrehorta

legendary

Activity: 1261

Merit: 1000

Sorry Jonh,

But for me it´s not clear. I´m trying to make a query, to know how much some addess sent to another address, but i´m not understanding the database. For exemple, i would like to know how much the address 12cbQLTFMXRnSzktFkuoG3eHoMeFtpTu3S sent to 1Q2TWHE3GMdB6BZKafqwxXtWAWgFt5Jvm3.

Quote from: John Tobey on February 12, 2012, 03:28:26 PM

Quote from: andrehorta on February 11, 2012, 02:18:46 PM

How can i get the destination address of the transaction? Because, only the table txout has foreign key to the table pubkey. The table txin don´t has any indication of the destination address.

txout.pubkey_id receives coins in transaction txout.tx_id.

txin.txout_id points to txout.pubkey_id that sends coins in txin.tx_id.

Is this clear?

John Tobey

hero member

Activity: 481

Merit: 529

someguy123,

I'm afraid I have little time to devote to new features--or much of anything beyond explaining how things work. Patches are welcome. I suggest starting a thread about the design of such a system--or integration into existing Python template systems, about which I know nothing.

-John

someguy123

sr. member

Activity: 336

Merit: 254

CEO of Privex Inc. (www.privex.io)

John, I hope you realize your template system is terrible, my python errors out if I enable it
Last time I managed to get it to work... I sat for at least an hour or two trying to figure out how to get it to work.
Since then I've never managed it again, Is there any chance that you're going to make the template system USABLE? It's a PITA to even add my donation addresses

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: andrehorta on February 11, 2012, 02:18:46 PM

How can i get the destination address of the transaction? Because, only the table txout has foreign key to the table pubkey. The table txin don´t has any indication of the destination address.

txout.pubkey_id receives coins in transaction txout.tx_id.

txin.txout_id points to txout.pubkey_id that sends coins in txin.tx_id.

Is this clear?

eurekafag

full member

Activity: 185

Merit: 100

I'm stuck on block 2261:

Code:

2012-02-12 13:08:28,659 [4554:MainThread] DataStore INFO - block_tx 2259 2289
2012-02-12 13:08:28,660 [4554:MainThread] DataStore INFO - block_tx 2260 2290
2012-02-12 13:08:28,661 [4554:MainThread] DataStore INFO - block_tx 2261 2291
2012-02-12 13:08:28,661 [4554:MainThread] DataStore ERROR - Failed to catch up {'blkfile_number': 1, 'dirname': '/tmp/ramdisk', 'chain_id': None, 'id': 1, 'blkfile_offset': 515807}
Traceback (most recent call last):
  File "/home/eurekafag/svn-soft/bitcoin-abe/Abe/DataStore.py", line 2141, in catch_up
    store.catch_up_dir(dircfg)
  File "/home/eurekafag/svn-soft/bitcoin-abe/Abe/DataStore.py", line 2162, in catch_up_dir
    store.import_blkdat(dircfg, ds)
  File "/home/eurekafag/svn-soft/bitcoin-abe/Abe/DataStore.py", line 2277, in import_blkdat
    store.import_block(b, chain_ids = chain_ids)
  File "/home/eurekafag/svn-soft/bitcoin-abe/Abe/DataStore.py", line 1609, in import_block
    block_id))
  File "/home/eurekafag/svn-soft/bitcoin-abe/Abe/DataStore.py", line 403, in sql
    store.cursor.execute(cached, params)
OverflowError: long too big to convert
2012-02-12 13:08:28,662 [4554:MainThread] __main__ INFO - Abe initialized.
2012-02-12 13:08:28,663 [4554:MainThread] __main__ WARNING - Listening on http://localhost:2750

My config:

Code:

dbtype = sqlite3
connect-args = abe.sqlite
port 2750
host localhost
datadir = /tmp/ramdisk
template = "



         href=\"%(dotdot)s%(STATIC_PATH)sabe.css\" />
    
    %(title)s


         src=\"%(dotdot)s%(STATIC_PATH)slogo32.png\" alt=\"Abe logo\" /> %(h1)s
    

    %(body)s
    
        
            Powered by %(APPNAME)s
        
        Tips appreciated!
        BTC
        NMC
    



"
commit-bytes = 10000
logging = {
    "version":1,
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "formatter": "full",
            "level": "DEBUG"}},
    "formatters": {
        "full": {
            "format": "%(asctime)s [%(process)d:%(threadName)s] %(name)s %(levelname)s - %(message)s"}},
   "root": {
        "handlers": ["console"],
        "level": "DEBUG"}}

andrehorta

legendary

Activity: 1261

Merit: 1000

I´m not using the webserver or the python, i´m using the database.

Ok, resolved. But i have another question:

How can i get the destination address of the transaction? Because, only the table txout has foreign key to the table pubkey. The table txin don´t has any indication of the destination address.

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: someguy123 on February 10, 2012, 03:26:12 PM

Scanning the block chain... but not starting the web server til it thinks its done...
There's a huge problem with that, it seems on a lot of systems, ABE takes a LOOONG time to scan blockchains of various whateverCoins, and during that time, it refuses to start the web server til it's done =_=
It'd be better if it would start the web server FIRST so that the users of an ABE block explorer don't have to wait HOURS for ANYTHING AT ALL to work, if the admin decides to add a new block chain...

Have you tried scanning with one process and serving with another?

Code:

python -m Abe.abe --config db.conf --config datadirs.conf --no-serve &
python -m Abe.abe --config db.conf --config server.conf --datadir ~/emptydir

Not as elegant as an option to do it all in one, I'll grant.

someguy123

sr. member

Activity: 336

Merit: 254

CEO of Privex Inc. (www.privex.io)

Scanning the block chain... but not starting the web server til it thinks its done...
There's a huge problem with that, it seems on a lot of systems, ABE takes a LOOONG time to scan blockchains of various whateverCoins, and during that time, it refuses to start the web server til it's done =_=
It'd be better if it would start the web server FIRST so that the users of an ABE block explorer don't have to wait HOURS for ANYTHING AT ALL to work, if the admin decides to add a new block chain...

John Tobey

hero member

Activity: 481

Merit: 529

Sorry, I think I confused you by linking to the algorithm going from hash to address.

To go from address to hash, you would reverse it: decode base58, discard last 4 bytes (checksum), discard first byte (version), and hex-encode the result.

If you want the date/time a transaction appeared in the chain, it is block.block_nTime (seconds since 0:00 1 Jan 1970 UTC). You find the block row by joining tx to block_tx to block to chain_candidate, restricting in_longest=1 and chain_id=1 (for BTC). Note that date is not a property of transactions. (tx has a lock_time but it is unused.) A transaction doesn't say when it was signed, and it doesn't know when it will become part of the chain. The joins could produce different results at different times because of reorganizations, which change the value of chain_candidate.in_longest.

andrehorta

legendary

Activity: 1261

Merit: 1000

Dears

How can i get the Date and Time of transaction?

Thank you

andrehorta

legendary

Activity: 1261

Merit: 1000

Why it´s not working?

I´m whating for (e5ec4ef0b857699ca315ea0a05645691a92369ff) but i´m seeing 5eFYXDVERjk3BfotTnR1SA5BD4sTzWTwMB7tTTbcDYU7

string sSourceData;
byte[] tmpSource;
byte[] tmpHash;
string final;
sSourceData = "1MxipvUHfKaZzWVrCGhrLKVPtNxwESRoq2";

tmpSource = BitCoinSharp.Base58.Decode(sSourceData);

System.Security.Cryptography.HMACSHA256 h = new HMACSHA256();
tmpHash = h.ComputeHash(tmpSource);
final = BitCoinSharp.Base58.Encode(tmpHash);

System.Security.Cryptography.RIPEMD160 r160 = new RIPEMD160Managed();
byte[] tmpHash2 = r160.ComputeHash(tmpHash);
final = BitCoinSharp.Base58.Encode(tmpHash);

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: andrehorta on February 09, 2012, 11:12:20 AM

OK, thank you! I understand.

How can i convert the address to a public key hash?

abe.py: decode_check_address or decode_address. They return two values: address_version and pubkey_hash. The "version" should be "\0" for BTC.

Also there is base58.py: bc_address_to_hash_160 returning only the hash.

Here's the algorithm in pseudocode: https://en.bitcoin.it/wiki/Protocol_specification#Addresses

Topic: [ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff - page 41. (Read 220986 times)

src=\"%(dotdot)s%(STATIC_PATH)slogo32.png\" alt=\"Abe logo\" /> %(h1)s