[ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff - page 40.

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: tiker on February 26, 2012, 08:46:07 PM

So I'm stuck at block 42156 on the testnet for now then? It doesn't sound like a quick upgrade / change to Abe.

To be compatible with Testnet, but incompatible with Namecoin and chains that use its proof-of-work format, remove or comment out these lines in Abe/deserialize.py, lines 175-176 in the latest commit:

Code:

  if d['version'] & (1 << 8):
    d['auxpow'] = parse_AuxPow(vds)

To be compatible with both, until I have a better fix, I suggest you run this query:

Code:

SELECT DISTINCT c.chain_name, b.block_version
FROM block b
JOIN chain_candidate cc ON (b.block_id = cc.block_id)
JOIN chain c ON (cc.chain_id = c.chain_id)

You will see something like:

Code:

 chain_name | block_version 
------------+---------------
 NcTestnet  |         65537
 NcTestnet  |             1
 Namecoin   |             1
 Namecoin   |         65537
 Namecoin   |         65793
 Bitcoin    |             1
 NcTestnet  |         65793

Make a list of the values under block_version that belongs to a network with Namecoin-style merged mining. For the above output, the list would be (1, 65537, 65793) corresponding to Namecoin and NcTestnet. Then add " and d['version'] in (LIST)" after "d['version'] & (1 << Cool

" on line 175, e.g.:

Code:

  if (d['version'] & (1 << 8)) and d['version'] in (1, 65537, 65793):
    d['auxpow'] = parse_AuxPow(vds)

Let us know how it goes.

tiker

sr. member

Activity: 459

Merit: 250

So I'm stuck at block 42156 on the testnet for now then? It doesn't sound like a quick upgrade / change to Abe.

John Tobey

hero member

Activity: 481

Merit: 529

The testnet contains a block with an unusual value in the version field. Abe thinks it is a merged-mining block. I mean to provide a workaround, especially since the main BTC chain is vulnerable to it, but haven't got to it yet. Abe will need a new kind of chain metadata with flags for features such as merged mining, rather than rely on a universal interpretation of block version.

tiker

sr. member

Activity: 459

Merit: 250

I've run into a new problem. Trying to import the bitcoin testnet, was working fine then this came up...

Code:


Failed to catch up {'blkfile_number': 1, 'dirname': '/home/ed/.bitcoin/testnet', 'chain_id': 20, 'id': Decimal('95'), 'blkfile_offset': 30108327}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2146, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2167, in catch_up_dir
    store.import_blkdat(dircfg, ds)
  File "Abe/DataStore.py", line 2279, in import_blkdat
    b = deserialize.parse_Block(ds)
  File "Abe/deserialize.py", line 176, in parse_Block
    d['auxpow'] = parse_AuxPow(vds)
  File "Abe/deserialize.py", line 153, in parse_AuxPow
    d = parse_MerkleTx(vds)
  File "Abe/deserialize.py", line 103, in parse_MerkleTx
    d = parse_Transaction(vds)
  File "Abe/deserialize.py", line 86, in parse_Transaction
    d['txIn'].append(parse_TxIn(vds))
  File "Abe/deserialize.py", line 45, in parse_TxIn
    d['sequence'] = vds.read_uint32()
  File "Abe/BCDataStream.py", line 71, in read_uint32
    def read_uint32(self): return self._read_num('  File "Abe/BCDataStream.py", line 110, in _read_num
    (i,) = struct.unpack_from(format, self.input, self.read_cursor)
OverflowError: Python int too large to convert to C long

Other chains continue to import fine so far from what I can tell. It seems to be just this current spot in this chain.

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: tiker on February 22, 2012, 09:42:56 AM

Edit: Now that I look at it a bit more.. I think it is an orphaned block but also a bug where the block version code isn't being applied to properly show the correct addresses.

Correct you are.

tiker

sr. member

Activity: 459

Merit: 250

Another question here...

http://blockexplorer.funkymonkey.org/block/00000000bc06d31f6fc31e99e3be7adfbc47c04e9bf56cddf1accda6cac69529

What does it mean when there are 2 "next block" links, one looking to be a valid block and 1 appearing to be part of a different chain? I'm would take a guess and say 1 got orphaned and the other was accepted but the incorrect "to" address makes it look like it's part of another chain.

Edit: Now that I look at it a bit more.. I think it is an orphaned block but also a bug where the block version code isn't being applied to properly show the correct addresses.

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: kstepyra on February 17, 2012, 09:05:47 PM

had to quit app and rerun it and after using ctrl-c i lack one record :

More likely it seems to me, you have one unassigned block_id. This is not the same as a missing block. block_id is an internal identifier, not visible in any HTML or API output. So it doesn't really matter what the block_id is. It will vary from one Abe instance to another for the same block. Abe allocated a block_id but hadn't used it when you stopped it, so it remains unused. This is not a problem.

By the way, the visible "block number" is called "height" internally. Often height and block_id are numerically close, depending on how you load the data, but they will diverge if, for example, your block file contains side chains or you load Testnet or alt chain data along with the main BTC block chain.

Quote from: kstepyra on February 17, 2012, 09:05:47 PM

Is here any possibility to recheck for missing data?

It would be nice to have a validation tool that checks for missing data. We don't have that. Part of it is covered by the foreign key constraints, and there is a tool (Abe/verify.py) that verifies the transaction Merkle hashes, but it could do much more.

If you haven't done anything like delete table rows, and barring any bugs, if Abe shows a block number, it has all data from the genesis through that block.

Quote from: kstepyra on February 17, 2012, 09:05:47 PM

Another thing - when i have bitcoin running - can i rerun app to check for new blocks, or i have to stop bitcoin?

I have never noticed a problem running both at once.

Quote from: kstepyra on February 17, 2012, 09:05:47 PM

And btw - how is this tool different from Bitcointools ? i am new to this tools, was using crappy json-rpc methods given in bitcoind : p

Abe borrows some low-level code from Bitcointools and adds many features, notably SQL database support, speed improvements through indexing, and the HTML interface similar to Bitcoin Block Explorer.

kstepyra

newbie

Activity: 52

Merit: 0

Hi there,

I've started using abe(initial check right now

), had to quit app and rerun it and after using ctrl-c i lack one record :

Code:

mysql> SELECT count(*) FROM `block` WHERE block_id=116245;
+----------+
| count(*) |
+----------+
|        1 |
+----------+
1 row in set (0.00 sec)

mysql> SELECT count(*) FROM `block` WHERE block_id=116246;
+----------+
| count(*) |
+----------+
|        0 |
+----------+
1 row in set (0.00 sec)

mysql> SELECT count(*) FROM `block` WHERE block_id=116247;
+----------+
| count(*) |
+----------+
|        1 |
+----------+
1 row in set (0.00 sec)

commit should prevent that, right? i am ussing InnoDB.

Is here any possibility to recheck for missing data? EDIT: commits are only for tx's so maybe just wrong id given while selecting last block in app after starting it? like 116247 instead of 116246 while having 116245 put in mysqlDb?

Another thing - when i have bitcoin running - can i rerun app to check for new blocks, or i have to stop bitcoin?

and most important: how is this tool different from Bitcointools ? i am new to this tools, was using crappy json-rpc methods given in bitcoind : p

tiker

sr. member

Activity: 459

Merit: 250

Quote from: John Tobey on February 15, 2012, 12:30:35 PM

Quote from: tiker on February 15, 2012, 12:20:24 PM

I've seen somewhere an Abe install with the SolidCoin2 chain but I can't seem to get it going on my install.

Code:

  File "Abe/DataStore.py", line 1493, in import_block
    raise MerkleRootMismatch(b['hash'], tx_hash_array)
MerkleRootMismatch: Block header Merkle root does not match its transactions. block hash=725ca2009ba1b949a820d3a48511dc21c7f8d76d06d8afb6abb5988f5a911a8b

I won't be surprised if SC2 has changed the Merkle tree hashing algorithm. If you trust your copy of the block chain, you may get around this by simply deleting these lines from Abe/DataStore.py:

Code:

        # Verify Merkle root.
        if b['hashMerkleRoot'] != util.merkle(tx_hash_array):
            raise MerkleRootMismatch(b['hash'], tx_hash_array)

A long-term solution would be to add support for different algorithms per chain, or at least an option to skip the verification in a given chain.

Just an FYI, after doing some checking it turns out the block chain for SC2 isn't downloading correctly and is getting stuck. The windows client is stuck at block 17999 (like the other posts on this board) so I can't trust anything coming down for SC2 at the moment. I'm leaving the above lines in the code.

Thanks.

John Tobey

hero member

Activity: 481

Merit: 529

Simple answer, Abe runs on many databases, all of them transactional. This knowledge permits one to code with a certain set of assumptions that may not hold in a MyISAM environment. Fearing inconsistencies in users' MyISAM tables, I put in the check that you discovered.

I think the changes needed to support MyISAM's semantics safely would be too invasive. To use MyISAM unsafely is easy, just disable the check in Abe.DataStore._test_transaction, but don't expect support when you start to see wrong data.

eurekafag

full member

Activity: 185

Merit: 100

Hmm, I didn't experience something like this but it could be crucial for abe. Thanks for your help and answers, keep up the good work!

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: eurekafag on February 15, 2012, 12:46:34 PM

Just out of curiosity why don't you support MyISAM engine? I tried replacing innodb with myisam in the source but then abe complains about inability to create a transactional database. Isn't MyISAM faster than InnoDB? I don't know just read some articles comparing them so I'm asking you.

As I understand, MyISAM is non-transactional. It does not support rollback. See: http://en.wikipedia.org/wiki/ACID

Abe relies on the ability to roll back the current transaction and recover when interrupted or in the event of an error. You may find MyISAM works for a while, but when something goes wrong, typically you will have to rebuild the database. With transactional tables, it normally suffices to restart Abe.

eurekafag

full member

Activity: 185

Merit: 100

Quote from: John Tobey on February 15, 2012, 12:08:38 PM

Please run the script in this post: https://bitcointalksearch.org/topic/m.707302

I've committed a small change that I hope will stop these MySQL duplicate key errors.

Thanks, now it works. Just out of curiosity why don't you support MyISAM engine? I tried replacing innodb with myisam in the source but then abe complains about inability to create a transactional database. Isn't MyISAM faster than InnoDB? I don't know just read some articles comparing them so I'm asking you.

tiker

sr. member

Activity: 459

Merit: 250

Thanks. Don't know if I trust it or not.. I'll send some SC from my windows client to this linux install and see if they show up.. guess that's the only way to tell.

Thanks.

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: tiker on February 15, 2012, 12:20:24 PM

I've seen somewhere an Abe install with the SolidCoin2 chain but I can't seem to get it going on my install.

Code:

  File "Abe/DataStore.py", line 1493, in import_block
    raise MerkleRootMismatch(b['hash'], tx_hash_array)
MerkleRootMismatch: Block header Merkle root does not match its transactions. block hash=725ca2009ba1b949a820d3a48511dc21c7f8d76d06d8afb6abb5988f5a911a8b

I won't be surprised if SC2 has changed the Merkle tree hashing algorithm. If you trust your copy of the block chain, you may get around this by simply deleting these lines from Abe/DataStore.py:

Code:

        # Verify Merkle root.
        if b['hashMerkleRoot'] != util.merkle(tx_hash_array):
            raise MerkleRootMismatch(b['hash'], tx_hash_array)

A long-term solution would be to add support for different algorithms per chain, or at least an option to skip the verification in a given chain.

tiker

sr. member

Activity: 459

Merit: 250

I've seen somewhere an Abe install with the SolidCoin2 chain but I can't seem to get it going on my install. I wasn't able to compile the SC2 from source so I'm using the pre-compiled binary from the SC2 site.

Here's what I'm getting when trying to start the import:

Code:


Failed to catch up {'blkfile_number': 1, 'dirname': u'/home/ed/.solidcoin2', 'chain_id': Decimal('19'), 'id': 77L, 'blkfile_offset': 0}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2141, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2162, in catch_up_dir
    store.import_blkdat(dircfg, ds)
  File "Abe/DataStore.py", line 2277, in import_blkdat
    store.import_block(b, chain_ids = chain_ids)
  File "Abe/DataStore.py", line 1493, in import_block
    raise MerkleRootMismatch(b['hash'], tx_hash_array)
MerkleRootMismatch: Block header Merkle root does not match its transactions. block hash=725ca2009ba1b949a820d3a48511dc21c7f8d76d06d8afb6abb5988f5a911a8b

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: eurekafag on February 15, 2012, 10:12:01 AM

I switched to MySQL for experiment.

Please run the script in this post: https://bitcointalksearch.org/topic/m.707302

I've committed a small change that I hope will stop these MySQL duplicate key errors.

eurekafag

full member

Activity: 185

Merit: 100

I switched to MySQL for experiment. During the scanning I had to stop abe and restart later. 2-3 times it was ok but now it shows me this:

Code:

Failed to catch up {'blkfile_number': 1, 'dirname': '/tmp/ramdisk', 'chain_id': None, 'id': Decimal('1'), 'blkfile_offset': 446201276}
Traceback (most recent call last):
  File "/home/eurekafag/svn-soft/bitcoin-abe/Abe/DataStore.py", line 2145, in catch_up
    store.catch_up_dir(dircfg)
  File "/home/eurekafag/svn-soft/bitcoin-abe/Abe/DataStore.py", line 2166, in catch_up_dir
    store.import_blkdat(dircfg, ds)
  File "/home/eurekafag/svn-soft/bitcoin-abe/Abe/DataStore.py", line 2281, in import_blkdat
    store.import_block(b, chain_ids = chain_ids)
  File "/home/eurekafag/svn-soft/bitcoin-abe/Abe/DataStore.py", line 1485, in import_block
    tx['tx_id'] = store.import_tx(tx, pos == 0)
  File "/home/eurekafag/svn-soft/bitcoin-abe/Abe/DataStore.py", line 1802, in import_tx
    store.intin(tx['lockTime']), len(tx['tx'])))
  File "/home/eurekafag/svn-soft/bitcoin-abe/Abe/DataStore.py", line 404, in sql
    store.cursor.execute(cached, params)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
IntegrityError: (1062, "Duplicate entry '2' for key 'PRIMARY'")
Abe initialized.
Listening on http://localhost:2750

...refusing to add left transactions. I experienced this before during configuring MySQL but then I just dropped the DB and created it again. Now it contains too many txs and I don't want to spend another 2 days rescanning the chain. I don't know how the transactional db can be so inconsistent but looks like it really has 2 equal keys in to different records. I can't figure out which tables to look at.

andrehorta

legendary

Activity: 1261

Merit: 1000

yes, thank you!

molecular

donator

Activity: 2772

Merit: 1019

Quote from: andrehorta on February 10, 2012, 12:54:31 PM

Dears

How can i get the Date and Time of transaction?

Thank you

does this help?

Quote

select to_timestamp(block_ntime) from block b inner join block_tx btx on btx.block_id = b.block_id inner join tx on tx.tx_id = btx.tx_id where tx.tx_hash = '97ddfbbae6be97fd6cdf3e7ca13232a3afff2353e29badfab7f73011edd4ced9';

this will get you the time the block the tx got in was mined

Topic: [ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff - page 40. (Read 220986 times)