[ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff - page 4.

internetbitcoins

newbie

Activity: 12

Merit: 0

Quote from: John Tobey on August 14, 2013, 08:47:19 AM

Quote from: internetbitcoins on August 13, 2013, 06:13:58 PM

I run

Code:

python -m Abe.abe --config abe-my.conf --commit-bytes 100000 --no-serve

but 2 or 3 times I had to restart the server, so now in the block table, I check on block_id and there are some missing blocks.

block_id is not the same as block number (a.k.a. block_height). The gaps in block_id are not a problem.

Hi, thanks. Finally I can return to the abe staff today. I launched the web server and I can get the data. For example, for this one:
http://localhost:2750/chain/Bitcoin/q/getreceivedbyaddress/19vDdUsaiQrra5CDDKNn8yooai2pF3XU6A
I get 0.43250932

And on http://blockexplorer.com/address/19vDdUsaiQrra5CDDKNn8yooai2pF3XU6A
I get 0.47998327

So I think there is something not updated on my database.

On launching abe web server I get this error:

Code:

python -m Abe.abe --config abe-my.conf
no chain_id
catch_up_rpc: abort
Opened /xx/xx/.bitcoin/blocks/blk00074.dat
Exception at 87267839
Failed to catch up {'blkfile_offset': 87139457, 'blkfile_number': 100074, 'chain_id': None, 'loader': None, 'dirname': '/xx/xx/.bitcoin', 'id': Decimal('1')}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2637, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2895, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "Abe/DataStore.py", line 3025, in import_blkdat
    store.import_block(b, chain_ids = chain_ids)
  File "Abe/DataStore.py", line 1737, in import_block
    tx['tx_id'] = store.import_and_commit_tx(tx, pos == 0)
  File "Abe/DataStore.py", line 2259, in import_and_commit_tx
    tx_id = store.import_tx(tx, is_coinbase)
  File "Abe/DataStore.py", line 2193, in import_tx
    pubkey_id = store.script_to_pubkey_id(txout['scriptPubKey'])
  File "Abe/DataStore.py", line 2573, in script_to_pubkey_id
    for opcode, data, i in deserialize.script_GetOp(script):
  File "Abe/deserialize.py", line 236, in script_GetOp
    opcode |= ord(bytes[i])
IndexError: string index out of range
Abe initialized.
Listening on http://localhost:2750

Any idea? Thanks (again) in advance!

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: unick on August 27, 2013, 10:48:42 AM

This is the output

Code:

This means you are getting the previously installed version. You want the one from from 'Abe/abe.py'. Either install the new version (as root, python setup.py install) or set PYTHONPATH to the git clone directory (bitcoin-abe, not the Abe subdirectory). Normally, sys.path starts with an empty string, so if you "cd" to the correct directory, Python should find the local modules.

unick

sr. member

Activity: 504

Merit: 254

Quote from: John Tobey on August 27, 2013, 10:19:31 AM

Quote from: unick on August 27, 2013, 01:58:25 AM

yep, Git tells me I'm up to date and I have manually checked for those changes and everything is there.

any thing else?

Perhaps you have an older version installed on your system? Otherwise, I don't see how Abe could produce that message. What does this show:

Code:

python -c 'import Abe.abe; print Abe.abe;'

This is the output

Code:

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: unick on August 27, 2013, 01:58:25 AM

yep, Git tells me I'm up to date and I have manually checked for those changes and everything is there.

any thing else?

Perhaps you have an older version installed on your system? Otherwise, I don't see how Abe could produce that message. What does this show:

Code:

python -c 'import Abe.abe; print Abe.abe;'

unick

sr. member

Activity: 504

Merit: 254

yep, Git tells me I'm up to date and I have manually checked for those changes and everything is there.

any thing else?

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: unick on August 26, 2013, 02:46:22 PM

Code:

RPC data not understood: block hash mismatch

Are you sure you are using the latest from the diamond branch? These changes should be reflected in your Python files:

https://github.com/jtobey/bitcoin-abe/commit/829890b0c3bd2599e66108c24d90d16a5a1b0ea8

In particular, the part about "block hash mismatch" should appear commented out.

unick

sr. member

Activity: 504

Merit: 254

Quote from: John Tobey on August 23, 2013, 04:22:56 PM

Quote from: unick on August 22, 2013, 10:23:01 PM

I am not sure if this is what you are looking for

https://github.com/diamondcoinproject/diamondcoin/blob/dd388c28bf89abeaaad761982bb4d04f18995c27/src/serialize.h#L59

Close enough, it's here, and the new field is strTxComment: https://github.com/diamondcoinproject/diamondcoin/blob/dd388c28bf89abeaaad761982bb4d04f18995c27/src/main.h#L452-462

I've downloaded some of the DMD chain and got Abe to load it, but only over RPC, since this coin uses strange block hashing rules. Loading from the blockfile resulted in only Block 0 being displayed, since Abe could not link any block to its parent. Also, I had to disable verification and make Abe trust the RPC server's block hashes. (If you have info about the differences between DMD and Bitcoin block hashing, I may be able to fix this.)

Please pull branch "diamond", copy Diamond.conf as bitcoin.conf, add "rpcport=17772" to that file if not already present, and use a config such as this:

Code:

datadir [{"dirname":"/root/.Diamond", "chain":"Diamond", "code3":"DMD", "address_version":"\u005a"}]
default-loader rpc

I am having the following error

Code:

RPC data not understood: block hash mismatch
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('9'), 'loader': None, 'dirname': u'/root/.Diamond', 'id': 36L}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2635, in catch_up
    raise Exception("RPC load failed")
Exception: RPC load failed
Abe initialized.

I have pulled the diamond branch from the repository
I have cp Diamond.conf. to bitcoin.conf (making sure the rpc port was on 17772)
change abe-my.conf accordingly

while the server listens... I can't seem to have the RPC loaded... did I miss something ?

b!z

legendary

Activity: 1582

Merit: 1010

Very cool. This looks like a good way to view the blockchain without relying on an external site.

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: tacoman71 on August 23, 2013, 05:48:59 PM

You should make a newbie friendly .exe version of this that they can just double click to install.

I have too little time. I would happily show another developer around, should one wish to package it nicely.

tacoman71

sr. member

Activity: 320

Merit: 250

You should make a newbie friendly .exe version of this that they can just double click to install.

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: unick on August 22, 2013, 10:23:01 PM

I am not sure if this is what you are looking for

https://github.com/diamondcoinproject/diamondcoin/blob/dd388c28bf89abeaaad761982bb4d04f18995c27/src/serialize.h#L59

Close enough, it's here, and the new field is strTxComment: https://github.com/diamondcoinproject/diamondcoin/blob/dd388c28bf89abeaaad761982bb4d04f18995c27/src/main.h#L452-462

I've downloaded some of the DMD chain and got Abe to load it, but only over RPC, since this coin uses strange block hashing rules. Loading from the blockfile resulted in only Block 0 being displayed, since Abe could not link any block to its parent. Also, I had to disable verification and make Abe trust the RPC server's block hashes. (If you have info about the differences between DMD and Bitcoin block hashing, I may be able to fix this.)

Please pull branch "diamond", copy Diamond.conf as bitcoin.conf, add "rpcport=17772" to that file if not already present, and use a config such as this:

Code:

datadir [{"dirname":"/root/.Diamond", "chain":"Diamond", "code3":"DMD", "address_version":"\u005a"}]
default-loader rpc

unick

sr. member

Activity: 504

Merit: 254

Quote from: John Tobey on August 22, 2013, 07:54:23 PM

Quote from: unick on August 22, 2013, 06:40:07 PM

Hey John Tobey, I am trying to set up the explorer for the DMD coin, I've read the issue about PPcoin branches, could you give any help on setting this up for DMD?

thanks

Sure, just post how far you got and any detailed error message. If the transactions have a new format, we may need to find how the coin's source code serializes CTransaction. Bitcoin example: src/core.h line 194.

This is the error message I get when trying to read the block chain

Code:


root@blockchains:/var/www/blockchain/bitcoin-abe/Abe# python -m Abe.abe --config abe-my.conf --commit-bytes 100000 --no-serve
failed to load /root/.Diamond/bitcoin.conf: [Errno 2] No such file or directory: u'/root/.Diamond/bitcoin.conf'
catch_up_rpc: abort
Opened /root/.Diamond/blk0001.dat
Exception at 72056513853653130
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('9'), 'loader': None, 'dirname': u'/root/.Diamond', 'id': 24L}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2637, in catch_up
    store.catch_up_dir(dircfg)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2895, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 3022, in import_blkdat
    b = store.parse_block(ds, chain_id, magic, length)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 3053, in parse_block
    d['transactions'].append(deserialize.parse_Transaction(ds))
  File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 88, in parse_Transaction
    d['txIn'].append(parse_TxIn(vds))
  File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 46, in parse_TxIn
    d['sequence'] = vds.read_uint32()
  File "/usr/local/lib/python2.7/dist-packages/Abe/BCDataStream.py", line 71, in read_uint32
    def read_uint32 (self): return self._read_num('  File "/usr/local/lib/python2.7/dist-packages/Abe/BCDataStream.py", line 110, in _read_num
    (i,) = struct.unpack_from(format, self.input, self.read_cursor)
OverflowError: Python int too large to convert to C long

I am not sure if this is what you are looking for

https://github.com/diamondcoinproject/diamondcoin/blob/dd388c28bf89abeaaad761982bb4d04f18995c27/src/serialize.h#L59

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: unick on August 22, 2013, 06:40:07 PM

Hey John Tobey, I am trying to set up the explorer for the DMD coin, I've read the issue about PPcoin branches, could you give any help on setting this up for DMD?

thanks

Sure, just post how far you got and any detailed error message. If the transactions have a new format, we may need to find how the coin's source code serializes CTransaction. Bitcoin example: src/core.h line 194.

unick

sr. member

Activity: 504

Merit: 254

Hey John Tobey, I am trying to set up the explorer for the DMD coin, I've read the issue about PPcoin branches, could you give any help on setting this up for DMD?

thanks

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: internetbitcoins on August 13, 2013, 06:13:58 PM

I run

Code:

python -m Abe.abe --config abe-my.conf --commit-bytes 100000 --no-serve

but 2 or 3 times I had to restart the server, so now in the block table, I check on block_id and there are some missing blocks.

block_id is not the same as block number (a.k.a. block_height). The gaps in block_id are not a problem.

internetbitcoins

newbie

Activity: 12

Merit: 0

Hi,

I run

Code:

python -m Abe.abe --config abe-my.conf --commit-bytes 100000 --no-serve

but 2 or 3 times I had to restart the server, so now in the block table, I check on block_id and there are some missing blocks.

Is there a way to recover them without restart all the process of reading all the blocks on the blockchain??
For example, any way to launch the

Code:

python -m Abe.abe --config abe-my.conf --commit-bytes 100000 --no-serve

only for missing ids?

Thanks in advances and for Abe!

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: johntawaki on July 24, 2013, 12:07:04 PM

nice one, but I heard it's not working for some new coins, such as Diamond (DMD).

Yes, several new coins trace their ancestry to PPCoin and need the "ppcoin" branch in Git. See: https://github.com/jtobey/bitcoin-abe/issues/19

Another one, CopperLark, has its own format, which the "copperlark" branch handles.

Ideally, Abe would automatically detect among known variants and handle them without code changes.

johntawaki

member

Activity: 65

Merit: 10

nice one, but I heard it's not working for some new coins, such as Diamond (DMD).

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: katriel on July 24, 2013, 10:59:33 AM

Abe is really terrific! I used it to create a transaction network and play around with that.

Good to hear!

Quote from: katriel on July 24, 2013, 10:59:33 AM

However, on my machine I left it running (writing to a SQLite database) for I think about two weeks and only got about 18 months into the blockchain (2.4 million transactions). The readme says it typically takes several days to get through the chain--is there something I'm probably doing wrong?

I don't use SQLite for the full block chain, so I don't really know. One user, K1773R, reported a fast load with the database file in tmpfs (RAM). (https://bitcointalksearch.org/topic/m.2639709) I think K1773R uses SQLite. If your system has enough memory, you could try that. Abe takes a lot of space, though. I estimate 4-5 times the size of all block files combined. (~/.bitcoin/blocks/blk00???.dat in Unix) You can reduce this somewhat by setting keep-scriptsig=false or by using the no-statistics branch and default-trim-depth. See the comments in abe.conf for what those options entail.

K1773R mentioned possibly distributing the SQLite file via torrent: https://github.com/jtobey/bitcoin-abe/issues/15

katriel

newbie

Activity: 3

Merit: 0

Abe is really terrific! I used it to create a transaction network and play around with that.

However, on my machine I left it running (writing to a SQLite database) for I think about two weeks and only got about 18 months into the blockchain (2.4 million transactions). The readme says it typically takes several days to get through the chain--is there something I'm probably doing wrong?

Topic: [ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff - page 4. (Read 28953 times)