[ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff - page 17.

dasource

hero member

Activity: 821

Merit: 1000

Quote from: John Tobey on March 25, 2014, 02:14:15 AM

Hi John, double checked all that and the DarkCoin/HiroCoin code has had no alterations on any of those classes on IMPLEMENT_SERIALIZE macro.
Any other ideas on what could be the issue?

ShadesOfMarble

donator

Activity: 543

Merit: 500

I haven't kept up with the development of Abe, but is P2SH/Multisig supported now?

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: unick on March 25, 2014, 12:41:18 AM

With current master version of ABE, If I try to load with rpc, it will work but the frontend crashes as int(ss) is of the NoneType.

My knee-jerk fix for this type of bug is to replace "int(ss)" with "None if ss is None else int(ss)".

John Tobey

hero member

Activity: 481

Merit: 529

unick

sr. member

Activity: 504

Merit: 254

Update:

So I tried to use ppcoin branch to get Abe to read the GrowthCoin database without success.

I also tried to manually load the genesis transaction without success (I had to hexdump the blk0001.dat file since getblock 0 returned and error in the client, similar to the bitcoin issue). ABE complained of an odd lenght transaction with import-tx parameter in abe.conf.

So if I load with the blkfile loader and merkleroot check commented out, I get up to 971 blocks then it fails with

Code:

block_tx 969 969
Skipped 72 bytes at block end
block_tx 970 970
Skipped 73 bytes at block end
block_tx 971 971
Skipped 73 bytes at block end
Exception at 266991
Failed to catch up {'blkfile_offset': 264462, 'blkfile_number': 1, 'chain_id': Decimal('8'), 'loader': None, 'dirname': u'/root/.GrowthCoin', 'id': 2L}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2631, in catch_up
    store.catch_up_dir(dircfg)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2895, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 3022, in import_blkdat
    b = store.parse_block(ds, chain_id, magic, length)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 3053, in parse_block
    d['transactions'].append(deserialize.parse_Transaction(ds))
  File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 89, in parse_Transaction
    d['txIn'].append(parse_TxIn(vds))
  File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 45, in parse_TxIn
    d['scriptSig'] = vds.read_bytes(vds.read_compact_size())
  File "/usr/local/lib/python2.7/dist-packages/Abe/BCDataStream.py", line 59, in read_bytes
    result = self.input[self.read_cursor:self.read_cursor+length]
MemoryError

With current master version of ABE, If I try to load with rpc, it will work but the frontend crashes as int(ss) is of the NoneType. I checked the database and I have NULL values after ~3k blocks mark.... So it will load the blocks but it misses some info.

With PPCOIN branch, it was prior to ABE's NovaCoin upgrade so I assumen datadir wasn't implemented yet at that point and it doesn't accept my growthcoin.conf file (it says bitcoin.conf) still trying to find were can I input the correct conf file but I am not sure it will work since I guess the block format is a lot different from NovaCoin yet it is a descendent of it (and Diamond).

Trying to contact the dev to have more info on that block format

EDIT: does this decription give you any clue if ABE can parse the blockchain data correctly
GrowthCoin - a fork of Diamond, with several algorithms fixed and improved. It is decendent of Novacoin/PPCoin with Pow/PoS. It combines the great features from Luckycoin (random blocks) and Florincoin (transaction message).

dasource

hero member

Activity: 821

Merit: 1000

dasource

hero member

Activity: 821

Merit: 1000

Quote from: John Tobey on March 24, 2014, 11:11:01 AM

If you mean all those NULLs, they result from the incorrect block hash algorithm. Abe does not see block_id 1 as the parent of block_id 2, so it can not compute chain statistics or even height (block number). See if ltc_scrypt helps.

Ok tried this with ltc_scrypt and policy:novacoin and now get this..
Dies at block 9590 rather than before it was getting to full 26000 .... Same output on the web interface, only shows block 0.

Code:

block_tx 9590 1
Skipped 92 bytes at block end
commit
Exception at 5777995176649507762
Failed to catch up {'blkfile_offset': 1917723, 'blkfile_number': 100000, 'chain_id': 11, 'loader': None, 'conf': None, 'dirname': u'/home/hiro/.hirocoin', 'id': 4L}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2789, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 3053, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "Abe/DataStore.py", line 3175, in import_blkdat
    b = chain.ds_parse_block(ds)
  File "Abe/Chain.py", line 121, in ds_parse_block
    d = Chain.ds_parse_block(chain, ds)
  File "Abe/Chain.py", line 59, in ds_parse_block
    d['transactions'].append(chain.ds_parse_transaction(ds))
  File "Abe/Chain.py", line 118, in ds_parse_transaction
    return deserialize.parse_Transaction(ds, has_nTime=True)
  File "Abe/deserialize.py", line 94, in parse_Transaction
    d['txOut'].append(parse_TxOut(vds))
  File "Abe/deserialize.py", line 67, in parse_TxOut
    d['value'] = vds.read_int64()
  File "Abe/BCDataStream.py", line 72, in read_int64
    def read_int64  (self): return self._read_num('  File "Abe/BCDataStream.py", line 110, in _read_num
    (i,) = struct.unpack_from(format, self.input, self.read_cursor)
error: unpack_from requires a buffer of at least 8 bytes
Abe initialized.

John Tobey

hero member

Activity: 481

Merit: 529

dasource

hero member

Activity: 821

Merit: 1000

Quote from: John Tobey on March 24, 2014, 10:39:32 AM

Quote from: dasource on March 24, 2014, 09:42:34 AM

Quote from: John Tobey on March 19, 2014, 10:52:59 AM

Quote from: agran on March 19, 2014, 05:47:41 AM

Shows only the first block. The thing I do not understand.

When Abe loads many blocks but shows only the genesis block, it means that Abe is using the wrong algorithm to calculate the block header hash as it appears in the next block's hashPrev field. By default, Abe uses double-SHA256, the same as Bitcoin. Many coins use scrypt. I would like to make this selectable by datadir=[{...,"hashPrev":"scrypt"}], but for now it requires a few lines of code. Look for "NovaCoin" in the sources for an example.

Hi John,
Could you elaborate abit further on this? much appreciated. What few lines of code are needed?

Come to think of it, if the format is exactly NovaCoin's, you might not need a code change. In datadir, add "policy":"NovaCoin" along with "chain" etc.

The changes are mostly in Abe/Chain.py: add a class defined like NovaCoin (or CryptoCash) and a corresponding conditional in the create function. In Abe/DataStore.py, where it says

Code:

    {"chain":"NovaCoin"},
    {"chain":"CryptoCash"},

add a similar line.

Edit: and if it works, post your changes, or better yet, create a GitHub pull request so I can add them.

Thanks John, will give this a shot. Actually trying this on HiroCoin which is a fork of litecoin.

EDIT : Am I being dumb or have I missed something ?

Code:

ImportError: No module named ltc_scrypt

Here is where I think the issue is (block_id 2) :

Code:

*************************** 1. row ***************************
             block_id: 1
           block_hash: ae94c51e61c87f32a2a6c755df00b1ffa130c73b1bd165aaccaa3f90f8d243a9
        block_version: 1
 block_hashMerkleRoot: b0019d92bc054f7418960c91e252e7d24c77719c7a30128c5f6a827c73095d2a
          block_nTime: 1394723131
          block_nBits: 504365040
         block_nNonce: 1234746574
         block_height: 0
        prev_block_id: NULL
      search_block_id: NULL
     block_chain_work: 0000000000000000000000000000000000000000000000000000000000000000000000100010
       block_value_in: 0
      block_value_out: 40000000000
 block_total_satoshis: 40000000000
  block_total_seconds: 0
block_satoshi_seconds: 0
       block_total_ss: 0
         block_num_tx: 1
   block_ss_destroyed: 0
*************************** 2. row ***************************
             block_id: 2
           block_hash: 54a642a1aabb71a5b9d57a6455ba4ab7f69641e2ebe24ca18dfdc4bd9d98e198
        block_version: 2
 block_hashMerkleRoot: cf42adf0dbda83114f23f3c771700b1daf107aaf57f935f1d6bfcc568f38aa4f
          block_nTime: 1394867933
          block_nBits: 504365040
         block_nNonce: 264378
         block_height: NULL
        prev_block_id: NULL
      search_block_id: NULL
     block_chain_work: NULL
       block_value_in: 0
      block_value_out: 40000000000
 block_total_satoshis: -1
  block_total_seconds: NULL
block_satoshi_seconds: NULL
       block_total_ss: NULL
         block_num_tx: 1
   block_ss_destroyed: NULL

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: dasource on March 24, 2014, 09:42:34 AM

Quote from: John Tobey on March 19, 2014, 10:52:59 AM

Quote from: agran on March 19, 2014, 05:47:41 AM

Shows only the first block. The thing I do not understand.

When Abe loads many blocks but shows only the genesis block, it means that Abe is using the wrong algorithm to calculate the block header hash as it appears in the next block's hashPrev field. By default, Abe uses double-SHA256, the same as Bitcoin. Many coins use scrypt. I would like to make this selectable by datadir=[{...,"hashPrev":"scrypt"}], but for now it requires a few lines of code. Look for "NovaCoin" in the sources for an example.

Hi John,
Could you elaborate abit further on this? much appreciated. What few lines of code are needed?

Come to think of it, if the format is exactly NovaCoin's, you might not need a code change. In datadir, add "policy":"NovaCoin" along with "chain" etc.

The changes are mostly in Abe/Chain.py: add a class defined like NovaCoin (or CryptoCash) and a corresponding conditional in the create function. In Abe/DataStore.py, where it says

Code:

    {"chain":"NovaCoin"},
    {"chain":"CryptoCash"},

add a similar line.

Edit: and if it works, post your changes, or better yet, create a GitHub pull request so I can add them.

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: unick on March 23, 2014, 03:34:17 PM

Ok tried what you said John, but I reposted that quote since I get the same message as previously when I tryed to comment out the MerkleRoot check.

It goes pass the 370th byte but fails at byte 1677721600398062

"Exception at " means an unexpected block format. Where Abe expects a block size or script length, there is some other data. Abe interprets it as a huge number, uses it as the blockfile pointer, and reports the error as coming from there. See: https://github.com/bitcoin-abe/bitcoin-abe/issues/4. Try either the ppcoin patch or RPC mode.

dasource

hero member

Activity: 821

Merit: 1000

Quote from: John Tobey on March 19, 2014, 10:52:59 AM

Quote from: agran on March 19, 2014, 05:47:41 AM

Shows only the first block. The thing I do not understand.

When Abe loads many blocks but shows only the genesis block, it means that Abe is using the wrong algorithm to calculate the block header hash as it appears in the next block's hashPrev field. By default, Abe uses double-SHA256, the same as Bitcoin. Many coins use scrypt. I would like to make this selectable by datadir=[{...,"hashPrev":"scrypt"}], but for now it requires a few lines of code. Look for "NovaCoin" in the sources for an example.

Hi John,
Could you elaborate abit further on this? much appreciated. What few lines of code are needed?

unick

sr. member

Activity: 504

Merit: 254

unick

sr. member

Activity: 504

Merit: 254

Quote from: John Tobey on March 22, 2014, 02:25:37 AM

Quote from: unick on March 21, 2014, 05:28:31 PM

Is there a reason why my questions don't get answered ?

Two possible reasons: bad timing (I am busy) and no obvious answer.

ok thanks, no offense, it seemed as my questions were overlooked, but I understand Wink

Quote from: John Tobey on March 22, 2014, 02:25:37 AM

Quote from: unick on March 21, 2014, 05:28:31 PM

Is there a reason why my questions don't get answered ?

You have several questions, and it is not clear what you are currently using in terms of configuration and modifications to the code. You mentioned commenting out the Merkle root check, but your latest error shows it still occurring. Are you able to load blocks without the check?

I have converted your hex dump back to binary, commented out the Merkle root check, installed the ltc_scrypt module, and loaded the block with this command:

Code:

python -m Abe.abe --dbtype sqlite3 --connect-args :memory: --port 2750 --datadir '[{"dirname":".", "chain":"GrowthCoin", "policy":"NovaCoin"}]'

I wasn't able to load after commenting out the merkle root check, I'll give it a shot like you did, maybe I passed too many parameters on datadir option.

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: unick on March 21, 2014, 05:28:31 PM

Is there a reason why my questions don't get answered ?

Two possible reasons: bad timing (I am busy) and no obvious answer.

You have several questions, and it is not clear what you are currently using in terms of configuration and modifications to the code. You mentioned commenting out the Merkle root check, but your latest error shows it still occurring. Are you able to load blocks without the check?

I have converted your hex dump back to binary, commented out the Merkle root check, installed the ltc_scrypt module, and loaded the block with this command:

Code:

python -m Abe.abe --dbtype sqlite3 --connect-args :memory: --port 2750 --datadir '[{"dirname":".", "chain":"GrowthCoin", "policy":"NovaCoin"}]'

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: unick on March 19, 2014, 06:30:34 PM

is it possible to get address balance from command line query?

I get the following error

Code:

python -m Abe.abe --query /chain/FairCoin/q/addressbalance/fc44xJawRecE9xrZwGHMUtAfxWeS3CJqtP
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/lib/python2.7/dist-packages/Abe/abe.py", line 2374, in 
    sys.exit(main(sys.argv[1:]))
  File "/usr/local/lib/python2.7/dist-packages/Abe/abe.py", line 2370, in main
    serve(store)
  File "/usr/local/lib/python2.7/dist-packages/Abe/abe.py", line 2236, in serve
    }, start_response)
  File "/usr/local/lib/python2.7/dist-packages/Abe/abe.py", line 276, in __call__
    handler(page)
  File "/usr/local/lib/python2.7/dist-packages/Abe/abe.py", line 415, in handle_chain
    chain = abe.chain_lookup_by_name(symbol)
  File "/usr/local/lib/python2.7/dist-packages/Abe/abe.py", line 397, in chain_lookup_by_name
    ret = abe.store.get_chain_by_name(symbol)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 819, in get_chain_by_name
    return store.chains_by.name.get(name, None)
AttributeError: 'DataStore' object has no attribute 'chains_by'

maybe I have the syntax wrong

I'm sorry for the cryptic message. The problem is that you have to give --config (or --dbtype and --connect-args) along with --query if the query involves the database, as this one does.

unick

sr. member

Activity: 504

Merit: 254

Is there a reason why my questions don't get answered ?

elcoco

member

Activity: 111

Merit: 10

what is the proper format for import-tx?

i'm getting this error after your fixes
File "Abe/DataStore.py", line 216, in __init__
store.maybe_import_binary_tx(chain_name, str(hex_tx).decode('hex'))
File "/usr/lib/python2.7/encodings/hex_codec.py", line 42, in hex_decode
output = binascii.a2b_hex(input)
TypeError: Odd-length string

from
import-tx += 010000006eb7dc52010000000000000000000000000000000000000000000000000000000000000 000ffffffff7604ffff001d020f274c6c4a616e2032302c20323031342031323a3430616d204544 542e204e65776567672074656173657220737567676573747320746865205553206f6e6c696e652 072657461696c206769616e74206d617920626567696e20616363657074696e6720626974636f69 6e20736f6f6effffffff0100000000000000000000000000

John Tobey

hero member

Activity: 481

Merit: 529

elcoco

member

Activity: 111

Merit: 10

is it safe to assume that when using the RPC loader rather than the blkfile loader that it doesn't matter the algo of the coin since bitcoin-abe is using rpc call to get data ?
I'm trying to setup fuguecoin using RPC here's my config
datadir += [{"dirname": "/home/azureuser/.fuguecoin",
"loader": "rpc",
"chain": "FugueCoin",
"code3": "FC",
"conf":"fuguecoin.conf",
"address_version" : "\u0023",
"magic" : "\u00f9\u00be\u00b4\u00d4"
}]

i has getting "RPC data not understood: block hash mismatch"
so i figured out the genesis transaction and block 0 hash
I tried this
import-tx += 010000006eb7dc52010000000000000000000000000000000000000000000000000000000000000 000ffffffff7604ffff001d020f274c6c4a616e2032302c20323031342031323a3430616d204544 542e204e65776567672074656173657220737567676573747320746865205553206f6e6c696e652 072657461696c206769616e74206d617920626567696e20616363657074696e6720626974636f69 6e20736f6f6effffffff0100000000000000000000000000
then got
store = DataStore.new(args)

File "Abe/DataStore.py", line 216, in __init__
store.maybe_import_binary_tx(chain_name, str(hex_tx).decode('hex'))
UnboundLocalError: local variable 'chain_name' referenced before assignment
so i changed my config file to this:
import-tx += {"chain":"FugueCoin" ,"tx":"010000006eb7dc52010000000000000000000000000000000000000000000000000000000000000 000ffffffff7604ffff001d020f274c6c4a616e2032302c20323031342031323a3430616d204544 542e204e65776567672074656173657220737567676573747320746865205553206f6e6c696e652 072657461696c206769616e74206d617920626567696e20616363657074696e6720626974636f69 6e20736f6f6effffffff0100000000000000000000000000"}
no i'm getting this

File "Abe/DataStore.py", line 2323, in maybe_import_binary_tx
chain = store.chains_by.name(chain_name)
TypeError: 'dict' object is not callable

changed that line from chain = store.chains_by.name(chain_name) to chain = store.chains_by.name.get(chain_name)
and getting
File "Abe/BCDataStream.py", line 110, in _read_num
(i,) = struct.unpack_from(format, self.input, self.read_cursor)
struct.error: unpack_from requires a buffer of at least 4 bytes

Huh

Topic: [ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff - page 17. (Read 220986 times)