Pages:
Author

Topic: [ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff - page 19. (Read 220986 times)

sr. member
Activity: 504
Merit: 254
Maybe it's an endiannes thing?

I'm not familiar with the concepet of endiannes. After relooking at the code, I had made a typo on the class name


This coin is new to Abe.  Please follow the example "datadir" syntax in abe.conf, and add datadir = [{....}] to your abe-mysql.conf file.  If the currency is scrypt-based, it may need more changes discussed recently in this thread.



Well I modified the Chain.py file as I did for FairCoin (it worked for FairCoin) so the abe-mysql.conf file would be a better option on the perspective of future upgrades, but the specs of the GrowthCoin blockchain were indeed entered. I will revert this to the config file when I sorted out the issue.

Meanwhile I triple checked the code and found that I made a typo on the class name.

That being said, I have another issue now. This time with the address version

Code:
root@grw:~/bitcoin-abe# python -m Abe.abe --config /home/abe/abe-mysql.conf --commit-bytes 100000 --no-serve
ddl_implicit_commit=true
create_table_epilogue=''
Abe/DataStore.py:536: Warning: Converting column 'a' from VARCHAR to TEXT
  store.cursor.execute(stmt)
Abe/DataStore.py:536: Warning: Converting column 'b' from VARCHAR to TEXT
  store.cursor.execute(stmt)
max_varchar=4294967295
clob_type=LONGTEXT
Abe/DataStore.py:536: Warning: Converting column 'test_varbit' from VARCHAR to TEXT
  store.cursor.execute(stmt)
binary_type=hex
int_type=int
Created silly table abe_dual
sequence_type=mysql
limit_style=native
Abe/DataStore.py:536: Warning: Converting column 'txout_scriptPubKey' from VARCHAR to TEXT
  store.cursor.execute(stmt)
Abe/DataStore.py:536: Warning: Converting column 'txin_scriptSig' from VARCHAR to TEXT
  store.cursor.execute(stmt)
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/root/bitcoin-abe/Abe/abe.py", line 2276, in
    sys.exit(main(sys.argv[1:]))
  File "/root/bitcoin-abe/Abe/abe.py", line 2270, in main
    store = make_store(args)
  File "/root/bitcoin-abe/Abe/abe.py", line 153, in make_store
    store = DataStore.new(args)
  File "Abe/DataStore.py", line 3446, in new
    return DataStore(args)
  File "Abe/DataStore.py", line 181, in __init__
    store.initialize()
  File "Abe/DataStore.py", line 1262, in initialize
    store.insert_chain(chain)
  File "Abe/DataStore.py", line 1301, in insert_chain
    chain.name, chain.decimals))
  File "Abe/DataStore.py", line 523, in sql
    store._execute(cached, params)
  File "Abe/DataStore.py", line 500, in _execute
    store.cursor.execute(stmt, params)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
_mysql_exceptions.OperationalError: (1048, "Column 'chain_address_version' cannot be null")

if I decode the address, this is what I get

Code:
root@grw:~/bitcoin-abe# python -m Abe.abe --query /q/decode_address/GTjYKganLhDtMTcWCinzbQZpsrSjXfB29m
26:6c7c51acb76a0b62c10b6cb17b39cc0982f5d289

and this is my GrowthCoin class

Code:
class GrowthCoin(NvcChain):
    def __init__(chain, **kwargs):
        chain.name = 'GrowthCoin'
        chain.code3 = 'GRW'
        chain.address_version = "\x26"
        chain.magic = "\xa5\xef\xdb\xfd"
        Chain.__init__(chain, **kwargs)

    datadir_conf_file_name = "GrowthCoin.conf"
    datadir_rpcport = 17178

I also added CHAIN_CONFIG = [{"chain":"GrowthCoin"}] in the DataStore.py file...

Everything looks ok, what could be the issue?
hero member
Activity: 481
Merit: 529
Code:
psycopg2.DataError: value too long for type character(3)

Either shorten your chain's "code3" to 3 characters or widen the database column:

Code:
ALTER TABLE chain ALTER COLUMN chain_code3 TYPE VARCHAR(4);
newbie
Activity: 4
Merit: 0
I am attempting to get this running on a postgresql database.  I have followed the instructions to the letter and this is the error that I get:

Code:
admin@ip-10-196-49-5:~$ python -m Abe.abe --config doge.conf --commit-bytes 100000 --no-serve
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/lib/python2.7/dist-packages/Abe/abe.py", line 2276, in
    sys.exit(main(sys.argv[1:]))
  File "/usr/local/lib/python2.7/dist-packages/Abe/abe.py", line 2270, in main
    store = make_store(args)
  File "/usr/local/lib/python2.7/dist-packages/Abe/abe.py", line 153, in make_store
    store = DataStore.new(args)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 3445, in new
    return DataStore(args)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 199, in __init__
    store._init_datadirs()
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 753, in _init_datadirs
    dircfg.get('policy', chain_name), decimals))
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 522, in sql
    store._execute(cached, params)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 499, in _execute
    store.cursor.execute(stmt, params)
psycopg2.DataError: value too long for type character(3)

I am on Debain Wheezy running python 2.7 and postgresql 9.1
hero member
Activity: 481
Merit: 529
ok I found where NovaCoin took it's config from in Chain.py... and like I said they both have the same magic number so I hardcoded the FairCoin names into that file, witch I find is the unelegant way to go but it works for now... If you have a better solution, I would love to know Smiley

edit: well after putting more thoughts to it... I just added a conditional statement for the FairCoin name, it's a bit more clean this way... still if there is a better solution (as an update to the code will wipe this setting) let me know (or maybe update the github with this conditional statement)

I'd put the FairCoin parameters in abe.conf datadir=[{...}] and not in Python.  That should survive upgrades.  See the datadir examples in abe.conf.

Given a simple directory name (datadir="...") Abe relies on the magic number.  We could support two chains with the same magic number, but we would need some way to distinguish them.  That's why I introduced the expanded syntax for datadir, a JSON array of objects.
hero member
Activity: 481
Merit: 529
Hi, I'm trying to setup GrowthCoin block explorer.

This is what I get

Code:
root@grw:~/bitcoin-abe# python -m Abe.abe --config /home/abe/abe-mysql.conf --commit-bytes 100000 --no-serve
Opened /root/.GrowthCoin/blk0001.dat
Chain not found for magic number a5efdbfd in block file /root/.GrowthCoin/blk0001.dat at offset 0.

This coin is new to Abe.  Please follow the example "datadir" syntax in abe.conf, and add datadir = [{....}] to your abe-mysql.conf file.  If the currency is scrypt-based, it may need more changes discussed recently in this thread.
legendary
Activity: 1862
Merit: 1011
Reverse engineer from time to time
sr. member
Activity: 504
Merit: 254
Hi, I'm trying to setup GrowthCoin block explorer.

This is what I get

Code:
root@grw:~/bitcoin-abe# python -m Abe.abe --config /home/abe/abe-mysql.conf --commit-bytes 100000 --no-serve
Opened /root/.GrowthCoin/blk0001.dat
Chain not found for magic number a5efdbfd in block file /root/.GrowthCoin/blk0001.dat at offset 0.

But we can clearly see the firt byte from

Code:
hexdump -n 300 -C /root/.GrowthCoin/blk0001.dat


are correct!


Code:
00000000  a5 ef db fd 69 01 00 00  01 00 00 00 00 00 00 00  |....i...........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 bf f2 79 9b  |..............y.|
00000030  bf ec 52 41 32 ea 94 54  5f 53 83 7c 5c eb 46 26  |..RA2..T_S.|\.F&|
00000040  b6 3d 18 70 76 01 0c cb  96 10 53 14 31 0b fa 51  |.=.pv.....S.1..Q|
00000050  ff ff 0f 1e ba 6e 15 01  01 02 00 00 00 1d 0b fa  |.....n..........|
00000060  51 01 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |Q...............|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080  00 00 ff ff ff ff b9 04  ff ff 00 1d 02 0f 27 4c  |..............'L|
00000090  af 4a 75 6c 79 20 33 31  2c 20 32 30 31 33 20 31  |.July 31, 2013 1|
000000a0  32 70 6d 20 45 44 54 3a  20 54 68 65 20 55 2e 53  |2pm EDT: The U.S|
000000b0  2e 20 65 63 6f 6e 6f 6d  79 20 67 72 65 77 20 31  |. economy grew 1|
000000c0  2e 37 25 20 69 6e 20 74  68 65 20 73 65 63 6f 6e  |.7% in the secon|
000000d0  64 20 71 75 61 72 74 65  72 2c 20 61 69 64 65 64  |d quarter, aided|
000000e0  20 62 79 20 73 6f 6c 69  64 20 63 6f 6e 73 75 6d  | by solid consum|
000000f0  65 72 20 73 70 65 6e 64  69 6e 67 20 61 6e 64 20  |er spending and |
00000100  61 20 72 61 6d 70 2d 75  70 20 69 6e 20 62 75 73  |a ramp-up in bus|
00000110  69 6e 65 73 73 20 69 6e  76 65 73 74 6d 65 6e 74  |iness investment|
00000120  2c 20 74 68 65 20 67 6f  76 65 72 6e              |, the govern|
0000012c


What did I miss to get this message?

thanks
newbie
Activity: 15
Merit: 0
I'm transferring Abe from one server to the next. After restoring the MySQL dump from the previous server and starting Abe, the block number no longer increases and it is stuck at block 289302 for Bitcoin. The MySQL dump was made for at block height 289302. The previous server's copy of Abe is able to go past block 289302.

On another pair of servers, the same transfer was made and Ltc had no problems updating the new blocks.

Any idea what has happened?
hero member
Activity: 481
Merit: 529
So I was finally able to get Dogecoin working.  I was using the wrong magic.  I have everything up on a live server now and it doesn't seem to be polling the database for the queries, instead it is opening up the block file every single time.

Please search for "no-serve" in this thread.
newbie
Activity: 4
Merit: 0
So I was finally able to get Dogecoin working.  I was using the wrong magic.  I have everything up on a live server now and it doesn't seem to be polling the database for the queries, instead it is opening up the block file every single time.

I am on the no-statistics branch using this config file:

Code:
dbtype sqlite3
connect-args doge.sqlite
port 2750
host 0.0.0.0
#upgrade
datadir = [{
        "dirname": "/home/ubuntu/.dogecoin",
        "chain": "Dogecoin",
        "code3": "DOGE",
        "address_version": "\u001e",
        "magic": "\u00c0\u00c0\u00c0\u00c0",
        "conf": "dogecoin.conf",
        "trim_depth": 40
        }]
commit-bytes = 100000
keep-scriptsig = false
int-type str

Here is what shows up on the server:

Code:
[13/Mar/2014 02:06:12] "GET /q/addressbalance/D9f7RRygjA7N91TLDoshLwNVoFxqCjvv7i HTTP/1.1" 200 83
failed to load /home/ubuntu/.dogecoin/bitcoin.conf: [Errno 2] No such file or directory: u'/home/ubuntu/.dogecoin/bitcoin.conf'
catch_up_rpc: abort
Opened /home/ubuntu/.dogecoin/blocks/blk00022.dat

member
Activity: 111
Merit: 10
I'm trying to run a blockchain explorer for HeavyCoin, by using the newest bitcoin-abe, but i keep getting this error
Opened /home/azureuser/.heavycoin/blocks/blk00000.dat
Exception at 433754408174527358
Failed to catch up {'blkfile_offset': 40726, 'blkfile_number': 100000, 'chain_id': 19, 'loader': u'blkfile', 'conf': None, 'dirname': '/home/azureuser/.heavycoin', 'id': Decimal('84')}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2686, in catch_up
    store.catch_up_dir(dircfg)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2950, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 3072, in import_blkdat
    b = chain.ds_parse_block(ds)
  File "/usr/local/lib/python2.7/dist-packages/Abe/Chain.py", line 59, in ds_parse_block
    d['transactions'].append(chain.ds_parse_transaction(ds))
  File "/usr/local/lib/python2.7/dist-packages/Abe/Chain.py", line 52, in ds_parse_transaction
    return deserialize.parse_Transaction(ds)
  File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 90, in parse_Transaction
    d['txIn'].append(parse_TxIn(vds))
  File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 46, in parse_TxIn
    d['sequence'] = vds.read_uint32()
  File "/usr/local/lib/python2.7/dist-packages/Abe/BCDataStream.py", line 71, in read_uint32
    def read_uint32 (self): return self._read_num('  File "/usr/local/lib/python2.7/dist-packages/Abe/BCDataStream.py", line 110, in _read_num
    (i,) = struct.unpack_from(format, self.input, self.read_cursor)
error: unpack_from requires a buffer of at least 4 bytes


I installed the ltc_scrypt module, by doing python setup.py install, but i'm not sure what to do next.
at first I was getting
raise MerkleRootMismatch(b['hash'], tx_hash_array)
but I changed lines in     
# Verify Merkle root.
    if b['hashMerkleRoot'] != util.merkle(tx_hash_array):
        raise MerkleRootMismatch(b['hash'], tx_hash_array)
from the Abe/DataStore.py

but now i'm getting the error posted
sr. member
Activity: 504
Merit: 254
ok I found where NovaCoin took it's config from in Chain.py... and like I said they both have the same magic number so I hardcoded the FairCoin names into that file, witch I find is the unelegant way to go but it works for now... If you have a better solution, I would love to know Smiley

edit: well after putting more thoughts to it... I just added a conditional statement for the FairCoin name, it's a bit more clean this way... still if there is a better solution (as an update to the code will wipe this setting) let me know (or maybe update the github with this conditional statement)
sr. member
Activity: 504
Merit: 254
Hi John,

I have a quick question (don't know if the answer would be as quick and easy) so here's the deal in a nutshell.

I am trying to setup a block explorer for FairCoin.

it reads the blockchain just fine but on the HTTP side, I can't get it to read the FairCoin chain name. It defaults to NovaCoin no matter what I do.

It seems that the magic number for NovaCoin is the same as the one for FairCoin. could that be it?

how can this be fixed/where should I look?

thanks in advance
hero member
Activity: 481
Merit: 529
What would be faster, using one DB for all the blocks
or each chain one DB

I guess you should use a separate database per chain.  It complicates the HTTP side but simplifies deleting a corrupt (or abandoned) chain and spreading an increasing load.  I don't know if it would be much faster.
member
Activity: 96
Merit: 10
are u accessing the database with your own queries?

no not my own queries, i dont know if your replying on this topic as a person that uses Abe, or someone that is intrested.

i apriciate both ,but to explain abe uses sql queries to recall data (blocks, hashes etc)
Now the blocks of all chains get put in one table of blocks and then the database identifies those blocks by their chain number(or something else not sure, but this is the most logic in my eyes)

now if you have 20 chains running next too each other and giving every block a different id, so their can be: block id=1 from doge, block=2 from bitcoin and block=5 from LiteCoin. and now i have just 2 chains one of 700k and one of 150k blocks, 2 of the biggest chains out there. but still the database has to check if not one of those 150k blocks is lost in those 700k blocks since i mybe wouldnt always start that wallet.

so thats my problem i might be wrong but if your going to run 20 coins or more(what i am planning) you can get yourself in deep shit since the queries has to look trough 1-5mil blocks or more if you have 20 big chains running.

so thats why i think of a db for every chains, since your mostly searching for 1 block within a chain than looking within 200k is easyer than looking within 5 mil. since it has only one critiria(block_id) instead of 2 (block_id+ chain_id)

ahh well i have just started to learn that stuff at uni about DB, atleast in the sence of going deeper, so anything i might just say is just complete BS.

thxn upfront, for any advice your going to give.

LT
newbie
Activity: 1
Merit: 0
are u accessing the database with your own queries?
member
Activity: 96
Merit: 10
Oke so i have managed to setup, my Abe blockexplorer.
will release it soon and its newly design after i have completed my work,
but i have this problem that i think its extremly slow.

I didnt want to host it on some server for 5 to 50$ a month
but hosted it on my pc trough a VM running ubuntu 12.04.

now my machine specs are
CPU: 6 core 3.4 ghz
RAM(dedicated to VM): 12GB 1800mhz
HDD: 200GB

now i think my problem lays with the internet connected since i am not directly connected to the router,
what means i use wifi of about 20mbs instead of the 100mb i should get from my fiber wire.

any tips or idea's on this, on how to make the website run faster.
or is it just abe that is slow?

also another question,
What would be faster, using one DB for all the blocks
or each chain one DB
LT
 
hero member
Activity: 481
Merit: 529
Code:
block_tx 1181 1
Skipped 95 bytes at block end
Exception at 235523
Failed to catch up {'blkfile_number': 100000, 'dirname': u'/home/ubuntu/.dogecoin', 'chain_id': 8, 'id': 1, 'blkfile_offset': 208572}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2422, in catch_up
    store.catch_up_dir(dircfg)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2475, in catch_up_dir
    store.import_blkdat(dircfg, ds, filename[0])
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2596, in import_blkdat
    b = store.parse_block(ds, chain_id, magic, length)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2630, in parse_block
    d['transactions'].append(deserialize.parse_Transaction(ds))
  File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 90, in parse_Transaction
    for i in xrange(n_vout):
OverflowError: Python int too large to convert to C long

Most likely the transaction format differs from Bitcoin's.  NovaCoin and CryptoCash also changed their transaction format, and Abe/Chain.py accommodates them (by passing has_nTime in PpcPosChain.ds_parse_transaction).  If Doge shared their format, it could do the same, but it seems different, since the default parser got to Block 1181.  It is a matter of translating the C++ definition of CTransaction into Python.
newbie
Activity: 4
Merit: 0
I have been having a heck of a time getting this to work with dogecoin.  I have downloaded the blockchain and rescanned.  Here is my abe.conf:

Code:
dbtype sqlite3
connect-args abe.sqlite
upgrade
port 7000

datadir = [{
        "dirname": "/home/ubuntu/.dogecoin",
        "chain":   "Dogecoin",
        "code3":   "DOGE",
        "address_version": "\u001e" }]

I have applied the fix located at: https://bitcointalksearch.org/topic/m.750417

This is the error I am getting when I run:

Code:
python -m Abe.abe --config abe.sqlite.conf --commit-bytes 100000 --no-serve

Code:
Skipped 95 bytes at block end
block_tx 1170 1
Skipped 95 bytes at block end
block_tx 1171 1
Skipped 95 bytes at block end
block_tx 1172 1
Skipped 95 bytes at block end
block_tx 1173 1
Skipped 96 bytes at block end
block_tx 1174 1
Skipped 95 bytes at block end
block_tx 1175 1
Skipped 95 bytes at block end
block_tx 1176 1
Skipped 95 bytes at block end
block_tx 1177 1
Skipped 96 bytes at block end
block_tx 1178 1
Skipped 95 bytes at block end
block_tx 1179 1
Skipped 96 bytes at block end
block_tx 1180 1
Skipped 96 bytes at block end
block_tx 1181 1
Skipped 95 bytes at block end
Exception at 235523
Failed to catch up {'blkfile_number': 100000, 'dirname': u'/home/ubuntu/.dogecoin', 'chain_id': 8, 'id': 1, 'blkfile_offset': 208572}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2422, in catch_up
    store.catch_up_dir(dircfg)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2475, in catch_up_dir
    store.import_blkdat(dircfg, ds, filename[0])
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2596, in import_blkdat
    b = store.parse_block(ds, chain_id, magic, length)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2630, in parse_block
    d['transactions'].append(deserialize.parse_Transaction(ds))
  File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 90, in parse_Transaction
    for i in xrange(n_vout):
OverflowError: Python int too large to convert to C long
newbie
Activity: 15
Merit: 0
Thank you for the quick update! I pulled the fix and reran Abe but the same error still appears.

If the error is still at
Code:
File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 247, in script_GetOp

then Python is using the old, installed version.  Either reinstall Abe or adjust PYTHONPATH to use the new code.


It works, thank you so much.
Pages:
Jump to: