[ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff - page 28.

mapleshadow

member

Activity: 79

Merit: 10

That I can build wdc of abe services?

Quote from: John Tobey on July 08, 2013, 01:02:30 PM

Quote from: mapleshadow on July 08, 2013, 12:17:00 PM

my conf is wdc

wdc.conf:
dbtype sqlite3
connect-args abe.sqlite
datadir = [{
"dirname": "/home/maple/.worldcoin",
"chain": "Worldcoin",
"code3": "WDC",
"address_version": "\u0049" }]
int-type str
upgrade
port 2750
host 12.16.2.24

DataStore.py:
{"chain":"Worldcoin",
"code3":"WDC", "address_version":"\x49", "magic":""},

ERROR-1:
failed to load /home/maple/.worldcoin/bitcoin.conf: [Errno 2] No such file or directory: u'/home/maple/.worldcoin/bitcoin.conf'
catch_up_rpc: abort
Opened /home/maple/.worldcoin/blocks/blk00000.dat
Abe initialized.
Listening on http://12.16.2.24:2750

WHY?

Abe tries to load data over RPC, and when that fails, it reads block files. (See default-loader in abe.conf to choose one or the other.)

Abe does not know about Worldcoin, so it looks for RPC credentials in bitcoin.conf. There is no bitcoin.conf, so it logs an error and tries to load blocks from files, starting with blocks/blk00000.dat. It did not find any blocks in that file, so perhaps the file is empty or the block format is not the same as Bitcoin's.

I am surprised that the configured dirname is in /home/mapleshadow but Abe looked for bitcoin.conf in /home/maple.

mapleshadow

member

Activity: 79

Merit: 10

Sorry, I was wrong
Paths are maple

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: mapleshadow on July 08, 2013, 12:17:00 PM

my conf is wdc

wdc.conf:
dbtype sqlite3
connect-args abe.sqlite
datadir = [{
"dirname": "/home/mapleshadow/.worldcoin",
"chain": "Worldcoin",
"code3": "WDC",
"address_version": "\u0049" }]
int-type str
upgrade
port 2750
host 12.16.2.24

DataStore.py:
{"chain":"Worldcoin",
"code3":"WDC", "address_version":"\x49", "magic":""},

ERROR-1:
failed to load /home/maple/.worldcoin/bitcoin.conf: [Errno 2] No such file or directory: u'/home/maple/.worldcoin/bitcoin.conf'
catch_up_rpc: abort
Opened /home/mapleshadow/.worldcoin/blocks/blk00000.dat
Abe initialized.
Listening on http://12.16.2.24:2750

WHY?

Abe tries to load data over RPC, and when that fails, it reads block files. (See default-loader in abe.conf to choose one or the other.)

Abe does not know about Worldcoin, so it looks for RPC credentials in bitcoin.conf. There is no bitcoin.conf, so it logs an error and tries to load blocks from files, starting with blocks/blk00000.dat. It did not find any blocks in that file, so perhaps the file is empty or the block format is not the same as Bitcoin's.

I am surprised that the configured dirname is in /home/mapleshadow but Abe looked for bitcoin.conf in /home/maple.

mapleshadow

member

Activity: 79

Merit: 10

my conf is wdc

wdc.conf:
dbtype sqlite3
connect-args abe.sqlite
datadir = [{
"dirname": "/home/mapleshadow/.worldcoin",
"chain": "Worldcoin",
"code3": "WDC",
"address_version": "\u0049" }]
int-type str
upgrade
port 2750
host 12.16.2.24

DataStore.py:
{"chain":"Worldcoin",
"code3":"WDC", "address_version":"\x49", "magic":""},

ERROR-1:
failed to load /home/maple/.worldcoin/bitcoin.conf: [Errno 2] No such file or directory: u'/home/maple/.worldcoin/bitcoin.conf'
catch_up_rpc: abort
Opened /home/mapleshadow/.worldcoin/blocks/blk00000.dat
Abe initialized.
Listening on http://12.16.2.24:2750

WHY?

Jouke

sr. member

Activity: 426

Merit: 250

Quote from: bbxx on July 05, 2013, 03:53:54 AM

i have some problems with bitcoin blockchain

database size is huge!
i am at block 228k and already it took 31.6 GB!
142,140,449 records inserted so far.

my cpu i5 3570, 4gb ram, 50gb assigned for vdi.
ubuntu server, mysql, 1gb left on ssd Sad

i need only one information, balance on given address.
can i clear some of tables to reduce mysql database size ?

should i use postgresql or maybe there is a fork with reduced db size needed ?
what i can do to reduce resource usage ?

https://bitcointalk.org/index.php?topic=22785.msg1554489;topicseen#msg1554489

bbxx

hero member

Activity: 728

Merit: 500

cryptoshark

i have some problems with bitcoin blockchain

database size is huge!
i am at block 228k and already it took 31.6 GB!
142,140,449 records inserted so far.

my cpu i5 3570, 4gb ram, 50gb assigned for vdi.
ubuntu server, mysql, 1gb left on ssd Sad

i need only one information, balance on given address.
can i clear some of tables to reduce mysql database size ?

should i use postgresql or maybe there is a fork with reduced db size needed ?
what i can do to reduce resource usage ?

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: crendore on July 02, 2013, 02:29:12 PM

Thanks for your awesome contribution to the open source community. I really appreciate the work you've done!

Happy to hear it.

Quote from: crendore on July 02, 2013, 02:29:12 PM

I have been wondering if there is any way to accomplish this blockexplorer.com feature using abe:
In blockexplorer.com i can do this:
http://blockexplorer.com/q/getreceivedbyaddress/ADDRESS/N
Where N specifies the number of confirmations before a deposit is considered valid at that address.

With Abe, i don't seem to be able to accomplish this functionality, eg:
http://bbq.cryptocoinexplorer.com/chain/BBQCoin/q/getreceivedbyaddress/ADDRESS/N
does the same thing as
http://bbq.cryptocoinexplorer.com/chain/BBQCoin/q/getreceivedbyaddress/ADDRESS

I checked in abe.py and it doesn't look like that extra parameter is handled at all.

I know a bit of python and wouldn't be too afraid to dive in and attempt to solve this myself, but before i do that i would like to hear any pointers / advice you would suggest on what the best approach would be to accomplish this (without making too much of a muck of your code).

Actually, this ought to be very simple. I'd do it right now, but I am not at my secure workstation. Where q_getreceivedbyaddress calls abe.store.get_received, it could pass the optional block_height parameter to ignore blocks after a certain number. That number would be something like:

Code:

abe.store.get_block_number(chain['id']) + 1 - N

K1773R

legendary

Activity: 1792

Merit: 1008

/dev/null

Quote from: robanswe on July 02, 2013, 04:27:56 PM

Hello, I really appreciate the work you are putting into this. I currently only got one "problem" with it. It takes forever to do the "initial data load". I'm trying to load the bitcoin blockchain into a mysql db and I'm currently at block 176100 and it takes around 5seconds/block is this normal or I'm I doing something wrong?

db @ tmpfs and its extreme fast Wink

crendore

sr. member

Activity: 363

Merit: 250

Quote from: robanswe on July 02, 2013, 04:27:56 PM

Hello, I really appreciate the work you are putting into this. I currently only got one "problem" with it. It takes forever to do the "initial data load". I'm trying to load the bitcoin blockchain into a mysql db and I'm currently at block 176100 and it takes around 5seconds/block is this normal or I'm I doing something wrong?

It takes a while, depending on your system, but that sounds about right. i think it took me almost 2 days to get it running the first time. What might make sense though, is having the database it is creating hosted seperately, like on mega.co.nz or something, so that people could just download the database directly instead of creating it everytime. Obviously any database dump would be a little bit outdated, but then you could run abe and update from that point.

robanswe

newbie

Activity: 24

Merit: 0

Hello, I really appreciate the work you are putting into this. I currently only got one "problem" with it. It takes forever to do the "initial data load". I'm trying to load the bitcoin blockchain into a mysql db and I'm currently at block 176100 and it takes around 5seconds/block is this normal or I'm I doing something wrong?

Jouke

sr. member

Activity: 426

Merit: 250

Quote from: John Tobey on July 02, 2013, 10:14:22 AM

Quote from: Jouke on July 02, 2013, 02:50:46 AM

Ah, I can look up mempool transactions, but it does not add the addresses in that transaction to the database? I was hoping I could do a /unspent/ on mempool transactions as well.

This is a little tricky to do. Abe does not delete transactions that can never get into the main chain because they contain inputs that are spent elsewhere. /unspent/ excludes these from its output by filtering on block_tx and chain_candidate.in_longest = 1. This also excludes mempool transactions. Can you see them on the address page?

I'll find some time to play with the code.

crendore

sr. member

Activity: 363

Merit: 250

Hey John,

Thanks for your awesome contribution to the open source community. I really appreciate the work you've done!

I have been wondering if there is any way to accomplish this blockexplorer.com feature using abe:
In blockexplorer.com i can do this:
http://blockexplorer.com/q/getreceivedbyaddress/ADDRESS/N
Where N specifies the number of confirmations before a deposit is considered valid at that address.

With Abe, i don't seem to be able to accomplish this functionality, eg:
http://bbq.cryptocoinexplorer.com/chain/BBQCoin/q/getreceivedbyaddress/ADDRESS/N
does the same thing as
http://bbq.cryptocoinexplorer.com/chain/BBQCoin/q/getreceivedbyaddress/ADDRESS

I checked in abe.py and it doesn't look like that extra parameter is handled at all.

I know a bit of python and wouldn't be too afraid to dive in and attempt to solve this myself, but before i do that i would like to hear any pointers / advice you would suggest on what the best approach would be to accomplish this (without making too much of a muck of your code).

Cheers,
Crendore

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: Jouke on July 02, 2013, 02:50:46 AM

Ah, I can look up mempool transactions, but it does not add the addresses in that transaction to the database? I was hoping I could do a /unspent/ on mempool transactions as well.

This is a little tricky to do. Abe does not delete transactions that can never get into the main chain because they contain inputs that are spent elsewhere. /unspent/ excludes these from its output by filtering on block_tx and chain_candidate.in_longest = 1. This also excludes mempool transactions. Can you see them on the address page?

Jouke

sr. member

Activity: 426

Merit: 250

Quote from: John Tobey on July 01, 2013, 10:03:55 AM

Quote from: Jouke on July 01, 2013, 05:59:23 AM

Quote from: John Tobey on June 30, 2013, 06:08:01 PM

Run this SQL and see if it is fixed:

Code:

UPDATE datadir SET chain_id = 1, loader = 'rpc' WHERE datadir_id = 1;

In the datadir table I have the following columns:
datadir_id
dirname
blkfile_number
blkfile_number
blkfile_offset
chain_id
datadir_loader

Whoops, I guess I meant:
UPDATE datadir SET chain_id = 1, datadir_loader = 'rpc' WHERE datadir_id = 1;

Quote from: Jouke on July 01, 2013, 05:59:23 AM

How does Abe keep track of blockfiles and blockfile offsets when it is updating from RPC?

Putting chain_id and datadir_loader in the table was bad design. I think they get their values from the "chain" and "loader" (or "default-loader") config elements when the datadir is first used, and thereafter, the config values are ignored. I plan to make it use the config values only, but meanwhile, I think you can copy the correct values into the table row, and it will work.

Thanks. I see it is adding mempool transactions now. ~~But I did understand correctly that it also updates blocks right?~~ It does.

Ah, I can look up mempool transactions, but it does not add the addresses in that transaction to the database? I was hoping I could do a /unspent/ on mempool transactions as well.

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: diatonic on July 01, 2013, 09:13:19 AM

I'll probably just pick one of those chains. I didn't suspect trying to get 30 chains loaded in Abe would require you to fix a bunch of stuff

Your reports are helping me improve the code. Thank you!!

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: diatonic on July 01, 2013, 12:19:37 PM

In trying to find out the cause of my issue, I found that LTC, FTC, CNC & GME all have the same genesis block hash. Would that cause my issue?

Yes, probably, but it is a bug in Abe, and I'll try to fix it. Thanks.

diatonic

sr. member

Activity: 271

Merit: 250

In trying to find out the cause of my issue, I found that LTC, FTC, CNC & GME all have the same genesis block hash. Would that cause my issue?

Line 32 of src/main.cpp on all of those:

Code:

uint256 hashGenesisBlock("0x12a765e31ffd4059bada1e25190f6e98c99d9714d334efa41a195a7e7e04bfe2");

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: Jouke on July 01, 2013, 05:59:23 AM

Quote from: John Tobey on June 30, 2013, 06:08:01 PM

Run this SQL and see if it is fixed:

Code:

UPDATE datadir SET chain_id = 1, loader = 'rpc' WHERE datadir_id = 1;

In the datadir table I have the following columns:
datadir_id
dirname
blkfile_number
blkfile_number
blkfile_offset
chain_id
datadir_loader

Whoops, I guess I meant:
UPDATE datadir SET chain_id = 1, datadir_loader = 'rpc' WHERE datadir_id = 1;

Quote from: Jouke on July 01, 2013, 05:59:23 AM

How does Abe keep track of blockfiles and blockfile offsets when it is updating from RPC?

Putting chain_id and datadir_loader in the table was bad design. I think they get their values from the "chain" and "loader" (or "default-loader") config elements when the datadir is first used, and thereafter, the config values are ignored. I plan to make it use the config values only, but meanwhile, I think you can copy the correct values into the table row, and it will work.

diatonic

sr. member

Activity: 271

Merit: 250

Quote from: John Tobey on June 30, 2013, 10:10:00 PM

@diatonic Please try the latest code and report back.

Quote from: John Tobey on June 30, 2013, 08:16:58 PM

Quote from: diatonic on June 30, 2013, 06:15:57 PM

The errors I posted above happen when I try to load Litecoin, CHNCoin or GameCoin. I've got about 27 chains loaded now I think at http://altcha.in .

I am working on a fix for this old bug.

Thanks John, I really appreciate the time you've put in to this. I was able to load litecoin by itself in a fresh database, so I dropped the abe database and loaded from scratch. CHNcoin loaded, but I got errors when litecoin loaded. Perhaps they share a genesis block or something that is leading to the duplicate key error I'm seeing:

Code:

failed to load /home/diatonic/.litecoin/bitcoin.conf: [Errno 2] No such file or directory: u'/home/diatonic/.litecoin/bitcoin.conf'
catch_up_rpc: abort
Opened /home/diatonic/.litecoin/blk0001.dat
Exception at 8
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('19'), 'loader': None, 'dirname': u'/home/diatonic/.litecoin', 'id': 33L}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2632, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2890, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "Abe/DataStore.py", line 3014, in import_blkdat
    if not store.offer_existing_block(hash, chain_id):
  File "Abe/DataStore.py", line 2501, in offer_existing_block
    store.offer_block_to_chains(b, frozenset([chain_id]))
  File "Abe/DataStore.py", line 2383, in offer_block_to_chains
    b['top'] = store.adopt_orphans(b, 0, chain_ids, chain_ids)
  File "Abe/DataStore.py", line 1970, in adopt_orphans
    stack.pop()()
  File "Abe/DataStore.py", line 1975, in doit
    store._adopt_orphans_1(stack)
  File "Abe/DataStore.py", line 2086, in _adopt_orphans_1
    store._populate_block_txin(int(next_id))
  File "Abe/DataStore.py", line 1919, in _populate_block_txin
    (block_id, txin_id, oblock_id))
  File "Abe/DataStore.py", line 508, in sql
    store._execute(cached, params)
  File "Abe/DataStore.py", line 485, in _execute
    store.cursor.execute(stmt, params)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
IntegrityError: (1062, "Duplicate entry '577171-1214092' for key 'PRIMARY'")

I'll probably just pick one of those chains. I didn't suspect trying to get 30 chains loaded in Abe would require you to fix a bunch of stuff

Jouke

sr. member

Activity: 426

Merit: 250

Quote from: John Tobey on June 30, 2013, 06:08:01 PM

Quote from: Jouke on June 27, 2013, 04:44:16 PM

I am trying to get Abe to connect through RPC, but I am getting the following error. The initial database loading was trough blkfile.

2013-06-27 23:39:43,338 [5344:MainThread] DataStore DEBUG - no chain_id
2013-06-27 23:39:43,339 [5344:MainThread] DataStore ERROR - Failed to catch up {'blkfile_offset': 40774615, 'blkfile_number': 100068, 'chain_id': None, 'loader': None, 'dirname': '/home/node/.bitcoin-fullindex', 'id': Decimal('1')}
Traceback (most recent call last):
File "/home/node/bitcoin-abe/bitcoin-abe-master/Abe/DataStore.py", line 2594, in catch_up
raise Exception("RPC load failed")
Exception: RPC load failed

Run this SQL and see if it is fixed:

Code:

UPDATE datadir SET chain_id = 1, loader = 'rpc' WHERE datadir_id = 1;

It's a bug, Abe should take the chain and loader from the config, not the database.

In the datadir table I have the following columns:
datadir_id
dirname
blkfile_number
blkfile_number
blkfile_offset
chain_id
datadir_loader

How does Abe keep track of blockfiles and blockfile offsets when it is updating from RPC?

Topic: [ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff - page 28. (Read 220986 times)