[ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff - page 50.

molecular

donator

Activity: 2772

Merit: 1019

First of all, John, thanks a lot for Abe, it's awesome as far as I can tell.

I'm currenlty importing blocks into a mysql db (at block 16000 after about 1.5 hours), it'll takes ages (as in: days), but that's ok for me.

One problem I found was with the datadir table:

Quote from: DataStore.py

"""CREATE TABLE datadir (
dirname VARCHAR(500) PRIMARY KEY,
blkfile_number NUMERIC(4),
blkfile_offset NUMERIC(20),
chain_id NUMERIC(10) NULL
)""",

MySql didn't like the VARCHAR(500) primary key. Says it's too long, something about 768 (?) bytes.

Reducing it to 128 helped.

Maybe you should consider using an INT PK here and an index on dirname if you need it? Something within me tells me that having a VARCHAR as Primary Key is somehow bad. Can't substantiate that, but in my own projects, I always use an INT as primary key (I always name it "id", too, but that's another matter). You wouldn't want a VARCHAR(500) as foreign key, would you? That's just wastefull. This is, of course, just a suggestion and I know Abe is currenlty not optimized at all and a change like this will probably be a bitch to upgrade.

Other than that, keep going! I sent my small donation

John Tobey

hero member

Activity: 481

Merit: 529

Hi Yanz,

This error

Code:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xf3' in position 0: ordinal not in range(128)

is fixed in the latest commit. Thanks!

But I don't think it explains the "lockup". Are you sure Abe isn't just loading data? For SQLite the full block chain can take a week or longer. SQLite is okay for small, experimental block chains, though even there, you could wait hours. (Optimization of the initial data load should be a high priority enhancement. Currently, it's slow but simple, since it uses the same mechanism as the run-time new block import, which can not be optimized the way I have in mind.)

Let me know how it goes.

Yanz

full member

Activity: 133

Merit: 100

Hi John,
I can't get nethash to show. It locks up the server and I see python take up 99% of the cpu. I thought it was the database being slow since I'm using sqlite. I installed the mysql handler for python and now it says

Code:

Traceback (most recent call last):
  File "abe.py", line 1741, in 
    sys.exit(main(sys.argv[1:]))
  File "abe.py", line 1735, in main
    store = make_store(args)
  File "abe.py", line 109, in make_store
    store = DataStore.new(args)
  File "/home/henry_root/abe/DataStore.py", line 1766, in new
    return DataStore(args)
  File "/home/henry_root/abe/DataStore.py", line 123, in __init__
    store._init_datadirs()
  File "/home/henry_root/abe/DataStore.py", line 392, in _init_datadirs
    store.binin(addr_vers)))
  File "/home/henry_root/abe/DataStore.py", line 195, in to_hex
    return None if x is None else binascii.hexlify(x)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf3' in position 0: ordinal not in range(128)

I check the database via phpmyadmin and I see that it made all the tables and stuff.

John Tobey

hero member

Activity: 481

Merit: 529

I've added the network hash rate estimates. It's pretty compatible with Block Explorer and adds optional start and stop arguments for fetching less than all history.

http://abe.john-edwin-tobey.org/q/nethash

Hoping someone converts this into graphs like these for alt chains...

John Tobey

hero member

Activity: 481

Merit: 529

Hi, thanks for the report. Update: Fixed in latest. ~~I've reproduced the error using the latest code. I don't have a good fix yet, but adding "store.commit()" after line 886 in DataStore.py gets me past this.~~

Code:

diff --git a/DataStore.py b/DataStore.py
index 2bb250e..12eb096 100644
--- a/DataStore.py
+++ b/DataStore.py
@@ -884,6 +884,7 @@ store._ddl['txout_approx'],
                 "CREATE TABLE abe_test_1 ("
                 " abe_test_1_id NUMERIC(12) PRIMARY KEY,"
                 " foo VARCHAR(10))")
+            store.commit()  # XXX
             id1 = store.new_id('abe_test_1')
             id2 = store.new_id('abe_test_1')
             if int(id1) != int(id2):

Sedo

newbie

Activity: 23

Merit: 0

I am getting this error after having installed Postgresql 8.4, python-crypto, python-psycopg2 and setting up the DB

Code:

Traceback (most recent call last):
  File "abe.py", line 1485, in 
    sys.exit(main(sys.argv[1:]))
  File "abe.py", line 1479, in main
    store = make_store(args)
  File "abe.py", line 87, in make_store
    store = DataStore.new(args)
  File "/home/max/workspace/abe-jtobey-bitcoin-9d671ec/DataStore.py", line 1662, in new
    return DataStore(args)
  File "/home/max/workspace/abe-jtobey-bitcoin-9d671ec/DataStore.py", line 102, in __init__
    store.initialize()
  File "/home/max/workspace/abe-jtobey-bitcoin-9d671ec/DataStore.py", line 538, in initialize
    store.configure()
  File "/home/max/workspace/abe-jtobey-bitcoin-9d671ec/DataStore.py", line 801, in configure
    "No known sequence type works")
Exception: No known sequence type works

any hints?

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: dserrano5 on July 15, 2011, 03:46:12 PM

Quote from: John Tobey on July 15, 2011, 11:21:02 AM

Ah, well, Abe tends to return "500 Server Error" rather than an incomplete page. This, too, fixes itself in a bit if you reload. Not sure which failure mode I prefer.

500 + handler explaining the most probable cause?

Getting better. Throw in an automatic retry or two?

dserrano5

legendary

Activity: 1974

Merit: 1030

Quote from: John Tobey on July 15, 2011, 11:21:02 AM

Ah, well, Abe tends to return "500 Server Error" rather than an incomplete page. This, too, fixes itself in a bit if you reload. Not sure which failure mode I prefer.

500 + handler explaining the most probable cause?

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: theymos on July 15, 2011, 04:33:03 AM

This is a bug unique to the BBE mirror: pages sometimes appear before all of the necessary data is fully committed to the database. It fixes itself in ~30 seconds.

Ah, well, Abe tends to return "500 Server Error" rather than an incomplete page. This, too, fixes itself in a bit if you reload. Not sure which failure mode I prefer.

theymos

administrator

Activity: 5222

Merit: 13032

Quote from: gentakin on July 05, 2011, 03:47:45 PM

Congrats, ABE processed block #134917 correctly: http://john-edwin-tobey.org:2750/block/0000000000000789fe6e014f62a3c32466e53abde8db62517b30e0648d9835c5
While blockexplorer shows "Transactions: " and "Fees: " without any numbers next to it: http://blockexplorer.com/block/0000000000000789fe6e014f62a3c32466e53abde8db62517b30e0648d9835c5

It was a 162 transactions block with lots of fees.

This is a bug unique to the BBE mirror: pages sometimes appear before all of the necessary data is fully committed to the database. It fixes itself in ~30 seconds.

John Tobey

hero member

Activity: 481

Merit: 529

Changes since July 4:

* I've created the Version 0.4 branch, where I intend to backport fixes.

* The chain summary page (the one listing several blocks in the same chain) loads much faster than before at high counts per page.

* Address search accepts an initial substring, still without storing addresses in the database.

* FastCGI support has matured. See README-FASTCGI.txt for setup.

* Abe supports Weeds currency natively.

* The datadir configuration directive can add a new currency without changes to Python code.

* auto-agpl provides a link to download the source directory: a license compliance aid for those not wishing to use a Github fork.

* /chain/Bitcoin/q/getblockcount: first of (I hope) many BBE-compatible APIs.

* Several small fixes and speedups.

John Tobey

hero member

Activity: 481

Merit: 529

Faster site, thanks to RylandAlmanza: http://abe.john-edwin-tobey.org/ Please use this in preference to http://john-edwin-tobey.org:2750/.

First API supported: /chain/{CHAIN}/q/getblockcount
E.g.: http://abe.john-edwin-tobey.org/chain/Bitcoin/q/getblockcount

New currencies: Testnet, Weeds.

Next up: speeding up page loading, starting with the list of blocks. Thanks to those brave enough to try it, even if the initial database load was too much for you.

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: gentakin on July 05, 2011, 03:47:45 PM

Congrats, ABE processed block #134917 correctly: http://john-edwin-tobey.org:2750/block/0000000000000789fe6e014f62a3c32466e53abde8db62517b30e0648d9835c5
While blockexplorer shows "Transactions: " and "Fees: " without any numbers next to it: http://blockexplorer.com/block/0000000000000789fe6e014f62a3c32466e53abde8db62517b30e0648d9835c5

I wonder what the problem was. It seems fixed now. But I can almost guarantee Abe will pick up bugs on its way to being as fast as BlockExplorer.

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: XIU on July 05, 2011, 04:59:46 PM

People actually send 15000BTC without a transaction fee? What if the transaction wouldn't be included?

They could try again with a fee. Either the same coin (from a wallet backup or custom software) or the change. If the second transaction depends on the first but has a big fee, a miner will gain by including the first so they can include the second.

XIU

member

Activity: 84

Merit: 10

Quote from: gentakin on July 05, 2011, 03:47:45 PM

Congrats, ABE processed block #134917 correctly: http://john-edwin-tobey.org:2750/block/0000000000000789fe6e014f62a3c32466e53abde8db62517b30e0648d9835c5
While blockexplorer shows "Transactions: " and "Fees: " without any numbers next to it: http://blockexplorer.com/block/0000000000000789fe6e014f62a3c32466e53abde8db62517b30e0648d9835c5

It was a 162 transactions block with lots of fees.

People actually send 15000BTC without a transaction fee? What if the transaction wouldn't be included?

gentakin

member

Activity: 98

Merit: 10

Congrats, ABE processed block #134917 correctly: http://john-edwin-tobey.org:2750/block/0000000000000789fe6e014f62a3c32466e53abde8db62517b30e0648d9835c5
While blockexplorer shows "Transactions: " and "Fees: " without any numbers next to it: http://blockexplorer.com/block/0000000000000789fe6e014f62a3c32466e53abde8db62517b30e0648d9835c5

It was a 162 transactions block with lots of fees.

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: error on June 28, 2011, 12:14:49 AM

Quote from: bitlotto on June 28, 2011, 12:10:09 AM

Quote from: error on June 27, 2011, 11:08:12 PM

AGPL? Shit. Guess I'll go back to coding my own.

I'm curious what your main complaints are about AGPL? It does guarantee is will be free forever but does limit the code it can be integrated in...if he keeps all the copyright I'm sure if someone gave him the right price he'd offer it under different terms while keeping the community one as is! Wink

With the software as is, it means (among other things) that I have to publish the source code to any changes made to it, such as changes to the site design. I'm not concerned about publishing any changes I make that fix bugs or add features; these I would just send in a pull request anyway even if it wasn't AGPL. But the license requires me to have a whole source code publishing infrastructure if I change a single byte, which is unavoidable.

This is now automated, as described in README.txt:

Quote

The Affero General Public License requires whoever modifies this code
and runs it on a server to make the modified code available to users
of the server. You may do this by forking the Github project (if you
received this code from Github.com), keeping your modifications in the
new project, and linking to it in the page template. Or you may wish
to satisfy the requirement by simply passing "--auto-agpl" to
"abe.py". This option makes all files in the directory containing
abe.py and its subdirectories available to clients. See the comments
in abe.conf for more information.

k

sr. member

Activity: 451

Merit: 250

excellent work. Just sent a small tip.

I really like the extra stats Average Coin Age, Cumulative Coin-days Destroyed etc.
I don't know how difficult it would be but something like the nethash query on BBE which produces a csv would be great - or if it was something similar (i.e. giving a csv of average stats for 100 blocks or whatever) but gave stats for Average Coin Age, Cumulative Coin-days Destroyed, average tx fee that would be super.

keep up the good work. will be following with interest.

John Tobey

hero member

Activity: 481

Merit: 529

Quote from: thallium205 on July 02, 2011, 10:05:12 PM

I have to ask - how in the heck did you acquire enough knowledge about the block chain in order to do this? I have been looking everywhere and can't seem to find much about it.

The block chain is just a sequence of "blocks" whose format is described here: https://en.bitcoin.it/wiki/Protocol_specification

I figured out that the blk0001.dat file is just a concatenation of blocks in wire format, each preceded by two four-byte words: the chain's "magic value" and the block length. I probably checked the C++ code to verify this. The Bitcointools project contains code to parse the blocks. Abe uses that code, only slightly modified to obtain each transaction's hash.

thallium205

newbie

Activity: 42

Merit: 0

I have to ask - how in the heck did you acquire enough knowledge about the block chain in order to do this? I have been looking everywhere and can't seem to find much about it.

Topic: [ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff - page 50. (Read 221044 times)