Pages:
Author

Topic: [ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff - page 47. (Read 220986 times)

member
Activity: 147
Merit: 11
The day to rise has come.
BTW it's working great now, keep up the good work.
hero member
Activity: 481
Merit: 529
Now I just have to figure out how I can rebase that with git, because I'm misusing abe as a webserver.

Btw: should be possible to add caching headers and gzip compression, right?
Provided it does not complicate the core stuff.  Are you thinking of working on this?  You could do it at a high level of abstraction ("squid") or a very low level (store compressed pages in db, add "last modified" columns to tx, pubkey, and block records) or in between, perhaps some WSGI middleware.  If you are going for low level, some refactoring might be in order beforehand.

To get cache headers on static content, I simply use FastCGI and configure static-path to something Apache can serve directly as described in README-FASTCGI.txt.
donator
Activity: 2772
Merit: 1019
Yup, just that.  No need to read SC source code to figure out magic number and address version byte, no need to touch Python code.  Just pass "--datadir ~/.solidcoin".

Btw, I've cut the stuff down a bit:
Code:
    {"chain":"SolidCoin",
     "code3":"SCN", "address_version":"\x7d", "magic":"\xde\xad\xba\xbe"},

Nice one!

Now I just have to figure out how I can rebase that with git, because I'm misusing abe as a webserver.

Btw: should be possible to add caching headers and gzip compression, right?
hero member
Activity: 481
Merit: 529
  • Native SolidCoin support.

What's that mean? I added SolidCoin before myself by adding all the magic and policy stuff in DataStore.py. That?

Yup, just that.  No need to read SC source code to figure out magic number and address version byte, no need to touch Python code.  Just pass "--datadir ~/.solidcoin".

Btw, I've cut the stuff down a bit:
Code:
    {"chain":"SolidCoin",
     "code3":"SCN", "address_version":"\x7d", "magic":"\xde\xad\xba\xbe"},
donator
Activity: 2772
Merit: 1019
Cool, a release!

  • Native SolidCoin support.

What's that mean? I added SolidCoin before myself by adding all the magic and policy stuff in DataStore.py. That?
hero member
Activity: 481
Merit: 529
Version 0.6 highlights:
  • Python packaging; abe.py moved; run as "python -m Abe.abe".
  • Big speed improvements (c. 10x) for MySQL and SQLite.
  • ODBC tested successfully.
  • IBM DB2 tested successfully.
  • HTTP API functions: getreceivedbyaddress getsentbyaddress.
  • Verify transaction Merkle roots on block import.
  • Show Namecoin-style network fees and name transaction outputs.
  • Adjust coins outstanding and coin-days destroyed for Namecoin-style network fees.
  • Native SolidCoin support.
  • Suppress display of empty chains on home page.
  • Show the search form on /chain/CHAIN pages.
  • Many minor improvements; see the Git log.

Next up are a couple of bug fixes and the test suite.  When the test suite passes, it is 1.0.
hero member
Activity: 481
Merit: 529
Thinking about also hosting a copycat of this... posting here to remind me to check up on it later.  Wink
If you do, it could be abe.john-edwin-tobey.org, since the hosting arrangement there is ending.  I'm switching DNS back to my pathetic excuse for a server, but if it can't take the load, I'll punt to whoever wants to run it.

Of course, anyone with serious uptime requirements is welcome to run a private or public instance.
donator
Activity: 2772
Merit: 1019
Just some minor reformatting which I did.  Thanks!

You're welcome. Love to be able to give back a little code, too.
hero member
Activity: 481
Merit: 529
It returned this.
Code:
C:\abe>python ab.py
at 169336263 of 698256440: '\x00\x00\x00\x00'
at 169394422: '\xf9\xbe\xb4\xd9'
Thanks.  I've committed a fix that I think will get you past this error.

I've had some problem with format_satoshis (worked around them by casting to UNSIGNED in the sql query)
The problem was that format_satoshis expected the caller to cast its first argument to int.  I decided it would be better for the function to do the casting.  (txout_value is in 1e-8 BTC, so all values are integer.)

I could put the sql string directly into the call to selectrow(...). The sql variable was only used for debugging output, which doesn't make too much sense any more anyways since the values are to filled in any more (% -> ?)
I like little variables like that, they help document the intent and aid future tinkerers.

Anything else?
Just some minor reformatting which I did.  Thanks!
donator
Activity: 2772
Merit: 1019
The joining looks correct.  My "way" to format decimals is
Code:
format_satoshis(satoshis, chain)
.

I've had some problem with format_satoshis (worked around them by casting to UNSIGNED in the sql query)

Code:
format_satoshis(decimal.Decimal('12.3456'), 1)
gives
Code:
1.23456E-7.012.3456

other than that, I made alle the changes you requested:

Code:

    def q_getreceivedbyaddress(abe, page, chain):
        """getreceivedbyaddress"""
        addr = wsgiref.util.shift_path_info(page['env'])
        if chain is None or addr is None:
            return 'returns amount of money received by given address (not balance, sends are not subtracted)\n' \
                '/chain/CHAIN/q/getreceivedbyaddress/ADDRESS\n'

        if ADDRESS_RE.match(addr):
          version, hash = decode_address(addr)
          sql = """
            SELECT
              CONVERT(COALESCE(SUM(txout.txout_value),0), UNSIGNED) from pubkey
            JOIN txout on txout.pubkey_id=pubkey.pubkey_id
            JOIN block_tx on block_tx.tx_id=txout.tx_id
            JOIN block b on b.block_id=block_tx.block_id
            JOIN chain_candidate cc on cc.block_id=b.block_id
            WHERE
              pubkey_hash=? AND
              cc.chain_id=? AND
              cc.in_longest=1
            ;"""
          row = abe.store.selectrow(sql, (binascii.hexlify(hash), chain['id']))
          ret = format_satoshis(row[0], chain);
        else:
          ret = 'ERROR: address invalid'

        return ret
        
    def q_getsentbyaddress(abe, page, chain):
        """getsentbyaddress"""
        addr = wsgiref.util.shift_path_info(page['env'])
        if chain is None or addr is None:
            return 'returns amount of money sent from given address\n' \
                '/chain/CHAIN/q/getsentbyaddress/ADDRESS\n'

        if ADDRESS_RE.match(addr):
          version, hash = decode_address(addr)
          sql = """
            SELECT
              CONVERT(COALESCE(SUM(txout.txout_value),0), UNSIGNED) from pubkey
            JOIN txout txout on txout.pubkey_id=pubkey.pubkey_id
            JOIN txin on txin.txout_id=txout.txout_id
            JOIN block_tx on block_tx.tx_id=txout.tx_id
            JOIN block b on b.block_id=block_tx.block_id
            JOIN chain_candidate cc on cc.block_id=b.block_id
            WHERE
              pubkey_hash=? AND
              cc.chain_id=? AND
              cc.in_longest=1
            ;"""
          row = abe.store.selectrow(sql, (binascii.hexlify(hash), chain['id']))
          ret = format_satoshis(row[0], chain);
        else:
          ret = 'ERROR: address invalid'

        return ret

Anything else?

EDIT: I could put the sql string directly into the call to selectrow(...). The sql variable was only used for debugging output, which doesn't make too much sense any more anyways since the values are to filled in any more (% -> ?)
member
Activity: 147
Merit: 11
The day to rise has come.
Code:
#!/usr/bin/env python
import BCDataStream
ds = BCDataStream.BCDataStream()
ds.map_file(open(r"C:\Users\user\AppData\Roaming\Bitcoin\blk0001.dat", "rb"),
            169336263)
offset = ds.read_cursor
magic = ds.read_bytes(4)
print "at %d of %d: %s" % (offset, len(ds.input), repr(magic))
if magic[0] == chr(0):
    ds.read_cursor = offset
    while ds.read_cursor < len(ds.input):
        b = ds.read_bytes(1)[0]
        if b != chr(0):
            ds.read_cursor -= 1
            break
    if ds.read_cursor + 4 <= len(ds.input):
        magic = ds.read_bytes(4)
        print "at %d: %s" % (ds.read_cursor, repr(magic))
    else:
        print "%d near end %d" % (ds.read_cursor, len(ds.input))
It returned this.
Code:
C:\abe>python ab.py
at 169336263 of 698256440: '\x00\x00\x00\x00'
at 169394422: '\xf9\xbe\xb4\xd9'
hero member
Activity: 481
Merit: 529
  • SQL Query correct? (longest chain only, correct joining of txout in getsentbyaddress?)
  • Handling of decimal values correct? Is it "your way"? Probably not since I needed to "import decimal"

I didn't implement the "confirmations" filter yet.

The joining looks correct.  My "way" to format decimals is
Code:
format_satoshis(satoshis, chain)
.

Couple of requests.  Instead of IFNULL, use the portable COALESCE with the same arguments.  Instead of '%s' and the % operator, use an unquoted question mark (?) and pass a tuple of bind values as the second argument to selectrow.  (selectrow is the same as selectall but returns just the first row.)  (Abe's database abstraction layer converts the standard qmark parameter style to whatever the driver needs.)  Please capitalize SQL keywords like SELECT FROM JOIN ON WHERE AND for easier readability.  Please prefix error messages with "ERROR:" as per BBE convention.  And please show the usage message if address is None, even if chain is given.  This is my convention where address is required, as in q_translate_address.

Thanks!
hero member
Activity: 481
Merit: 529
Quote
chain not found for magic '\x00\x00\x00\x00' in block file C:\Users\user\AppData\Roaming\Bitcoin\blk0001.dat at offset 169336263
Though i am not running bitcoin at the moment.

Hmm, running will not help with this error.  Have you redownloaded the block chain since the last abe.py run?  If so,
Code:
UPDATE datadir SET blkfile_number = 1, blkfile_offset = 0
will force a rescan.

If not, well, now would be a good time to implement scanning for the magic number sequence, but that is not trivial.  Maybe skipping past '\x00' would solve it here.  I am curious what this program prints if you run it in the bitcoin-abe directory:
Code:
#!/usr/bin/env python
import BCDataStream
ds = BCDataStream.BCDataStream()
ds.map_file(open(r"C:\Users\user\AppData\Roaming\Bitcoin\blk0001.dat", "rb"),
            169336263)
offset = ds.read_cursor
magic = ds.read_bytes(4)
print "at %d of %d: %s" % (offset, len(ds.input), repr(magic))
if magic[0] == chr(0):
    ds.read_cursor = offset
    while ds.read_cursor < len(ds.input):
        b = ds.read_bytes(1)[0]
        if b != chr(0):
            ds.read_cursor -= 1
            break
    if ds.read_cursor + 4 <= len(ds.input):
        magic = ds.read_bytes(4)
        print "at %d: %s" % (ds.read_cursor, repr(magic))
    else:
        print "%d near end %d" % (ds.read_cursor, len(ds.input))
member
Activity: 147
Merit: 11
The day to rise has come.
Quote
chain not found for magic '\x00\x00\x00\x00' in block file C:\Users\user\AppData\Roaming\Bitcoin\blk0001.dat at offset 169336263
Though i am not running bitcoin at the moment.
donator
Activity: 2772
Merit: 1019
John,

have you implemented q/getreceivedbyaddress and q/getsentbyaddress yet?

I have:

    import decimal.Decimal

    def q_getreceivedbyaddress(abe, page, chain):
        """getreceivedbyaddress"""
        if chain is None:
            return 'returns amount of money received by given address (not balance, sends are not subtracted)\n' \
                '/chain/CHAIN/q/getreceivedbyaddress/ADDRESS*\n'
        addr = wsgiref.util.shift_path_info(page['env'])

        if ADDRESS_RE.match(addr):
          version, hash = decode_address(addr)
          sql = """
            select
              ifnull(sum(txout.txout_value),0) from pubkey
            join txout on txout.pubkey_id=pubkey.pubkey_id
            join block_tx on block_tx.tx_id=txout.tx_id
            join block b on b.block_id=block_tx.block_id
            join chain_candidate cc on cc.block_id=b.block_id
            where
              pubkey_hash='%s' and
              cc.chain_id=%s and
              cc.in_longest=1
            ;""" % (binascii.hexlify(hash), chain['id'])
          #ret = NETHASH_HEADER + '\n' + sql + '\n\n' # debugging
          rows = abe.store.selectall(sql)
          ret = (rows[0][0] / decimal.Decimal('1E8')).to_eng_string()
        else:
          ret = 'address invalid'

        return ret
       
    def q_getsentbyaddress(abe, page, chain):
        """getsentbyaddress"""
        if chain is None:
            return 'returns amount of money sent from given address\n' \
                '/chain/CHAIN/q/getsentbyaddress/ADDRESS\n'
        addr = wsgiref.util.shift_path_info(page['env'])

        if ADDRESS_RE.match(addr):
          version, hash = decode_address(addr)
          sql = """
            select
              ifnull(sum(txout.txout_value),0) from pubkey
            join txout txout on txout.pubkey_id=pubkey.pubkey_id
            join txin on txin.txout_id=txout.txout_id
            join block_tx on block_tx.tx_id=txout.tx_id
            join block b on b.block_id=block_tx.block_id
            join chain_candidate cc on cc.block_id=b.block_id
            where
              pubkey_hash='%s' and
              cc.chain_id=%s and
              cc.in_longest=1
            ;""" % (binascii.hexlify(hash), chain['id'])
          #ret = NETHASH_HEADER + '\n' + sql + '\n\n' # debugging
          rows = abe.store.selectall(sql)
          ret = (rows[0][0] / decimal.Decimal('1E8')).to_eng_string()
        else:
          ret = 'address invalid'

        return ret



If you want to use it, please take a look at the following things especially:

  • SQL Query correct? (longest chain only, correct joining of txout in getsentbyaddress?)
  • Handling of decimal values correct? Is it "your way"? Probably not since I needed to "import decimal"

I didn't implement the "confirmations" filter yet.


legendary
Activity: 1400
Merit: 1005
Thinking about also hosting a copycat of this... posting here to remind me to check up on it later.  Wink
member
Activity: 147
Merit: 11
The day to rise has come.
Anyway i updated my bitcoin blk file and copied it then ran the command and waiting the abe.py to update.

Code:
UPDATE datadir SET blkfile_offset=0 WHERE dirname='C:\Users\user\AppData\Roaming\Bitcoin'
hero member
Activity: 481
Merit: 529
Come to think of it, in Linux Abe reads the block file even if Bitcoin is running, but Windows may not allow that.  Anyway, your next step would be to run the client, let it catch up, and let Abe keep loading.

And of course please report whether this is indeed the problem.  Abe will resume loading if you either restart it with the same arguments as before or try to load a page in your browser.  Sometimes Abe stops loading because of a miscellaneous error such as a full disk, and it would output the error message but serve the incomplete history as you observed.  So we have to rule that out.
hero member
Activity: 481
Merit: 529
Now that you mentioned it, It shows block 124783

Ah!  I guess it's not clear: Abe will not download the block chain but relies on a recent run of bitcoin.exe or bitcoind.exe to have done so.  Another item for the readme.

Come to think of it, in Linux Abe reads the block file even if Bitcoin is running, but Windows may not allow that.  Anyway, your next step would be to run the client, let it catch up, and let Abe keep loading.
member
Activity: 147
Merit: 11
The day to rise has come.
Now that you mentioned it, It shows block 124783
Pages:
Jump to: