Complete chainstate export from 2018-10-15 as torrent.

hackbyte

copper member

Activity: 282

Merit: 111

Doing cryptocurrency stuff since 2010.

Quote from: aplistir on October 16, 2018, 05:04:28 PM

Quote from: hackbyte on October 15, 2018, 02:00:33 PM

So, is there any demand for regular exports of this data?

I worked some days on the first one i made at 2018-10-10, the latest one was from 2018-10-15, is there any demand for regular updated files?
I will usually put them just in a torrent because i don't want to set up a website for it or so....
Aynone? Wink

There is demand.
I am one of those, who is interested in downloading that data regularly. But only about once in 3-6 months.

Quote from: NeuroticFish on October 15, 2018, 12:51:10 PM

it looks like a nice project, although I find it mostly "helpful" to the dreamers that think they can brute force wallets

Examining blockchain has other uses too. I used the data in studying "Satoshis mining pattern" and things like that. Was quite interesting.

I also find it interesting to monitor really old bitcoin addresses for unusual traffic. Eg. Following old zombie coins coming back to life, which sometimes happen.

For those things I have separate files of old addresses that are easy to go through with a script comparing them to the "current" chainstate.

For my small needs it would be enough to get a smaller database of only the addresses containing more than 10 BTC (most really old addresses have more than 10BTC in them.) Filesize of such a smaller database is only about 7Mb

Unfortunately I am currently in a country that has expensive internet and wont be downloading much until things get better. (I have to pay 1€/Gb for the net.) Otherwise I would have my own node and get the data myself.

But yes. there is demand, and thanks for the good work

Well, i should be able to change my little python thingie to not only write down any address but even add up the exact deposited value...

Should not take up that much space at all. In the end i would be able to get out a simple list of any address holding up any value.

But, there are already walletwatchers out there and just checking the rpc outputs of a full bitcoin node might be way more fast in indicatin that a particular address has changed....

Altough, having already a list of addresses and their values would help a lot to that.....

I check if i can write that in my python fubar... Wink

Regards,

hacky

aplistir

full member

Activity: 378

Merit: 197

Quote from: hackbyte on October 15, 2018, 02:00:33 PM

So, is there any demand for regular exports of this data?

I worked some days on the first one i made at 2018-10-10, the latest one was from 2018-10-15, is there any demand for regular updated files?
I will usually put them just in a torrent because i don't want to set up a website for it or so....
Aynone? Wink

There is demand.
I am one of those, who is interested in downloading that data regularly. But only about once in 3-6 months.

Quote from: NeuroticFish on October 15, 2018, 12:51:10 PM

it looks like a nice project, although I find it mostly "helpful" to the dreamers that think they can brute force wallets

Examining blockchain has other uses too. I used the data in studying "Satoshis mining pattern" and things like that. Was quite interesting.

I also find it interesting to monitor really old bitcoin addresses for unusual traffic. Eg. Following old zombie coins coming back to life, which sometimes happen.

For those things I have separate files of old addresses that are easy to go through with a script comparing them to the "current" chainstate.

For my small needs it would be enough to get a smaller database of only the addresses containing more than 10 BTC (most really old addresses have more than 10BTC in them.) Filesize of such a smaller database is only about 7Mb

Unfortunately I am currently in a country that has expensive internet and wont be downloading much until things get better. (I have to pay 1€/Gb for the net.) Otherwise I would have my own node and get the data myself.

But yes. there is demand, and thanks for the good work

hackbyte

copper member

Activity: 282

Merit: 111

Doing cryptocurrency stuff since 2010.

Strangely the original posting which i found some days back which brought me to this thing is no longer writeable?!?

Mkay happens. Anyways, over there the programmer which wrote the chainstate tool said he had set up a website hosting his very own daily exports behind a paywall (which i can totally understand). URL https://balances.syndevio.com/

So, is there any demand for regular exports of this data?

I worked some days on the first one i made at 2018-10-10, the latest one was from 2018-10-15, is there any demand for regular updated files?

I will usually put them just in a torrent because i don't want to set up a website for it or so....

Aynone? Wink

Regards,

hacky

hackbyte

copper member

Activity: 282

Merit: 111

Doing cryptocurrency stuff since 2010.

Quote from: NeuroticFish on October 15, 2018, 12:51:10 PM

I will help seeding this, I'll start tomorrow with a computer that stays online a lot.

And maybe in some week-end I'll play a bit with it, it looks like a nice project, although I find it mostly "helpful" to the dreamers that think they can brute force wallets

Yeah, you're welcome. Right now finally my seedhost is catching up on the file (651MB of 2,2GB uploaded), it will take another 2 hours or so until it is fully uploaded.

But then, it will get seeded from a nice vps somehwere in europe with somewhat thick inet connection (100mbit/s uplink iirc, we have still somewhat around 5tb traffic/month free .. so you're welcome anyways! Wink

Hacky

P.s.: The trackers i added are:
udp://public.popcorn-tracker.org:6969/announce and
udp://tracker.opentrackr.org:1337/announce

hackbyte

copper member

Activity: 282

Merit: 111

Doing cryptocurrency stuff since 2010.

Quote from: almightyruler on October 15, 2018, 01:44:22 AM

Hmm, interesting. I've been doing it (very slowly) the naiive way, by querying everything through RPC calls to the Bitcoin Core client. Should be done by next month if I'm lucky. Cheesy

By the way ... doing it this "slow native way" is not a real problem at all, if you already have most of the data exported using the chainstate tool...

After that, i's just checking for updates and sending them into the database. On a plain CSV it might a bit more complicated, if you want to delete old entries which are no longer holding any coins.

Possible alternatives are, using sqlite - even maybe a in-memory only table if you have enought ram - or like i do a postgresql db..

NeuroticFish

legendary

Activity: 3668

Merit: 6382

Looking for campaign manager? Contact icopress!

I will help seeding this, I'll start tomorrow with a computer that stays online a lot.

And maybe in some week-end I'll play a bit with it, it looks like a nice project, although I find it mostly "helpful" to the dreamers that think they can brute force wallets

hackbyte

copper member

Activity: 282

Merit: 111

Doing cryptocurrency stuff since 2010.

Quote from: aliashraf on October 15, 2018, 04:01:24 AM

Although I merrited this topic and OP's commitment deserves more appreciation imo, a more general approach to this problem would be more appropriate.

Thanks for the merit.

Quote from: aliashraf on October 15, 2018, 04:01:24 AM

We have UTXO commitment proposals/implementations trending right now with their most important application being pruning and having full nodes with minimum resources and fast bootstrapping, etc. @spartacusrex started a topic in btctalk few days ago for instance and Paul Sztorc's first version of Drivechain released even later (couple of days ago) has already, implemented UTXO snapshotting.

Two most important differences:
1- UTXO snapshots include output scripts as well.
2- A Merkle Tree like and its root hash is generated and maintained for all UTXOs such that commitment to the state at a given height is feasible.

We got advanced algorithms/data structures like MMR and a complete history of interest in UTXOs.

Hhhmm.. interesting... I'm just "for fun" on this topic whilst doing completely unrelated stuff anyways. But i will look in to the links you mentioned. After all, i still have the full node running here anyways and want to play a bit even with the json-rpc and blockchain data format stuff anyways... So maybe i can dig a bit deeper in this chainstate export code to get the output-scripts too.

By the way. On a different run, i just took a list of all my addresses and got any data which is related to it from blockchain.info/com.
There are input and outputs included, but yeah .. calling the blockchain.com api for somewhat more than 1k adresses is not really a nice thing, so i'm already thinking about finding a better way. As i wrote already, getting the data from the bitcoind directly is probably the next thing i try.. Wink

Hacky

hackbyte

copper member

Activity: 282

Merit: 111

Doing cryptocurrency stuff since 2010.

Quote from: almightyruler on October 15, 2018, 01:44:22 AM

Hmm, interesting. I've been doing it (very slowly) the naiive way, by querying everything through RPC calls to the Bitcoin Core client. Should be done by next month if I'm lucky. Cheesy

Based on the github page for chainstate, the bulk of the output format would be: ;

;

Is this correct? If not, please post a small sample of the output.

Yeah, that's probably what i will do as next step. I found 'bitcoin-abe' over on git which should be able to read blkNNNN.dat files natively and put them into a SQL db for use as blockchain explorer like thing. But sadly the code is somewhat old, won't run on python3 and .. yeah, overall, it's old.. Wink

( you can find it here: https://github.com/bitcoin-abe/bitcoin-abe or using 'pip install abe')

My "problem" with bitcoin-abe is, it wants to run directly on the blockchain .dat files, which means, i should shut down my bitcoin knots node as long as abe is running on the files .... or "just copy the whole blockchain over to somewhere else" .. mhmm

I'm not happy doing that anyways, so .. yeah ... connecting the bitcoind and reading anything i want over the RPC thing might be not really fast but more convenient to me. Wink

The sample of the output is in the posting above this one. Wink

Have fun,

hacky

hackbyte

copper member

Activity: 282

Merit: 111

Doing cryptocurrency stuff since 2010.

Quote from: ?? on ??

So, how do we use the parsed chainstate? Do we use https://github.com/mycroft/chainstate and follow the instruction on "How to Run" section?

Well, i don't know how you want or can use it at all. For me, i just wrote some bits python down to write every address i find in the original csv in to a database table..

Code:

#!/usr/bin/python
import sys, time, psycopg2

# chainstate output
sourcefile = "chainstate.output"

# oeffne chainstate export
filehandle = open(sourcefile, 'r')

# open database connection
dbcon = psycopg2.connect('host=** dbname=** user=** password=**')
dbcur = dbcon.cursor()

counter=1
tablename="btc_address"
fieldname="address"

for line in filehandle:

# read data from csv line
txid, dunno, address, value = line.strip().split(';')

# put it into the db, don't ask, just try and skip if it fails
try:
dbcur.execute("INSERT INTO " + tablename + " (" + fieldname + ") VALUES ('"+str(address)+"')")
dbcon.commit()
print("\033[K %08d written %s\n" % (counter,address), end='', flush=True)
counter += 1
continue
except:
print("\033[K %08d skipped %s\r" % (counter,address), end='', flush=True)
counter += 1
continue

print("\n")

So, whatever you want to do, just have fun. I'm just playing around for educational reasons. Wink

Oh btw, the format of the chainstate.output file is as follows (first 5 lines):

Code:

questron|hackbyte|18:01:48|~| 0> head -n 5 chainstate.output
033e83e3204b0cc28724e147f6fd140529b2537249f9c61c9de9972750030000;0;1KaPHfvVWNZADup3Yc26SfVdkTDvvHySVX;65279
4aa1defd46f10c27ff4d6c2565e1e0e611429cd673b0f7342f30820690030000;0;3PPqDTQgndCHiDxqeB3zayJxXB6UeKAiod;996900
b58cf0f52c86022b046838472fc308cb55c9756071399840ad38de8166040000;0;1NXznFqccAinm9RLEf9eTtppZG9tWJQp8M;259531
e1c9467a885a156e56a29d9c854e65674d581ad75611b02290454b4862060000;1;1LpCmEejWLNfZigApMPwUY9nZTS8NTJCNS;9466355
a1f28c43f1f3d4821d0db42707737ea90616613099234f905dfc6ae2b4060000;1;1FuphZ7xVPGrxthQT1S8X7nNQNByYxAT3V;339500

Have fun

Hacky

hackbyte

copper member

Activity: 282

Merit: 111

Doing cryptocurrency stuff since 2010.

Update; Had to set up some portforwarding and added 2 trackers for the torrent.. Wink

aliashraf

legendary

Activity: 1456

Merit: 1175

Always remember the cause!

Although I merrited this topic and OP's commitment deserves more appreciation imo, a more general approach to this problem would be more appropriate.

We have UTXO commitment proposals/implementations trending right now with their most important application being pruning and having full nodes with minimum resources and fast bootstrapping, etc. @spartacusrex started a topic in btctalk few days ago for instance and Paul Sztorc's first version of Drivechain released even later (couple of days ago) has already, implemented UTXO snapshotting.

Two most important differences:
1- UTXO snapshots include output scripts as well.
2- A Merkle Tree like and its root hash is generated and maintained for all UTXOs such that commitment to the state at a given height is feasible.

We got advanced algorithms/data structures like MMR and a complete history of interest in UTXOs.

almightyruler

legendary

Activity: 2268

Merit: 1092

Hmm, interesting. I've been doing it (very slowly) the naiive way, by querying everything through RPC calls to the Bitcoin Core client. Should be done by next month if I'm lucky. Cheesy

Based on the github page for chainstate, the bulk of the output format would be: ;

;

Is this correct? If not, please post a small sample of the output.

hackbyte

copper member

Activity: 282

Merit: 111

Doing cryptocurrency stuff since 2010.

Heya people,

from this post: https://bitcointalksearch.org/topic/get-list-of-all-addresses-with-a-balance-over-x-267618 i've seen that there might be some interest in some chainstate export which includes any bitcoin utxo address and it's particular value.

Well, here you go:

magnet:?xt=urn:btih:d9a56bd02a0f9c50462b0fab32ed3223d36ff182&dn=chainstate-201810150125CEST.7z

Code:

fornax|hackbyte|01:06:56|/currencies/src/chainstate|0$ time ./chainstate >chainstate.output 2>chainstate.errors

real 17m49.092s
user 13m47.253s
sys 2m49.484s
fornax|hackbyte|01:25:15|/currencies/src/chainstate|0$ 7z a chainstate-201810150125CEST.7z chainstate.output

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,6 CPUs AMD FX(tm)-6300 Six-Core Processor (600F20),ASM,AES-NI)

Scanning the drive:
1 file, 5478234093 bytes (5225 MiB)

Creating archive: chainstate-201810150125CEST.7z

Items to compress: 1

Files read from disk: 1
Archive size: 2261586261 bytes (2157 MiB)
Everything is Ok
fornax|hackbyte|01:43:40|/currencies/src/chainstate|0$ ls -lah chainstate-201810150125CEST.7z
-rw-r--r-- 1 hackbyte hackbyte 2.2G Oct 15 01:43 chainstate-201810150125CEST.7z

I will host this one for some few weeks and will have a somewhat capable seed host on it too.

Have fun..

Oh, how i got it?

Using https://github.com/mycroft/chainstate and my very own full blown bitcoin knots node...

Regards,

hacky

Topic: Complete chainstate export from 2018-10-15 as torrent. (Read 328 times)