Topic: Exact binary map of database blockchain?! - page 2. (Read 3417 times)

Nakamoto/BTC vs. bookkeeping
As I wrote

Satoshi Nakamoto writes:

Abstract. A purely peer-to-peer version of electronic cash would allow online payments to be sent directly from one party to another without going through a financial institution. Digital signatures provide part of the solution, but the main benefits are lost if a trusted third party is still required to prevent double-spending.

We propose a solution to the double-spending problem using a peer-to-peer network.
The network timestamps transactions by hashing them into an ongoing chain of hash-based proof-of-work, forming a record that cannot be changed without redoing the proof-of-work. The longest chain not only serves as proof of the sequence of events witnessed, but proof that it came from the largest pool of CPU power. As long as a majority of CPU power is controlled by nodes that are not cooperating to attack the network, they'll generate the longest chain and outpace attackers. The network itself requires minimal structure. Messages are broadcast on a best effort basis, and nodes can leave and rejoin the network at will, accepting the longest proof-of-work chain as proof of what happened while they were gone.
...
In this paper, we propose a solution to the double-spending problem using a peer-to-peer distributed timestamp server to generate computational proof of the chronological order of transactions. The system is secure as long as honest(??) nodes collectively control more CPU power than any cooperating group of attacker nodes.

9. Combining and Splitting Value (unbelievable!!))
Although it would be possible to handle coins individually (), it would be unwieldy to make a SEPARATE TRANSACTION FOR EVERY CENT() IN A TRANSFER.
To allow value to be split and combined,() transactions contain multiple inputs and outputs. Normally there will be either a single input
from a larger previous transaction or multiple inputs combining smaller amounts, and at most two outputs:
one for the payment, and one returning the change, if any, back to the sender. ?
(unbelievable!!)

12. Conclusion
We have proposed a system for electronic transactions without relying on trust.
[cmnt: misleading! we rely on trust. Trusted "party" is just like a bank the public blockchain!!]

We started with the usual framework of coins made from digital signatures, which provides strong control of ownership, but is incomplete without a way to prevent double-spending.
To solve this, we proposed a peer-to-peer network using proof-of-work to record a public history of transactions that quickly becomes computationally impractical for an attacker to change if honest(?) nodes control a majority of CPU power.
The network is robust in its unstructured simplicity. Nodes work all at once with little coordination. They do not need to be identified, since messages are not routed to any particular place and only need to be delivered on a best effort basis. Nodes can
leave and rejoin the network at will, accepting the proof-of-work chain as proof of what
happened while they were gone. They vote with their CPU power, expressing their acceptance of
valid blocks by working on extending them and rejecting invalid blocks by refusing to work on
them. Any needed rules and incentives can be enforced with this consensus mechanism.
========================

It might be a sacred cow, but I can't help it:
The basic logic of BTC as explained by Nakamoto himself, is wrong.

This does not mean, BTC is not usable at all.
But most complications have nothing todo with encryption or the public peer-to-peer database/network.
These are secondary technical questions, used except the public database in all "normal" electronic banking.

The problems have todo with the fact, that Satoshi Nakamoto as cryptologist forgot to implement the basics of normal bookkeeping.
He ignored the fact, that his blockchain is logically nothing else than a normal bank and should work like that, of course.
His main factual interest ("prevent double-spending") would be -since logically impossible- no special problem at all.

Satoshi Nakamoto, naming himself the "founder" of BTC might be an expert in cryptology.

This might explain that all what he writes in his basic article
"Bitcoin: A Peer-to-Peer Electronic Cash System"
http://bitcoin.org/bitcoin.pdf
has NOTHING todo with basic knowledge of money and bookkeeping.

His main factual problem is his so called "prevent double-spending".
Normally "a trusted third party [bank] is required to prevent double-spending" he tells us.

Sorry.
Never heard that "double-spending" i.e. spending the same money (or other things) twice is possible at all.

In case of real coins (and all other things) each child knows that, I hope.
In case of cashless payments its exactly the same since each payment reduces my disposable fund.

Where no money, there no payment.
Controlling that by software designed to manage whichever currency is normally the easiest thing at all.

Security and privacy are very important, but totally different questions.
They have as in all other cases (cars, gold or potatoes) logically nothing todo with the things protected.

But Satoshi Nakamoto sees that quite different, stylizing prevent of "double-spending" of BTC to the main task BTC has to care for.

Talking about Bitcoins in its title the central topic BTC is mentioned in his paper not even ONCE.
Instead he only speaks about important but secondary problems, about hashes, keys, "proof-of-work" etc.
But first comes the basic logic, then security and encryption - everywhere.

This might explain that even the basic logic of BTC is at least for me not understandable at all.
Perhaps he didn't understand it himself.

@etotheipi

Be sure, any halfway experienced developer having to communicate with other Computers/datas
first of all studies the BASICS, often called "protocols".

This was one of the first things I did, of course - weeks ago.
Spent hours and days searching usable, complete and reliable infos, starting with Satoshis paper (in this respect totally unusable).
I really couldn't and can't believe it.

I am working with databases since decades, first programming them myself (yes!), then using half a dozen of all the very similar DBMS (dBase, SQL, MySQL, BdB ...).

The basics are logically always the same: Headers, records, fields, indexfiles.
If variable field lens are used, we need delimiters or len specifiers for them.
Any encryption, hashes, Endianness, compression...  are next to be explained.

I normally have no problem with all that. Normal electronic banking belongs to the easiest things I do.
Since databases never are loaded into memory they normally are not splitted as in BTC with its numerous blkxxx.dat + revxxx.dat
This and much more I would like to understand.

Seearching for infos about BTC I found a mess of superficial, inconstistent mixtures of this and that, advertising and promotion mixed with technical infos, even "Messages" and DB-structures -although complete different things- mixed together, nothing fully explained.

Why James, Bitrick, etotheipi tried to provide what I am searching for?
Thats why I asked my silly questions.

Where eg. is the date/time of transfers ? The "lock_time" is a different thing, mostly zero.
The block creation time might be interesting for miners or other internals, not for applications.

Just because you mentioned it above, lets take another example:
Above you write about "BlockHeader(80)"
whereas in the https://en.bitcoin.it/wiki/Protocol_specification
the field sizes of Block Headers are 81.
You know it yourself:
One byte difference on this level is enough to crash a complete app.

Lvm, what bank do u work for? And what is your job there? I expect you do functional maintenance or something similar? I work at a bank as well, and I develop software as well. I concur that the bitcoin specs aren't anything near to the specs for like a SEPA transfer, but you forget one thing: because banks all have their own software (most still written in cobol :-p), banks have a need for perfect specs. Bitcoin is bitcoin. One client which uses the wallet and blockchain. There can never be discussion between bitcoin and itself, it simply works.

Thats what I, kjj and ethotheipi tried to explain, ethotheipi even went much farther than that giving you the closest thing to specs I've ever seen on the subject (thanks for that btw!)

Bottom line, if you want exact specs, read the code, bitcoin is not a bank and is not by any means "required" to hand out specs or api's. If you have a specific question (other than gimme the specs) because of a project you are working on, ask the questions instead of the specs.

Lastly, don't try harder, try different ;-)

Ahh, fuck it.  I was trying to cut you some slack because I figured english wasn't your first language, but you found it out.

All of it is top secret.  The published source code is fake, the extensive documentation on the wiki is all fake, even the posts on the forum and to the mailing list are fake.  Blocks are really just incomprehensible blocks of random data

LvM, I don't know what the problem is.  That protocol page very precisely lays out all of the serializations of block data and messages.  And it's accurate.  Everyone who has developed any Bitcoin software has used the information on that page, and errors subsequently flushed out if they were found.  It's not 100% complete, but it's accurate in what it says, even if it's not the most visually pleasing/organized descriptions possible. 

What kjj was alluding to was that there's deeper meanings to a lot of these things.  And while documentation could be improved, a lot of you just get from experience.  If the documentation is lacking, help fix it.  But for where you're at, there is more than enough there. It describes the binary maps exactly as you wanted.  Like I said, just open blocks/blk00000.dat and look at the first 300 bytes in a hex editor and compare against the data on that page.  You'll get it.  But it can't be spoon-fed to you.

The question is specific, absolutely precise, simple, quite normal and most important if third party datas must be used.
I would and could answer the same question regarding my datas/databases within minutes.

The blockchain must be used by each client, by all users of BTC.
So I can't help but assume: the details are unknown or hidden deliberately.
Top secret

Maybe you would be better off asking specific questions.  All of the information has been provided, but we are apparently unable to translate our answers into a form that is simple enough for you. 

LvM - the point was that you are going to have to put in some elbow grease like the rest of us and figure out the minutiae.  Once you look at some binary maps and match them up against the serializations described on the protocol page, it will seem a lot simpler.  Only so much of it can be spelled out for you.  Trying to spell it out even further isn't all that productive, because you're not going to "get it" until you dive in anyway, and then those documents will seem entirely sufficient.  

100% of the details may not be there.  But most of them are.  It's accurate.  Ask questions.  Look at other implementations.  

Open blocks/blk00000.dat and read the first 300 bytes.  Then match it up against the protocol page and what I wrote above.  You'll see the magic bytes, the numBytes, the 80-byte header, the numTx, then a list of Tx.  You can then jump to an arbitrary block.  Jump to block 170 which has the first real transaction (it has two tx: a coinbase tx and then are regular pay-to-address transaction).  

Bitcoins are money.

The basic logic of money transfers from A to B normally belongs to the easiest things in the world.
Technical things like encryption or the structure of involved databases change nothing in the very, very simple logic of money transfers.

In short:
A minus for the payer is a plus for the receiver.
At least time and reason of transfers are always added normally.
All is collected and balanced in the accounts of payers/receivers.
Easiest bookkeeping.

Some of these things either do not exist in BTC or simply are somehow hidden.
Where e.g. is the field for the transaction date/time?
I assume that there IS one, since Armory restored by the blockchain date and time of my transactions.

Lost after a restore were my manual "Comments" of transactions only.
BTC does indeed not have a field for the reason of transfers.
Other things like the strange "changings" not to mention....
Sorry, but even after reading quite a lot I am far away to understand that.

@kjj
If its really all so "simple" - why isn't it likewise simply explained?

Ah!
What do you mean with "bitcoin is nested" ?? What do you mean with "a lot of the complicated bits are in the interactions"
To begin with: what exactly is that "interaction"?
So far I only heard from transfers/transactions.
If you mean the same, what exactly do you mean with "complicated bits" (bits?)?

I reject your claim that nothing in the wiki is usable.  It really does contain everything that you need to understand a block.  Just takes some getting used to... 

The "surrounding block" is described exactly above.  There is some confusion on exactly what parts are block and which parts are message.  Most people consider the block proper to be "header + NumTx + TX_0...NumTx".  In practice, you'll pretty much never find those stored or communicated without the magic number and the size, except when using the RPC calls that return block bodies.

Keep in mind that bitcoin is nested, and a lot of the complicated bits are in the interactions.  The block itself is very simple, a short header followed by a list of transactions.  Transactions are simple too, a header (really just the version field), a list of inputs, a list of outputs, and a footer (just the locktime field).  Inputs and outputs are where things start getting strange, but even those are pretty simple in representation.

Thanks.  I did a lot of this because of the learning curve to get into this stuff.  Bitcoin is complicated.   But if you want to dig into it, you'll have to do a little messing around on your own.  I don't want to ruin all the fun for you

The rest of the serialization is trivial once you know the transactions.  The header is exactly 80 bytes, 6 values.  And you can look up how VAR_INTs are read from that protocol page.  The magic bytes are f9beb4d9.  The protocol page isn't pleasant, but it's not useless, either.

So far I could not use anything in the BTC wiki.
From the technical point of view we find quite a lot of more or less superficial descriptions of this and that.

But this for the transactions looks much better, thank you!
http://dl.dropboxusercontent.com/u/1139081/BitcoinImg/TxBinaryMap.png

Still missing an equally exact description of the "surrounding" "block".

And btw many thanks for your other good work, Alan.
Armory is definitely the best wallet I know.
Very well designed, very well explained and due to its even me convincingly explained "deterministic" architecture almost foolproof.

All the complaints of users having lost their money due to password problems with things like bitcoin-qt
https://bitcointalk.org/index.php?topic=85495.0;all
https://bitcointalk.org/index.php?topic=191401.0;all
have an end with Armory.

I can vouch that they are extremely consistent.  Armory does a raw scan of them on every load (which will be changed soon, thank god).  Despite being used by thousands of people over the course of the last 12-18 months, there is rarely any problems with it.   Some of the recent Armory problems have actually had to do with memory usage on 32-bit architectures, and not due to actual blk*.dat corruption (switching to the 64-bit version solved it).

Because the block files are externally indexed, I usually tell people not to make assumptions about what is in them.  The -loadblocks code is a very good example of a parser that can tolerate all manners of garbage between blocks, corrupt blocks, etc.

That said, when I was ripping my files for the bootstrap torrent, I found exactly zero cruft in what my node had accumulated over the last few years.

The serialization of the headers and the transactions are documented on the protocol specification page.  What I wrote above is how those individual pieces come together in the blk*.dat files which contain the entire blockchain.  I did make an exact binary map of the transactions, though it was made before P2SH.  You can see it here.  There's a few other nice inkscape drawings here. 

The only caveat is that now the blk*.dat files are padded to 16MB chunks, so the end of the files may contain lots of zeros.  But this is easy to detect, because blocks are never split between files.  So you only have to check for (isEndOfFile() OR magicBytes==0x00000000) to determine that there are no more blocks in the file.

Yes, thank you.
But searching around I already saw your equal posting in the last year:
https://bitcointalksearch.org/topic/m.1111214
Its not what I mean, namely something like the perfect description of your own Armory wallet.

Using a third party database developers must exactly know everything about its structure, each of its fields...

We get those infos from each normal Bank.
But in BTC this seems to be top secret.

The blk*.dat files stored in the Bitcoin home directory are quite simple -- just a straight concatenation of blocks of the form:


The "BlockSizeBytes" field includes the BlockHeader and NumTx, so you only have to read that and skip that many bytes if you want to jump to the next block.  This is is also how blocks are serialized peer-to-peer. 

Yeah! I am really sorry that an experienced banking developer is so silly.

Sorry, I really need these infos somehow on the lowest level.
Functions/APIs already working somehow with a database are much too high for me.