natural order of blocks in blk-files?

amaclin

legendary

Activity: 1260

Merit: 1019

Quote from: pepethefrog on November 03, 2016, 04:23:04 AM

Now, if they are not

They are not.

Quote

, it will require you to either know the index structure to be able to quickly look up the position of the next block,

Too difficult for single-pass parser

For example, the block order is ... 99, 100, 101, 200, 102, 103, ... 199, 201, 202, 203...

The easiest way for blockchain parser is to keep in memory the blocks which can not be added to
a ~~memory-chain~~ memory-tree for processing them later. I wanted to eliminate this routine
by asking bitcoin client save them in right order. OK, it is not possible today. I've already written a
code for linearization-on-the-fly. The topic can be closed.

pepethefrog

member

Activity: 120

Merit: 13

Pepe is NOT a hate symbol

Quote from: amaclin on November 03, 2016, 03:30:55 AM

Quote from: pepethefrog on November 03, 2016, 03:07:58 AM

That is good to know.
Pepe has seen some altcoins that have added a 4 byte index to the block headers to number the blocks according to their "height".
This way you can atleast know (during the linear parsing of the chain) that you stay "on track".

This is not a problem. Blocks are linked together with their hashes. No need to check the height field (and this field doesn't exists in early blocks)
Checking "staying on track" is very easy. Some pseudocode:

Code:

void process ( const Block& b )
{
  static Hash256 prev;
  assert ( block.getPrev ( ) == prev );
  prev = block.getKey ( );
// ...your code here...
}

The problem is "what to do with the block loaded from blk-file if it is not possible to link it right now to its parent, because the parent is below in blk-file"

Yes ofcourse, but since the blocks are already in the blk-files, we can assume that the full bitcoin node has already checked and validated their hashes.
Pepe thought that this thread is about whether you can assume that the blocks inside the blk-files are stored in linear order or not?
Now, if they are not, it will require you to either know the index structure to be able to quickly look up the position of the next block,
or, to have a numbered index in the block header of the block to quickly check the linear progression.
Looking up a number is way faster than calculating and comparing hashes.

Just Pepe's 2 cents

amaclin

legendary

Activity: 1260

Merit: 1019

Quote from: pepethefrog on November 03, 2016, 03:07:58 AM

That is good to know.
Pepe has seen some altcoins that have added a 4 byte index to the block headers to number the blocks according to their "height".
This way you can atleast know (during the linear parsing of the chain) that you stay "on track".

This is not a problem. Blocks are linked together with their hashes. No need to check the height field (and this field doesn't exists in early blocks)
Checking "staying on track" is very easy. Some pseudocode:

Code:

void process ( const Block& b )
{
  static Hash256 prev;
  assert ( block.getPrev ( ) == prev );
  prev = block.getKey ( );
// ...your code here...
}

The problem is "what to do with the block loaded from blk-file if it is not possible to link it right now to its parent, because the parent is below in blk-file"

pepethefrog

member

Activity: 120

Merit: 13

Pepe is NOT a hate symbol

Quote from: gmaxwell on October 30, 2016, 07:16:31 PM

Quote from: amaclin on October 29, 2016, 06:29:15 AM

Is there a way to force client 0.13.1 to store blocks in blk-files in their "natural order"?

No. Not reliably, but you can use the script in contrib/linearize to get a linearized copy.

You shouldn't generally assume that you can parse Bitcoin's internal datastructures, they're not a supported interface and may change from version to version.

That is good to know.
Pepe has seen some altcoins that have added a 4 byte index to the block headers to number the blocks according to their "height".
This way you can atleast know (during the linear parsing of the chain) that you stay "on track".

amaclin

legendary

Activity: 1260

Merit: 1019

Quote from: gmaxwell on October 30, 2016, 07:16:31 PM

Quote from: amaclin on October 29, 2016, 06:29:15 AM

Is there a way to force client 0.13.1 to store blocks in blk-files in their "natural order"?

No. Not reliably, but you can use the script in contrib/linearize to get a linearized copy.

linearizing 80+gb is too boring for everyday usage Grin

gmaxwell

staff

Activity: 4284

Merit: 8808

Quote from: amaclin on October 29, 2016, 06:29:15 AM

Is there a way to force client 0.13.1 to store blocks in blk-files in their "natural order"?

No. Not reliably, but you can use the script in contrib/linearize to get a linearized copy.

You shouldn't generally assume that you can parse Bitcoin's internal datastructures, they're not a supported interface and may change from version to version.

2112

legendary

Activity: 2128

Merit: 1073

Quote from: amaclin on October 29, 2016, 06:29:15 AM

Is there a way to force client 0.13.1 to store blocks in blk-files in their "natural order"?

Did you try using "-connect" to force talking to just one peer?

I haven't tried 0.13.1, but with some older versions that resulted in "reproducible ordering", I don't recall if it was "natural order".

amaclin

legendary

Activity: 1260

Merit: 1019

Is there a way to force client 0.13.1 to store blocks in blk-files in their "natural order"?

Topic: natural order of blocks in blk-files? (Read 1051 times)