[BETA] Bitcoin blockchain torrent - page 2.

K1773R

legendary

Activity: 1792

Merit: 1008

/dev/null

Quote from: ibno on October 28, 2012, 05:44:52 AM

Two ideas for improvement:

1) Compression: this will easy shave off 25% of the file size. My 2.6 G blockchain file uncompressed became 1.9 G compressed with zip.

2) Split it up in multiple files. If I stop using bitcoin for a couple of months and then start using it again I might still have a large chunk of the blockchain left on my computer. I really only need the new block. The current mainline has a new way of storing blockfiles (.bincoin/blocks/blk000?.dat instead of .bitcoin/blk000?.dat) each with size 128 MB (instead of the much larger .bitcoin/blk000?.dat files). These smaller, 128 MB, files would be nice to have available via a torrent, making it possible to limit the download to only the latest blocks.

im sry but zip compression totally sucks...
read this fora nice list: http://www.maximumcompression.com/index.html

ibno

newbie

Activity: 11

Merit: 0

Two ideas for improvement:

1) Compression: this will easy shave off 25% of the file size. My 2.6 G blockchain file uncompressed became 1.9 G compressed with zip.

2) Split it up in multiple files. If I stop using bitcoin for a couple of months and then start using it again I might still have a large chunk of the blockchain left on my computer. I really only need the new block. The current mainline has a new way of storing blockfiles (.bincoin/blocks/blk000?.dat instead of .bitcoin/blk000?.dat) each with size 128 MB (instead of the much larger .bitcoin/blk000?.dat files). These smaller, 128 MB, files would be nice to have available via a torrent, making it possible to limit the download to only the latest blocks.

jgarzik

legendary

Activity: 1596

Merit: 1100

Quote from: rini17 on October 26, 2012, 05:09:59 AM

Quote from: ?? on ??

I stopped seeding. There were no leechers for few days now, problem discovered is not fixed, and I'm solo mining (50+ connections). Lips sealed

That may be problem with your client or firewall. I'm seeding only since yesterday evening and now it has > 300M uploaded already, using Deluge 1.3.5 .

Under ideal conditions, where you have enough seeders to make each new download saturate the client's download link, that produces a situation where you have several seeders and many periods of idle time, punctuated by short bursts of network activity.

I'm definitely seeing daily downloads here, but we could use some more seeders nonetheless.

Bittorrent also helps more for unexpected bursts of network traffic -- like those accompanying a new bitcoin version release, for example.

rini17

sr. member

Activity: 340

Merit: 250

GO http://bitcointa.lk !!! My new nick: jurov

Quote from: ?? on ??

I stopped seeding. There were no leechers for few days now, problem discovered is not fixed, and I'm solo mining (50+ connections). Lips sealed

That may be problem with your client or firewall. I'm seeding only since yesterday evening and now it has > 300M uploaded already, using Deluge 1.3.5 .

Richy_T

legendary

Activity: 2576

Merit: 2267

1RichyTrEwPYjZSeAYxeiFBNnKC9UjC5k

Quote from: 2112 on October 25, 2012, 12:29:26 PM

Quote from: jgarzik on October 24, 2012, 08:41:10 AM

We will work around this in the next release of the bitcoin client.

Theoretically it would be sufficient to replace "unsigned int" and "fseek" by "fpos_t" and "fsetpos". But because of "CAutoFile" and "FILE *" mixing the actual fix may be quite complex.

It's been a while since I ran into a similar issue myself but if I recall correctly, it was pretty much a drop-in fix.

2112

legendary

Activity: 2128

Merit: 1073

Quote from: jgarzik on October 24, 2012, 08:41:10 AM

We will work around this in the next release of the bitcoin client.

Theoretically it would be sufficient to replace "unsigned int" and "fseek" by "fpos_t" and "fsetpos". But because of "CAutoFile" and "FILE *" mixing the actual fix may be quite complex.

The only clean install that I have now is Visual Studio 2012, therefore I can't really test it. But I clearly need a Windows development refresher.

jgarzik

legendary

Activity: 1596

Merit: 1100

This 2GB bootstrap.dat import problem should be present on all 32-bit systems, as a consequence of using fseek() on a signed 32-bit long internally.

We will work around this in the next release of the bitcoin client. Until then, 32-bit platform will simply cease importing earlier than expected, but still import a fair amount of the block chain.

Nicolai Larsen

sr. member

Activity: 331

Merit: 250

Earthling

Quote from: K1773R on October 24, 2012, 12:19:50 AM

Quote from: deepceleron on October 23, 2012, 08:17:37 PM

Quote from: deepceleron on October 23, 2012, 09:34:57 AM

....I'm running the similar command on x64 Linux, we'll see after 2 more hours or so if that works vs windows.

I just got finished doing an import of the bootstrap.dat file on ubuntu 12.04 with 1.7.0 PPA Bitcoin install, and it had the same problem - the import stopped at block 189205. It looks like this problem is platform-independent.

Quote from: Nicolai Larsen on October 23, 2012, 04:07:57 PM

I started seeding Wink

Will this also work with web seeds?

What's a "web seed"? If you are seeding, that's better than a torrent hit-it-and-quit-it. If you are using Azureus/Vuze, you should also install the Mainline DHT plugin so your seeding is more easily discoverable.

webseeds are dedicated or VPS seeds Wink

I use uTorrent and I'm seeding. What would be the point of downloading bootstrap.dat and then remove it after? Tongue

A web seed is a file you can link directly to. like, domain.com/downloads/bootstrap.dat - You can add that to the torrent and it will also download from that link.

K1773R

legendary

Activity: 1792

Merit: 1008

/dev/null

Quote from: deepceleron on October 23, 2012, 08:17:37 PM

Quote from: deepceleron on October 23, 2012, 09:34:57 AM

....I'm running the similar command on x64 Linux, we'll see after 2 more hours or so if that works vs windows.

I just got finished doing an import of the bootstrap.dat file on ubuntu 12.04 with 1.7.0 PPA Bitcoin install, and it had the same problem - the import stopped at block 189205. It looks like this problem is platform-independent.

Quote from: Nicolai Larsen on October 23, 2012, 04:07:57 PM

I started seeding Wink

Will this also work with web seeds?

What's a "web seed"? If you are seeding, that's better than a torrent hit-it-and-quit-it. If you are using Azureus/Vuze, you should also install the Mainline DHT plugin so your seeding is more easily discoverable.

webseeds are dedicated or VPS seeds Wink

deepceleron

legendary

Activity: 1512

Merit: 1036

Quote from: deepceleron on October 23, 2012, 09:34:57 AM

....I'm running the similar command on x64 Linux, we'll see after 2 more hours or so if that works vs windows.

I just got finished doing an import of the bootstrap.dat file on ubuntu 12.04 with 1.7.0 PPA Bitcoin install, and it had the same problem - the import stopped at block 189205. It looks like this problem is platform-independent.

Quote from: Nicolai Larsen on October 23, 2012, 04:07:57 PM

I started seeding Wink

Will this also work with web seeds?

What's a "web seed"? If you are seeding, that's better than a torrent hit-it-and-quit-it. If you are using Azureus/Vuze, you should also install the Mainline DHT plugin so your seeding is more easily discoverable.

Nicolai Larsen

sr. member

Activity: 331

Merit: 250

Earthling

I started seeding Wink

Will this also work with web seeds?

2112

legendary

Activity: 2128

Merit: 1073

Quote from: deepceleron on October 23, 2012, 02:24:12 PM

Answer: >2GB file access possible, but less portable coding specifically addressing large file access must be used.

Thanks for letting me know. I just double-checked on the (private) code I used to maintain: I'm was using a custom build of GCC/MinGW/Cygwin made by/for a global data integrator.

Quote from: deepceleron on October 23, 2012, 02:24:12 PM

How about try to replicate the problem yourself? Betcha have it too...

Again, you are right. I most likely wouldn't notice it because of the custom modifications made in 2003-2005 to our Windows build environment. It is almost 2013 and I already forgot what was modified since Microsoft Visual Studio 6.0. It's been almost a decade of patches!

You've prompted me to make a new, truely clean, install of the Windows XP and the associated development tools.

jgarzik

legendary

Activity: 1596

Merit: 1100

At least one C API uses a 32-bit signed integer ("long") on 32-bit platforms, for file offsets and such. It is potentially a limitation due to the seeking that LoadExternalBlockFile() performs in the source code.

deepceleron

legendary

Activity: 1512

Merit: 1036

How about try to replicate the problem yourself? Betcha have it too...

There is a definite reason for the number 2GB coming up; why do you think the blockchain files are split before they reach 2GB? Look at these threads concerning MinGW (which is what Bitcoin is built with):

http://stackoverflow.com/questions/9026896/get-large-file-size-in-c
http://stackoverflow.com/questions/4003405/32-bit-windows-and-the-2gb-file-size-limit-c-with-fseek-and-ftell

Answer: >2GB file access possible, but less portable coding specifically addressing large file access must be used.

This likely hasn't been noticed because everyone using the bootstrap is connected to the Internet and doesn't look at the logs. It takes so long to import blocks (~1.5 hours) that it is not noticed that Bitcoin prematurely hits the network 400MB early for the rest of the blockchain.

2112

legendary

Activity: 2128

Merit: 1073

Quote from: deepceleron on October 23, 2012, 09:34:57 AM

although it is likely caused by the Win32 C libraries not being able to access >2GB in files.

Hmm. I'm not aware of any actual Microsoft Win32 implementation having 2GB limit. It was always 4GB-1. Many "compatible" re-implementations of FAT (or SMB) did have the 2GB limit, but Microsoft never had this problem. (Never here means at least Win95 OSR2 or WinNT 3.11 .)

Various 3rd party add-ons, like on-line antivirus, could create this situation.

My bet is on a flaky disk drive, especially if it is MLC SSD. Please run the usual battery of tests: chkdsk /f /r; smartctl and badblocks -n if you can reboot to Linux.

deepceleron

legendary

Activity: 1512

Merit: 1036

Quote from: jgarzik on October 23, 2012, 12:49:58 PM

Is this on FAT32 or NTFS also?

Windows 7 Service Pack 1 64 Bit.

>fsutil fsinfo ntfsinfo C:
NTFS Volume Serial Number : 0xeebxxxxxxxxxxxx
Version : 3.1
Number Sectors : 0x000000000770d7ff
Total Clusters : 0x0000000000ee1aff
Free Clusters : 0x00000000005a0838
Total Reserved : 0x00000000000007c0
Bytes Per Sector : 512
Bytes Per Physical Sector : 512
Bytes Per Cluster : 4096
Bytes Per FileRecord Segment : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length : 0x0000000009200000
Mft Start Lcn : 0x00000000000c0000
Mft2 Start Lcn : 0x0000000000870d7f
Mft Zone Start : 0x00000000000c9200
Mft Zone End : 0x00000000000c9ec0

jgarzik

legendary

Activity: 1596

Merit: 1100

Is this on FAT32 or NTFS also?

deepceleron

legendary

Activity: 1512

Merit: 1036

Quote from: deepceleron on October 23, 2012, 09:34:57 AM

I just started it up with no -loadblock with the torrent in the empty datadir just for shit's 'n giggles...

Still didn't work right, block importing stops at 189205. It is surprising that the whole blockchain torrent thing and inclusion of block importing into Bitcoin got this far without anyone noticing...

deepceleron

legendary

Activity: 1512

Merit: 1036

Quote from: ?? on ??

Quote from: deepceleron on October 23, 2012, 06:59:45 AM

I just downloaded the torrent and imported it into a new datadir with 0.7.1 with this command:

bitcoind.exe -datadir=C:\datadir -loadblock=C:\bootstrap.dat -connect=127.0.0.1 -detachdb -printtoconsole

Quote from: jgarzik on October 11, 2012, 10:17:49 PM

If the file "bootstrap.dat" is found in the bitcoin data directory, it will validate and import all blockchain data found in that file.

Try without -loadblock, after moving bootstrap.dat to datadir (this might be the stupidest suggestion ever, though).

I just started it up with no -loadblock with the torrent in the empty datadir just for shit's 'n giggles, although it is likely caused by the Win32 C libraries not being able to access >2GB in files. I'm running the similar command on x64 Linux, we'll see after 2 more hours or so if that works vs windows.

The easy solution will probably be to do a binary split of the current bootstrap.dat file on a block boundary below block 189205, and import the two new files with -loadblock=bootstrap.001 -loadblock=bootstrap.002. Having Bitcoin look for multiple dat files when auto-importing would be a smaller change than altering the compile environment. Easier still is just get a fully built dat/idx set and stick it in the datadir.

deepceleron

legendary

Activity: 1512

Merit: 1036

I just downloaded the torrent and imported it into a new datadir with 0.7.1 with this command:

bitcoind.exe -datadir=C:\datadir -loadblock=C:\bootstrap.dat -connect=127.0.0.1 -detachdb -printtoconsole

However, 193000 blocks were not imported, it stopped after 189205. I verified the hash is correct with SHA256:
a3f258e7af030165360596e4cb0b9beb24b4ce97352c22e65349b89ad5fc5d3e *bootstrap.dat

SetBestChain: new best=0000000000000214aa18 height=189201 work=388855085476008295362 date=07/15/12 15:45:16
ProcessBlock: ACCEPTED
SetBestChain: new best=00000000000005478db7 height=189202 work=388862608030743090363 date=07/15/12 16:41:47
ProcessBlock: ACCEPTED
SetBestChain: new best=0000000000000854e370 height=189203 work=388870130585477885364 date=07/15/12 16:53:22
ProcessBlock: ACCEPTED
SetBestChain: new best=00000000000009441aeb height=189204 work=388877653140212680365 date=07/15/12 16:56:00
ProcessBlock: ACCEPTED
SetBestChain: new best=0000000000000802f60d height=189205 work=388885175694947475366 date=07/15/12 17:12:54
ProcessBlock: ACCEPTED
Loaded 189205 blocks from external file in 5073035ms
Loading addresses...
ERROR: CAddrman::Read() : open failed
Invalid or missing peers.dat; recreating
Loaded 0 addresses from peers.dat 4ms
RandAddSeed() 196108 bytes
mapBlockIndex.size() = 189206
nBestHeight = 189205
setKeyPool.size() = 100
mapWallet.size() = 0
mapAddressBook.size() = 1
Done loading
ThreadRPCServer started
Error: To use bitcoind, you must set a rpcpassword in the configuration file:
C:\datadir\bitcoin.conf

Does the torrent really have blocks up to (193000, uint256("0x000000000000059f452a5f7340de6682a977387c17010ff6e6c3bd83ca8b1317"))? The python script says it should. This looks like a bitcoin bug.

Reference: https://bitcointalksearch.org/topic/m.1266479

Additional Theory: bitcoind x32 win can't read bootstrap files past 2GB, the generated blk0002.dat is only 49M, blk0001.dat+blk0002.dat=2.00GB which corresponds to import dying after reading 2.00GB of the 2.32 GB bootstrap.

Topic: [BETA] Bitcoin blockchain torrent - page 2. (Read 57756 times)