Topic: sweet mother of god, the blockchain is HUGE (Read 4119 times)
Mine took well over a bloody week!
so the applets are the security problem, not the underlying program?
can I just download the application without the servlets and applets?
can I just download the application without the servlets and applets?
Yes, u can. But I'm not sure if Multibit doesn't require to use applets.
so the applets are the security problem, not the underlying program?
can I just download the application without the servlets and applets?
can I just download the application without the servlets and applets?
Java isn't a browser extension. There is a Java runtime that can be used a browser plug-in, but I don't think this is the same Java required by Multibit.
not to interrupt the discussion on how to best serve the blockchain off ramdisk (not being sarcastic, very pleased if the thread has some actual utility) but can anyone clarify or expand on the above statement?
Java has different types of executable programs:
1. Just an application (ran via command line on a local computer) [Multibit]
2. Servlets (ran on a server inside a web server)
3. Applets (ran in a web browser like Flash) [browser plug-in]
Java isn't a browser extension. There is a Java runtime that can be used a browser plug-in, but I don't think this is the same Java required by Multibit.
not to interrupt the discussion on how to best serve the blockchain off ramdisk (not being sarcastic, very pleased if the thread has some actual utility) but can anyone clarify or expand on the above statement?
I'm testing this here and if you want to suggest some other configuration I might test it too.
Here are some basic pre-tests.
Download speed:
Write speed on SSD (this varies greatly, from ~90 MB/s to ~200 MB/s):
Write speed on ramdisk:
This is a Xeon X3440 with 8GB ram, not sure if specific details are required for such tests that are barely considering stddev, and so on.
Sample code for getting block count each minute:
Block count evolution on SSD, first 60 minutes. Ending count at 161266
Block count evolution on RAMdisk, first 21 minutes. Ending count at 175481
The x axis describes the minute the block count was collected. RAMdisk beats SSD hands down, but I cannot afford running bitcoin there. These results may vary greatly, maybe if someone suggests a set of specific nodes to connect to this could be better controlled.
Here is the raw data for SSD and RAMdisk:
Here are some basic pre-tests.
Download speed:
Code:
$ wget -v http://mirrors.syringanetworks.net/debian-cd/7.2.0-live/amd64/iso-hybrid/debian-live-7.2-amd64-standard.iso
Length: 483393536 (461M) [application/octet-stream]
Saving to: ‘debian-live-7.2-amd64-standard.iso’
100%[============================================================>] 483,393,536 18.0MB/s in 23s
2013-11-14 06:39:06 (20.4 MB/s) - ‘debian-live-7.2-amd64-standard.iso’ saved [483393536/483393536]
Write speed on SSD (this varies greatly, from ~90 MB/s to ~200 MB/s):
Code:
$ dd if=/dev/zero of=test bs=100k count=1k conv=fdatasync
1024+0 records in
1024+0 records out
104857600 bytes (105 MB) copied, 0.682985 s, 154 MB/s
Write speed on ramdisk:
Code:
$ dd if=/dev/zero of=test bs=100k count=1k conv=fdatasync
1024+0 records in
1024+0 records out
104857600 bytes (105 MB) copied, 0.0436794 s, 2.4 GB/s
This is a Xeon X3440 with 8GB ram, not sure if specific details are required for such tests that are barely considering stddev, and so on.
Sample code for getting block count each minute:
Code:
import time
import json
from getpass import getpass
import subprocess
from clirpc import make_basic_auth, generic_cb, AuthClientRPC
btc_host = raw_input("Host: ")
btc_port = int(raw_input("Port: "))
btc_user = getpass("User: ")
btc_pwd = getpass("Password: ")
btc_auth = make_basic_auth(btc_user, btc_pwd)
out = open('result', 'w')
data = (time.time(), 0)
out.write("%s, %d\n" % data)
out.flush()
print 'starting'
subprocess.Popen(["./bitcoind"]) # Adjust for passing other parameters as needed.
time.sleep(60)
rpc = AuthClientRPC(btc_host, btc_port, btc_auth, protocol='http')
print rpc
while True:
try:
count_now = json.loads(rpc.getblockcount().result())['result']
except Exception, e:
print 'error', e
time.sleep(0.5)
continue
data = (time.time(), count_now)
out.write("%s, %d\n" % data)
out.flush()
print data
time.sleep(60)
Block count evolution on SSD, first 60 minutes. Ending count at 161266
Code:
180000 ++----------------------------------------------------------------++
| |
160000 ++ *******************+
| ***************** |
140000 ++ **** ++
| ****** |
120000 ++ ******** ++
| ** |
100000 ++ ** ++
| ** |
80000 ++ ** ++
| * |
60000 ++ ** ++
| * |
40000 ++ ** ++
| * |
20000 ++ ** ++
| * |
0 ++-+-+-++-*-+-+-+-+-++-+-+-+-+-+-++-+-+-+-+-+-++-+-+-+-+-+-++-+-+-++
+ + + + + + + +
50 00 10 20 30 40 50 00
Block count evolution on RAMdisk, first 21 minutes. Ending count at 175481
Code:
180000 ++---------------------------------------------------------***----++
| ******* |
160000 ++ ****** ++
| ******* |
140000 ++ ************ ++
| *** |
120000 ++ ********* ++
| **** |
100000 ++ * ++
| * |
80000 ++ * ++
| **** |
60000 ++ * ++
| * |
40000 ++ * ++
| * |
20000 ++ ********* ++
| **** |
0 ++--*-+--+--+--+-+--+--+-+--+--+--+-+--+--+-+--+--+-+--+--+--+-+--++
+ + + + + + + + + + + + +
26 28 30 32 34 36 38 40 42 44 46 48 50
The x axis describes the minute the block count was collected. RAMdisk beats SSD hands down, but I cannot afford running bitcoin there. These results may vary greatly, maybe if someone suggests a set of specific nodes to connect to this could be better controlled.
Here is the raw data for SSD and RAMdisk:
Code:
1384430385.86, 0
1384430445.9, 20000
1384430505.96, 40500
1384430566.02, 59000
1384430626.09, 78907
1384430686.15, 92907
1384430746.21, 104259
1384430806.26, 113859
1384430866.32, 116434
1384430926.36, 118314
1384430986.37, 119080
1384431046.38, 119140
1384431106.4, 119325
1384431166.42, 119640
1384431226.48, 119640
1384431286.54, 123832
1384431346.61, 129617
1384431406.67, 132636
1384431466.73, 134400
1384431526.76, 134405
1384431586.79, 134905
1384431646.84, 135299
1384431706.91, 137985
1384431766.95, 140577
1384431827.01, 143117
1384431887.08, 145873
1384431947.1, 148941
1384432007.15, 150073
1384432067.21, 150562
1384432127.28, 150754
1384432187.33, 151057
1384432247.37, 151437
1384432307.41, 151908
1384432367.42, 152267
1384432427.49, 152679
1384432487.54, 153062
1384432547.58, 153062
1384432607.64, 153146
1384432667.68, 153417
1384432727.71, 153591
1384432787.76, 153937
1384432847.83, 154722
1384432907.89, 156051
1384432967.95, 156793
1384433028.01, 157349
1384433088.07, 157463
1384433148.12, 157568
1384433208.18, 157664
1384433268.24, 157744
1384433328.3, 158079
1384433388.3, 158079
1384433448.35, 158079
1384433508.36, 158283
1384433568.4, 158357
1384433628.43, 158418
1384433688.49, 158499
1384433748.54, 159061
1384433808.6, 159160
1384433868.66, 159365
1384433928.68, 160766
1384433988.72, 161266
Code:
1384435634.01, 0
1384435694.07, 7677
1384435754.13, 16201
1384435814.2, 17500
1384435874.2, 23000
1384435934.3, 70184
1384435994.36, 106500
1384436054.37, 121000
1384436114.43, 121000
1384436174.49, 124135
1384436234.54, 131073
1384436296.2, 135577
1384436356.26, 138574
1384436416.33, 141458
1384436476.39, 144266
1384436536.46, 147723
1384436596.52, 150015
1384436656.58, 155441
1384436716.64, 160925
1384436776.7, 166218
1384436836.77, 170953
1384436896.83, 175481
new OS on a new box here, been...oh I don't know...year and a half since I have to set bitcoin QT up from scratch
sweet baby zombie jesus, hours and hours, I got hours to go, still 15 weeks back and the weeks are obviously getting bigger all the time
obviously this has to be a tired topic so forgive me, what are the strategies that have been proposed to deal with this? Couple years ago it was inconvenient, now it is ridiculous, couple years from now it will be untenable.
mass adoption indeed, lol
sweet baby zombie jesus, hours and hours, I got hours to go, still 15 weeks back and the weeks are obviously getting bigger all the time
obviously this has to be a tired topic so forgive me, what are the strategies that have been proposed to deal with this? Couple years ago it was inconvenient, now it is ridiculous, couple years from now it will be untenable.
mass adoption indeed, lol
Mine was done after 7 days and 6 hours while a few months back it took 1.5 days.
Hmmm... memcached blockchain implementation for bitcoind anyone? 
memory increase is worth far more than upgrading any other component.
Desktop with 8 GiB and 1 Mb/s spent 2+ weeks to download blockchain.
Virtual machine with 256 (or 512, not sure) MiB and 100 Mb/s spent 1.5 days for the same.
RAM is not the silver bullet.
RAMdisk, not only RAM. The biggest bottleneck is disk access, when disk == ram, that goes away.
Indeed. And, as I think I mentioned before, the biggest download speed enhancement is having a RAMdisk on the sending side. Writing the block file out sequentially isn't hard for spinning disks to do. Seeking backwards through it one un-aligned jump at a time is murder.
RAMdisk -> magnetic drive isn't much slower than RAMdisk -> RAMdisk, but it beats the crap out of magnetic drive -> anything.
memory increase is worth far more than upgrading any other component.
Desktop with 8 GiB and 1 Mb/s spent 2+ weeks to download blockchain.
Virtual machine with 256 (or 512, not sure) MiB and 100 Mb/s spent 1.5 days for the same.
RAM is not the silver bullet.
RAMdisk, not only RAM. The biggest bottleneck is disk access, when disk == ram, that goes away.
memory increase is worth far more than upgrading any other component.
Desktop with 8 GiB and 1 Mb/s spent 2+ weeks to download blockchain.
Virtual machine with 256 (or 512, not sure) MiB and 100 Mb/s spent 1.5 days for the same.
RAM is not the silver bullet.
YES, making a temporary ramdisk and using -datadir makes it sync much faster. This might not be an option for everyone, but for anyone using bitcoind on a heavy duty server, its a no-brainer.
This. I skipped the temporary part and just left the blockchain on ramdisk with backup to fixed disks. The good news is memory is cheap, insanely cheap these days. If you are looking at doing any serious bitcoin backend work $100 or $200 on memory increase is worth far more than upgrading any other component.
set maxconns to 1
then connect=5.9.24.81 (and change it back after it's finished)
i have plenty of bandwidth unless some DoS is going on
then connect=5.9.24.81 (and change it back after it's finished)
i have plenty of bandwidth unless some DoS is going on
Hopefully that box is serving the chain from a RAM disk. Bootstrapping a new node from a spinning disk causes a TON of thrashing. SSDs are decent, but DRAM is king.
YES, making a temporary ramdisk and using -datadir makes it sync much faster. This might not be an option for everyone, but for anyone using bitcoind on a heavy duty server, its a no-brainer.
Java isn't a browser extension. There is a Java runtime that can be used a browser plug-in, but I don't think this is the same Java required by Multibit.
set maxconns to 1
then connect=5.9.24.81 (and change it back after it's finished)
i have plenty of bandwidth unless some DoS is going on
then connect=5.9.24.81 (and change it back after it's finished)
i have plenty of bandwidth unless some DoS is going on
Hopefully that box is serving the chain from a RAM disk. Bootstrapping a new node from a spinning disk causes a TON of thrashing. SSDs are decent, but DRAM is king.
That, bezzeb, was certainly a better executive summary than I expected or deserved.
Thank you for taking the time sir.
As to the other comment of multibit being the "default" client. I tried installing that and got a popup that it required java...well, that swiss cheese pile of excrement is not going on my box period. How is it that the "default" client requires a known insecure browser extension?
Thank you for taking the time sir.
As to the other comment of multibit being the "default" client. I tried installing that and got a popup that it required java...well, that swiss cheese pile of excrement is not going on my box period. How is it that the "default" client requires a known insecure browser extension?
new OS on a new box here, been...oh I don't know...year and a half since I have to set bitcoin QT up from scratch
sweet baby zombie jesus, hours and hours, I got hours to go, still 15 weeks back and the weeks are obviously getting bigger all the time
obviously this has to be a tired topic so forgive me, what are the strategies that have been proposed to deal with this? Couple years ago it was inconvenient, now it is ridiculous, couple years from now it will be untenable.
mass adoption indeed, lol
Trying wiping the entire block chain and starting over from null. There are not really enough active nodes to download the entire chain in less than a week on even the fastest connection because the rates from the nodes is so low. sweet baby zombie jesus, hours and hours, I got hours to go, still 15 weeks back and the weeks are obviously getting bigger all the time
obviously this has to be a tired topic so forgive me, what are the strategies that have been proposed to deal with this? Couple years ago it was inconvenient, now it is ridiculous, couple years from now it will be untenable.
mass adoption indeed, lol
set maxconns to 1
then connect=5.9.24.81 (and change it back after it's finished)
i have plenty of bandwidth unless some DoS is going on
new OS on a new box here, been...oh I don't know...year and a half since I have to set bitcoin QT up from scratch
sweet baby zombie jesus, hours and hours, I got hours to go, still 15 weeks back and the weeks are obviously getting bigger all the time
obviously this has to be a tired topic so forgive me, what are the strategies that have been proposed to deal with this? Couple years ago it was inconvenient, now it is ridiculous, couple years from now it will be untenable.
mass adoption indeed, lol
sweet baby zombie jesus, hours and hours, I got hours to go, still 15 weeks back and the weeks are obviously getting bigger all the time
obviously this has to be a tired topic so forgive me, what are the strategies that have been proposed to deal with this? Couple years ago it was inconvenient, now it is ridiculous, couple years from now it will be untenable.
mass adoption indeed, lol
Hey, you can read up a bit on potential improvements in the milestone .9 release which is in the pipe:
https://bitcoinfoundation.org/blog/?p=290
I mention it because it has some very interesting stuff related to this problem. Of note: Headers first downloading, and Provably prunable outputs. I'll pause while every reads and absorbs the wisdom.
... ... ...Of note for your question:
Headers first is a breakthrough that will allow true bit-torrent style parallel downloading of the blockchain. The reason it's quite slow now is that it must be downloaded in serial sequence from block 0 to now. Headers only means you can essentially just grab all the block headers to build a "table of contents" so to speak for the whole chain. Then AFTER this, you can fetch the transaction details of each block to fill in all the gaps in a massively parallel way.
Proveably Prunable outputs is a new way to help mitigate future size explosion in the block chain by allowing a legitimate way for messages to be inserted into the blockchain - and most importantly to let nodes ignore these messages if they don't want to archive them. It creates the potential to have "archival" nodes on heavy duty servers that store everything for everyone, and then "normal" nodes which just store transactions. Currently we all store everything which is not sustainable for average users as you have discovered.
Not mentioned but coming hopefully before release 1.0: methods to remove spent transactions from the blockchain to start to prune the size down and hopefully level off at some plateau size that is more stable and less exponential. There have been many proposals on how to do this, but i suspect it would involve the spin off of another "type" of bitcoin node which is intended for desktop computer use which has a version of the blockchain that has fully spent transactions summarized by some cryptographic method like merkle trees and then removed to save space. This is a really gnarly / ugly problem though so i think it's been lowered in priority compared to all the other key useability features they are working on.
As for mass adoption: the future will not look like today. Pulling out my crystal ball, i see: Full heavy weight nodes will shift into serious players hands - ie: serious investors, financial institutions and hard core hobbyists. "Light chain" clients with compressed or summarized block chains will fill some middle ground. Light weight clients with no local block chain (like Multibit) will fill another niche. And then for the unrully mobs and unwashed masses - they will use banks with possibly bitcoin backends (using full heavy nodes).
Why? The idea of having the responsibility to keep wallets safe and not letting them get stolen for example by malware is a responsibility which most average folks do not want to have. They will prefer some protection from a bank-like institution to help keep their money safe for some modest fee - kind of like online wallets - or the idealized version of the current banking system from old hollywood films.
according to the devs, the "default" client is no longer bitcoin-qt but Multibit
It takes less than a minute to be fully connected after downloading the client (it's an SPV client,
it doesn't download the blockchain). It'd be somewhat longer if you have to import
old private keys (the older they are, the longer it takes to synchronise) but it's still incomparable to downloading
the whole blockchain.
It takes less than a minute to be fully connected after downloading the client (it's an SPV client,
it doesn't download the blockchain). It'd be somewhat longer if you have to import
old private keys (the older they are, the longer it takes to synchronise) but it's still incomparable to downloading
the whole blockchain.
new OS on a new box here, been...oh I don't know...year and a half since I have to set bitcoin QT up from scratch
sweet baby zombie jesus, hours and hours, I got hours to go, still 15 weeks back and the weeks are obviously getting bigger all the time
obviously this has to be a tired topic so forgive me, what are the strategies that have been proposed to deal with this? Couple years ago it was inconvenient, now it is ridiculous, couple years from now it will be untenable.
mass adoption indeed, lol
Trying wiping the entire block chain and starting over from null. There are not really enough active nodes to download the entire chain in less than a week on even the fastest connection because the rates from the nodes is so low. sweet baby zombie jesus, hours and hours, I got hours to go, still 15 weeks back and the weeks are obviously getting bigger all the time
obviously this has to be a tired topic so forgive me, what are the strategies that have been proposed to deal with this? Couple years ago it was inconvenient, now it is ridiculous, couple years from now it will be untenable.
mass adoption indeed, lol
Jump to: