Block chain size/storage and slow downloads for new users - page 6.

ranochigo

legendary

Activity: 2954

Merit: 4158

Quote from: zvs on February 23, 2015, 09:02:34 PM

Quote from: crazyearner on February 21, 2015, 07:39:04 PM

Block chain download needs to be shrunk down and only have say at least 1 years of active transactions and then to be archived into another system where your wallet can access it by typing a command in and then calling to download archived transactions this way it would reduce the active size of the chain as it does not need to have such a large amount of data to download 20Gb and counting how many people wanting to download that on to their system. I dont think many like this and why they use places liek blockchain.info but poses more of a risk that way online wallet.

Hmm, it's more like 35GB (aha! just read above, oops)

I purchased my first 100GB + HDD probably a decade or more ago?

Bandwidth, well, I guess it depends on your location, but generally all these cable and DSL providers are all about downstream and give jack shit for upstream, at least in the US.

Blockchain download would only use download bandwidth, correct me if I'm wrong. It will use upload speed only if you set it to relay transactions and block.

zvs

legendary

Activity: 1680

Merit: 1000

https://web.archive.org/web/*/nogleg.com

Quote from: crazyearner on February 21, 2015, 07:39:04 PM

Block chain download needs to be shrunk down and only have say at least 1 years of active transactions and then to be archived into another system where your wallet can access it by typing a command in and then calling to download archived transactions this way it would reduce the active size of the chain as it does not need to have such a large amount of data to download 20Gb and counting how many people wanting to download that on to their system. I dont think many like this and why they use places liek blockchain.info but poses more of a risk that way online wallet.

Hmm, it's more like 35GB (aha! just read above, oops)

I purchased my first 100GB + HDD probably a decade or more ago?

Bandwidth, well, I guess it depends on your location, but generally all these cable and DSL providers are all about downstream and give jack shit for upstream, at least in the US.

danielpbarron

full member

Activity: 212

Merit: 100

Daniel P. Barron

Quote from: crazyearner on February 21, 2015, 07:39:04 PM

Block chain download needs to be shrunk down and only have say at least 1 years of active transactions and then to be archived into another system where your wallet can access it by typing a command in and then calling to download archived transactions this way it would reduce the active size of the chain as it does not need to have such a large amount of data to download 20Gb and counting how many people wanting to download that on to their system. I dont think many like this and why they use places liek blockchain.info but poses more of a risk that way online wallet.

It's a little more than 20 gigs.

Code:

$ du -h .bitcoin/
14.4M .bitcoin/database
39.1G .bitcoin/

If this is too much, bitcoin speculation isn't for you. In the hypothetical future you have described, the poor people will just have to get bitcoin checking accounts or something; we're not compromising the network so that they can have easy blockchain access.

lemay697

newbie

Activity: 1

Merit: 0

yes very interesting stuff. . Thanks for the info

crazyearner

legendary

Activity: 1820

Merit: 1001

Block chain download needs to be shrunk down and only have say at least 1 years of active transactions and then to be archived into another system where your wallet can access it by typing a command in and then calling to download archived transactions this way it would reduce the active size of the chain as it does not need to have such a large amount of data to download 20Gb and counting how many people wanting to download that on to their system. I dont think many like this and why they use places liek blockchain.info but poses more of a risk that way online wallet.

danielpbarron

full member

Activity: 212

Merit: 100

Daniel P. Barron

Quote from: Bernie87 on February 21, 2015, 02:51:02 PM

Quote from: danielpbarron on February 21, 2015, 08:34:12 AM

Quote from: Bernie87 on February 21, 2015, 12:25:32 AM

If demand is sufficient, a better solution would be flash memory/usb flash drive(s) sold with the initial blockchain already written on them

That would defeat the purpose of bitcoin. What good is a blockchain that you didn't receive from the network through the lengthy process of consensus verification? You might as well just use a thin client.

That would defeat absolutely nothing. Once thoroughly verified, old versions of the blockchain NEVER change, so that would be very stupid to insist that the only way to get the whole blockchain is to download it all through slow network connections and very slow servers, knowing also that the blockchain tends to grow faster with time; what a stupid waste of network bandwidth and time! Yes, one can of course use a thin client instead but that was not the point of my post.

False. When you download it the usual way, you get it from many different sources. When you get one pre-verified chunk, you get it from one source. If that one source is your own full node, then no problem. If that source is some stranger on the internet, you have a big problem. Maybe at this stage in the game it isn't an issue. But these are certainly the considerations to make if you see bitcoin having any sort of future.

Bernie87

newbie

Activity: 3

Merit: 0

Quote from: danielpbarron on February 21, 2015, 08:34:12 AM

Quote from: Bernie87 on February 21, 2015, 12:25:32 AM

If demand is sufficient, a better solution would be flash memory/usb flash drive(s) sold with the initial blockchain already written on them

That would defeat the purpose of bitcoin. What good is a blockchain that you didn't receive from the network through the lengthy process of consensus verification? You might as well just use a thin client.

That would defeat absolutely nothing. Once thoroughly verified, old versions of the blockchain NEVER change, so that would be very stupid to insist that the only way to get the whole blockchain is to download it all through slow network connections and very slow servers, knowing also that the blockchain tends to grow faster with time; what a stupid waste of network bandwidth and time! Yes, one can of course use a thin client instead but that was not the point of my post.

alishahid

newbie

Activity: 13

Merit: 0

2112

legendary

Activity: 2128

Merit: 1065

Quote from: ranochigo on February 21, 2015, 09:01:02 AM

A normal thumbdrive would be able to last quite sometime even with high IO usage. A few years is already considered long and flash drives would be much cheaper then.

"A few years"? More like "a few months" or maybe even "a few weeks". A database with write-ahead-logging is a perfect example of pessimal application for flash storage: lots of small writes, forced buffer flushes to permanent storage and never read back (unless the database application crashed).

http://en.wikipedia.org/wiki/Write-ahead_logging

If "normal thumbdrive" means the typical cheap drive with controller and wear-leveling optimized for FAT32 file system then I wouldn't be surprised if the device died even before the full current blockchain synchronization was complete (when formatted using any modern file system).

ranochigo

legendary

Activity: 2954

Merit: 4158

Quote from: 2112 on February 21, 2015, 01:05:25 AM

Quote from: Bernie87 on February 21, 2015, 12:25:32 AM

Flash media is ideal for that usage because it's cheap, has standard capacities similar to blockchain size, is efficient with random read-only access, and the blockchain copy can easily be updated from time to time with new blocks accumulated on the hard disk. That's also better for your hard disk health and availability.

Unfortunately the very high write amplification caused by bitcoind will also kill any flash device in a few years.

There's quite bit of work that needs to be done to safely run Bitcoin on a flash device for extended time periods. I'm not aware of any open-source flash-specific database engines.

A normal thumbdrive would be able to last quite sometime even with high IO usage. A few years is already considered long and flash drives would be much cheaper then. A better idea is to use your old harddisk and hook it up to your computer. If you aren't in need of the features available only in Bitcoin Core/ other full chain client, don't use it. A normal SPV client is secure enough for daily usage.

danielpbarron

full member

Activity: 212

Merit: 100

Daniel P. Barron

Quote from: Bernie87 on February 21, 2015, 12:25:32 AM

If demand is sufficient, a better solution would be flash memory/usb flash drive(s) sold with the initial blockchain already written on them

That would defeat the purpose of bitcoin. What good is a blockchain that you didn't receive from the network through the lengthy process of consensus verification? You might as well just use a thin client.

2112

legendary

Activity: 2128

Merit: 1065

Quote from: Bernie87 on February 21, 2015, 12:25:32 AM

Flash media is ideal for that usage because it's cheap, has standard capacities similar to blockchain size, is efficient with random read-only access, and the blockchain copy can easily be updated from time to time with new blocks accumulated on the hard disk. That's also better for your hard disk health and availability.

Unfortunately the very high write amplification caused by bitcoind will also kill any flash device in a few years.

There's quite bit of work that needs to be done to safely run Bitcoin on a flash device for extended time periods. I'm not aware of any open-source flash-specific database engines.

btcminer021

member

Activity: 98

Merit: 10

Mine hard!

Too many words... what's the TL;DR? My new install is taking DAYS to sync.

Bernie87

newbie

Activity: 3

Merit: 0

Quote from: danielpbarron on February 17, 2015, 03:37:57 PM

Quote from: Borisz on February 17, 2015, 03:29:32 PM

I realize that online services are most likely not the best practice and I haven't used them myself as they were going down and popping up, but on my current system I cannot afford 30 something GB of missing space.

I might look into lightweight clients that do not download the entire chain, however I am not sure how secure those are. I will have to read up on that topic as well.

Oh, perhaps you should familiarize yourself with this. It's a $20 device that you can plug a used laptop hard-drive into, plug into your router, and forget about. These instructions will do for now, until we figure out how to get it working without the systemd nightmare. The thing isn't really meant to be your wallet, but you can probably figure out how to connect it to your thin client. You shouldn't rely on someone else's full node; you run a similar risk to relying on someone else's javascript.

Interesting device (that actually costs $50, not $20, for the basic version), but the router still needs to download the whole blockchain and you need a spare disk, so that will not solve the download time problem. If demand is sufficient, a better solution would be flash memory/usb flash drive(s) sold with the initial blockchain already written on them, and a full node software update to allow it to directly read blocks from the flash media, in addition to the newer blocks on the hard disk. Flash media is ideal for that usage because it's cheap, has standard capacities similar to blockchain size, is efficient with random read-only access, and the blockchain copy can easily be updated from time to time with new blocks accumulated on the hard disk. That's also better for your hard disk health and availability.

Borisz

sr. member

Activity: 476

Merit: 251

Quote from: danielpbarron on February 17, 2015, 03:37:57 PM

Quote from: Borisz on February 17, 2015, 03:29:32 PM

I realize that online services are most likely not the best practice and I haven't used them myself as they were going down and popping up, but on my current system I cannot afford 30 something GB of missing space.

I might look into lightweight clients that do not download the entire chain, however I am not sure how secure those are. I will have to read up on that topic as well.

Oh, perhaps you should familiarize yourself with this. It's a $20 device that you can plug a used laptop hard-drive into, plug into your router, and forget about. These instructions will do for now, until we figure out how to get it working without the systemd nightmare. The thing isn't really meant to be your wallet, but you can probably figure out how to connect it to your thin client. You shouldn't rely on someone else's full node; you run a similar risk to relying on someone else's javascript.

I know about the PogoPlug, wanted to use it for another project. I have a raspberry Pi and managed to get an altcoin client "partially" running (some node connections cut out, didn't have the time to properly troubleshoot) I think I could get bitcoin working with that. The question of storage still remains, 2.5" external HDD would do yes.

In this case I would be still concerned that I have to remotely connect to my wallet. Even though it is on my server. But I get it, it is a step better than someone else's service on the internet.

irfansial

newbie

Activity: 1

Merit: 0

what is the bitcoin an which aresa is about to be handel rth orgidnistiom is post by bitcoin

danielpbarron

full member

Activity: 212

Merit: 100

Daniel P. Barron

Quote from: Borisz on February 17, 2015, 03:29:32 PM

I realize that online services are most likely not the best practice and I haven't used them myself as they were going down and popping up, but on my current system I cannot afford 30 something GB of missing space.

I might look into lightweight clients that do not download the entire chain, however I am not sure how secure those are. I will have to read up on that topic as well.

Oh, perhaps you should familiarize yourself with this. It's a $20 device that you can plug a used laptop hard-drive into, plug into your router, and forget about. These instructions will do for now, until we figure out how to get it working without the systemd nightmare. The thing isn't really meant to be your wallet, but you can probably figure out how to connect it to your thin client. You shouldn't rely on someone else's full node; you run a similar risk to relying on someone else's javascript.

Borisz

sr. member

Activity: 476

Merit: 251

Quote from: danielpbarron on February 17, 2015, 03:11:43 PM

And it could happen again..

Your coins would be safe offline -- as in, the private keys are never on a networked machine.

I know that the private key is safe if it is offline, as long as given malware is not interfering. Which has probably similar chances as people trying to inject on my connection.

Quote from: danielpbarron on February 17, 2015, 03:11:43 PM

I'm not sure how you skip from "javascript wallets aren't safe" to "cold wallets aren't safe." The point is that you re-download your wallet software every single time you open up blockchain.info. It's not a one-time thing.

"If even cold wallets"...
I tried to imply here that just because your private key is on a computer - that was supposedly never connected to the internet- it doesn't mean that it is completely safe. Coupling directly to the previous statement.

I realize that online services are most likely not the best practice and I haven't used them myself as they were going down and popping up, but on my current system I cannot afford 30 something GB of missing space.

I might look into lightweight clients that do not download the entire chain, however I am not sure how secure those are. I will have to read up on that topic as well.

danielpbarron

full member

Activity: 212

Merit: 100

Daniel P. Barron

Quote from: Borisz on February 17, 2015, 03:06:39 PM

Quote from: danielpbarron on February 17, 2015, 02:39:51 PM

Quote from: Borisz on February 17, 2015, 02:18:36 PM

Quote from: danielpbarron on February 17, 2015, 01:30:40 PM

Quote from: Borisz on February 17, 2015, 01:18:26 PM

I think I'll move my coins to blockchain.info, I cannot/ do not really want to run a core node consuming 30+ GB of HDD space.

That is a terrible idea.

Quote from: BingoBoingo

On their blog, Blockchain.info has disclosed that a routine update left them serving insecure code to customers using their wallet between 12:00 AM and 2:30 AM GMT today. All customers who used the Blockchain.info web wallet to interface to create wallets, generate addresses, or send transactions are reported to be affected. The problem given the scope appears to be that Blockchain.info was serving weak pseudo-random number generating software.

Based on the description this doesn't affect me. I guess there are plenty of people who are happy with their service overall.

Of course it effects you, unless it's possible to keep your own copy of the javascript and never let them serve you another (a practice I doubt you were employing). Every time you open up that site in your browser you're hoping that someone doesn't give you malicious javascript. It doesn't have to be on their end; if someone is able to compromise your communications channel they can serve you any code they want. And by 'they' I mean USG.

The incident linked happened once, for 2.5 hours.

Also based on this I should also hope that nobody has installed an undetectable malware on my computer that logs my password for my wallet.

If even cold wallets can be compromised where would my coins be safe anyway?

And it could happen again..

Your coins would be safe offline -- as in, the private keys are never on a networked machine. I'm not sure how you skip from "javascript wallets aren't safe" to "cold wallets aren't safe." The point is that you re-download your wallet software every single time you open up blockchain.info. It's not a one-time thing.

Borisz

sr. member

Activity: 476

Merit: 251

Quote from: danielpbarron on February 17, 2015, 02:39:51 PM

Quote from: Borisz on February 17, 2015, 02:18:36 PM

Quote from: danielpbarron on February 17, 2015, 01:30:40 PM

Quote from: Borisz on February 17, 2015, 01:18:26 PM

I think I'll move my coins to blockchain.info, I cannot/ do not really want to run a core node consuming 30+ GB of HDD space.

That is a terrible idea.

Quote from: BingoBoingo

On their blog, Blockchain.info has disclosed that a routine update left them serving insecure code to customers using their wallet between 12:00 AM and 2:30 AM GMT today. All customers who used the Blockchain.info web wallet to interface to create wallets, generate addresses, or send transactions are reported to be affected. The problem given the scope appears to be that Blockchain.info was serving weak pseudo-random number generating software.

Based on the description this doesn't affect me. I guess there are plenty of people who are happy with their service overall.

Of course it effects you, unless it's possible to keep your own copy of the javascript and never let them serve you another (a practice I doubt you were employing). Every time you open up that site in your browser you're hoping that someone doesn't give you malicious javascript. It doesn't have to be on their end; if someone is able to compromise your communications channel they can serve you any code they want. And by 'they' I mean USG.

The incident linked happened once, for 2.5 hours.

Also based on this I should also hope that nobody has installed an undetectable malware on my computer that logs my password for my wallet.

If even cold wallets can be compromised where would my coins be safe anyway?

Topic: Block chain size/storage and slow downloads for new users - page 6. (Read 228612 times)