unbearable blockchain size | Bitcointalksearch.org

RawDog

legendary

Activity: 1596

Merit: 1026

Quote from: Pursuer on January 19, 2017, 07:37:29 AM

it is not exponentially it is linearly, and there is a big difference.

That is not true. Even a linear line with slope zero is 'exponential' - the exponent being 0. 'exponential' is a bullshit term that is overused. Basically, everything is 'exponential'.

franky1

legendary

Activity: 4410

Merit: 4766

Quote from: Yakamoto on January 20, 2017, 07:48:24 PM

Quote from: franky1 on January 20, 2017, 10:17:18 AM

Quote from: morantis on January 20, 2017, 10:09:23 AM

i personnally am not bothered by the wait, i was just stating that, as of this moment, the only issue created by block chain size is the initial setup of full wallet, by this i am implying that there is no problem, as not many people need that full node on their pc. those that want to run the node should expect some time to be used. you want to start a new domain, you expect a possible 24 wait for DNS propagation. technical aspects to all project take longer than the standard use version

and my reply was that you can have the cake and eat it.

rather than having the implementation useless until syncing. thus holding people up and the sync becomes not discrete and noticable enough to frustrate

you can be a full node and not have to wait. simply by devs releasing a implementation that emulates a litenode instantly while it discretely syncs in the background, unnoticed because its no longer a hindrance to utility.

So could something like this be one way to solve the issue of node count that the community has expressed concern over recently?

I'm not well versed in what happens when you run a node and how a lot of the technical aspects affect other portions, but this could be one way to maintain node numbers in my opinion, assuming that you can take the blockchain from the node and sync it to the wallet.

---my reply below is about an earlier concept that deals with the problem of a node waiting to sync.

as i said. it would be a full node. this extra code to grab the utxo set from its connected peers first. and then syncs the blockchain second.
its not an electrum node with a few tweaks. its a full node with a couple tweaks

so seeing as it would be a full node, with all rules. and also as it syncs it is checking the data like all full nodes.
all that changes is that its actually able to spend funds at syncing from block 0 instead of waiting for block 420k+ to be properly usable.

its not like electrum or other lite wallets that rely on a central server. my idea is a proper full node but just things in a different order.. grabbing utxo data from nodes, probably safe to get utxo from all connections to have multiple opinions. and then build the blockchain after.

thus solving the long wait where users have to wait for the node to build its own utxo set

---to address morantis's new scenario about one source of data that multiple softwares can use.
you could set up a bitcoin Deamon on the data store. and then get other 'client' software to RPC call the data from it
but then its not really what i would call a full node anymore for the 'client' implementations

morantis

hero member

Activity: 868

Merit: 503

Quote from: Yakamoto on January 20, 2017, 07:48:24 PM

Quote from: franky1 on January 20, 2017, 10:17:18 AM

Quote from: morantis on January 20, 2017, 10:09:23 AM

i personnally am not bothered by the wait, i was just stating that, as of this moment, the only issue created by block chain size is the initial setup of full wallet, by this i am implying that there is no problem, as not many people need that full node on their pc. those that want to run the node should expect some time to be used. you want to start a new domain, you expect a possible 24 wait for DNS propagation. technical aspects to all project take longer than the standard use version

and my reply was that you can have the cake and eat it.

rather than having the implementation useless until syncing. thus holding people up and the sync becomes not discrete and noticable enough to frustrate

you can be a full node and not have to wait. simply by devs releasing a implementation that emulates a litenode instantly while it discretely syncs in the background, unnoticed because its no longer a hindrance to utility.

So could something like this be one way to solve the issue of node count that the community has expressed concern over recently?

I'm not well versed in what happens when you run a node and how a lot of the technical aspects affect other portions, but this could be one way to maintain node numbers in my opinion, assuming that you can take the blockchain from the node and sync it to the wallet.

what i am calling a full node is not the correct term. a full installation of bitcoin core will(after downloading and syncing and all that) will start the wallet software and the GUI upon start up. almost every normal user can operate like that. you have the ability to use the GUI to perform all functions and can also use the CLI functions from the command line(that is what we have to have). there are various ways to tell the core to act as a server and this activates the daemon or server. that is a full node. your software is performing three functions as a full node, the wallet, the server and the in between parsing of data and commands and what not that make the real bitcoin software do something when you click the GUI

a full node is simply the core running everything and announcing to the network that it is there. do not confuse this with generation or hash mining.

Yakamoto

legendary

Activity: 1218

Merit: 1007

Quote from: franky1 on January 20, 2017, 10:17:18 AM

Quote from: morantis on January 20, 2017, 10:09:23 AM

i personnally am not bothered by the wait, i was just stating that, as of this moment, the only issue created by block chain size is the initial setup of full wallet, by this i am implying that there is no problem, as not many people need that full node on their pc. those that want to run the node should expect some time to be used. you want to start a new domain, you expect a possible 24 wait for DNS propagation. technical aspects to all project take longer than the standard use version

and my reply was that you can have the cake and eat it.

rather than having the implementation useless until syncing. thus holding people up and the sync becomes not discrete and noticable enough to frustrate

you can be a full node and not have to wait. simply by devs releasing a implementation that emulates a litenode instantly while it discretely syncs in the background, unnoticed because its no longer a hindrance to utility.

So could something like this be one way to solve the issue of node count that the community has expressed concern over recently?

I'm not well versed in what happens when you run a node and how a lot of the technical aspects affect other portions, but this could be one way to maintain node numbers in my opinion, assuming that you can take the blockchain from the node and sync it to the wallet.

morantis

hero member

Activity: 868

Merit: 503

Ok, stupid is as stupid does. the purpose of doing all this was to not have to download a wallet, the block chain and wait for sync as a full node if the PC crashes. the idea was to store the block chain and wallet file on a second hard drive and try to force a software wallet to use those. the real solution is much simpler. the wallet software and all its files will reside on the second hard drive. if your primary drive data gets lost or destroyed, you restore the OS to the primary hard drive and the second drive's data is still untouched. Use a desktop link to the wallet exe as either a simple shortcut or as a network link, whatever. using software or a simple script have the wallet.dat file copy to a backup location.

addmenum:franky posted as i was posting this and his question was pretty much answered here. we do a lot of technical playing with bitcoin core and the issue was the fact that everytime there is a computer crash, the block chain and the synced data is gone.

franky1

legendary

Activity: 4410

Merit: 4766

cant you just have a bitcoin node set up to be an electrum server. and the the other users in your intranet, just use electrum to connect to your 'server'

other people have already got other local wallet systems that API call specific servers to grab the unspent data required to locally make a tx on their own system away from the 'server'

..

may i ask why do you want to have a 'full node' that doesnt have its own full dataset..
is it so you can sybil attack by having hundreds of 'full nodes' appearing on the network but only having one data store or is there a specific reason for requiring a fullnode without having the data locally.

morantis

hero member

Activity: 868

Merit: 503

Quote from: DannyHamilton on January 20, 2017, 01:55:14 PM

Quote from: morantis on January 20, 2017, 11:46:37 AM

i have never thought of it or tried, but in some situations, like a small office, could multiple users share the same block chain file. say, three users on separate pc's each with their own wallet.date file, but the actual downloaded block chain data in a central network share. there may be "file in use" errors kicked back at times like a block update, but the internal block update function in their wallet core could be used to stagger the wallets accessing the file, wallet one goes straight there, the block update fires a busy program that delays wallet two from updating and so on.

If this is possible, then most people could, even by themselves, download the chain once, store the file on a separate drive and access it as a network share and auto backup the wallet.dat to the shared directory. then, a full bore computer crash would only mean downloading the fresh wallet software and moving the backed wallet.dat file in and pointing the wallet to the block chain data file

Bitcoin Core is not currently written to support this. It would be possible to do if someone wanted to put the effort into writing the code. There is nothing in the protocol that prevents this.

Another option would be to run a single server as a full node with the ability to connect to other full nodes on the internet. You could then store regular backups of the blockchain on this full node server. Have the other computers on the same network fire-walled so that they can ONLY connect to the local full node server. All these other computers could all have pruning turned on so that they don't store the most recent blocks from the blockchain. Bitcoin Core already supports this setup.

that would work well for a small network that actually used multiple users, but is a little much as a simple backup area. i tempted to puruse the code and find where the daemon declares the location of the file, but that would just piss me off. even a tiny simple code change and backups files or not, i have forked and live in my own little bitcoin world

EDIT:I will look it up later and have a thought. Not sure if Windows has a like function, but Linux can use symbolic links, i wonder if when the code names the bootstrap file if I could somehow create a symbolic link with the same file name. got a few things to look up

franky1

legendary

Activity: 4410

Merit: 4766

Quote from: Kprawn on January 20, 2017, 11:38:13 AM

Even though storage media are also growing linear to block size increases, you will still find that most average computer users will not be able to

sacrifice hard disk space to store the Blockchain. Most of the people I know, have on average 3TB storage capacity. { Eg. 1 TB for OS & Apps and

2 TB on external hard drives for data archiving or backups } Most of these users are power users, and I expect the average computer user will only

have a 500GB to 1TB hard drive, so sacrificing 120GB+ to store the Blockchain, would be a big ask for them. Roll Eyes

this is like saying
World of warcraft or Call of duty should not make another release because each release adds another 60gb of data to a hard drive each year..

halting 5600 interested and active people that want to be nodes, with speculation about what they may or may not want is foolish

blockstream done this foolish attempt in 2015. '2mb is bad because increasing the possible bloat of 52gb a year to 104gb a year is a big ask...'
but now blockstream are suddenly happy with 4mb because they need 3mb of weight area to fit in their clunky confidential payments feature in the near future.

strange isnt it... how they suddenly stop doomsdaying 'little people cant handle 2mb' the very second they realise their own features require 4mb weight blocks

DannyHamilton

legendary

Activity: 3472

Merit: 4801

Quote from: morantis on January 20, 2017, 11:46:37 AM

i have never thought of it or tried, but in some situations, like a small office, could multiple users share the same block chain file. say, three users on separate pc's each with their own wallet.date file, but the actual downloaded block chain data in a central network share. there may be "file in use" errors kicked back at times like a block update, but the internal block update function in their wallet core could be used to stagger the wallets accessing the file, wallet one goes straight there, the block update fires a busy program that delays wallet two from updating and so on.

If this is possible, then most people could, even by themselves, download the chain once, store the file on a separate drive and access it as a network share and auto backup the wallet.dat to the shared directory. then, a full bore computer crash would only mean downloading the fresh wallet software and moving the backed wallet.dat file in and pointing the wallet to the block chain data file

Bitcoin Core is not currently written to support this. It would be possible to do if someone wanted to put the effort into writing the code. There is nothing in the protocol that prevents this.

Another option would be to run a single server as a full node with the ability to connect to other full nodes on the internet. You could then store regular backups of the blockchain on this full node server. Have the other computers on the same network fire-walled so that they can ONLY connect to the local full node server. All these other computers could all have pruning turned on so that they don't store the most recent blocks from the blockchain. Bitcoin Core already supports this setup.

morantis

hero member

Activity: 868

Merit: 503

i have never thought of it or tried, but in some situations, like a small office, could multiple users share the same block chain file. say, three users on separate pc's each with their own wallet.date file, but the actual downloaded block chain data in a central network share. there may be "file in use" errors kicked back at times like a block update, but the internal block update function in their wallet core could be used to stagger the wallets accessing the file, wallet one goes straight there, the block update fires a busy program that delays wallet two from updating and so on.

If this is possible, then most people could, even by themselves, download the chain once, store the file on a separate drive and access it as a network share and auto backup the wallet.dat to the shared directory. then, a full bore computer crash would only mean downloading the fresh wallet software and moving the backed wallet.dat file in and pointing the wallet to the block chain data file

Kprawn

legendary

Activity: 1904

Merit: 1074

Even though storage media are also growing linear to block size increases, you will still find that most average computer users will not be able to

sacrifice hard disk space to store the Blockchain. Most of the people I know, have on average 3TB storage capacity. { Eg. 1 TB for OS & Apps and

2 TB on external hard drives for data archiving or backups } Most of these users are power users, and I expect the average computer user will only

have a 500GB to 1TB hard drive, so sacrificing 120GB+ to store the Blockchain, would be a big ask for them. Roll Eyes

franky1

legendary

Activity: 4410

Merit: 4766

Quote from: morantis on January 20, 2017, 11:19:35 AM

Quote from: franky1 on January 20, 2017, 11:15:17 AM

Quote from: morantis on January 20, 2017, 11:07:11 AM

very true, and i am willing to wait a day of two for a secure wallet for storage. and if i am intending to deposit coins into that wallet to keep them safe, i do not want to try to monkey around the download time at the risk of being any less than 100% secure

like i said.. by downloading red first instead of last. your getting that
the wallet is local. the data is not coming from a central server but from the same diverse nodes. its just downloading the red color first instead of last.

ok, let me say that i am not disagreeing with your idea, simply saying i personally do not need it. if the demand for such changes becomes great enough, you or someone else will make it happen. it is not a bad idea, simply one that i do not need to take advantage of.

im not saying you personally need it. but you raised a point

Quote from: morantis on January 20, 2017, 06:39:18 AM

this is the only time is using bitcoin when there is a huge issue because the the size. i think it is just going to get to the point when fewer and fewer people choose that type of wallet. someone intentionally setting up a full node will have to deal with it

and i informed you that its not actually an issue that cannot be solved in a few lines of code.. because it can be solved with just a few lines of code

morantis

hero member

Activity: 868

Merit: 503

Quote from: franky1 on January 20, 2017, 11:15:17 AM

Quote from: morantis on January 20, 2017, 11:07:11 AM

very true, and i am willing to wait a day of two for a secure wallet for storage. and if i am intending to deposit coins into that wallet to keep them safe, i do not want to try to monkey around the download time at the risk of being any less than 100% secure

like i said.. by downloading red first instead of last. your getting that
the wallet is local. the data is not coming from a central server but from the same diverse nodes. its just downloading the red color first instead of last.

ok, let me say that i am not disagreeing with your idea, simply saying i personally do not need it. if the demand for such changes becomes great enough, you or someone else will make it happen. it is not a bad idea, simply one that i do not need to take advantage of.

franky1

legendary

Activity: 4410

Merit: 4766

Quote from: morantis on January 20, 2017, 11:07:11 AM

very true, and i am willing to wait a day of two for a secure wallet for storage. and if i am intending to deposit coins into that wallet to keep them safe, i do not want to try to monkey around the download time at the risk of being any less than 100% secure

like i said.. by downloading red first instead of last. your getting that
the wallet is local. the data is not coming from a central server but from the same diverse nodes. its just downloading the red color first instead of last.

morantis

hero member

Activity: 868

Merit: 503

Quote from: BillyBobZorton on January 20, 2017, 11:00:24 AM

Quote from: morantis on January 20, 2017, 10:40:23 AM

Quote from: franky1 on January 20, 2017, 10:17:18 AM

Quote from: morantis on January 20, 2017, 10:09:23 AM

i personnally am not bothered by the wait, i was just stating that, as of this moment, the only issue created by block chain size is the initial setup of full wallet, by this i am implying that there is no problem, as not many people need that full node on their pc. those that want to run the node should expect some time to be used. you want to start a new domain, you expect a possible 24 wait for DNS propagation. technical aspects to all project take longer than the standard use version

and my reply was that you can have the cake and eat it.

rather than having the implementation useless until syncing. thus holding people up and the sync becomes not discrete and noticable enough to frustrate

you can be a full node and not have to wait. simply by devs releasing a implementation that emulates a litenode instantly while it discretely syncs in the background, unnoticed because its no longer a hindrance to utility.

for me, as a trader and wallet holder, several wallet types without the need for a download are fine. the reason for me ever setting up a wallet deeper than that is because i am going to do some major work in bitcoin. at a bare minimum, i am choosing the run a node and help the network, on the other end of the spectrum, I might be manually constructing transactions via cli. When i am going to work within the network, i am not just not bothered by the setup time, i want that full block chain. what you are talking about is great for many people, but to me it is like starting a painting with only red paint because i am too impatient to wait for all the colors to arrive. getting around a full block chain sync is bypassing some of the security of the ledger and i see no reason to do that. if i need an immediate solution and a full wallet, i will simply download electrum and the official bitcoin core and use electrum while the other syncs

Trades should learn the habit to download the entire blockchain and store coins locally. Yes, you may not worry about altcoins, as a trader you will likely not hold altcoins long term, but your main stack of bitcoins should be held locally un bitcoin core's wallet, everything else is just not as safe.

very true, and i am willing to wait a day of two for a secure wallet for storage. and if i am intending to deposit coins into that wallet to keep them safe, i do not want to try to monkey around the download time at the risk of being any less than 100% secure

BillyBobZorton

legendary

Activity: 1204

Merit: 1028

Quote from: morantis on January 20, 2017, 10:40:23 AM

Quote from: franky1 on January 20, 2017, 10:17:18 AM

Quote from: morantis on January 20, 2017, 10:09:23 AM

i personnally am not bothered by the wait, i was just stating that, as of this moment, the only issue created by block chain size is the initial setup of full wallet, by this i am implying that there is no problem, as not many people need that full node on their pc. those that want to run the node should expect some time to be used. you want to start a new domain, you expect a possible 24 wait for DNS propagation. technical aspects to all project take longer than the standard use version

and my reply was that you can have the cake and eat it.

rather than having the implementation useless until syncing. thus holding people up and the sync becomes not discrete and noticable enough to frustrate

you can be a full node and not have to wait. simply by devs releasing a implementation that emulates a litenode instantly while it discretely syncs in the background, unnoticed because its no longer a hindrance to utility.

for me, as a trader and wallet holder, several wallet types without the need for a download are fine. the reason for me ever setting up a wallet deeper than that is because i am going to do some major work in bitcoin. at a bare minimum, i am choosing the run a node and help the network, on the other end of the spectrum, I might be manually constructing transactions via cli. When i am going to work within the network, i am not just not bothered by the setup time, i want that full block chain. what you are talking about is great for many people, but to me it is like starting a painting with only red paint because i am too impatient to wait for all the colors to arrive. getting around a full block chain sync is bypassing some of the security of the ledger and i see no reason to do that. if i need an immediate solution and a full wallet, i will simply download electrum and the official bitcoin core and use electrum while the other syncs

Trades should learn the habit to download the entire blockchain and store coins locally. Yes, you may not worry about altcoins, as a trader you will likely not hold altcoins long term, but your main stack of bitcoins should be held locally un bitcoin core's wallet, everything else is just not as safe.

franky1

legendary

Activity: 4410

Merit: 4766

Quote from: morantis on January 20, 2017, 10:54:42 AM

i see no major problem that requires repair

Quote from: morantis on January 20, 2017, 06:39:18 AM

this is the only time is using bitcoin when there is a huge issue because the the size. i think it is just going to get to the point when fewer and fewer people choose that type of wallet. someone intentionally setting up a full node will have to deal with it

morantis

hero member

Activity: 868

Merit: 503

Quote from: franky1 on January 20, 2017, 10:49:59 AM

Quote from: morantis on January 20, 2017, 10:40:23 AM

for me, as a trader and wallet holder, several wallet types without the need for a download are fine. the reason for me ever setting up a wallet deeper than that is because i am going to do some major work in bitcoin. at a bare minimum, i am choosing the run a node and help the network, on the other end of the spectrum, I might be manually constructing transactions via cli. When i am going to work within the network, i am not just not bothered by the setup time, i want that full block chain. what you are talking about is great for many people, but to me it is like starting a painting with only red paint because i am too impatient to wait for all the colors to arrive. getting around a full block chain sync is bypassing some of the security of the ledger and i see no reason to do that. if i need an immediate solution and a full wallet, i will simply download electrum and the official bitcoin core and use electrum while the other syncs

what im saying is you get the cake and eat it.. imagine it as electrum and core combined.. a hybrid

that way your still waiting for all the colors of the painting to download. but your not having to wait for red to download at the end because its given to you at the start.

thus not needing 2 wallets
thus not needing to copy and paste privkeys from one to the other. or spending the funds of one wallet to move it across.. its all one and the same

just saying that i have no issue that both options are there in different forms. i see no reason to break a working system in an effort to reduce the number of tools i can use from two to one. if i needed ten versions of the bitcoin wallet and you said recoding and compiling would reduce me from ten tools to one, awesome. bitcoin gets enough people under the hood playing around and i see no major problem that requires repair

franky1

legendary

Activity: 4410

Merit: 4766

Quote from: morantis on January 20, 2017, 10:40:23 AM

for me, as a trader and wallet holder, several wallet types without the need for a download are fine. the reason for me ever setting up a wallet deeper than that is because i am going to do some major work in bitcoin. at a bare minimum, i am choosing the run a node and help the network, on the other end of the spectrum, I might be manually constructing transactions via cli. When i am going to work within the network, i am not just not bothered by the setup time, i want that full block chain. what you are talking about is great for many people, but to me it is like starting a painting with only red paint because i am too impatient to wait for all the colors to arrive. getting around a full block chain sync is bypassing some of the security of the ledger and i see no reason to do that. if i need an immediate solution and a full wallet, i will simply download electrum and the official bitcoin core and use electrum while the other syncs

what im saying is you get the cake and eat it.. imagine it as electrum and core combined.. a hybrid.. but the extra advantage of not neding electrums central server

that way your still waiting for all the colors of the painting to download. but your not having to wait for red to download at the end because its given to you at the start.

thus not needing 2 wallets
thus not needing to copy and paste privkeys from one to the other. or spending the funds of one wallet to move it across.. its all one and the same

it also wont rely on a electrum server.. it would download the color red from other full nodes on the network like it does with syncing. so it becomes better than electrum by not relying on electrums central server.. simply by downloading red first. rather than waiting to collate the data to get red last

morantis

hero member

Activity: 868

Merit: 503

Quote from: franky1 on January 20, 2017, 10:17:18 AM

Quote from: morantis on January 20, 2017, 10:09:23 AM

i personnally am not bothered by the wait, i was just stating that, as of this moment, the only issue created by block chain size is the initial setup of full wallet, by this i am implying that there is no problem, as not many people need that full node on their pc. those that want to run the node should expect some time to be used. you want to start a new domain, you expect a possible 24 wait for DNS propagation. technical aspects to all project take longer than the standard use version

and my reply was that you can have the cake and eat it.

rather than having the implementation useless until syncing. thus holding people up and the sync becomes not discrete and noticable enough to frustrate

you can be a full node and not have to wait. simply by devs releasing a implementation that emulates a litenode instantly while it discretely syncs in the background, unnoticed because its no longer a hindrance to utility.

for me, as a trader and wallet holder, several wallet types without the need for a download are fine. the reason for me ever setting up a wallet deeper than that is because i am going to do some major work in bitcoin. at a bare minimum, i am choosing the run a node and help the network, on the other end of the spectrum, I might be manually constructing transactions via cli. When i am going to work within the network, i am not just not bothered by the setup time, i want that full block chain. what you are talking about is great for many people, but to me it is like starting a painting with only red paint because i am too impatient to wait for all the colors to arrive. getting around a full block chain sync is bypassing some of the security of the ledger and i see no reason to do that. if i need an immediate solution and a full wallet, i will simply download electrum and the official bitcoin core and use electrum while the other syncs

Topic: unbearable blockchain size (Read 2213 times)