Topic: Bitcoin Core blockchain impossible to download now, too slow and too big (Read 498 times)

Learning as I go folks! So no, as I found out, an upgrade does not require the blockchain to be downloaded again. Once downloaded and unpacked, the upgrade is really just the bitcoin-qt file plus a couple more files. Basically copied them into my bitcoin folder and viola! everything runs just fine.

Thanks,

v1.0

p.s. new core version released today:  v0.17.0

 "New info escalates importance: upgrading to 0.16.3 is REQUIRED"

Does the entire blockchain have to be downloaded again now??

Hi, I'm tired of waiting for a long sync and stumbled upon the blocks with a torrent and there are still coins torrented
https://krypto-wallet.org/en/coin/download-blockchain-bitcoin-torrent-file.html

Why is this a problem? I must be missing something.

Excellent- thanks!

Taking this a bit further to rejected transactions...on another altcoin I was mining (not bitcoin, as individual users no longer can mine), apparently I mined my first block ever ( super excited) and got paid, it showed the number of coins I had in my wallet. But then apparently the crypto "forked". I had to upgrade my software to continue mining. However, after the upgrade, my payment for my newly mined block disappeared! I inquired to the sysop about it, he responded that likely my new mined block was mined during a "fork", thereby rejected by all the nodes I was attempting to connect to.

If this is inappropriate to ask about in a bitcoin thread, by all means just tell me I wont bring it up again. But I'm thinking perhaps the same thing could have occurred in the past with Bitcoin itself during a fork? Mined blocks would not be valid?

***

So, after a couple weeks of testing, ie. having my wallet and the entire chain offline on a removable drive, than plugging in and starting the resync all over again, happy to report ( I know I know I'm preaching to the choir) that everything works perfectly! Both times Bitcoin Core has synced back up to the network and brought the chain up to date even after being offline for a week. So---I can happily say that: It is NOT impossible to download now, but still is definitely too slow and too big!

What is holding back implementing this kind of idea? There must be trade-offs of some sort, otherwise it would have been done by now?

They almost immediately filter into non-existence.

Every node on the network verifies every transaction it receives before it adds the transaction to it's own mempool, or relays the transaction to any other connected peers.

Therefore, if anyone tries to connect to any nodes and broadcast an invalid transaction, every one of those connected peers will reject the invalid transaction.  It won't be accepted into their own mempools, AND it won't be relayed to ANY other nodes.

Furthermore, if a node is found to be repeatedly sending invalid data, then the nodes that it is connected to will close that connection so that they don't need to repeatedly check invalid data.

It's done, it is not impossible. You all were right. And I've been won over. So now I shall view the title to the thread as merely now an exasperation of sorts, more of "because it took so long, it seemed impossible". Man, does that feel great though, after such a long time blocks are up to date and finally syncs.

So now what. Well, I exited it the program and disconnected the drive. Replugged, started up the core again, and it was up and synced fairly quick. So, that was about a week ago. I've since unplugged the drive again and its been sitting cold for a week. I want to see how long it takes to resync after a couple weeks. I think its going to annoy me enough to the point of upgrading some hardware..it always spirals into ridiculousness though. For example, if I want more ram, I'll have to upgrade my MB to support it. MY new MB might not be compatible with my current cpu. So then I'll have to change that. etc But what I'm really looking forward to is putting Core on an SSD as many on here have suggested.

DannyHamilton thank you for the video suggestion, I am going to look at it. This is going to be a slow and steady race for me, but eventually I hope to at least grasp enough of the basics to have a proper dialogue about it. The rest of your response I'm going to have to sort through at least 5 or 6 times, there is such great detail in there.

So a newer Bitcoin Core version can process SegWit transactions? I will have to check my version. That is interesting to know about how an older version handles (or doesn't, to be correct) a transaction from a SegWit wallet.

Also, invalid transactions seem important to understand. I want to know what kind of reprocussions it causes in the ecosystem, if any, or do they just eventually filter into non-existance?

Thanks again for all the responses/help.

the main gripe is that people cant really spend their funds until the chain is uptodate. which can be days-weeks to set it up

if only there was a smart dev that could think of some simple solution..
like

an uptodate UTXO set of (lets say current blockheight minus 1) gets its own hash. and that hash is set in the current block data.
making the UTXO set of 1confirms+ verifiable as part of block checks

thus people can then have their wallet grab a very recent UTXO set FIRST. so they can actually go ahead and spend their funds, the same day.
after all sending out a tx using the UTXO set wont hurt the network. if it contains a input that is actually spend or non existent it wont get relayed/accepted into a new block anyway..

...
this way it is also good for light wallet users that just want the UTXO set have some faith that the UTXO set has some checks and validations by the network.

and then it makes downloading the blockchain if they want to part of the networks full validation/data archiving, more of a background feature..
thus then alleviates the requirement for some people to not have to wait 2 weeks or so to download the blockchain before they are upto date enough just to spend funds.

It seems you are lacking some understanding of what bitcoin is and how it works.  As such, you're unable to put your frustrations with the time to synchronize a new wallet into a mental framework that allows you to understand why it takes as long as it does, and what the future is likely to look like.

To get a grasp on the general concepts, I suggest pretending that you don't know anything about bitcoin (or at least assuming that everything you know is wrong) and then watching this video as if it is about something completely different than your current understandings:

https://www.youtube.com/watch?v=bBC-nXj3Ng4

Some of the specific details are generalized or simplified in that video, so it won't give you an exact understanding of every technical detail, but it is a good introduction to the more basic concepts.

Then once you've watched it, let me know if you still have some questions.

Note: That video was created before SegWit was implemented, so it won't cover anything about SegWit at all.


Transactions are that are not yet in a block are not a reliable indication of transfer of value. A transaction without "confirmations" simply means that the transaction is not yet in the blockchain. Those transactions may make it into a block eventually, but they also may be replaced with some other transaction.  This is why it is important to wait for "confirmations" when receiving a transaction from someone that you don't already have a trust relationship with.

So, when there is a HUGE volume of transactions (transactions are being created at a faster rate than they fit into a block) then the miners/pools choose to include the transactions with the highest fees in their blocks (since they get to keep the fees). Transactions with lower fees sit around waiting for later blocks and become part of a backlog of unconfirmed transactions.  If the rate of new high fee transactions eventually drops off, then the backlog of lower fee transactions fills in the block space that becomes available.

There has pretty much been a continuous backlog of transactions waiting for confirmation for a bit more than a year now. As older low fee transactions become confirmed, or abandoned, newer low fee transactions get broadcast and take their place waiting for confirmation. Prior to 2016 blocks were nearly never full.  Bitcoin was brand new and there just wasn't enough use to result in full blocks yet.  As such, the growth of the blockchain from 2009 through 2016 was exponential.  Then starting sometime in 2016, most blocks were nearly full and transaction fees started to increase.  As such, growth of the blockchain throughout 2016 and 2017 was linear.  Then in late 2017 SegWit was introduced.  With that change, the blocks are no longer limited by "size", but instead by "weight" where the "weight" of the SegWit portion of a transaction counts for less than the "weight" of the non-SegWit portion.  As such, the effective size of a block will depend on how much of it is SegWit transactions, however even if it were 100% SegWit, the block size would not exceed 4 megabytes, with the typical size closer to 2.6 megabytes.

SegWit transactions ARE backward compatible, however you won't be able to take advantage of the features of SegWit (such as increased probability of confirmation in any given block for the same transaction & fees) unless you are using SegWit receiving addresses in a SegWit enabled wallet.  Furthermore, a pre-SegWit full node (such as an old version of Bitcoin Core) will not fully validate transactions that spend bitcoins that were previously received at a SegWit address.  Instead it will rely on the hope that someone else on the network is running a SegWit enabled full node which will filter out invalid transactions before they are relayed.

Harddrives are not that expensive anymore, considering that a 1 Tb HDD costs less than $40. I don't agree that an average person can't afford 0.15*40=$6 to store the full blockchain. Also downloading the full blockchain is not impossible either, you just need to have some patience. It took my less than 2 days to download the full blockchain, while I don't have the best hardware. Besides that, you don't really need to have the full blockchain on your computer to use the bitcoin ecosystem. It's your own choice whether you contribute to decentralization by having a full node or use Bitcoin by trusting other's blockchain. Last thing, deframenting your HDD will speed up the syncing process, although Windows does this automatically each week.

Thanks for the encouragement! Gave it a couple hours of rest, and went back to it..there is hope its at 96.99% now.

Going to grind through everyones responses now, fell behind on the weekend.

So, in terms of linear vs exponential growth- I apologize in advance if I keep rehashing this- and someone responded with a graph. So yes looks like linear to me, and it appears my definitions of linear and exponential are somewhat skewed. Anyone wish to elaborate in detail about how Bitcoin growth in terms of the blockchain is not exponential? Would linear growth not indicate that there are never massice fluctuations in increase?

Ok, so if the blocks ARE the ledger, what diffirentiates mined blocks vs a day when suddenly there's 500,000 transactions vs a day of only 20,000? Does this not contribute toward determining a linear vs exponential growth of the blockchain?

@ DannyHamilton said:

"Looks pretty linear to me.

Regardless, the math for the future is pretty simple.  Maximum 4 MB per block.  Average 1 block per 10 minutes.  That's linear growth."

Ok, so you're saying there is also a transaction limit per block?

Hmm, maybe you already answered my question, with your response of the Segwit introduction. Great, now I'm even more confused. Is Segwit implementation backwards-compatible with transactions pre-Segwit? Is there anything I have to do with the Bitcoin core client or will everything just run? I vaguely remember when I set up Electrum it asking me something about if I wanted a classic wallet or Segwit wallet, no idea what it was talking about so I went classic.

Thnx @ CodeWrench, atliens99 and emuLOAD too!

DannyHamilton, your Satoshi references are awesome! Thanks so much. I can't believe he had foresight to know that the average user would actually NOT need the entire blockchain! Especially the quote "The more burden it is to run a node, the fewer nodes there will be." which rings true. People would just get discouraged, and interest would wane off. Also, I appreciate your further detail about pruning without having to download the entire blockchain first- thanks.

Dammit, my response is all over the place and completely disorganized, ugh! Apologies. Usually not my MO.

Don't unplug! Keep going. Once you are up to date it really is easy and smooth. You won't remember the hassle of getting up to date!

Ok- but I won't hold my breathe 

80% to 84% in 10 hours boy o boy. I turned it off and unplugged the drive to give it rest, maybe getting too worked. I should have known better to get an SSD as poster DannyHamilton said. I honestly didn't think it would matter. Oh well, lesson learned.

Good to know about the 10 minute update/week thats not bad at all.

Thanks for response!

Be patient - if you are new, it will take at least a week to download the entire blockchain from scratch.

BUT - once downloaded, you just need to update your wallet for ten minutes a week to keep updated and it should function as smoothly as butter.

Wow- really appreciate all the detailed clarifications plus responses! Going to grind through them some more on the weekend.

At about 80.5% with the blocks..so is it impossible? We shall see. I want to back it up before 90%.

Currently kept on seperate drive than my Windows OS. So just a matter of copying entire Bitcoin folder (subdirectories+files) to my backup drive, correct? Exit the client first..

Looks pretty linear to me.

Regardless, the math for the future is pretty simple.  Maximum 4 MB per block.  Average 1 block per 10 minutes.  That's linear growth.


They are not the best choice for everyone, but if you can't handle synchronizing and maintaining the blockchain they are a workable option.  They don't require an outside source at all for securing your bitcoins, nor do they require an outside source for sending/spending bitcoins.  They ONLY require an outside source for determining if (and how many) bitcoins you've received.  That risk can be significantly reduced by using multiple methods to check.


There was a lot of growth in block sizes for the first 6 or 7 years of Bitcoin's existence, because there was not enough transaction volume to fill the blocks.  For about 2 years now the blocks have been full at 1 megabyte per block.  So, you'll see it slow down until it is processing the transactions from the year 2016.  Then the speed will level off for the remaining 2+ years of data. The introduction of SegWit 6 months ago has allowed for a bit more growth in block size.  The maximum was bumped up to 4 megabytes, but the average size isn't likely to grow to be much more than 2.6 megabytes.


It is only ridiculous if you are using insufficient hardware.  As I said, I recently started up a new node. It took less than 5 days for me to get completely caught up.


The blockchain is only needed during initial synchronization.  After that, storing it is only useful for assisting others in starting up a new node.  

If you WANT to help others start up new nodes, that's great.  But, if you are unable to, that's okay. There are many of us that can and will.


Bitcoin Core implemented pruning specifically because they were aware that it isn't necessary for everybody to store a complete copy of the blockchain forever.

Additionally, Satoshi himself said:


AND


AND



Incorrect.

The ledger IS the mined blocks.  The mined blocks ARE the ledger.  The blockchain is a chain of blocks of transactions which have a valid proof-of-work.


No.  You can set pruning mode before you start synchronization.  It will prune old blocks as it downloads the newer blocks.  You still need to process the entire blockchain, but you don't need to store it all. That way you don't need as big of a hard drive, and it is easier to stop your node and create a backup once a day during synchronization.  Then, if your node crashes in a disasterous way (due to Windoze?) you can pick up from where you left off less than a day ago.


Correct.  Backup what you have so far. That way, if you have a disaster, you don't have to start all over from the beginning.


Yes.  The synchronization process accesses the storage a LOT.  A faster drive (or better yet SSD) makes a HUGE difference.

The blocks are the ledger! The pre-Segwit 1MB block limit meant that the ledger can only grow by 1MB every ~10 minutes. Now with Segwit, the ledger can theoretically grow 4MB in ~10 minutes, though the practical rate is about 2MB/~10min and the current actual rate is only slightly over 1MB/~10 min.

** Some awesome responses on here by experienced coiners, I want to address some of them, particularly by DannyHamilton **

** This is all from my point of view and experience to date, which is <2 months **

1) I keep seeing "linear growth" mentioned. This is incorrect IMO. The data is growing exponentially, you can see this by checking the blockchain growth charts.

2) Lightweight wallets are fine and dandy yes, but they require 100% dependency on an outside source :shudder:

3) Modern computer w/ RAM, SSD, some good cores etc---> I really appreciate this info, I had no idea the Bitcoin Core client was so dependant on this with the block processing, I had assumed the software itself was limited in how fast it accomplished this.. Silly oversight on my part, software is ALWAYS dependent on the quality of the hardware it resides in. It's like asking "why can't my Balckberry Playbook run Autocad?"

4) The download is still impossible. Because of the exponential growth, it gets slower and slower as it inches closer to the last year and a half, to the point of ridiculousness.

5) Pray, do tell, why the average person doesn't need to store the blockchain? Again, imagine if this type of thinking was the modus operative of the Bitcoin Core initiative.

6) I learned something new---> blocks "filling up". So, please correct me if I'm wrong. The size of the blockchain is the sum of two parts: the mined blocks & the ledger

7) Pruning mode--> neat! But you still need the entire blockchain first before entering this mode.

Backups are very important. If you aren't backing up your data, don't cry when its lost. But I'm not going to back up something I don't even have yet. Unless, I suppose, you do a backup of the partial data.

9) If its processing slower now than my old drive, than yes I suppose the specs are worse...I didn't realize how important this was, and should have wend SSD.

10) Silicone-chip based storage w/ fee percentages, never mind. A discussion for a different time. Typical newbie blurt.

I never considered this- good to know. My connection speed certainly wouldn't be a factor @ 25/10 megabits..