Indexing new node has slowed to a crawl

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: doctrbob on March 19, 2024, 03:04:34 PM

I'm down to 361,355 blocks left to index, but it says 10 days until completion. It's been running 30 hours. I was at 47 hours last night and decreasing quickly. Thanks in advance.

This is extremely slow. Last time i had to perform reindex, it took about 1-2 days where all files stored on HDD.

Quote from: LoyceV on March 21, 2024, 04:14:28 AM

Quote from: doctrbob on March 19, 2024, 03:40:39 PM

Thank you for the replies. I am running a PC with 8gb ram and and an i7 processor and 400 Mbps internet connection. I have an attached Samsung T7 2tb external drive to run a full node, not pruned. It just seemed like it was indexing faster and faster than all of a sudden barely moving. If it's still downloading the blockchain regardless of speed, does that mean there was no corruption in the blockchain that is downloaded so far?

There are 2 things I would improve: get more RAM, and don't use an external drive. It's very easy to accidentally disconnect the USB cable, and that will lead to file corruption. Chainstate when fully synced is 11 GB, that doesn't fit your RAM which will lead to terabytes of data written to your disk.

Adding RAM will improve your overall computing experience, and is not very expensive anyway.

And if you don't use your PC for other task, you can close most application and background application so you can let Bitcoin Core use a bit more of RAM.

Quote from: promise444c5 on March 23, 2024, 07:20:15 AM

But I got a crashed os luckily I know how I could use the cmd-pmt for the retrieval of my file before resetting because i do keep my files hidden in the program data directory, what could have triggered it because It's was my first time experiencing it and that was when I started running the Bitcoin core

I recommend you to create new thread about your problem, rather than continue discussing on this thread.

promise444c5

sr. member

Activity: 476

Merit: 299

Learning never stops!

Quote from: LoyceV on March 23, 2024, 04:59:12 AM

Quote from: promise444c5 on March 23, 2024, 03:22:52 AM

Explains why my OS crashed without giving any sign

Not really: the OS should kill the program that runs out of memory instead of crashing.

But I got a crashed os luckily I know how I could use the cmd-pmt for the retrieval of my file before resetting because i do keep my files hidden in the program data directory, what could have triggered it because It's was my first time experiencing it and that was when I started running the Bitcoin core

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: promise444c5 on March 23, 2024, 03:22:52 AM

Explains why my OS crashed without giving any sign

Not really: the OS should kill the program that runs out of memory instead of crashing.

promise444c5

sr. member

Activity: 476

Merit: 299

Learning never stops!

Quote from: LoyceV on March 23, 2024, 02:56:51 AM

Without enough RAM, that sucks Tongue

If the 11 GB from chainstate doesn't fit in your file cache, you're going to be scraping that HDD.

Explains why my OS crashed without giving any sign
Almost lost my important files though

Or does Windows sucks entirely??

Quote

I have to ask: if your system still runs on an old HDD, why don't you replace it for an internal SSD instead of external?

Yes
Never cared about HDD until I was running a bitcoin Core on the PC

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: promise444c5 on March 22, 2024, 10:12:54 AM

What If I use a no sdd drive??

Without enough RAM, that sucks Tongue

If the 11 GB from chainstate doesn't fit in your file cache, you're going to be scraping that HDD.

I have to ask: if your system still runs on an old HDD, why don't you replace it for an internal SSD instead of external? I made that switch years ago, and I'll never go back! Buy a decent one: some (budget) SSDs have low performance on sustained writes.

Quote

Just asking because I got an external ssd drive because my disk functionality was been used up entirely to 100% while synchronising.

I did another IBD test recently, with 8 GB RAM, Linux, and SSD, and the SSD was the bottleneck. It's was reading/writing at maximum capacity for hours in a row.

Quote

Does the RAM provides a way of not getting the disk used up ??

Yes.

Cricktor

hero member

Activity: 714

Merit: 1010

Crypto Swap Exchange

Quote from: promise444c5 on March 22, 2024, 10:12:54 AM

Does the RAM provides a way of not getting the disk used up ??

What is a "no SSD drive", a spinning magnetic platter aka HDD drive?

What kind of an interface does your external drive use? Most common would be USB3 or up and USB protocol, especially when UASP is not used (your USB adapter needs to support it and the driver must have it enabled, that's not always on by default, depending what OS you use), slows down the performance of a SSD quite a bit. That's what LoyceV summerized with "internal drives are preferable", especially for SSDs.

More RAM means, more data can be kept in RAM where access speed is still orders of magnitude faster than for internal and to greater extend for external drives. Less often data needs to be flushed out to storage. The OS has more space for file I/O buffers, can better optimize how data is written out to storage (moving R/W heads for HDDs is snail slow compared to flash storage with no moving parts).

promise444c5

sr. member

Activity: 476

Merit: 299

Learning never stops!

Quote from: LoyceV on March 21, 2024, 04:14:28 AM

Quote from: doctrbob on March 19, 2024, 03:40:39 PM

Thank you for the replies. I am running a PC with 8gb ram and and an i7 processor and 400 Mbps internet connection. I have an attached Samsung T7 2tb external drive to run a full node, not pruned. It just seemed like it was indexing faster and faster than all of a sudden barely moving. If it's still downloading the blockchain regardless of speed, does that mean there was no corruption in the blockchain that is downloaded so far?

There are 2 things I would improve: get more RAM, and don't use an external drive. It's very easy to accidentally disconnect the USB cable, and that will lead to file corruption. Chainstate when fully synced is 11 GB, that doesn't fit your RAM which will lead to terabytes of data written to your disk.

Adding RAM will improve your overall computing experience, and is not very expensive anyway.

What If I use a no sdd drive?? Just asking because I got an external ssd drive because my disk functionality was been used up entirely to 100% while synchronising.
Does the RAM provides a way of not getting the disk used up ??

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: doctrbob on March 19, 2024, 03:40:39 PM

Thank you for the replies. I am running a PC with 8gb ram and and an i7 processor and 400 Mbps internet connection. I have an attached Samsung T7 2tb external drive to run a full node, not pruned. It just seemed like it was indexing faster and faster than all of a sudden barely moving. If it's still downloading the blockchain regardless of speed, does that mean there was no corruption in the blockchain that is downloaded so far?

There are 2 things I would improve: get more RAM, and don't use an external drive. It's very easy to accidentally disconnect the USB cable, and that will lead to file corruption. Chainstate when fully synced is 11 GB, that doesn't fit your RAM which will lead to terabytes of data written to your disk.

Adding RAM will improve your overall computing experience, and is not very expensive anyway.

promise444c5

sr. member

Activity: 476

Merit: 299

Learning never stops!

Firstly, check the usage of your hardisk if it's getting to 100% it will definitely be slow no matter what .
I've been unable to download mine due to low bandwidth in my country but I found a great trick that makes is fast .
Get an external SSD drive
Create a copy of your bitcoin data in the new SSD drive and rename the initial to another name to serve as backup incase of any error
Now follow this step as directed from @NotATether if you're using the GUI mode

Quote

you need to move the datadir to be on the partition that has your external drive.

You will need about 650-700 GB free space on the external drive.

The datadir is located in C:\Users\\AppData\Local\Bitcoin (or maybe it's in AppData\Roaming\Bitcoin, I don't remember which). You need to shut down Core and move all the contents of the folder into a new folder like D:\Bitcoin

Then in the C drive, go to the original folder like AppData\Local\Bitcoin or like that and create a file called bitcoin.conf and use a text editor to add the following:

Code:

datadir=D:\Bitcoin (or whatever your new folder is called)

Additionally, you can also add this setting:

Code:

dbcache=(write down 50% of ram, in MB: for example just write 1024 to use 1 GB or 2048 for 2GB and so on)

If you have a lot of RAM like at least 8 or 16 gigs then using this option will make syncing faster because it is not constantly writing the chainstate to the disk.

Then you can open Bitcoin Core again and it will store everything in the datadir.

In addition , the file is the bitcoin and you should save it as a conf file not as a text (note this)

Amphenomenon

sr. member

Activity: 700

Merit: 470

Hope Jeremiah 17vs7

Quote from: doctrbob on March 19, 2024, 03:40:39 PM

Thank you for the replies. I am running a PC with 8gb ram and and an i7 processor and 400 Mbps internet connection. I have an attached Samsung T7 2tb external drive to run a full node, not pruned. It just seemed like it was indexing faster and faster than all of a sudden barely moving. If it's still downloading the blockchain regardless of speed, does that mean there was no corruption in the blockchain that is downloaded so far?

I don't think you have to restart syncing, try Increasing the database cache size to

Code:

dbcache=2048

since you are using an external drive this would increase the syncing process

Quote

When you say start resyncing, you mean delete everything and start over?

yes but you don't have to use this

Cricktor

hero member

Activity: 714

Merit: 1010

Crypto Swap Exchange

Increasing dbcache= might help a bit (add in your bitcoin.conf file).

blocksonly=1 during initial blockchain download (IBD) lets your node only deal with block data over the network, don't bother at that time with unconfirmed transations.

External storage via USB isn't ideal, USB protocol eats some performance and with an UTXO set of around 16GB size your storage has to shuffle quite some data around when it doesn't fit all in your RAM during blockchain sync.

With optimized settings for a Raspi 4B with 8GB RAM and a 1TB SSD attached via USB3 a full IBD took about 4 days some months ago (my internet speed is 100MBit/s downstream).
I was kind of surprised, I expected more time as a Raspi 4B isn't really fast compared to normal PC CPUs. I did this with an Umbrel node as an experiment and I hope Umbrel doesn't use some sort of shortcuts for IBD. I might still have the debug.log of this experimental IBD.

doctrbob

newbie

Activity: 2

Merit: 0

Thank you for the replies. I am running a PC with 8gb ram and and an i7 processor and 400 Mbps internet connection. I have an attached Samsung T7 2tb external drive to run a full node, not pruned. It just seemed like it was indexing faster and faster than all of a sudden barely moving. If it's still downloading the blockchain regardless of speed, does that mean there was no corruption in the blockchain that is downloaded so far?

When you say start resyncing, you mean delete everything and start over?

Mia Chloe

sr. member

Activity: 448

Merit: 560

Crypto Casino and Sportsbook

What you are experiencing is normal.
Firstly you don't download the node rather the computer you are using runs as the node. You are actually downloading all of Bitcoin's block chain data and it's actually over 500Gb except you are trying to run a prune node by download some data and omitting some from a particular time frame.
Depending on the computation power of your computer and network speed, downloading the Blockchain data from the Genesis block till date and also syncing, could take a couple of days. The faster you computational power and network strength, the faster the syncing process.

Make sure you have a storage of up to 1tb preferably SSD to prevent issues due to space as it syncs and also when you attempt running it.

doctrbob

newbie

Activity: 2

Merit: 0

I'm trying to download a bitcoin node. Bitcoin Core Ver 26.0 Indexing started out fast but has now slowed to a crawl. Bitcoin core did close down twice on its own (once because the hard drive went to sleep before I changed the setting and not sure why the second time). Did that possibly corrupt the indexing? It started up fine when restarted but now is barely moving. I closed the program with control q and let it gracefully shut down and rebooted the computer, but that didn't help. I'm down to 361,355 blocks left to index, but it says 10 days until completion. It's been running 30 hours. I was at 47 hours last night and decreasing quickly. Thanks in advance.

Topic: Indexing new node has slowed to a crawl (Read 196 times)