Tough problems during running a Full node

DaveF

legendary

Activity: 3500

Merit: 6320

Crypto Swap Exchange

Quote from: BlackHatCoiner on October 01, 2021, 12:27:33 AM

I think it's solely a Pi thing or something related with Pi. On my main computer, which has obviously better CPU than Pi, it was syncing the block chain must faster than that, but with an SSD. Could the fault be the HDD drive?

Tough to say. If you were taking two weeks to sync instead of one then I might lean towards it being the issue, or part of one.
But, three weeks I don't think so. Yes, an SSD gives a big boost, but it should not be that much.

Honestly, I am out of ideas of what could be causing it.

-Dave

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Quote from: DaveF on September 30, 2021, 05:47:54 PM

If you don't mind me asking where in the world are you? And who are you using for internet connectivity?

I live in Greece and my internet provider is Vodafone, but I doubt that's the problem. My internet speed is quite impressive in contrast with my previous ones:

Quote from: DaveF on September 30, 2021, 05:47:54 PM

Just wondering if it's an issue with something upstream of you throttling something.

I think it's solely a Pi thing or something related with Pi. On my main computer, which has obviously better CPU than Pi, it was syncing the block chain must faster than that, but with an SSD. Could the fault be the HDD drive?

DaveF

legendary

Activity: 3500

Merit: 6320

Crypto Swap Exchange

Quote from: BlackHatCoiner on September 30, 2021, 11:21:22 AM

Quote from: DaveF on September 29, 2021, 05:31:12 PM

So, yes there seems to be a real performance penalty for syncing with TOR.

My experience with Umbrel was another story. For some strange reason I sometimes had 0 connections. Thus, it stopped syncing often. I'm sure that if I was still running Umbrel, it wouldn't have finished; it'd be around 70%. I also created a thread in their community page. No one responded so I got rid of it.

Quote from: DaveF on September 29, 2021, 05:31:12 PM

Now, as to why it took you a month. No idea.

As I said, not exactly one month, but around 3 weeks. Well, I won't use Tor again if I'll have to reset it.

If you don't mind me asking where in the world are you? And who are you using for internet connectivity?
Just wondering if it's an issue with something upstream of you throttling something.

-Dave

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Quote from: DaveF on September 29, 2021, 05:31:12 PM

So, yes there seems to be a real performance penalty for syncing with TOR.

My experience with Umbrel was another story. For some strange reason I sometimes had 0 connections. Thus, it stopped syncing often. I'm sure that if I was still running Umbrel, it wouldn't have finished; it'd be around 70%. I also created a thread in their community page. No one responded so I got rid of it.

Quote from: DaveF on September 29, 2021, 05:31:12 PM

Now, as to why it took you a month. No idea.

As I said, not exactly one month, but around 3 weeks. Well, I won't use Tor again if I'll have to reset it.

DaveF

legendary

Activity: 3500

Merit: 6320

Crypto Swap Exchange

Quote from: BlackHatCoiner on September 26, 2021, 01:26:30 AM

Quote from: DaveF on September 25, 2021, 04:27:41 PM

So long as you are sure. Is the case a big heatsink or is there a fan?

There's a fan.

Quote from: DaveF on September 25, 2021, 04:27:41 PM

Actually TOR might be the issue, I am re-syncing 2 nodes on the exact same hardware one TOR on clearnet and as of now the clearnet is at 55% and the TOR is below 40% done.

You should have told me this before I waited over three weeks to finish. Tongue

Anyway, now that I know what is probably the issue, I'll change the configuration file if I ever need to resync it again. I need Tor to connect with my node outside home. I don't know if it's possible to run both on clearnet and connect with Tor.

How did my testnet node finish syncing within hours, though? It runs over Tor too.

So. The umbrel setup that I mentioned here: https://bitcointalksearch.org/topic/m.58008085 just finished it's sync on TOR
Took from about 2 hours before I made the post till about 10 minutes ago. So about 6 days and a few hours.

During that same amount of time, not using umbrel but raspiblitz with the same hardware setup 2 complete syncs on clearnet (noticed I had the wrong drive installed after I finished the 1st sync)

So, yes there seems to be a real performance penalty for syncing with TOR. Have not seen much of a discussion about it, but it is there.
Now, as to why it took you a month. No idea.

As a test, I am going to wipe the raspiblitz and resync on TOR to see how long that takes. Don't know if different configurations may act differently.
*All behind the same firewall so it's not a bandwidth issue.

-Dave

n0nce

hero member

Activity: 882

Merit: 5834

not your keys, not your coins!

Quote from: BlackHatCoiner on September 26, 2021, 01:26:30 AM

How did my testnet node finish syncing within hours, though? It runs over Tor too.

Sorry for answering your question towards Dave, but maybe just the much smaller size Wink

Quote from: Trojane on September 25, 2021, 07:59:34 PM

~

BTC blockchain is around 380GB, yes, it needs some time, even on a good connection. 4 hours is nothing, most of us need at least a day to sync the blockchain, if not days or weeks depending on the hardware and internet connection. Just get a 500 or 1TB SSD and let it run. It's normal that your device gets a bit hot, it needs to perform lots of computations (verification), it's not a plain download. Every block is verified.

This is where 'verify' in our mantra 'don't trust, verify' comes from.

For more info about the IBD (initial block download), refer to this good page:

Quote from: https://en.bitcoin.it/wiki/Block_chain_download

Downloader Behavior
When receiving an inv (Inventory Message)
For each block
If already have as orphan, send getblocks(locator(chainHead), orphanRoot(block))
If not already have
send getdata(block)
If already have and this is last block in message (NOTE: this triggers specific behavior in peer getdata)
send getdata(block)

When receiving a block
If already have, ignore
Else if have parent not in main chain
store as orphan
getblocks(locator(chainHead), orphanRoot(block))
Else
Accept block
Accept orphan blocks that depend on this one

When starting download from a peer
getblocks(locator(chainHead), 0)

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Quote from: DaveF on September 25, 2021, 04:27:41 PM

So long as you are sure. Is the case a big heatsink or is there a fan?

There's a fan.

Quote from: DaveF on September 25, 2021, 04:27:41 PM

Actually TOR might be the issue, I am re-syncing 2 nodes on the exact same hardware one TOR on clearnet and as of now the clearnet is at 55% and the TOR is below 40% done.

You should have told me this before I waited over three weeks to finish. Tongue

Anyway, now that I know what is probably the issue, I'll change the configuration file if I ever need to resync it again. I need Tor to connect with my node outside home. I don't know if it's possible to run both on clearnet and connect with Tor.

How did my testnet node finish syncing within hours, though? It runs over Tor too.

Trojane

member

Activity: 294

Merit: 28

Looks as if my internet connections were too poor .
Since am trying this action for the first time,
Downloading the block chain took almost 4 hrs and after this ,the connections crumbled.
After much stress and heat radiation
I had to search for the current block chain size
And it is https://ibb.co/z6qxPhN

I need to get strong internet connections and prolly a new flash drive,bigger memory space too

How about exothermic reactions?
Is it normal ?
Thanks for support fam

DaveF

legendary

Activity: 3500

Merit: 6320

Crypto Swap Exchange

Quote from: BlackHatCoiner on September 25, 2021, 11:13:45 AM

Quote from: DaveF on September 25, 2021, 08:42:06 AM

How is your cooling? Thermal throttle can be a big issue with RPis at times.

The case I have covers the cooling part.

So long as you are sure. Is the case a big heatsink or is there a fan?

Quote from: BlackHatCoiner on September 25, 2021, 11:13:45 AM

Quote from: DaveF on September 25, 2021, 08:42:06 AM

What type of USB drive are you using?

A 1TB external one with USB 3.0. Specifically, this one. (Yes, I plugged it on the 3.0 port)

That is more then fine. It's actually newer then the one I had on my old node in a box setup.

Quote from: BlackHatCoiner on September 25, 2021, 11:13:45 AM

Quote from: DaveF on September 25, 2021, 08:42:06 AM

Are you downloading over clearnet or TOR?

I'm downloading over Tor, but I don't think that's the issue. I believe that what's taking so long is the verification of those blocks.

Actually TOR might be the issue, I am re-syncing 2 nodes on the exact same hardware one TOR on clearnet and as of now the clearnet is at 55% and the TOR is below 40% done.

Quote from: n0nce on September 25, 2021, 12:29:12 PM

Quote from: BlackHatCoiner on September 25, 2021, 11:13:45 AM

Have you ever experienced a hard drive issue from a power surge?

I did! And it needed to re-verify the whoooole damn thing. I don't keep a backup of the blockchain though, since verification takes much longer than the actual downloading part, no matter the setup, so I don't see the need.

It saves the bandwidth and a bit of time. And you don't have to have it online so you can do it anywhere with any device and just come back later.
If you want to say put up 3 more nodes to play with, you don't have to download the entire blockchain 3 more times, its just copy - paste - wait.

-Dave

n0nce

hero member

Activity: 882

Merit: 5834

not your keys, not your coins!

Quote from: BlackHatCoiner on September 25, 2021, 11:13:45 AM

Quote from: DaveF on September 25, 2021, 08:42:06 AM

How is your cooling? Thermal throttle can be a big issue with RPis at times.

The case I have covers the cooling part.

Tricky - it might not be enough cooling. Did you check your temps and frequencies at all? Like, if the frequency doesn't stay pegged at its max rating, when under load, it might be throttling.

Quote from: BlackHatCoiner on September 25, 2021, 11:13:45 AM

Quote from: DaveF on September 25, 2021, 08:42:06 AM

What type of USB drive are you using?

A 1TB external one with USB 3.0. Specifically, this one. (Yes, I plugged it on the 3.0 port)

Quote from: DaveF on September 25, 2021, 08:42:06 AM

Are you downloading over clearnet or TOR?

I'm downloading over Tor, but I don't think that's the issue. I believe that what's taking so long is the verification of those blocks.

Quote from: DaveF on September 25, 2021, 08:42:06 AM

I would make a backup of the blockchain files someplace for now, so if you ever have to or want to do it again you have a known clean copy of the blockchain up to today.

Well, a backup would require another 1TB disk. Is it that necessary? Have you ever experienced a hard drive issue from a power surge?

I must say: HDD is not the best choice, because as far as I know for every block, Core has to look up whether the inputs are valid UTXOs, so it needs to 'look back' in the blockchain, requiring random disk reads which are the slowest of disk read operations.^{(I might be wrong about the way the verification works)}

However, I was able to sync quite quickly with a HDD in an old laptop. It only has 4GB of RAM, so maybe less than your Pi, but the computing power is higher and the cooling is good. So in hindsight, I'm not able to tell you 100% if your computing power is limiting you due to cooling constraints or if it's a disk bottleneck. I'd bet on the former though, since my HDD worked fine (around a week) in a laptop node.

Quote from: BlackHatCoiner on September 25, 2021, 11:13:45 AM

Have you ever experienced a hard drive issue from a power surge?

I did! And it needed to re-verify the whoooole damn thing. I don't keep a backup of the blockchain though, since verification takes much longer than the actual downloading part, no matter the setup, so I don't see the need.

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Quote from: DaveF on September 25, 2021, 08:42:06 AM

How is your cooling? Thermal throttle can be a big issue with RPis at times.

The case I have covers the cooling part.

Quote from: DaveF on September 25, 2021, 08:42:06 AM

What type of USB drive are you using?

A 1TB external one with USB 3.0. Specifically, this one. (Yes, I plugged it on the 3.0 port)

Quote from: DaveF on September 25, 2021, 08:42:06 AM

Are you downloading over clearnet or TOR?

I'm downloading over Tor, but I don't think that's the issue. I believe that what's taking so long is the verification of those blocks.

Quote from: DaveF on September 25, 2021, 08:42:06 AM

I would make a backup of the blockchain files someplace for now, so if you ever have to or want to do it again you have a known clean copy of the blockchain up to today.

Well, a backup would require another 1TB disk. Is it that necessary? Have you ever experienced a hard drive issue from a power surge?

n0nce

hero member

Activity: 882

Merit: 5834

not your keys, not your coins!

Quote from: DaveF on September 25, 2021, 08:42:06 AM

What type of USB drive are you using? A slower 5400RPM with limited cache might cause some slowdowns as it's indexing and downloading and verifying 2 blcokchains.

I had a quite bad experience with 5400 rpm normal USB HDD (RPi 3B+ though). It took a month and then something got corrupted due to a bad powered USB hub..

Quote from: DaveF on September 25, 2021, 08:42:06 AM

I would make a backup of the blockchain files someplace for now, so if you ever have to or want to do it again you have a known clean copy of the blockchain up to today.

I don't think it helps much, if it's a drive / verification issue. In my case, I tried downloading the blockchain with PC and then connecting it to the Pi as well, but the verification was what took so long, so it didn't help much. I finally solved my issues by switching to a more powerful machine with more RAM. However, this issue should be solved by using an SSD; people used to run Bitcoin nodes on RPi 3's all the time when that was the latest offering, but they did use solid state.

DaveF

legendary

Activity: 3500

Merit: 6320

Crypto Swap Exchange

Quote from: BlackHatCoiner on September 25, 2021, 12:46:27 AM

Quote from: DaveF on September 24, 2021, 08:39:40 PM

Was this an older 2GB unit? Slow internet? something seems off.

Well, no. I'm not having an older 2GB unit neither a slow internet connection. Does it matter if I've set txindex=1? I'm running both main net and test net, but that shouldn't be the issue. I really don't know what went that wrong. The synchronization started in 30st of August and ended yesterday. (Almost a month)

The txindex=1 might slow it down a tiny bit like add two or three days onto since you are indexing as you go but not weeks of time.

Running both mainnet and testnet really should not matter much, once again add a day or two onto the sync and once the testnet is fully done it should really make no resource difference.

How is your cooling? Thermal throttle can be a big issue with RPis at times.

What type of USB drive are you using? A slower 5400RPM with limited cache might cause some slowdowns as it's indexing and downloading and verifying 2 blcokchains.

Are you downloading over clearnet or TOR? I have had some painfully slow downloads over TOR for the initial sync at times.

Could also be a combination of things. None of which add a lot of time. But 3 days here and 3 days there and another few days due to something else....

I would make a backup of the blockchain files someplace for now, so if you ever have to or want to do it again you have a known clean copy of the blockchain up to today.
Does not help you figure out why it took so long this time, but if you have to do it again it's one less thing to worry about.

-Dave

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Quote from: Trojane on September 24, 2021, 06:12:31 PM

I just needed few clips from articles i could read for more details on how i could achieve what i wanted

But, how can we know those few clips if we don't firstly understand what's your issue? You said you're having some tough problems when you try to run a full node; please elaborate!

Quote from: DaveF on September 24, 2021, 08:39:40 PM

Was this an older 2GB unit? Slow internet? something seems off.

Well, no. I'm not having an older 2GB unit neither a slow internet connection. Does it matter if I've set txindex=1? I'm running both main net and test net, but that shouldn't be the issue. I really don't know what went that wrong. The synchronization started in 30st of August and ended yesterday. (Almost a month)

DannyHamilton

legendary

Activity: 3472

Merit: 4794

Quote from: Trojane on September 24, 2021, 06:12:31 PM

You know what?
Anything that has to do with TECHNOLOGY and CONNECTIONS might go faulty anytime,

Yes, it might. But you never described anything faulty.

Quote from: Trojane on September 24, 2021, 06:12:31 PM

thats not because it wasn't working previously

Maybe it was, maybe it wasn't. If you took the time to actually describe what your problem was, rather than complaining about mysterious "tough problems" you probably would have gotten better help a lot faster.

Quote from: Trojane on September 24, 2021, 06:12:31 PM

I made this topic according to the the FUNCTION that i needed to perform with the NODE and the output i got during several times of trials ...

You never posted any output. You never described any output. You never explained what happened during any trials.

Quote from: Trojane on September 24, 2021, 06:12:31 PM

Please we are all in the process of learning,

And there are many people here willing to help and to teach. But first, you need to actually ask a question or describe a problem.

Quote from: Trojane on September 24, 2021, 06:12:31 PM

nobody knows everything, even in the different fields that we specialise.

I never said anybody knew everything. I just asked you to actually explain what your problem is.

Quote from: Trojane on September 24, 2021, 06:12:31 PM

I felt very bad for your own opinion. I don't think i should be mocked .,not at all!

I'm not mocking you. I'm just asking you to explain what your problem is so that we can help you (and help others that may have a similar problem in the future and may see this discussion.

Quote from: Trojane on September 24, 2021, 06:12:31 PM

I just needed few clips from articles i could read for more details on how i could achieve what i wanted,

Glad to hear you figured it out. It would still be good to explain what the problem was and how you resolved it so others can learn from your experience.

Quote from: Trojane on September 24, 2021, 06:12:31 PM

not being quoted IGNORANT because i am already.

I never said anyone was ignorant. Where did you get that nonsense from?

Quote from: Trojane on September 24, 2021, 06:12:31 PM

It looks simple to you because you have been doing it OVERTIME
Angry

Where did I say anything was simple? I'm just trying to understand what your problem is.

DaveF

legendary

Activity: 3500

Merit: 6320

Crypto Swap Exchange

Quote from: BlackHatCoiner on September 24, 2021, 12:54:24 PM

With my Raspberry Pi 4 it has taken around a month to finish. It should take less to you if you're having a better CPU than my Pi's. (Which you'll most likely have)

That is a REALLY long time. This config: https://bitcointalksearch.org/topic/m.58008085 did it in under a week.
Was this an older 2GB unit? Slow internet? something seems off. Unless you are doing something else with the Pi at the same time.
Or if you don't have good cooling it might throttle a lot. A bad power supply might also cause some issues, but a month still seems long.

-Dave

n0nce

hero member

Activity: 882

Merit: 5834

not your keys, not your coins!

Quote from: Trojane on September 24, 2021, 06:12:31 PM

[...] and the output i got during several times of trials ... [...]

I don't think you actually did post any 'output'. You never really posted what actually happened or what you tried, just said that it was 'impossible to download' the blockchain.

Is your issue solved now?

Quote from: Trojane on September 24, 2021, 06:00:55 PM

All thanks to @nonce
I had to read through the link and i got some useful documentaries.
I have gone above my difficulties

What do you mean? Why don't you just ask a question similar to the StackOverflow recommended format, which I quite like (https://stackoverflow.com/help/how-to-ask)?
Then we could actually help and not have to guess what your issue is and what you mean by 'ITS IMPOSSIBLE', 'ITS IMPOSSIBLE' etc...

Trojane

member

Activity: 294

Merit: 28

Quote

You haven't described any complexity yet. You told us what you wanted to do, then you told us that you didn't want to do what you wanted to do, then you said that it's a problem that you don't want to do what you want to do.

You know what?
Anything that has to do with TECHNOLOGY and CONNECTIONS might go faulty anytime,thats not because it wasn't working previously
I made this topic according to the the FUNCTION that i needed to perform with the NODE and the output i got during several times of trials ...
Please we are all in the process of learning,nobody knows everything, even in the different fields that we specialise.
I felt very bad for your own opinion. I don't think i should be mocked .,not at all!

I just needed few clips from articles i could read for more details on how i could achieve what i wanted,not being quoted IGNORANT because i am already.
It looks simple to you because you have been doing it OVERTIME
Angry

Trojane

member

Activity: 294

Merit: 28

All thanks to @nonce
I had to read through the link and i got some useful documentaries.
I have gone above my difficulties

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Quote from: n0nce on September 24, 2021, 02:29:06 PM

I mean I guess they could find someone with a torrent of a snapshot or even a zip, download it to the blocks/ folder and then start core;

But, that wouldn't make the process finish sooner. Once you would have downloaded all the blocks it'd require you to verify them which is the main reason it takes so much time.

You can change the way your node will synchronize with the network, but not the time it'll take. There's no faster way other than running bitcoind from your cmd.

Topic: Tough problems during running a Full node (Read 380 times)