Pages:
Author

Topic: Tough problems during running a Full node (Read 410 times)

legendary
Activity: 3500
Merit: 6320
Crypto Swap Exchange
October 01, 2021, 05:40:52 AM
#29
I think it's solely a Pi thing or something related with Pi. On my main computer, which has obviously better CPU than Pi, it was syncing the block chain must faster than that, but with an SSD. Could the fault be the HDD drive?

Tough to say. If you were taking two weeks to sync instead of one then I might lean towards it being the issue, or part of one.
But, three weeks I don't think so. Yes, an SSD gives a big boost, but it should not be that much.

Honestly, I am out of ideas of what could be causing it.

-Dave
legendary
Activity: 1512
Merit: 7340
Farewell, Leo
October 01, 2021, 12:27:33 AM
#28
If you don't mind me asking where in the world are you? And who are you using for internet connectivity?
I live in Greece and my internet provider is Vodafone, but I doubt that's the problem. My internet speed is quite impressive in contrast with my previous ones:


Just wondering if it's an issue with something upstream of you throttling something.
I think it's solely a Pi thing or something related with Pi. On my main computer, which has obviously better CPU than Pi, it was syncing the block chain must faster than that, but with an SSD. Could the fault be the HDD drive?
legendary
Activity: 3500
Merit: 6320
Crypto Swap Exchange
September 30, 2021, 05:47:54 PM
#27
So, yes there seems to be a real performance penalty for syncing with TOR.
My experience with Umbrel was another story. For some strange reason I sometimes had 0 connections. Thus, it stopped syncing often. I'm sure that if I was still running Umbrel, it wouldn't have finished; it'd be around 70%. I also created a thread in their community page. No one responded so I got rid of it.

Now, as to why it took you a month. No idea.
As I said, not exactly one month, but around 3 weeks. Well, I won't use Tor again if I'll have to reset it.

If you don't mind me asking where in the world are you? And who are you using for internet connectivity?
Just wondering if it's an issue with something upstream of you throttling something.

-Dave
legendary
Activity: 1512
Merit: 7340
Farewell, Leo
September 30, 2021, 11:21:22 AM
#26
So, yes there seems to be a real performance penalty for syncing with TOR.
My experience with Umbrel was another story. For some strange reason I sometimes had 0 connections. Thus, it stopped syncing often. I'm sure that if I was still running Umbrel, it wouldn't have finished; it'd be around 70%. I also created a thread in their community page. No one responded so I got rid of it.

Now, as to why it took you a month. No idea.
As I said, not exactly one month, but around 3 weeks. Well, I won't use Tor again if I'll have to reset it.
legendary
Activity: 3500
Merit: 6320
Crypto Swap Exchange
September 29, 2021, 05:31:12 PM
#25
So long as you are sure. Is the case a big heatsink or is there a fan?
There's a fan.

Actually TOR might be the issue, I am re-syncing 2 nodes on the exact same hardware one TOR on clearnet and as of now the clearnet is at 55% and the TOR is below 40% done.
You should have told me this before I waited over three weeks to finish.  Tongue

Anyway, now that I know what is probably the issue, I'll change the configuration file if I ever need to resync it again. I need Tor to connect with my node outside home. I don't know if it's possible to run both on clearnet and connect with Tor.

How did my testnet node finish syncing within hours, though? It runs over Tor too.

So. The umbrel setup that I mentioned here: https://bitcointalksearch.org/topic/m.58008085 just finished it's sync on TOR
Took from about 2 hours before I made the post till about 10 minutes ago. So about 6 days and a few hours.

During that same amount of time, not using umbrel but raspiblitz with the same hardware setup 2 complete syncs on clearnet (noticed I had the wrong drive installed after I finished the 1st sync)

So, yes there seems to be a real performance penalty for syncing with TOR. Have not seen much of a discussion about it, but it is there.
Now, as to why it took you a month. No idea.

As a test, I am going to wipe the raspiblitz and resync on TOR to see how long that takes. Don't know if different configurations may act differently.
*All behind the same firewall so it's not a bandwidth issue.

-Dave
hero member
Activity: 882
Merit: 5834
not your keys, not your coins!
September 26, 2021, 09:26:46 AM
#24
How did my testnet node finish syncing within hours, though? It runs over Tor too.
Sorry for answering your question towards Dave, but maybe just the much smaller size Wink

~
BTC blockchain is around 380GB, yes, it needs some time, even on a good connection. 4 hours is nothing, most of us need at least a day to sync the blockchain, if not days or weeks depending on the hardware and internet connection. Just get a 500 or 1TB SSD and let it run. It's normal that your device gets a bit hot, it needs to perform lots of computations (verification), it's not a plain download. Every block is verified.

This is where 'verify' in our mantra 'don't trust, verify' comes from.

For more info about the IBD (initial block download), refer to this good page:
Downloader Behavior
    When receiving an inv (Inventory Message)
        For each block
            If already have as orphan, send getblocks(locator(chainHead), orphanRoot(block))
            If not already have
                send getdata(block)
            If already have and this is last block in message (NOTE: this triggers specific behavior in peer getdata)
                send getdata(block)

    When receiving a block
        If already have, ignore
        Else if have parent not in main chain
            store as orphan
            getblocks(locator(chainHead), orphanRoot(block))
        Else
            Accept block
            Accept orphan blocks that depend on this one

    When starting download from a peer
        getblocks(locator(chainHead), 0)
legendary
Activity: 1512
Merit: 7340
Farewell, Leo
September 26, 2021, 01:26:30 AM
#23
So long as you are sure. Is the case a big heatsink or is there a fan?
There's a fan.

Actually TOR might be the issue, I am re-syncing 2 nodes on the exact same hardware one TOR on clearnet and as of now the clearnet is at 55% and the TOR is below 40% done.
You should have told me this before I waited over three weeks to finish.  Tongue

Anyway, now that I know what is probably the issue, I'll change the configuration file if I ever need to resync it again. I need Tor to connect with my node outside home. I don't know if it's possible to run both on clearnet and connect with Tor.

How did my testnet node finish syncing within hours, though? It runs over Tor too.
member
Activity: 294
Merit: 28
September 25, 2021, 07:59:34 PM
#22
Looks as if my internet connections were too poor .
Since am trying this action for the first time,
Downloading the block chain took almost 4 hrs and after this ,the connections crumbled.
After much stress and heat radiation
I had to search for the current block chain size
And it is https://ibb.co/z6qxPhN

I need to get strong internet connections and prolly a new flash drive,bigger memory space too

How about exothermic reactions?
Is it normal ?
Thanks for support fam
legendary
Activity: 3500
Merit: 6320
Crypto Swap Exchange
September 25, 2021, 04:27:41 PM
#21
How is your cooling? Thermal throttle can be a big issue with RPis at times.
The case I have covers the cooling part.

So long as you are sure. Is the case a big heatsink or is there a fan?

What type of USB drive are you using?
A 1TB external one with USB 3.0. Specifically, this one. (Yes, I plugged it on the 3.0 port)

That is more then fine. It's actually newer then the one I had on my old node in a box setup.

Are you downloading over clearnet or TOR?
I'm downloading over Tor, but I don't think that's the issue. I believe that what's taking so long is the verification of those blocks.

Actually TOR might be the issue, I am re-syncing 2 nodes on the exact same hardware one TOR on clearnet and as of now the clearnet is at 55% and the TOR is below 40% done.

Have you ever experienced a hard drive issue from a power surge?
I did! And it needed to re-verify the whoooole damn thing. I don't keep a backup of the blockchain though, since verification takes much longer than the actual downloading part, no matter the setup, so I don't see the need.

It saves the bandwidth and a bit of time. And you don't have to have it online so you can do it anywhere with any device and just come back later.
If you want to say put up 3 more nodes to play with, you don't have to download the entire blockchain 3 more times, its just copy - paste - wait.

-Dave
hero member
Activity: 882
Merit: 5834
not your keys, not your coins!
September 25, 2021, 12:29:12 PM
#20
How is your cooling? Thermal throttle can be a big issue with RPis at times.
The case I have covers the cooling part.
Tricky - it might not be enough cooling. Did you check your temps and frequencies at all? Like, if the frequency doesn't stay pegged at its max rating, when under load, it might be throttling.

What type of USB drive are you using?
A 1TB external one with USB 3.0. Specifically, this one. (Yes, I plugged it on the 3.0 port)

Are you downloading over clearnet or TOR?
I'm downloading over Tor, but I don't think that's the issue. I believe that what's taking so long is the verification of those blocks.

I would make a backup of the blockchain files someplace for now, so if you ever have to or want to do it again you have a known clean copy of the blockchain up to today.
Well, a backup would require another 1TB disk. Is it that necessary? Have you ever experienced a hard drive issue from a power surge?
I must say: HDD is not the best choice, because as far as I know for every block, Core has to look up whether the inputs are valid UTXOs, so it needs to 'look back' in the blockchain, requiring random disk reads which are the slowest of disk read operations.(I might be wrong about the way the verification works)

However, I was able to sync quite quickly with a HDD in an old laptop. It only has 4GB of RAM, so maybe less than your Pi, but the computing power is higher and the cooling is good. So in hindsight, I'm not able to tell you 100% if your computing power is limiting you due to cooling constraints or if it's a disk bottleneck. I'd bet on the former though, since my HDD worked fine (around a week) in a laptop node.

Have you ever experienced a hard drive issue from a power surge?
I did! And it needed to re-verify the whoooole damn thing. I don't keep a backup of the blockchain though, since verification takes much longer than the actual downloading part, no matter the setup, so I don't see the need.
legendary
Activity: 1512
Merit: 7340
Farewell, Leo
September 25, 2021, 11:13:45 AM
#19
How is your cooling? Thermal throttle can be a big issue with RPis at times.
The case I have covers the cooling part.

What type of USB drive are you using?
A 1TB external one with USB 3.0. Specifically, this one. (Yes, I plugged it on the 3.0 port)

Are you downloading over clearnet or TOR?
I'm downloading over Tor, but I don't think that's the issue. I believe that what's taking so long is the verification of those blocks.

I would make a backup of the blockchain files someplace for now, so if you ever have to or want to do it again you have a known clean copy of the blockchain up to today.
Well, a backup would require another 1TB disk. Is it that necessary? Have you ever experienced a hard drive issue from a power surge?
hero member
Activity: 882
Merit: 5834
not your keys, not your coins!
September 25, 2021, 11:06:54 AM
#18
What type of USB drive are you using? A slower 5400RPM with limited cache might cause some slowdowns as it's indexing and downloading and verifying 2 blcokchains.
I had a quite bad experience with 5400 rpm normal USB HDD (RPi 3B+ though). It took a month and then something got corrupted due to a bad powered USB hub..

I would make a backup of the blockchain files someplace for now, so if you ever have to or want to do it again you have a known clean copy of the blockchain up to today.
I don't think it helps much, if it's a drive / verification issue. In my case, I tried downloading the blockchain with PC and then connecting it to the Pi as well, but the verification was what took so long, so it didn't help much. I finally solved my issues by switching to a more powerful machine with more RAM. However, this issue should be solved by using an SSD; people used to run Bitcoin nodes on RPi 3's all the time when that was the latest offering, but they did use solid state.
legendary
Activity: 3500
Merit: 6320
Crypto Swap Exchange
September 25, 2021, 08:42:06 AM
#17
Was this an older 2GB unit? Slow internet? something seems off.
Well, no. I'm not having an older 2GB unit neither a slow internet connection. Does it matter if I've set txindex=1? I'm running both main net and test net, but that shouldn't be the issue. I really don't know what went that wrong. The synchronization started in 30st of August and ended yesterday. (Almost a month)

The txindex=1 might slow it down a tiny bit like add two or three days onto since you are indexing as you go but not weeks of time.

Running both mainnet and testnet really should not matter much, once again add a day or two onto the sync and once the testnet is fully done it should really make no resource difference.

How is your cooling? Thermal throttle can be a big issue with RPis at times.

What type of USB drive are you using? A slower 5400RPM with limited cache might cause some slowdowns as it's indexing and downloading and verifying 2 blcokchains.

Are you downloading over clearnet or TOR? I have had some painfully slow downloads over TOR for the initial sync at times.

Could also be a combination of things. None of which add a lot of time. But 3 days here and 3 days there and another few days due to something else....

I would make a backup of the blockchain files someplace for now, so if you ever have to or want to do it again you have a known clean copy of the blockchain up to today.
Does not help you figure out why it took so long this time, but if you have to do it again it's one less thing to worry about.

-Dave
legendary
Activity: 1512
Merit: 7340
Farewell, Leo
September 25, 2021, 12:46:27 AM
#16
I just needed few clips from articles i could read for more details on how i could achieve what i wanted
But, how can we know those few clips if we don't firstly understand what's your issue? You said you're having some tough problems when you try to run a full node; please elaborate!

Was this an older 2GB unit? Slow internet? something seems off.
Well, no. I'm not having an older 2GB unit neither a slow internet connection. Does it matter if I've set txindex=1? I'm running both main net and test net, but that shouldn't be the issue. I really don't know what went that wrong. The synchronization started in 30st of August and ended yesterday. (Almost a month)
legendary
Activity: 3472
Merit: 4801
September 24, 2021, 09:08:03 PM
#15
You know what?
Anything that has to do with TECHNOLOGY and CONNECTIONS might go faulty anytime,

Yes, it might. But you never described anything faulty.

thats not because it wasn't working previously

Maybe it was, maybe it wasn't.  If you took the time to actually describe what your problem was, rather than complaining about mysterious "tough problems" you probably would have gotten better help a lot faster.

I made this topic according to the the FUNCTION that i needed to perform with the NODE and the output i got during several times of trials ...

You never posted any output. You never described any output. You never explained what happened during any trials.

Please we are all in the process of learning,

And there are many people here willing to help and to teach.  But first, you need to actually ask a question or describe a problem.

nobody knows everything, even in the different fields that we specialise.

I never said anybody knew everything.  I just asked you to actually explain what your problem is.

I felt very bad for your own opinion. I don't think i should be mocked .,not at all!

I'm not mocking you. I'm just asking you to explain what your problem is so that we can help you (and help others that may have a similar problem in the future and may see this discussion.

I just needed few clips from articles i could read for more details on how i could achieve what i wanted,

Glad to hear you figured it out. It would still be good to explain what the problem was and how you resolved it so others can learn from your experience.

not being quoted IGNORANT because i am already.

I never said anyone was ignorant.  Where did you get that nonsense from?

It looks simple to you because you have been doing it OVERTIME
 Angry Embarrassed

Where did I say anything was simple?  I'm just trying to understand what your problem is.
legendary
Activity: 3500
Merit: 6320
Crypto Swap Exchange
September 24, 2021, 08:39:40 PM
#14
With my Raspberry Pi 4 it has taken around a month to finish. It should take less to you if you're having a better CPU than my Pi's. (Which you'll most likely have)

That is a REALLY long time. This config: https://bitcointalksearch.org/topic/m.58008085 did it in under a week.
Was this an older 2GB unit? Slow internet? something seems off. Unless you are doing something else with the Pi at the same time.
Or if you don't have good cooling it might throttle a lot. A bad power supply might also cause some issues, but a month still seems long.

-Dave
hero member
Activity: 882
Merit: 5834
not your keys, not your coins!
September 24, 2021, 07:14:29 PM
#13
[...] and the output i got during several times of trials ... [...]
I don't think you actually did post any 'output'. You never really posted what actually happened or what you tried, just said that it was 'impossible to download' the blockchain.

Is your issue solved now?

All thanks to @nonce
I had to read through the link and i got some useful documentaries.
I have gone above my difficulties
What do you mean? Why don't you just ask a question similar to the StackOverflow recommended format, which I quite like (https://stackoverflow.com/help/how-to-ask)?
Then we could actually help and not have to guess what your issue is and what you mean by 'ITS IMPOSSIBLE', 'ITS IMPOSSIBLE' etc...
member
Activity: 294
Merit: 28
September 24, 2021, 06:12:31 PM
#12
Quote
You haven't described any complexity yet.  You told us what you wanted to do, then you told us that you didn't want to do what you wanted to do, then you said that it's a problem that you don't want to do what you want to do.

You know what?
Anything that has to do with TECHNOLOGY and CONNECTIONS might go faulty anytime,thats not because it wasn't working previously
I made this topic according to the the FUNCTION that i needed to perform with the NODE and the output i got during several times of trials ...
Please we are all in the process of learning,nobody knows everything, even in the different fields that we specialise.
I felt very bad for your own opinion. I don't think i should be mocked .,not at all!

I just needed few clips from articles i could read for more details on how i could achieve what i wanted,not being quoted IGNORANT because i am already.
It looks simple to you because you have been doing it OVERTIME
 Angry Embarrassed
member
Activity: 294
Merit: 28
September 24, 2021, 06:00:55 PM
#11
All thanks to @nonce
I had to read through the link and i got some useful documentaries.
I have gone above my difficulties
legendary
Activity: 1512
Merit: 7340
Farewell, Leo
September 24, 2021, 03:10:47 PM
#10
I mean I guess they could find someone with a torrent of a snapshot or even a zip, download it to the blocks/ folder and then start core;
But, that wouldn't make the process finish sooner. Once you would have downloaded all the blocks it'd require you to verify them which is the main reason it takes so much time.

You can change the way your node will synchronize with the network, but not the time it'll take. There's no faster way other than running bitcoind from your cmd.
Pages:
Jump to: