Topic: [Guide] How to run a Bitcoin Core full node for under 50 bucks! - page 5. (Read 2141 times)

All you need is mapping transaction hash to block hash (you can also use block number, but then you cannot check stale blocks). If you have that, then you can query raw transactions on a pruned node. The reason why it is not available by default is that "pruning" means "I want to reduce my disk usage", but "txindex" means "I want to store everything". It is kind of illogical to "store everything" and to "reduce disk usage" at the same time, so it is not allowed by default.

But as I said: just map transaction hashes to block hashes and you will get that functionality. Or scan UTXO set (there is even scanutxoset command; or even importprunedfunds if you really want).

In theory it probably could.
All of us keep going around in the same circle here with people and I hate to be the grouchy person BUT used drives have gotten so cheap I am at this point just e-wasting & recycling 500GB and below because just about anywhere in the Americas (North - Central -South) and most of the EU and ASIA it's actually cheaper to find them locally then to have me ship them.

If you want to run a node get a 1TB drive and....run a node. If you don't want to, then don't.

-Dave

What's the problem with keeping everyone's UTXOs and dumping the blocks? I'm just saying that you can have the same benefits even if you're planning to run an SPV server, without keeping the entire chain, but only the outputs.

Sure, if it's personal use set it to 0.

Edit: My bad. You're right. I hadn't ever tried it, to be honest, but I thought it would work. Could you explain me why a pruned node can't index all the transactions? The blocks/index is less than 40 GB and it could be used if one wanted to run an SPV server.

You cannot use those two options together, you have to choose one of them. You can kind of reach that by importing every output as yours, but you probably should not and focus on UTXO scanning instead, if you really need pruning.

Not wanting true decentralization doesn't mean hell-bent to lose self-custody. I believe the majority of the users, excluding those who're here just to trade, use SPV clients as it's the easiest solution with the least risks.

Those who leave them on centralized exchanges haven't understood what they've bought either way.

I think the storage cost has little matter. Running non-pruned full nodes definitely helps the network, but if the user doesn't want to buy an extra hard drive, they can enjoy the same features by running a pruned node with txindex=1.

Yeah, although its not for everyone. As most of us now by now, not many people want true decentralisation since it usually comes with a cost. By that I mean, when we ask the majority of the population they probably aren't going to be willing to pay for something just to use their own money, they would rather have third parties which handle that for them. Obviously, we don't need everyone to be running a node.

As for the costs, to be honest except for rising electricity costs I would like to think the total cost will continue to go down. The only issue that could become a problem is the storage, so far we've been pretty much following Moore's law with our increases in hardware capability, although I've definitely noticed the last year or so that most hardware is now becoming too powerful for our means, and therefore we won't use it a lot. Obviously, hard drives are a little different to CPU's, and graphics cards. Although, I do think that the general consumer won't be requiring much more than four terabyte (TB) hard drives in the future. Hell, I probably have over 50TB among all the hard drives I've collected over the years, but I don't use half of them. My main machine only has a 1TB hard drive in it, and that's no where near full.

So, storage is probably the biggest concern for Bitcoin self hosted nodes, at least full nodes is that the storage requirements might outpace what general consumers usually have or require. I don't think it'll be an issue for the capability we have in manufacturing larger hard drives, but it'll be what is available to the general consumer at a decent price, which the lesser demand probably means a lot of people could be out priced. Thoughts?

Shameless bump (not really): I just stumbled across a very old thread.

It's funny to look back at things and see how predictions by highly influential people eventually panned out.

This was basically exactly 7 years ago now; and Bitcoin was around for a bit over 6 years at that point in time. So kind of half-way between today and the beginning of the Bitcoin network.

Poor Lauda..
At the moment of writing, blocks are mostly full and I'm not feeling lots of pain.

I, for one, am very happy that we can still run Bitcoin on $50 worth of hardware after such a long time. And I'm still convinced this is what really sets Bitcoin apart; true decentralization.
Sorry for the off-topic ramble, but I figured this historic little piece of information does fit in this topic and maybe it will motivate someone to build their own $50 node over the weekend!

That's the great thing about all of this... there are plenty of ways to skin the proverbial cat   Plenty of options to get up and running in a way that suits your budget, your physical space requirements, power limits, mobility requirements etc.

Pi's tucked away behind your monitor, an old repurposed desktop sitting in your closet, you daily driver laptop etc etc... modern technology is great

It depends. I've had terrible experiences with certain laptops when running them for extended period of time. I would rather not stress laptops (particularly their battery) by having them run 24/7, because they are not designed with that in mind. You are far more likely to have it fail on you than a brand-new Pi and even if it does then you only have to replace the USB charger.

The reason why Pi has passive cooling is because it doesn't really need active cooling system, perhaps save for a little extra heatsink on top of it. You are more likely to save more money going with a Pi, at least I've never really seen any computers that would be far cheaper than Pi (less than $50??) and isn't a headache to use. You're probably either going to find it grossly inadequate for Core or have a certain component fail on you after a while, that is the problem with buying old hardware, I've had quite a fair bit of experience in this regard.

---

Anyways regarding the cost, if you need to be concerned about the cost of running a node then you should probably spend your money elsewhere. Not everyone has to run a node, especially if you can't really afford it financially.

I know all of this. But the whole point of this little 'guide' is to make it as cheap and accessible as possible. Especially in developing countries, brand new hardware (even Raspberry Pis) is prohibitively expensive. On the other hand, for example schools almost anywhere, give away their old hardware, which may be Pentium desktops or something like that, which can be upgraded for a few bucks with more storage and RAM to be able to run a Bitcoin node.

A couple of other advantages that the single board units like the Pi have is "size" and "power draw"... Even with a case on it, the Pi is a very compact unit, and easily fits on my desk behind my monitor etc. I don't have a lot of room in my workstation area (or anywhere else in my apartment for that matter) and having a unit like a laptop or old desktop in addition to my actual PC isn't really practical.

Then there is the power draw... the Pi runs on a 5V USB charger. It uses next to nothing in terms of power draw... at max load, the CPU is only consuming something like 6 or 7W! And an external drive will only be adding ~5W more at most.

Yes with a but or no with a however.
If you are talking new and good that is one thing. We are discussing old used laptops and PCs.
Power supplies that have been through who knows what. Fans that may or may not have been abused, etc.
I did treat my old laptop well, but in the end set it up to mine some crap CPU coins. Had it screaming for months.
Could have sold it on eBay and the buyer would never have known. Not my style to do something like that, but other people.....

-Dave

I would set it between 50-80% of free/available RAM.


SSD have far more impact if most/all UTXO set can't be stored on RAM. Even if you move chainstate directory to SSD and create symbolic link to bitcoin directory (on HDD), you still can feel the difference.

I don't think that's right. These are all general-purpose computers in different shapes and sizes. And you just said Pi only has passive cooling out of the box, while random old laptops or PCs have active cooling all set up and ready to go. Computers don't really care if they run 24/7 or not, as long as they're not running at the thermal limit all the time, and even then, they're quite durable.

Yeah; SSD is also a good speedup, but since everyone's hopping on the 'GET AN SSD' train and nobody mentions RAM, I wanted to show how much of an effect more memory can have. It may be significantly cheaper e.g. to get a 8GB Pi and (maybe existing) HDD instead of a 4GB Pi and an SSD.

I will add that info, thanks! I noticed myself; 500 is enough for now for just Core, but Lightning and Electrum need a lot of space as well.

As discussed in another thread, you should have 1TB of disk space.

1/2TB will run out in less than the next 12 months.

For all intents and purposes, Pi will synchronize just fine. Just note that while the processor is relatively decent, it is built with passive cooling in mind and the TDP isn't a lot. It won't provide for a blazing fast synchronization but should definitely be better than using something that isn't technically designed to be powered-on 24/7.

I'm not sure whether the dbcache allocation is dynamic or fixed, but IIRC it is fixed. Your Core wouldn't use more than that for synchronization no matter how much ram you have. I could be wrong though. I've found that HDD vs SSD still makes quite a huge difference in synchronization; with similar parameters, the SSD took 5 hours to synchronize while the HDD took more than a day.

Been there, done that: https://bitcointalksearch.org/topic/how-to-initially-sync-bitcoin-as-fast-as-possible-5246078
Really is kind of interesting how some conversations keep coming around and around.

And, yes I know I never finished the testing in the other thread, as the hardware was needed elsewhere and it was the middle of the pandemic and getting stuff was hard.
I didn't go back since it would have to be started from scratch for a new baseline.

-Dave

I'm not sure aobut dbcache since I haven't used it yet; so I don't know in which range it should be, but I'll look into it in the future for sure.

For now, I would like to share my experience with a node that I've setup a few days ago. It is one of my nodes that needed a bit of maintenance; it was quite cluttered and had outdated software so I rebuilt it from scratch. I will also post a guide about it soon (OpenSUSE node walkthrough).
The hardware is a laptop motherboard with 4GB RAM and 2 500GB HDDs.
After it had taken almost a week to achieve ~40% sync, it was going super slow; around 1-2% a day, so I thought it may be a good idea to just plop in a second stick of RAM and see if it does anything. I kind of expected something to happen, but I was astonished at the speed it was going at afterwards! The HDD arm was moving much less now (audible difference); I suspect it was swapping a ton before, and the log was literally flying.

Here's a graph of some measurements I took; I let you guys guess at which point in time I upgraded the node from 4GB to 8GB



The speed at which it's going, makes me think it should be able to do a full sync in 2 days or so (from scratch); even though it's using a HDD.
Sooo.. all these modern hardware - based super fast syncing nodes.... (Pi 4 8GB + SSD + nice case) I think they actually gain most of their speed from the larger RAM and not from the SSD (which I was considering buying for this node actually)! Very impressive; that HDDs are actually this well suited for a fast IBD. I had not expected it.

You could prevent aggressive swap it if you reduce dbcache value (on Bitcoin Core) and swappiness value (on the OS). If you only need to run OS and Bitcoin Core, 1GB can do the job.

You're right - I made the (maybe wrong) assumption that everyone these days has an unlimited data plan anyway. If it's a huge issue though; it should be possible to do IBD in a place with free internet like a friend who has unlimited data plan, then bring the node home. If you aren't able to serve many GB of data to the network, Core can be configured so that it's still useful to you (e.g. when running ElectrumX as well => privacy) with minimal data downloads per month. At ~1.5MB/block, just downloading new blocks and not seeding should be 6,480MB per month / 6GB per month.

From my experience, it doesn't make much sense to use less than 4GB since 4GB is very cheap to buy and it's much better in terms of performance than 2GB. I actually tried it once on a VPS and it was using swap like crazy You don't really want to rely this hard on swap if you're already pushing disk I/O like you are during initial block download..