Topic: Exploring alternative full node implementations - Gocoin (Read 437 times)

I could give a try if i only need to recompile gocoin and set environment variable. But since i don't use rolling release distro, it'll take some time before golang 1.19 available on the repository.

If i understood the release note correctly, it require modification Gocoin source code. That means i only could wait until Golang 1.19 available on my OS repository and Golang developer decide to use such feature. Even so, i doubt it'll bring big impact since Golang load all UTXO to RAM and i expect UTXO size continue to grow.

You'll have to recompile with 1.19 to be on the new runtime, but I don't think changing Gocoin's source code is necessary. The soft memory limit can be configured via the new GOMEMLIMIT environment variable [1].

Like I said, I haven't experimented with this, it just caught my eye and I remembered this thread.

[1] https://go.dev/doc/gc-guide#Memory_limit

Since a good amount of the discussion has been about excessive memory usage, I just wanted to leave this excerpt from the release notes [1] of Go 1.19 (released ~6 days ago):


I haven't tried it, but maybe this will be of help to someone looking to get Gocoin's memory usage under control.

[1] https://go.dev/doc/go1.19#runtime

There is little that can be optimized in the UTXO data format, for example it certainly cannot be compressed using zlib or zstd because these only give paltry returns in space, but require at least linear time to decompress every time you need to check a UTXO set.

At the end of the day, it really is down to the language you use. Even interpreters that are built on top of C++ tend to use much more resources for small programs than the equivalent codebase written directly in C++ (and even using a modest number of class objects).

It's not impossible if you ask me. Deserialized transaction on Bitcoin Core mempool is far higher than 2x. Looking at https://statoshi.info/d/000000020/memory-pool at 2022-07-30 09:10:00, totalTxBytes is 993KB but it's mempool memory usage is 4.94MB.

After searching a bit online, it seems that Go has a built-in garbage collector that can indeed use memory a bit lavishly. However it can be improved, such as described here.
https://medium.com/safetycultureengineering/analyzing-and-improving-memory-usage-in-go-46be8c3be0a8

I can't say for sure, but it's possible that these Bitcoin Go projects that have higher memory usage than their C and Rust counterparts are simply not optimized like that. So in that case you would be right that it depends on the developers. Would love to see them get more optimized so they run on cheaper hardware, which is where people generally like to run their nodes (be it Raspberry Pis at MSRP or used x86 hardware).

That's handy!

You think it takes more than double the amount of memory to store it deserialized? So if I were to set my node's dbcache to something like 5GB in Bitcoin Core, it wouldn't fit the whole UTXO set?
And why is it 8.5GB instead of <5.. Another multiple GB wasted - I think there's a lot of optimizing potential for this software before being widely used.

I wish I knew more about GO to be able to give a technical answer but I still believe that optimizing code depends mostly on the developer not the language.
For example if you implement UTXO like this in C# you'll get a lot of overhead because of the way the byte array is stored elsewhere in the memory then a reference to its position is stored inside the struct with a bunch of other overhead that increases the total memory consumption.

To give you an idea about severity of this in C#, I was running an analysis on block headers and the program was consuming more than 200 MB. With some tweaks I brought it back down below 100 MB and closer to the file size itself which is about 60 MB.
I use this example because people generally think C# is slow too which is not correct.

P.S. FWIW the live UTXO size could be fetched from here: https://statoshi.info/d/000000009/unspent-transaction-output-set

What I don't understand is that the UTXO set is not this large. I am checking again, but it should be in the 4-5GB range; at least it was last I checked.


This would fit on my system in RAM by just setting -dbcache=5000 or something like that, even though I only have 8GB of system memory. Keep in mind I also have electrs and lightningd running.

It could actually have something to do with the choice of programming language after all, as I mentioned before. Similar issues (high system resource utilization, hard to optimize / reduce) were reported by various people when using lnd (Go) instead of c-lightning (C).

I can see this becoming an operational problem if the UTXO set doubles or triples at any given time, then GoCoin will simply run out of memory before it can load all the UTXO sets. Why not just load the most recent UTXOs, if short of RAM? It's more likely that recent coins will be spent before dormant coins anyway.

I finally finished writing my experiment with Gocoin, it took longer than expected. Does anyone have suggestion what alternative implementations i should check next? But please don't suggest one of these,
1. Mako. It's still on heavy development, https://github.com/chjj/mako#development-status.
2. Bitcoin Knots. It's fork of Bitcoin Core, so i don't expect major difference with Bitcoin Core.
3. Libbitcoin version 4. I have negative experience with it, https://bitcointalksearch.org/topic/m.56811006.

---

That makes sense, but currently Gocoin RPC only limited to mining-related feature. People who need instant access to all UTXO need to find workaround with Gocoin's Text UI.

It is, though. For instance, a (modern) interpreted language like Python will be slower than a compiled C binary that does the same task.
So in general, I wouldn't say that speed and efficiency are completely unrelated to chosen language.

I am not knowledgeable enough about Go to explain well why it performs worse than C, but in a similar task (running a Lightning node), the Go implementation has been causing lots of issues for people and I was under the impression that Go is harder to optimize than C or Rust since it's more higher level / more abstract. But I may be wrong, that's just what I read online.

Okay, I mean it does have a benefit then. I guess it will depend on your application. It's going to be rare, but whoever runs services that need 'instant access to all UTXOs' may want to actually run Gocoin. Especially since if they are optimizing Bitcoin Core for similar UTXO access performance, they may increase dbcache to a similar value anyway.

Gocoin simply load all UTXO to RAM.


I don't see the point, since it's heavily based on your dbcache value. IIRC it only use 1-2GB RAM although dbcache value is far higher.

---

If you could show tutorial without installing lots of application or recompiling kernel, i could do a quick test.

Why is all that RAM needed, to make one giant dbcache? The majority of servers have only 16GB of RAM, so I don't see this client being used by a lot of nodes.

Technically speaking speed and efficiency is not decided based on what language was used (at least for most modern languages this is true). The implementation and the developer's expertise decides whether the code is fast or not.

Interesting, thanks for looking it up. A bit rushed from my side not to check myself.

One of my nodes was a laptop with 4GB of RAM and upgrading it to 8GB gave the initial block download an amazing speed boost. But for normal operation it would get along just fine with 4. I know people for whom 8GB is totally fine on their daily driver machine, so yeah, for maximum accessibility (think: lower income countries, etc.) requiring 16GB is definitely too high. My belief is that low system requirements are needed for the best shot at true decentralization.

But I guess it comes with the choice of programming language (Go). It would be great seeing a Rust implementation, since that should in theory give almost just (?) as performant code as C. In fact, Core Lightning is moving away from C and rewriting things in Rust.

According to their github page yes, you need AT LEAST 16GB of RAM


Seems very excessive. Yes it does a lot but still 16GB for most people that makes it a non starter. If I'm running it on my 'main PC' that would mean I need more then 16GB so I could do other stuff.
If I am running it on an older spare PC, that probably means I need to buy more RAM or just run core. Which can run fine on a much lower end system.

Still nice to have more options.

-Dave

Exactly my question. I assume it uses what is available, as project has already several years I am not sure if having more than 16GB was so obvious years ago. I really wonder how it would behave on machine with 8GB total (which means it would have around 6GB free, to use).
Would it be possible for you to launch it with limited memory settings? (maybe using cgroups?)

Thanks for the experiment and the great writeup. I've been toying with the idea of spinning up another node with an alternative implementation a while ago.

This is the only thing worrying me a bit: 19GB of memory usage is a lot. Not that RAM is expensive, but some nodes simply don't have that much built-in.
Do you know whether Gocoin bases its RAM usage on how much total RAM is available or would it be creating a 3+GB pagefile on a 16GB machine?

And can you compare with Bitcoin Core's memory usage on the same system?

WebUI

To access WebUI, just type 127.0.0.1:8833 or localhost:8833 on your Browser. You'll see homepage of Gocoin which show some statistic about your node. Most parts is easy to understand if you already use Bitcoin Core with primary intention to run full node. I don't what SPB refers though.



"Transactions" tab shows your node mempool. Compared with Bitcoin Core, Gocoin show more detail information about your node mempool statistic. You can enable/disable memory pool and transaction relay here. You also can upload and broadcast transaction as well. I haven't test this feature yet, so i don't know accepted transaction file format.



If you scroll down, you also can see list of transaction on your node mempool. It's quite detailed where it shows total signature operation and time to verify the transaction in milliseconds (ms). You also can choose how much transaction to be shown and how it's sorted.



"Blocks" tab show some newest blocks. It's quite interesting to see total signature operation of each block, which rarely shown on Bitcoin block explorer. The rest is something you can see on most block explorer.



"Miners" tab show statistic about miner during last 24 hours. I like how Gocoin show how much BTC earned by miner/pool from transaction fees.



"Counters" tab show lots of statistic about your node. Most of you won't find it's interesting, unless you're trying to analyze something.



TextUI

TextUI is available after you start Gocoin from terminal, load all UTXO/mempool to RAM and optionally start WebUI server. When you enter help, there are 56 available commands. But i'll just show few command here. By the way, the TextUI doesn't support key up to show previous typed command.


allbal or ab shows Bitcoin distribution on each address. I just found out there are at least 11500 Bitcoin stored on Taproot address.


balance or a show balance of an address or raw public key. The result is quite detailed which show when and how much an address receive Bitcoin. I really like this feature since it's fast and not exist on Bitcoin. But i wonder why the developer chose a as short hand option rather than b, bl or bal.


richest or r show Bitcoin address with most Bitcoin. It also can show richest Bitcoin addressed based on address type. I wouldn't use it much, but it's good this feature exist.


Gocoin configuration

As i said earlier, Gocoin can be configured through gocoin.conf file. But i just realized you can edit it from WebUI homepage. But what surprise me is there are few field which isn't mention on Gocoin config documentation[1] which are RPC., TXPool.Debug and Memory.FreeAtStart. The RPC is limited to experimentation mining API though[2], so it's not interesting feature for me.



Reducing RAM

Since Gocoin RAM usage is quite high, i tried to reduce it's RAM consumption. Based on Gocoin documentation page[1,3]. there are few action i could do. First action is compressing UTXO records. With tools i compiled earlier, it only reduce UTXO.db file from 8.5GB to 8.2GB on the HDD. Second action is modifying gocoin.conf which involve memory, you can see my modification below.


I managed to reduce RAM consumption to ~14.9GB. IMO it's rather small reduction since Gocoin usually use 17-19GB RAM. But there's nothing much i could do since Gocoin still store all UTXO to RAM.



Conclusion

Aside from RAM usage, Gocoin appear to be decent Bitcoin full node implementation. It's very impressive single developer write Gocoin alone. I don't need instant access to UXO though, so Gocoin isn't suitable for me. But even if i do, the RPC currently limited to mining-related feature.

Some things i like,
* Minimum dependency with easy compilation process
* Lots of features on TextUI
* Neat WebUI, although some feature only available on TextUI
* Lots of external tools
* Fast sync time, which only hold back by my HDD


Some things i don't like,
* Load all UTXO to RAM
* Incomplete documentation
* RPC only support mining-related feature, even though many feature exist on TextUI

---

[1] https://gocoin.pl/gocoin_manual_config.html
[2] https://gocoin.pl/gocoin_issues.html
[3] https://gocoin.pl/gocoin_tweaks.html