Topic: Starting preliminary 0.94 testing - "Headless fullnode" - page 6. (Read 15098 times)

I changed an aggressive lock around key addressing that was meant to cover certain stuff for supernode, and is unnecessary for fullnode. Now it's limited to supernode. I guess that untangled some other part of the process.

6e839bc fixed something for me unexpectedly, now getting 288 headers and 288 block files processed whereas before was getting either 285 or 286. Another difference is that the OS handles the interrupt overload when tx scanning alot more gracefully, background tasks in other domains are now given near full opportunity to take as contiguous processing cycles as they need, and the scanning appears to be hammering the CPU just as heavily. i.e. OS is usable simultaneously with scanning

I expect it will be that much of an improvement, OpenSSL is painfully slow at ECDSA.

And the bitcoin core developers will likely improve that balance when they eventually get around to using libsecp256k1 for transaction verification as well as signing. That's probably the most significant bottleneck on these ARM systems, lack of clockspeed to handle compute-expensive crypto functions. Cutting the demand for CPU cycles by 6 or 8 as is claimed could cut the intial sync time by a similar factor.

You can't compare the resource cost of initial synchronization with that of maintenance. I expect it would take your board over 2 weeks to sync the blockchain from scratch, yet it can process new blocks at an acceptable pace.

You probably had to sync the chain on an actual computer then move the DB and data over to your RPi (or whatever the board is). Same goes with Armory, maintenance cost is a fraction of what it takes to build the original database. Considering 0.94 hardware requirement, I expect that board of yours can run a fully sync'ed armory daemon just fine in fullnode.

As for the letstalkbitcoin guys, they were probably using an old version that had no scalability. Pre 0.9x kept all data in RAM, pre 0.93 was single threaded, relied on LevelDB (what legos are to structural steel) and had serious scalability issues which made the backend come to a dead stop once it reached critical mass. As for 0.93, it scales with RAM (lots of it). 0.94 is the first version to reduce hardware requirements (120MB disk space from 30GB+,  <2GB RAM from 8+) while increasing performance AND increasing scalability.

This is finally approaching good functionality and stability. It is also trivial to toggle RAM and CPU usage in this version, so I will most likely add command args to tune that for expert users.

You'll find this version will be comparatively faster on anything with more than one core, I think that includes most ARM SoC's these days. And because the database for Armory drops down to 120 MB, it's much less concerning to think about the future disk space.

Perhaps your FAQ should list minimum and recommended hardware specs.

I am running my full node on a $35 ARM single board computer with 1 GB ram.

The letstalkbitcoin guys say they used to use armory, and created a new tipping address for every episode, but stopped using it because it became too slow.

As with most transactional DB, there is only 1 write transaction active at any given time, so you can't have several threads writing to the same DB concurrently. In this aspect, multi-threading the build code seems pointless. Because of this, I simply grand-fathered the original build code into the backend overhaul. It relied on mirroring the full content of the blockchain, and that made storage bandwidth the only significant factor in build speed, so I ignored most optimizations in this area. Now that it skips the blockchain copy in fullnode, a meaningful portion of the building phase is taken by block parsing and processing, so there is enough performance to gain to justify the effort of multi-threading this part.


That's the same mentality that brings large IT companies to state that tablets and mobile devices will replace PCs in every home. For their use pattern, they don't need that kind of oomph, and in a way I believe no one but gamers and professionals needed that kind of oomph until large scale P2P networks appeared.

Now with Bitcoin things are quite different. With the blockchain eventually moving into big data territory, not only are big PCs relevant, but the implementation performance matters too. The faster the software, the lower the barrier to entry. If anything, my intention with these changes is for the code to scale with whatever hardware it is thrown at, so that a 10yo computer still has a chance at syncing a wallet against a full node, and that I won't have to rework everything from the ground up 10 years from now either.


My current machine is 6 core cpu (12 execution threads with HT), 32GB ddr4, SATA SSD RAID0 and fiber internet (~250Mb/s). And I'm thinking I should go with 64GB RAM soon. I had thought of buying a dual CPU board and go with a couple Haswell v3 Xeons but I backed off. Can't blow all my bitcoins on a PC =P.

For info, it takes 280~300 sec to scan the DB with ~5k addresses, about 80sec to build, and another ~100 sec for the header reading and what not (that could be reduced to 15sec with some effort).


That would be njaard (I try to avoid referring to people by their name on public forums unless they make a point of doing so). He was part of the original effort in remodeling the backend, which was deployed in 0.93. Since then he was given other responsibilities and I'm the only left with the task of completing this todo list. It still has some large entries to deal with, like blocks over P2P, lite node, full granularity coin selection, Trezor.

Andy is a Python dev. You can see some of his recent work in the hotfix release (0.93.2)

That sounds impressive, I just assumed the building workload didn't translate well to a multi-threaded design.

You know those people that say stuff like "3 GHz? 8 GB RAM? You can't even use that spec for anything! More processing than anyone could use!". Well, I'm imagining a machine with, say, an 8 core CPU, 32 GB RAM, a PCI express SSD with NVMe, plus a 100 Mbit internet connection. That's probably a mid level dev machine here and now, and I think it could setup Armory 0.94 and Core 0.10 in about 2 hours, going flat out. And the performance of both those pieces of software are set to improve further in the future? I'm slightly amazed that so much efficiency can be gained (and that we have hardware suited to the task )


Lol, I have clearly forgotten the name. You have another dev who is a Linux type, Andy?

Keep in mind that most the effort was in the scanning code. The building code is still single threaded so a lot of speed can be squeezed there as well (in due time).


Who's Nick o.o"

DB build performance seems now to be outpacing the increase to the blockchain size. With my 4 year old quad core machine, OS and blockchain running from an SSD, I'm getting ~45 minutes to set up Armory, and that's including transaction scanning for several wallets that pretty much covers every bitcoin transaction I've made since 2012. I dare not try, but I suspect that 93.x might take 2-3 times longer to do that job.

Great stuff, goatpig (and I think Nick was involved too?). Thanks.

Looks like we are going with this for the testing release + a couple minor bugs I'm still working on.

Thanks for your invaluable help guys, in particular Carlton Banks =). I guess etotheipi will create a brand new topic for the testing builds, sometimes this week.

Looks good. Made it through 3 full build/scans cleanly. Thread toggler only adjusts once, whereas it typically did so at least twice with the commit it was last enabled for (the thread toggler seemed to provoke the crashes before). CPU still gets tortured. It handles being interrupted during the scan OK too; it picked up where it left off fine, did 2 thread adjustments (the only time that happened) and finished the job. If this isn't good enough for the testing release, then it's probably pretty close.

I plan on adding at least a command line arg to limit the amount of available cores for the scan. The current setting is to squeeze as much CPU time as available.

My CPU heatsink is doing the same

Aight, time to re-enable the thread toggling. Bracing myself =D

Current push is stable enough to build and scan Db 4 times completely today, using extensive/varied wallets. Zero failures so far.

Forgot to initialize the atomic flag used to signal termination (I'm an idiot). Unlike to most std templates, atomic objects are constructed with an unknown state (which actually makes sense). Obviously I knew all that and I forgot to clear the flag in the ctor...

If this proves to be stable, I'll reactivate the thread toggling (which relies heavily on this flag).

Previous commit did produce one successful scan, but it mostly generated the deadlock that happens when/just before the scan begins. Scan completed through.

19c1a72e7f seems to reliably avoid the scanning deadlock, or at least it's done so twice consecutively so far. Continuing testing, trying to get a few more data points. Scanning completes.

Let's see what the latest push can do