Topic: The size of the blockchain... - page 2. (Read 4119 times)

And fast storage tech 10 years hence will not have the capacity and cell programming limitations of present-day flash technology. This sort of factor is never accounted for when arguments are made against the scalability of the blockchain. But significant near-term breakthroughs in both mechanical and solid state storage are mature enough to hit the market this year and next, and this is after both technologies have been bumping their heads against a ceiling for a couple of years now.

A couple years back I donated a 2GB SD card made by SanDisk to science.  I setup a simple loop which continually wrote to the card with pseudo random sequences until it filled the card.  It would then read it back, verify it was correct, erase the card, and start over.  I got 11TB worth of writes before the card started showing failures.  I have no idea if this is typical but lets assume it is.  A 64GB SD card would be more like 700 TB of writes, which should be sufficient for a decade of blockchain usage even if the tx volume increases by a factor of ten or more.  

Of course one advantage of a dedicated device is you can strike a compromise between storage writes and memory usage.  One example would be to only write confirmed tx to the memory "disk", the memory pool would remain in memory.  Blocks are occasionally orphaned so with enough memory you could delay writing the block to storage until it is x blocks deep in the blockchain (each block deeper becomes exponentially less likely to be orphaned).  Other temporary less critical data like information on current peers, can be batch updated to storage.  If the node goes offline between updates it isn't a critical that the stored information about peers is slightly stale.  The fact that Bitcoin is a decentralized network makes it easier to implement delayed writing.  You can always request recent information from peers again.

The main scenario to avoid with flash is using it as a high throughput "RAM", but even a memory constrained device, will have some memory so there is no need to write everything to flash.

Thinking about this some more, it doesn't really improve your DOS resistance having UTXOs in RAM as connected nodes can just send lots of badly signed (but otherwise valid) transactions instead.


As long as you are just writing new data and not writing over old data, they should last well.

As long as you can limit your erase/write cycles, this analysis earlier in the thread suggests it should work:

The power supply required is intimately related to the device's power consumption.  The kind of device I'm imagining could run from a simple walwart, power-over-ethernet, or directly from a low-voltage outlet if LV wiring was available (as is becoming more common--take for example AC outlets that now include a 5V USB charging port).  



You are thinking too specific and perhaps PoS was not the best example.  But imagine a tiny low-power, low-cost bitcoin node with trivial set-up.  I think some people would find innovative uses for these, while others would buy them simply to help support the network.



You wouldn't mine with an ARM processor.  You would mine with a bitcoin miner that employs 1 or more SHA256 ASICs.  If the ARM-core bitcoin node was cheap and simple enough, the "hashing device" could be transformed by default into a "plug-and-play P2P miner" by the manufacturer.  This may help to decentralize mining.  

SD Flash will die pretty quick if you continuously write to it, like for a blockchain database.

So ok for backup, but not really for a node.

That is a good point.  For a set consisting of 10 million objects 256 bits is not necessary to avoid excessive collisions.  Taking the left x bits should be sufficient as this isn't being used for cryptographic purposes (and NIST certifies SHA-2 to be used for truncated hashes in some applications anyways).  The index on the other hand requires a little more thinking.  The protocol allows more than 256 outputs per transaction and is stored as a VarInt which can potentially be up to 8 bytes long (but realistically I couldn't more than 3 is probably never going to happen).  I guess one option would be to to use x bytes for the lookup hash where x = (8 - VarIntSize).  Worse case scenario you could use a full 8 bytes for the hash and full 8 bytes for the index but that would double the memory requirements.

Thanks for this.  Do you think it is possible to keep the UXTO stored in some way that I can find a specific output in O(log n) time, where n is the number of unspent outputs?  I am not really qualified to say because I don't even know how the UXTO are currently organized by bitcoind.  But it seems that right now I could sort all the UXTO in canonical order to permit O(log n) look-ups.  I imagine there would be some way to keep the UXTO organized such that this remains the case moving forward in time too...

Personally, this is just a thought experiment for me at the moment, as my hands are full with a much simpler hardware project.  But it is interesting to consider what could be done with existing technology, time, and money.   

He means the size of the blockchain itself.

No it wouldn't.  Miners connect to the node (just like they do any node running p2pool) over TCP/IP.  The mining software is on the miner.   Instead of pointing it at a pool wallet you point it at your p2pool node, exactly as you do now if p2pool is running on a desktop.

As for not useful well that depends.  Just as a hardware which runs a full node for the network and does nothing for the user well your right that is beyond stupid.  Why would one buy one?  However how about this.   Your wallet is on your computer, you don't want to run bitcoind locally for a couple of reasons such as you have multiple computers running bitcoind on all of them it excessive, or you don't use your wallet everyday and hate the fact that when it syncs up you need to wait.

So instead you use a light wallet which connects to your local bitcoin node (running on that <1W device connected to your local network).  Got 8 wallets spread across multiple computers no problem they can all point to the same local bitcoind.  Now you could use a SPV client but maybe you don't like the security compromise and the fact that it doesn't strengthen the network.

Bitcoin transactions arrive at up to about 10 per second so checking the inputs are valid by reading flash storage is not too difficult. The problem is you want to avoid DOS attacks by having quick access to transaction hashes There are about 10 million unspent outputs, if you use a modified encoding which hashes transaction ids down to just 7 bytes and 1 byte for the output count this means you need 80MB of memory plus a few MB for the mempool and recent blocks.

This would require USB or serial support, and mining software. This gets increasingly complex as soon as someone starts looking for real use cases, which IMHO means the base design isn't really that useful

True but it could be useful to have high power ASIC miners connect to the node which is connected to p2pool.

If the flash (and controller) is sufficiently fast it may be possible to use that as the UXTO lookup storage.  This would drop the memory requirements to near negligible amounts (enough to hold memory pool, current block, and if used as a wallet the user's personal pubkeys).  A lot would come down to how long lookup of a random output from the UXTO would take.  A good place to start would be getting a development board.  Instead of working with a ARM CPU directly you could work with a microcontroller which is based on an ARM CPU.

I am a big fan of STI Micro.  This is their STM32F4 series. 
http://www.st.com/web/en/catalog/mmc/FM141/SC1169/SS1577/LN1806

The 429/439 is the most powerful microcontroller they offer.  It gives you 256KB SRAM, 2MB of flash, ethernet & usb connectivity, plus a hot of other stuff you probably won't need.  The 439 is the 429 with hardware crypto acceleration but sadly does not support the scep256k1 curve (I called to make sure ).  The main differences between the pin packages is the larger ones support more analog and digital IO pins. 

There is a development board available for rapid prototyping
http://www.st.com/web/catalog/tools/FM116/SC959/SS1532/PF259090

I'm not concerned about power usage, but about the necessity to supply power to the device, which requires dedicated hardware. Power supplies nowadays are indeed quite small, but one of them would still probably be bigger than the device itself.


This hypotetical device and a POS have completely different (and actually opposite) design goals: a POS must act as a wallet but doesn't require full node capabilities, while this project wants to create a full node without wallet functions.

Mining is a completely moot point. Given the current difficulty, even a million ARM processors wouldn't even be noticed by the network. They would only waste power.

LOL you began answering my next question before I posted it. 

This all makes sense to me then, and explains why Cypherdoc's bitcoind node uses so much RAM: it is mostly to permit fast look-ups from the UXTO.  If the UXTO can be organized and stored efficiently on a SD card, perhaps with the help of some intelligent caching system like you said, this might work.


(I also think people are underestimating how fast SD cards (especially nowadays). SD cards do not force a file-system like FAT, they are basically just an SPI-interface to a huge array of flash.  So I can read small blocks out of flash with an SPI command to address the memory I want and then clock in even short blocks of data at tens of megabytes per second.  There is perhaps less overhead here than it may at first appear.)

Thanks DeathAndTaxes.  If there is a "gotcha," I think it is this.  Earlier I proposed storing the UXTO on the SD card rather than RAM, organized in some efficient manner for fast cross-checking.  The UXTO would be updated each block, and earlier in the thread I showed that erase/write cycles every 10 min should still provide the memory card with several years of life.  

Am I missing something?  Is there a reason I need to store the UXTO in RAM that I cannot see?  It seems to me that even 64 MB or RAM would be sufficient (this is available on a 6x8mm ball-grid array chip for $1.50).  64 MB gives lots of room to pool the unconfirmed transactions, accept new blocks from peers, etc.  


Like you said, this certainly isn't a trivial project.  For the time being, it is a thought experiment: "how simple, cheap, small, and low-power can a useful bitcoin node be made?"

The memory usage is primarily storing the UXTO in memory.  The memory pool is also kept in memory but it is much smaller than the UXTO. You could probably reduce the memory footprint with some "intelligent" caching system scoring each output based on the likelihood it will be in the next transaction (i.e. the unspent output from the coinbase of block2 is probably not going to be used in the next tx so dropping it from the in memory cache won't be an issue).

Remember reads from disk are very slow (compared to memory) and you can't validate a transaction until you lookup the outputs.  If none of the unspent outputs are in memory well you have now made 100% of tx validations requiring a round trip to the disk.  Right now throughput of the network is slow enough that you won't fall behind however for bootstraping a new node it is going to be painfully slow.

One of the big appeals of ARM-core processors are their low power consumptions.  For example, a cortex-M4 core processor with 1.2V core consumes only 33 uW / MHz.  At 1.25 MIPS/MHz and 150 MIPS operation, this gives a power consumption of ~4 mW.  I believe a large portion of the energy requirements would go towards ECDSA signature verification.  I estimated earlier in this thread that an ARM-core processor running at 150 MHz could execute secp256k1 verifies in 20 ms, which at 4 mW represents 80 microjoules per verify.  Assuming 2 signature checks per TX and 10 TXs / s, this is 1600 uJ/s or 1.6 mW.  So this is very low power.  

Of course there are many other activities that will consume power too (reading and writing to the SD card, TCP/IP communication), but I think it is generally agreed that performing the signature checks is the most work and this process can be made fairly low power.  


This is actually quite straight forward, as there are both chips that implement TCP/IP and processors with integrated ethernet modules.


Yes, you would need this.  


The TCP/IP stack can come as part of an IC, or as a code library for the processor.  Attaching a microcontroller to the internet is routine nowadays.  However, implementing all the networking features of bitcoin would still certainly be a lot of work.  


I'm starting to feel confident that it is feasible too.  But how do you know how complex or costly I think the R&D process would be?  I assume it would be a great deal of work.  But I am interested in the question "how simple, cheap and low-power can a useful bitcoin node by made?"  I think this is an important question.


From my experience, once a project like this is complete and working, people find all sorts of applications.  

For example, a lower-power, small and inexpensive bitcoin node could be integrated in certain point-of-sales hardware, or it could come with bitcoin miners and run P2P pool by default (helping to decentralize mining efforts).  

It's been said before, but bears repeating: internet routers make the ideal candidate for a low cost bitcoin node device. The trouble is that both RAM and storage of a typical 2014 device are not in anywhere close to the right kind of league to do the job.

But maybe several development cycles of a mass produced standalone device (as I believe to be what OP suggests) could bring us somewhere close, and perhaps then in ~5 years they could be cheap enough to be packaged together in a slightly premium priced router product.

It certainly makes sense to sell such a device in the interim, as many a local network will be running more than one Bitcoin Core or Armory wallet as and when adoption ramps up. There will be a demand for just such a device that stores, maintains and serves one blockchain copy per building/household. Firing up bitcoin core once every few days, or even only once a week can get tiresome. Better to have an appliance that ensures you're ready to send or receive current transactions at all times (and one that has a dedicated amount of disk space, running out of storage is not an uncommon experience when running a bitcoin node)