Topic: [Project Showcase] spruned - Bitcoin Light Client & Full Node Emulator (Read 216 times)

Version 0.0.8 released.

Changelog:

- add database migrations. This is tricky and may fail. In case, delete the storage and sync headers again (I hope not).
- improve the efficiency of SQLite DB (fewer calls, less size) by changing data serialization and queries. reduces the size by roughly a half.
- improve the efficiency on the P2P side when new block headers are notified (should fix race conditions of issue #85 )
- persist and index raw transactions when fetched via getblock (reduce network calls)
- improve the efficiency of the getblock verbose calls by using the per-block transactions index, avoid to load the entire block data
- increase the cache agent speed by using the per-block transactions index instead of the leveldb iterator

There are a lot of changes that may break the code, but they are very needed and useful.
The basic idea is to improve data indexing so that it can also be used in the future for other purposes (layerthree).
The release is available on pypi and docker hub.

Sources: https://github.com/gdassori/spruned/releases/tag/0.0.8
Pypi: https://pypi.org/project/spruned/
Dockerhub: https://hub.docker.com/r/gdassori/spruned

I've implemented the more efficient serialization. As expected the new database is nearly half the previous:

71632 -rw-r--r-- 1 1000 1000  73347072 apr 13 16:15 headers.db

So it fits ((8+8+32+80)*571000)/1024, with this schema:

        id INTEGER PRIMARY KEY ASC,
        blockheight INTEGER,
        blockhash BLOB NOT NULL,
        data BLOB NOT NULL

As told in the previous post, I preferred to implement a migration system by myself, so I didn't add new third-party dependencies, and is something I like.

https://github.com/gdassori/spruned/commit/6767d89aaa9dc13f4800aec15b582d035069d1ba

Thanks for noticing the discrepancy :-)

Fair. I confirm.

I don't know if a moderator can move the post from this account to my original one and lock this account.
It would be appreciated.

Can you confirm this one last time from your coindax-account?

I'm the OP (coindax). I had this account locked for years without knowing why.
When I went back in the forum with a new account, to show the project, a user told me I may had the account recovered now, and so it is.
I'll continue to post from here and cease to use the other account, which was just created to post about spruned.

Hi,
it's because in first place I used the electrum network to download the block headers, and this implementation is still in the codebase today.

The electrum network uses the jsonrpc serialization, so the data is transmitted as an hex string. Citing the Electrum protocol page:
<<... now past height 500,000, the headers form over 40MB of raw data which becomes 80MB if downloaded as text from Electrum servers.>>
This explains why the bandwidth usage is higher than it could be (You didn't asked explicitly, but I think it was part of a complete answer :-))

The design fault here is, instead, to store the the blockhash as hex in an sqlite table.
A spruned instance will so have an headers db as the following, for mainnet:
142,172,160  headers.db
Because of the related indexes (by hash and height).

(On the leveldb side, developed months later I did, instead, a better job avoiding unefficient data serialization).

It's in development a larger usage of the P2P network and a refactor of the repository to further reduce the bandwidth and storage usage, that will lead to a smaller database.
Changing the repos, however, implies to add a migration system, which is something that I'm working on carefully. Alembic would be a good and easy choice, but I'm trying to use new dependencies as less as possible.

API are quite stable today and I don't see a lot of effort, in the upcoming releases, on creating new features.
Now is exactly the time to polish this kind of naive implementations and increase efficiency, before adding more stuff.

this seems like an interesting project but i am a bit confused about what exactly it downloads. you call it "block headers chain" and claim it is "about 150 MB on mainnet" which is where my confusion is. every bitcoin "block header" is 80 bytes and by the time of writing there are 570849 blocks so the size of the file should be 45,667,920 bytes or 45 MB (which is the same as the blockchain_headers file that Electrum downloads).
so what is the extra ~100 MB for?

spruned bitcoin client

Links:

- spruned+BTC-RPC-Explorer howto
- spruned @ Github
- spruned documentation @ readthedocs (often out of date, contributors wanted!)
- spruned mailing list \ groups.io


Hi everyone fellow bitcoiners!

spruned is a Bitcoin Light Client that emulates the bitcoind RPC API. I present it here for the first time! The project is on development for a year, with intermittency. Now it's quite mature and stable.

spruned is:

- a bitcoin client
- some sort of pseudo-node
- something that runs on small low-end computers, embedded devices, oversold VPS, cheap boxes, small containers, everywhere :-)

spruned is not:

- a real bitcoin node
- a wallet
- something that participates to keep the network healthy

The spruned behavior is quite different from a full node, and is the following:

- A full node downloads the entire block headers chain, and even spruned does. This is about 150mb on mainnet, 300mb on testnet.
- A full node downloads the entire blockchain. spruned potentially doesn't download even a single block, until needed. When the "getblock" command is used, spruned go on the P2P network to download blocks from the full nodes.
- A full node do relay, spruned doesn't (this is going to change)
- A full node has big reliable indexes of data to provide transactions over the API. spruned doesn't, it uses the electrum network on demand.
- A full node does a lot of math on the data and ensures its coherence (concatenation, cryptography, pow, scripts legalness). spruned does just some checks to ensure the provided transactions actually exists in the blockchain with Merkle proofs.

Given those compromises:

- A full node doesn't run on raspberry with a 2GB SD with the full tx and blocks API. spruned does :-)

Some crypto-checks are done:

- Block Headers with Checkpoints verified since the Genesis Block to the last one useful (1 week behind at the time of any release, new checkpoints will be added from Bitcoin Core).
- PoW verified for each block header. The difficulty is NOT enforced yet.
- Every transaction presented by spruned API is verified with merkle proofs from the electrum servers to the local downloaded headers chain. spruned will never present transactions fake claims about transactions blocks inclusion.

Some non-crypto-check are done:

- Quorum Mechanism. The connections pool must agree when information is out of the checkpoints scope.
- Quorum Based Fee Estimation with the exclusion of the out-of-range sources.
- Cache FIFO with custom size, to avoid downloading data twice.

Sources decentralization:

- No block explorers, centralized sources
- DNS seed bootstrap, peers discovery.

Privacy:

- The privacy model is the same as electrum.
- You can use your own servers disabling the peer discovery for both Electrum and the P2P network.
- Electrum Trust Model merged with data from the P2P network.
- Configurable Tor support: Two modes: only as a proxy, and use peers on clearnet, or --tor, to enforce only hidden services usages, no exit nodes.

Compatibility

- spruned emulates the bitcoind RPC API. this means that a lot of bitcoind-based software could be also backed by spruned. LND, clightning, opentimestamps, btc-rpc-explorer and more, are compatible with spruned. whenever the network subset of the bitcoind RPC API is needed, spruned may help.
- zeromq implemented. same 4 channels as bitcoind emulated, to grant better compatibility also outside the RPC segment.

Dream list (aka Todo):

- exists :-) (brand, website, binaries, installers for windows, APK with Kivy [spruned is Python based])
- decentralize the source of trust :-) [yeah, I said dream list.]. I'd like spruned to keep a subset of the information the electrum servers provide and relay them over a dedicated P2P net. To go on the real Electrum network would be a fallback.
- random P2P blocks "sharding", indexes propagation over Kademlia (maybe, Kademlia, maybe there's something better out there, IPFS, and much more. quite a lot of new stuff popped out since the last time I saw that world. We need something with supernodes)
- uber efficient serialization because...
- requests obfuscation (asks more than you need)
- bitcoind RPC API wallet subset with an HD one :-)

Support spruned!

- You can donate! Even a small one is very appreciated (well, over the dust threshold:P). The Bitcoin donation address (same in the README.md on Github for double checks) is: 3FVGopUDc6tyAP6t4P8f3GkYTJ5JD5tPwV
- Participate! Programmers, designers, people willing to write documentation and guides. Everyone is welcome.
- Use spruned :-) Maybe the best contribute is using it and open issues if new bugs are encountered.
- Request features!
- Star the project on Github. Starred projects mean interest, reputation, and it's easy to find more contributors (and so speed up the development).
- Talk about spruned! Do you like the idea and you see a future for that? Do you like the idea of some sort of a decentralized "layer 2" on top of the P2P to exchange provable information out of the scope of the legacy P2P net? Make some presentation :-)
- Subscribe to the spruned mailing list: groups.io \ mailing list! It's quite dormant, but if people start posting, it will not be dormant anymore :-)

Contacts:

- freenode khs9ne \ mn3mnonic
- twitter khs9ne
- telegram gdassori