Topic: Huge increase in satoshidice spam over the past day - page 7. (Read 8875 times)

Hi Matt,

Thanks for that headsup. Having custom bloom filters on the server end of the connection would be far more efficient bandwidth wise. Will keep an eye on that discussion.

Jim

Its an interesting idea, but it largely defeats the purpose of running a full node.  If you are gonna relay a full block, you really should check its transactions, or you are running the same trust model as an SPV node while telling other nodes that you are running a standard satoshi-client trust model, which could make it easier for miners to outright lie for short chains of blocks if it becomes to widespread.

I dont think anyone is saying we arent gonna move to more lightweight nodes, but the network needs a significant number of full nodes, or needs a broader base of full nodes with strong connections and are able to manage a ton of connections.

Its entirely true that Bitcoin is designed to operate with a number of SPV clients on the network, and the bloom filter in p2p protocol options makes that more viable.  However, merchants and those who want an better trust model are still encouraged and should still run full nodes.  Also note that a huge increase in SPV clients will starve listening full nodes of resources and connection slots, making it harder and harder for clients to get connected, causing further issues with node-isolation attacks and similar issues.  In the end, though its /possible/ to run the network largely on SPV nodes, its far from ideal, especially when the only reason this is happening is because of one, maybe two sites who insist on being lazy and, frankly, stupid.

See http://sourceforge.net/mailarchive/forum.php?thread_name=CANEZrP0kNZDByHpK2%3DUjP%2Bag0X1KmqHxnJdm%3De_pWMitP4QvvA%40mail.gmail.com&forum_name=bitcoin-development

That is not at all true.  The point of blockchain pruning is that most, if not all nodes on the network no longer need the full chain.  As nodes upgrade to the alt-chain and drop full blocks from the normal chain, nodes which do not want to use the new alt-chain will have a harder and harder time bootstrapping or even getting full blocks from any nodes, thus forcing them all onto the alt-chain.  Even if you argue that old nodes will always exist and be a fair percentage of the network (which they currently are, which is really a security issue for the network) connection mechanisms get complicated as nodes are increasingly trying more and more connections before finding a working one, which could lead to increased sybil attack possibilities and generally higher load on listening nodes.


Absolutely, I wasnt saying it wasnt possible or really /that/ difficult, but merging all of that code into the satoshi client is quite a bit of work, or at least quite a bit of code.  And with the client already so complex and poorly abstracted, it introduces huge possibilities for bugs.

Meh.
You dont have to force anyone to include anything additional in blocks to do pruning.  If you do pruning, you can simply drop transactions and branches in merkle trees.  Of course you have to snapshot a pruned chain and nodes cant do the pruning themselves, but that is true up to checkpoints anyway.

I dont think anyone wants a hardfork, nor do I think its worth doing without serious issues, and I dont even think this is worth it.

I'm not even sure developers themselves should worry with this. This is a problem to pool operators mainly. It's up to them to come out with a solution, or eventually finance one by offering bounties.
I find "securing your private key(s)" a much more urgent matter than this.

What a drama, come on...

This will not kill bitcoin. Even without pruning, professional pool operators should be able to handle much more traffic than this. https://en.bitcoin.it/wiki/Scalability

It seems you people are thinking "ordinary users" should be full nodes. That's not the case. Only solo-miners and pool operators have such need.

But anyway, since that's what being debated here, I always wondered if there couldn't be a "relay mode". Something lighter than the full mode of bitcoin-qt, but heavier than the lightweight mode of BitcoinJ.
You store the whole blockchain, but in its raw format. You only validate its headers, not its transactions. You index only the transactions that concern you, plus a simple index to locate the blocks in the big raw file. You relay everything as a full node would.
The greatest charge that a full node imposes today is the indexation of the database. Storing in raw format would allow those that want to contribute their nodes as relays (as me and probably you) to keep doing it for a longer time, I guess.
But, honestly, I'm not even sure we should encourage this. For it to scale, the bitcoin network will have to change to a network where only some full nodes exists and all the others connect to them as lightweight nodes. So maybe SatoshiDice is just encouraging us to take this step.

Exactly I don't see 25GB being an impossible barrier for merchants, miners, exchanges, and hardcore bitcoin supporters. 

Honestly even when pruning happens at least a significant number of need to carry fully blocks so that nodes which wan't/need full block have access to them. 

So, currently the block size limit is a maximum of 500KiB:

1year/10minutes * 500KiB = 25GiB/year

So yeah, it's not tiny. But seriously, 25GiB/year really isn't all that much storage, and that's the hard maximum, actual is maybe half that. Sure the regular bitcoin-qt client becomes annoying to use for some guy who just wants to move some coin around, but anyone doing anything serious isn't going to have a problem. Myself, I run bitcoin-qt, but my actual wallet which I actualy use is on the excellent blockchain.info

The real issue is if the number of transactions makes it difficult for people to get their transactions verified, but as long as miners do fee-based priority, that's fixed by telling people to increase their fees.

My request for the devs: work on pruning, but no rush.

Bitcoin will die unless this is solved, its not much but I pledge 15 BTC to whoever creates a viable beautiful pruning-client.

Something which can operate with parts of the chain and only get the parts it needs to confirm incoming otherwise.

Should work like a torrent file where the full file could be distributed.


Basically my client should operate as before thinking it has the full chain, only it doesn't - it gets the parts it doesn't when needed from the cloud/peers.

After using such parts they are deleted.

It would be extra cool if parts of blocks could be downloaded instead of full blocks.


Maybe it could send a request to the network to get an incoming transaction verified by the nodes that have that block section?

Say my node has a block with A->B someone sends me a transaction with B->C, I check my block and verify B address has enough then okays it.

To clarify:
1. Blocks are verified as today via their expensive hash/tree root thingie.
2. All the transactions IN that block are verified by the network via the distributed block chain.

Time line:
1. Block is received ->
2. Full prune node (FPN) checks root hash against its own mostly empty chain ->
3. If okay, it checks the transactions it CAN ->
4. If okay it broadcasts the block while assuming the other transactions are fine ->
5. If NOT okay it does NOT broadcast it, but instead reports it including info about the invalid transaction and its block ->
6. Other nodes get the report, check the root hash of that transaction against their empty chain ->
7. If bad report is true they re-broadcast it ->
8. Bad blocks get propagated WAY slower AND are immediately turned down by super nodes ->
9. No major changes to core BTC logic, SMALL additional bandwidth usage for reporting.

If this is not done, BTC is dead.

Another idea:
"Allow transmission of half-blocks."
root hash <- 2nd hashes <- 3rd hashes etc.

You basically treat the second or third hash as THE block - the benefit is that slower connections can keep up with very large blocks as they only look at one branch of the blocks.
They would however need to check their own users addresses via 3rd service.

This is however not critical at present.

They don't really have to, AFAIK.

It's up to miners to do something, if they want to do something. Maybe they can handle the volume and are happy with the fees.
EDIT: And when I say miners I mean solo-miners and pool operators, the only ones who really need the entire blockchain to operate.

SatoshiDice is certainly pushing Bitcoin into new territory.

I thought I would mention a data structure I am going to experiment with in MultiBit as it is related to blockchain management.

Currently the MultiBit/ bitcoinj blocks that are stored to disk contain:


i.e 100 bytes per block.

It would be very useful to have a mapping of 'bitcoin address -> blocks the address appears in' so that you can get the transactions for an arbitary address/ private key/ public key quickly.

I was thinking of adding a Bloom filter for each block (indexed by bitcoin addresses that appear in that block) into the locally stored blockchain.
The idea would be that to get the transactions for an address or set of addresses:
1) You scan the locally stored blockchain bloom filters to get a list of blocks that you know contain relevant transactions.
2) You download just those blocks from a Satoshi client.
3) You then parse the blocks' transactions to get the tx data.

You can store the presence of an object in a Bloom filter with approx 10 bits.
According to this I estimate there are about 2,000,000 distinct addresses on the current blockchain and each appears, on average, about 4 times. This would mean that the total size of the Bloom filters for bitcoin address indexing would be about:
10 x 2,000,000 x 4 bits = 80,000,000 bits i.e. about 10MB

Not much space for what it gives you. It is small enough that an up-to-date version can be included in the MultiBit installation download (I already include a snapshot of the current 100-byte-per-block blockchain).

Of course this does not solve the disk requirements for the Satoshi bitcoinds. It is only usable for the end-user client as it needs a full client to download the (specific) blocks from.

Looking at your statistics, it seems that it's not you. Sorry.

Some big miner is creating blocks near 500 kB without increasing fees.

Why do you think so ?

[For reference:  the idea I proposed is a special blockchain pruning method that allows nodes to verifiably query any address balance with only a couple kB download -- and the pruned data would be maintained & enforced on a second/alternate blockchain using merged mining]

Theoretically, the alt-chain created for this purpose is strictly optional.  No one is "forced" to do anything -- they only use it if they want to participate in creating/exchanging/verifying/downloading address-balance information.  One major benefit of the idea is how perfectly non-disruptive it is.

I agree that it would be complex.  But there's a lot of alt-chains already in existence that use merged mining, which probably provides 90% of the template that would be needed.  I don't mean to imply that there's anything easy or quick about it... just that a lot of work has already been done.

Otherwise, there's two options that both have serious downsides (compared to using an alt-chain).  But maybe some brainstorming can resolve it:
(1) Overlay network like what stratum is trying to do.  Issue:  requires some trust of other nodes, and malicious nodes can really muck up the network.
(2) Modify the main blockchain, forcing block-headers to include a valid balance-tree hash to be accepted.  Issue:  requires a hard fork. 

However, there's a dozen other things that could go in as long as we're doing a hard-fork, anyway.  If there's any inclination to believe it will have to be done at some point, earlier is better...

It seems that Deepbit doesn't dynamically increase tx fees like the Satoshi client does. The Satoshi client would keep blocks at around 250 kB.

The same result can easily be achieved without forcing users to add at least 2 transactions per bet.

There is an advantage in using blockchain such as transparency, anyone can verify how the site operates.

EDIT:I never used Satoshi Dice, but I like the idea of honest lottery.

I have to admit I havent spent a huge amount of time thinking about it, but its definitely an interesting concept.  It does carry with it some of the issues of alienating old nodes which may have issues finding peers which have a full chain to give them, but I believe that is inherent in any chain-pruning method anyone has come up with so far.  It does complicate bitcoin as a whole very greatly, forcing everyone to follow multiple chains (or...I cant think of a way to switch over to a new chain cleanly without hard-forking) and heavily complicating verifying the initial checking (probably ending up moving the initial download verification to the same trust-model as SPV nodes, which is ok, but not ideal).  These complications only further the effort alt-client devs have to put in, which is already pretty huge and alt-clients are pretty important for the health of the network.  In the end, I dont really see anyone jumping to implement it, so I dont see it happening any time soon, but if it does happen, it will provide some interesting new features.

Matt, can you comment on the armory dev's proposal for chain pruning facilitated by an balance alt-chain?