Topic: Removing incentive to mine in pools - page 2. (Read 3041 times)

You can manage miners with much less resources than running a full node.

You can even get a router to do it (if it can be flashed).

This kind of hardware won't support a full node.


Changing the protocol means changing the reference client. 

In my view, they should split the client into a server (bitcoind) and a client mode. 

The server would just verify transactions and blocks, and wouldn't be able to create new ones.  You pass it blocks and transactions and it tells you what is valid.

It simply defines what counts as valid transactions. 

This would be a much simpler piece of software.

Making changes is hard though.  It has been described as redesigning a plane while in flight.

They want to keep risks as low as possible (which is reasonable).

A formal/official p2p mining pool system means that they don't have to update the official client.


Mining against a centralised pool means just pointing their hardware at the pool.

Any p2p system has a larger overhead than that.

A very lightweight p2p mining system might be acceptable.  Miners can use a proxy to actually connect to the pool.

The p2p pool would have to run on those proxies.


Their POW is used just as much as a larger miners.  Virtually all hashes that are performed are worthless.


Right, in fact, there is a disincentive.  You have to run a p2pool node too.

Currently, miners don't have to run a full node.  They can connect direct to a mining pool.

How does adding p2pool capability to the reference client help? 

As far as I can see, you are making the reference client more complex.  There is no real benefit and now the reference client is more complex and more difficulty to maintain.


Fundamental changes to the protocol pretty much create an alt-coin.


That is a hark-fork change right there.


Maths please.  How do you work that out without looking outside the network?


You get a 30 second target by reducing POW by 20.  If the minting fees were also scaled down by 20, then everything remains in balance.

However, again, it is a hard-fork.


Ok, so you are proposing an alt-coin explicitly then?

Ideally, if you want changes in the official protocol, you need to do it in a way so that old clients will still accept new blocks.

This is called a soft-fork.  If you make the rules more strict, then old clients will still accept the new blocks, since the rules are more strict than their requirements.

If a majority of the miners follow the new rules, then blocks which meet the old rules, but fail the new (stricter) rules will be rejected by miners, so never get into the chain.

So, you need to understand the protocol.  Maybe what you want needs a hard-fork change (fails backwards compatibility).  But, you should try to find a soft-fork way of getting your ideas accepted.

I'm not sure what you're arguing TierNolan.

I agree, running a full node is low complexity, I run one off my laptop whenever it's turned on and my I notice absolutely no difference unless it's been turned off for a couple days and takes a couple minutes to download and verify everything.

As for the second point. I'm not talking about overloading the official reference client, I'm talking about the whole protocol - the way we generate blocks, verify them, etc... I think we can do better to make it scale better for the massive amounts of computing power that are moving into mining. Now it's nearly impossible to get payout without joining a pool - we can structure it so that people only compete with those of similar mining power to them, and have incentive to do so. We can also structure it so that someone with an insane amount of mining power would have to distribute that power in a way that makes it harder to attack the network.

We're still the early adopters, and as more people adopt, more people will buy mining ASICs and do a quick Google search "which mining pool should I go with?" - The big pools will be the first hits - and so the big pools will slowly grow their computing power. Sure it's a stable system theoretically - but when things like incentive get involved people start doing funny things. We need to provide incentive for everyone to do what's best for the network - that's the beautiful thing about much of the bitcoin setup is it's in everyone's best interests to support the network. It just seems like a flaw to have block rewards go to the ONE MINER who happened to get lucky enough to find the correct block, despite all the other work others could have done, it's not a distributed rewards system to match the distributed network (I understand that it's not luck, and that in the limit this proof of work system smooths everything out - but we don't live in that limit, not even close. We live in the here and now where people with 10Gh/s of power could get a block in 25 years! Meanwhile all their proof-of-work they've donated to Bitcoin goes to waste, it's not recorded anywhere. People live and die in 25 years).

Pooling solves that problem, but creates another more serious problem of individual pools having crazy amounts of network power - we shouldn't be shooting for "not 51%", we should be shooting for "not 2%". There is incentive to pool though - otherwise no payout. So p2pool is good, because it mitigates the problem of crazy big centralized pools being built up, BUT THERE IS NO INCENTIVE to use p2pool over normal pooling (other than "well it's good for the network", which we can't and shouldn't count on people to think about or follow). In fact, it's arguably HARDER to use p2pool (as of right now, not because it's actually harder, but because there is less entry-level documentation about it), so there is incentive not to use it.

***************************************
Full warning - Wall of text ahead. Most of my posts are wall of texts anyway. But this idea for a protocol is getting pretty refined in my head. I think it's shaping up pretty well.
***************************************

A system that rewards you for mining benevolently and intelligently (regarding stabilizing the network) is what we need. The protocol itself is the one thing everyone using the network agrees upon, so if everyone agrees upon a protocol for benevolent mining, then it will be enforced by the network. All I'm trying to do is come up with a mining protocol which allows any entry-level miner to turn a profit without having to join a pool (and hopefully a larger profit than they would get in a pool) - while I'm at it I think we can make the network more secure against 51% attacks.

Here's the idea so far (it's mutated even more today - keeps changing as I think about it).
Until now I've been thinking about building blocks smaller and smaller rather than bigger and bigger. Now that I think about it it makes much more sense to go bigger and bigger.

So, first block size (the one that gathers transactions), has a quick target rate, something like once every 30 seconds - but more importantly it has a target one-block per week-miner hash-cost rate (for instance, a miner who can spend 1BTC on his rig would get 1 block every week from that rig) - which can be made easier to enforce as just a pure hash-rate - say a miner who has 10Gh/s (achievable with 1BTC as of now), would get one block at this level every week on average. This is like the normal block-chain today (except the faster target rate), mined in the same way. People get block-rewards and transaction fees for these blocks, small block rewards, maybe 1/20th the current size? Who knows, it doesn't actually matter in the end, since eventually the currency would stabilize relative to other things anyway. Maybe in this new crypto-currency these could be 1coin rewards.

Then you have miners who gather those blocks (calling them sub-blocks now), and does a similar thing that the current miners do with transactions - they group the sub-blocks together in the order they were generated, hash them, hash the previous super-block at his mining level, and try for a difficulty. These are super-blocks, and are harder to find than the sub-blocks. Same algo can be used to generate these blocks, but the target difficulty could be some set number higher than the sub-difficulty, but even better it could be that the difficulty of the block you generate has to be greater than or equal to the sum of the difficulties of the sub-blocks you're including in your super-block (or some fraction there-of). This allows for really organic determination of the difficulty necessary to group a bunch of sub-blocks. Block reward is calculated similarly (as a sum of sub-rewards). A study of how to sum difficulties is necessary.

Sub-blocks have to be mined/included in the order that they were generated - just like transactions have to be grouped in order (to prevent double spending). So you have miners who are mining the main chain, which is collecting and verifying transactions, and you also have miners who are mining off of that chain, collecting the blocks up and generating super-blocks. The two chains are developed parallel to each other, and most importantly if the super-chain stops being generated you can still generate the sub-chain. You can't generate a super-chain if the sub-chain stops though. Every time someone mines a super-block, it adds that total difficulty to the main-chain (the transaction chain), meaning if you wanted to undo transactions from earlier you would have to recalculate previous sub-blocks and whatever super-blocks have been generated on top of those blocks - and you would have to do it sequentially (limiting you more), because you can't generate super-blocks before you generate the sub-blocks - making it very difficult to 51% attack a network. This is enforced by the network protocol - if a miner sees a chain + super-chains that has heavier difficulty he switches to that chain instead.

This seems like a really good way to make it so that everyone can contribute to the network security (by adding total difficulty to the hash-chain) including the very small miners who just want to run a single ASIC or two. As the super-chain starts to catch up or even try to outrun the sub-chain, people will move to another level super-chain, which will throw even more difficulty weight onto the already existing block-chain, making it even harder to fork.

With this scheme though, sub-block inclusion fees would be near-impossible to enforce. If the miner decided not to grab a sub-block because it didn't have an inclusion fee then that super-chain would be stopped altogether right there. There is an aging algorithm already in place for transactions though, yes? As a transaction gets older its priority is bumped up so it will be included without a fee even? (*not sure about that*) Maybe a similar thing could be done with sub-block-inclusion fees. But really, the inclusion-fee would turn into a "you better put a small incentive for me to mine a level higher than you or I'll absolutely decimate your ability to find blocks" fee, as the super-miner could easily threaten to descend to the sub-miners level and drive difficulty up if they don't feel fairly compensated for moving their mining power out of the smaller pond.

There is no end to how far up the hierarchy we allow people to mine, eventually we could have it be so that transactions are bundled in hundreds of levels of blocks. The important thing is that the smallest chain, the first chain, is where transactions happen. Nothing that happens in the other chains can stop this chain from being generated, so you can't DoS the entire network by stopping generation somewhere higher up in the chain-hierarchy. An interesting idea is to try to utilize the super-chains as they are generated to make it so you don't have to store the contained sub-blocks in them anymore or something. Pruning the block-chain based on proof-of-work - making it so the super-miners provide a two-fold service - 1) effectively doubling the total hashing power needed to attack the network with every block they find and 2) pruning the block-chain by finding super-blocks.

This allows mining to scale with technology in a way that doesn't encourage pool mining. It also allows smaller miners to contribute to the total safety of the network (if I'm mining at 1Gh/s solo, and I never find a block, my proof-of-work currently does nothing to contribute to the safety of the network). One issue is the payout - you can halve the block-reward at the transaction block level every so often (like we do), then the reward at the super-block level will also halve every so ofter, etc...but because infinite levels of super-blocks can be created you could generate many more units than originally anticipated, in fact infinite. It's an interesting problem.....but honestly who knows if a deflationary currency is what's best? At least here the inflation is controlled by how fast technology marches forward - just like with gold the inflation is controlled by how fast mining advances. And in todays/futures society, how fast technology grows is probably a good estimate/approximation to how fast commerce is growing, meaning that prices should stay relatively stable.

Interesting thing, this could be implemented in bitcoin. It doesn't invalidate any of the previous blocks/transactions. We would just add functionality to the protocol. So no one would have to switch over to a new crypto-coin. Such a sweeping change would probably upset some people though.

Let me know what you think....I was looking for a project anyway

Miners often have relatively low complexity computers managing their miners.  A bitcoind node requires around 20GB of hard drive space (and growing)


Right, it is 2 separate things. 

Overloading the official reference client with even more functionality is the wrong way to go.

Ok, I'm back.

To the first comment. Yes you could have pool mining at any level of sub-blocks, but because people will naturally find a place to "fit in" their hashing power, any attempt to do so would be spotted as an attack on the network much more easily. The miners working on the main-chain aren't inherently pooling though, they are just working on a similar problem as each other.

As for p2pool - yes I've looked at it. I think this system has several advantages:
a) Built into the Bitcoin protocol. No need to rely on good-Samaritans donating to the miners and such in the long run. I don't think we can count on that continuing to happen forever, and even if we think we can we shouldn't. A lot of Bitcoins success as a decentralized protocol is in providing incentive (built directly in) to support the protocol (mining rewards, etc...), so we should build in protective measures too. If we can utilize game theory to make people support the network, we can utilize it to make people want to decentralize too. People making the protocol understand the importance of making it non-abusable, but people looking to just make money by mining or whatever means and are willing to throw money at it to make it happen don't understand that importance (especially in the future when people look at Bitcoin like cars - it does what it does and they don't look under the hood).
b) Weight of smaller problems solved is added to the block chain (as the sub-blocks). In pool mining, the share-chain generated has weight for the people operating in the share-mining operation, but you still can only add weight to the block chain in big main-block-sized chunks. Here you can have weight added to the chain much quicker by allowing smaller problems being solved to directly contribute to the main-chain. Then if someone tries to fork the block-chain by ONLY mining the main chain, they'll have a much harder time of it because they'll have to generate main-chain blocks faster enough to outweigh the original main chain + the sub-blocks off the main-chain.

After a bit more discussion though, these are the conclusions I've drawn:
1) This can't be done with Bitcoin, it would have to be an alt-coin. Bitcoin mainchain supports collecting transactions, not really collecting sub-blocks as much. Maybe it can be done, but would require a well-coordinated shift in the protocol.
2) It's hard to give the miners incentive to not refuse the sub-chain transactions - the argument I made before was that they would want the heaviest main-chain so that they could be that much closer to guaranteeing their block rewards wouldn't be reversed. It's tough to tell if that's enough incentive. Instead, I thought up this: whatever level of sub-block is the last sub-block is the one that collects transactions and validates them. Then those sub-blocks are the ones that also collect transaction fees. The super-blocks above them would demand a "sub-block-inclusion" fee - whether this is based on a hard number or a percentage - who knows. This would happen all the way up to the main-chain. Additionally, instead of awarding the main-chain blocks the block reward, you split it among the smallest sub-blocks. Super-blocks won't get money unless they include sub-blocks, sub-blocks won't get included (therefore won't get money when it comes reward time - which would be every main-block find) if they don't include an inclusion fee.

Problems I see now:
DoS attack: Someone with a ton of hashing power and ill-will towards Bitcoin (or whatever alt this is), takes their hashing power to the lowest level and makes tons of blocks, huge amounts of them - raising the difficulty to mine at that level - then doesn't include any transactions in his blocks. So he's still getting his block-rewards and can still pay up the to the super-blocks to get his validated and included. Maybe this can be avoided by simply saying "you get your share of the block reward based on the percentage of the last x transactions you included" or something. Then he won't have the block reward necessary to put his transactions on the block chain - but could supply his own money by making transactions back and forth between two of his accounts to generate the block-reward distribution and the transaction fee distribution - this would get expensive though and isn't sustainable. And it would help to distribute his wealth among the super-block miners in the long-run.

You should take a look at p2pool which provides distributed mining while giving consistency similar to pooled mining as compared to solo mining.

seems to me that, in jour vision, "main" miners will act as a pool.
But wait to someone less newbie than me for a smarter and more complete response 

*** If you're just now seeing this thread, I apologize for the walls of text.
These are my ideas for improving the mining scheme of Bitcoin. Every post I put up it changes pretty drastically, so if you want to read through the whole evolution of the idea then start here. If you just want to read the idea as it is now read my last (currently 3rd of mine) post in the thread****

Here's the basic idea:

Right now it's tough for small miners to not join a pool simply because they don't have hashing power and will never dream of finding a block. If we had a more granular way of awarding block rewards built in, they wouldn't need to pool.

So, instead of making them mine in pools where they solve easier problems, let them just solve easier problems directly off the current blockchain.

Here's how it goes in my head at the moment:
1) Major miner finds a block on the main chain
2) Minor miners start mining an easier problem (with a different hashing algo to make it so you have to pick the easier or the harder problem, but not both) while the major miners search for the next big block
3) A sub-block is found (tuned to be faster than the main-blocks with a sub-difficulty) and the sub-chain from the previous main-block is started - all sub-blocks for a super-block have to be on the same chain (so all minor minors start working on same sub-chain and it's easier for a super-block-miner to sweep the sub-chain into the next super-block) - the heaviest summed difficulty sub-chain is what will be included by the major minors in the next main-block
4) After a sub-block is found, you could have ANOTHER level of granularity added, with an even easier problem (still another different algo) built off the first sub-block, obviously this idea continues as far as is needed until the smallest miner can get payouts without joining a large pool
5) Once a block is found on the super-chain, the sub-chain from the previous super-block (and all sub-sub-blocks off that one) is swept into that block
6) Only when blocks are found on the main chain are payouts calculated and handed out


Now we have a limited tree structure going on (limited so that it's easier to sweep in sub-chains), that can go to any depth. The summed difficulty of the entire tree is increased, making it more necessary to have > 51% of the hashing power in every algorithm in order to fork the main chain, or to have much greater than 51% of the hashing power of just the main chain to fork the chain. Payouts go to the main-block that was found, and all the sub-blocks built off the previous main blocks.

Some issues that came up:
1) How do you give main-miners incentive to include sub-chains when that means they have to split the payout? Simple, if they don't, another miner that will can find another main block even after the first miner who will include all the sub-chains - the added summed difficulty of the second miners main-block will beat out the difficulty of the original miners main-block.
2) How do you verify that blocks you're creating on sub-chains are valid? Well, they're built on an existing block, not the about-to-be-created block. It's up to them to broadcast the sub-chains/sub-blocks enough for the super-miners to hear about it and include it in the super-block. It's in the best interest of the super-miners to include the sub-chains (or at least we can tune it to be so by adjusting the relative difficulties and payouts for mining a sub/super block). This seems to be the biggest leap to me, but I think it can be tuned so it's true. User Jorge7777 on Reddit kept mentioning that we have these middle blocks that can't be fully verified as included in the chain - making only one sub-chain per super block makes it easier to sweep them in, and here we're making the same tradeoff that Bitcoin inherently does with transactions just with sub-blocks - we're trading "instantly verified but potentially invalid" for "verified a bit in the future but agreed upon to be valid".
3) How do you deal with the insane block-chain bloat that will occur? Well, how insane will it be? The sub-blocks that are created will be transaction-less, just proof of work with the address of the miner who found it or something. And we can tune the sub-difficulties so that they are generated every minute or so - surely one sub-chain running a scrypt algo wouldn't add that much bloat, it just needs block headers and a little info.
4) If we include transactions in the first sub-block, it could be that the main-block miner doesn't hear about a whole sub-block of transactions and doesn't include them in his main-block, we'd have to have a way of dealing with these "dangling" transactions as you can't push a sub-block from one main-block to the next - it won't be valid.

Some ideas:
1) Put the transaction bundling in the first sub-block off the main chain. Then people can get their "verification" earlier. It is a bit less secure of a verification, but for most transactions it's probably good enough. If you want to make a big transaction, you wait for a new main-block to be created, if you are just buying a pack of smokes or something, you only need to wait for a sub-block verification. Additionally, if your transaction is included in a sub-block, it means that the main-block you are basing that sub-block on already exists. This means if someone forked the previous main block they would have to generate both a main block and a sub-block and whatever sub-sub-blocks also exist (which could be different algos -> need different hardwares for each) which are valid to reverse your transaction, because otherwise they would have less weight in their version of the blockchain. Additionally, after the next main-block is put on top of the current sub-block where you transaction sits, you may be able to (though we probably need to do some math to say this) say with MORE security that your transaction has been validated, because you have all the additional weight of the sub-chains to back up the new main-block.
2) If transaction bundling is off the main chain, then the main chain is only for paying out rewards to miners and such.

Here's an ascii illustration of the time-ordering:
O - main-chain block, here rewards are calculated and spread out to the main-block finder and sub-block finders from the previous main block
v - first sub-block, here transactions are swept in? (potentially), but otherwise just proof of work for smaller miners
c - second sub-block, just proof of work for even smaller miners
(O1) - (v11) - (v12) - (c111) - (c112) - (c113) - (v13) - (c121) - (c122) - (v13) - (O2) - (v21) - (c211) - (c212) - (v22) - (c221) - (O3) ..... etc...

Anyway, poke it all full of holes. Figured it would be good to start a discussion though
Edit: Gotta head out for a bit, be back in a few hours - if anyone reads this.

Original concept/discussion thread here: (it has mutated a bit since then): http://www.reddit.com/r/Bitcoin/comments/1v9gp5/removing_the_incentive_to_mine_in_pools/