Pages:
Author

Topic: Pooled/Remote Mining - Open Source - Updated 2010-12-24 - page 8. (Read 59069 times)

LZ
legendary
Activity: 1722
Merit: 1072
P2P Cryptocurrency
Thank you, puddinpop! Someone succeeded to generate a block?
member
Activity: 103
Merit: 17
Toss up a couple of windows binaries and I'll host a server if anyone would like to join to test it out.

The first post has been updated with binaries.  Because of the issue I was having with the hash generation, I had to turn optimizations off for the bitcoin and bitcoind binaries.

I've added the best hash to the results when the client sends back the metahash, and as I thought, the results are highly erratic.  Averaging over 10 minutes doesn't reduce the error to an acceptable level.  Averaging over longer periods would either mean clients must stay connected longer, or the server must save the client's work between client connections, and I'm not prepared to write all the code involved in tracking and saving client state.
So... have clients send back their best N hashes (and the average should be N times better).


Now you're getting closer to just sending every hash the client generates.  I thought the argument was against that.
legendary
Activity: 1652
Merit: 2301
Chief Scientist
I've added the best hash to the results when the client sends back the metahash, and as I thought, the results are highly erratic.  Averaging over 10 minutes doesn't reduce the error to an acceptable level.  Averaging over longer periods would either mean clients must stay connected longer, or the server must save the client's work between client connections, and I'm not prepared to write all the code involved in tracking and saving client state.
So... have clients send back their best N hashes (and the average should be N times better).

RE: detecting server cheating:  Over a very long period of time, clients should be able to figure out approximately how many hash/sec the server's network is generating.  So they should be able to detect blatant cheating.  I should've taken statistics in college, seems like it'd be an interesting problem to work out the chances that a server is lying based on how many blocks it has generated over the last week...
member
Activity: 103
Merit: 17
The original post has been updated with the latest source.

I've added the best hash to the results when the client sends back the metahash, and as I thought, the results are highly erratic.  Averaging over 10 minutes doesn't reduce the error to an acceptable level.  Averaging over longer periods would either mean clients must stay connected longer, or the server must save the client's work between client connections, and I'm not prepared to write all the code involved in tracking and saving client state.

I think a dual approach would work best.  Use the metahash for accurate hash rate and complete verifiability of hashes reported by a suspicious client, and use the best hash to quickly check a client's work.  This will allow both scalability of verification and accuracy of the hash rate.

I've also added the full block to the message sent to the client when sending work for the client to do.  This way the client can verify the block has an output to their address.

Also, I've added the ability to ban clients by adding them to a file named banned.txt.

As far as verifying that the server isn't lying, I don't think that's possible with a multi client, single server architecture.  The server can lie about anything to any client, and each client will be none the wiser.  The only way around that is if each client has perfect knowledge of the other clients connected to the server.  One way to do this is setting up something similar to a torrent tracker, where the server and each client register with the tracker, and the clients can use the tracker to verify the work they are doing and verify the server is also accurately reporting the work other clients are doing.
sr. member
Activity: 416
Merit: 277
Thanks to ribuck for putting me right about client cheating. I have deleted the incorrect section from my post.

I think it's possible to create a coinbase transaction with one generation input and several outputs.

Magic! I've just looked at the code and it seems to agree. How did you know this esoteric fact?

In this case the scheme runs as follows:
The server publishes its bitcoin address. Any clients wishing to participate start work hashing a block with the coinbase partly crediting themselves and partly crediting the server - possibly half and half?. After some period of time or after a transaction is received, whichever comes first the clients send their best hash signed by the public key they want credited to the server which calculates their share of the hashing effort using the quality of their hash. The server then publishes all the signed best hashes and a new block to be hashed incorporating all the new received transactions (if any) and the coinbase split among the the clients according to their share of the work. The server possibly takes a cut. The clients verify all the published signed hashes and verify that the coinbase output credits them fairly. A winning client can distribute the winning hash block to the Bitcoin network itself.


The server has to publish the signed hashes and or the new block in a fashion so that everyone can see that everyone else sees the same thing otherwise the following server fraud is possible.
Suppose there are 2 normal clients A and B and one server owned client X. They all hash at the same speed so obviously the coinbase should be split into equal thirds. However the server sends signed hashes and new block targets to A, omitting B's signed hashes and similarly pretends to B that A doesn't exist. No matter who generates the winning hash, X walks off with half the coinbase instead of the third he deserves.

ByteCoin
administrator
Activity: 5222
Merit: 13032
I believe standard Bitcoin software rejects blocks attempting to spend coinbase that hasn't matured. If this could be changed to allowing the spend but making those transactions 0/unconfirmed until the coinbase matures then the whole problem goes away and we're all very happy.

I think it's possible to create a coinbase transaction with one generation input and several outputs.
donator
Activity: 826
Merit: 1060
The client can cheat if it generates the winning hash in a naive implementation by failing to transmit the winning hash to the server and just transmitting it directly like a normal Bitcoin client and taking the 50BTC for itself.
No, I don't think so.

The block being hashed includes the 50BTC transaction awarded to the generator. So a client must decide (before looking for hashes) whether it is hashing a block that will pay itself, or a block that will pay the server. It can't find a hash first, and then decide who it is for.

So there's no risk of cheating. If the client is looking for hashes that will pay itself, it's not participating in the shared mining and won't have any low-difficulty results to send back to the server either.
sr. member
Activity: 416
Merit: 277
I don't think that trusted groups are needed if the probabalistic scheme is used.

Agreed.

{Deleted incorrect stuff about clients cheating. Thanks to ribuck for putting me right.}

Server cheating schemes and their prevention:

The server can say that a lot of of not quite good enough hashes were submitted by clients owned by the server operator. To prevent this, the server must publish all the hashes which were used in the work calculation.

The server can take the proof of work hashes of one or more of the genuine clients and attribute them to a client owned by the server operator. To prevent this the clients sign their hashes with the public key of the address which they want to be credited with their portion of the pot. The hashes and signatures are all published by the server.

The server can omit to publish the signed hashes of a subset of the clients which are not controlled by him and which did not find the winning hash. This enhances his share of the pot. To prevent this, the clients periodically send signed hashes to the server and the server signs a recipt and returns it to the client. The client checks the recipt is valid before continuing work. If the server omits to publish a subset of the signed hashes then the client has proof that the server recieved them and can publish this to discredit the server. If the server signature is invalid then the client stops work and the amount of computation wasted is limited.

I imagine that gavinandresen's "best hash within a time limit" will be a low bandwidth and predictable method of measuring hash speed in this scheme.

Finally, the server can fail to distribute the 50BTC to the clients. To prevent this I thought the cunning idea was to have as the transactions in the new bock include the distribution of the newly minted 50BTC coinbase to the participants. This would have also prevented all the above attacks in one step. The clients would stop working on hashing the block if the block transactions did not divide the proceeds in a fashion the client thought was fair.
I believe standard Bitcoin software rejects blocks attempting to spend coinbase that hasn't matured. If this could be changed to allowing the spend but making those transactions 0/unconfirmed until the coinbase matures then the whole problem goes away and we're all very happy.

If the server gives the clients credit transactions which don't involve the prospectively minted coinbase but instead some other source of money then the clients can just submit those transactions to the network and get the credit before doing all the work thereby defrauding the server. Otherwise the client just has to take the server's word for it that the merkle root they're using includes transactions crediting them appropriately. There's no way of proving it. This is a problem.

There might be a solution involving nifty scripting.

ByteCoin
donator
Activity: 826
Merit: 1060
One of the most important things about the pooled mining will be participating in trusted groups.
I don't think that trusted groups are needed if the probabalistic scheme is used.

Each client, modified or not, knows the difficulty of the hashes it has produced and can calculate its expected long-term earnings, although of course there will be short-term fluctuations. A dishonest server will get caught out (and quite soon if several of the clients choose to share their statistics). A dishonest client doesn't get more than its real share of the earnings anyway.
sr. member
Activity: 416
Merit: 277
You seem to value the accuracy provided by your "metahash" scheme while apparently tolerating a significant amount of fraud in real-use scenarios. I believe that the probablistic schemes advocated by everyone else are sufficiently accurate, quite possibly fraud proof and result in essentially zero server load. If we can agree some plausible real world parameters beforehand and I crank through the maths I believe I can prove that the random errors produced by the probablistic rate measurement are considerably smaller than the distortions caused by inevietable fraud. Would you change your mind if I proved this?

Erroneous results would be a factor for any method of client verification.  The solution is to allow a certain amount of error.

This strikes me as tricky without using lots of bandwidth. Are you saying you do this at the moment or you plan to do it? How does it work?

There remain a couple of issues from your previous posts which I queried but which remain unaddressed.

The second issue is that each miner would now need to check the calculated hash against the target after every hash.  This is another operation the client must perform and it will slow down generation.

This was in response to a suggestion from tatcm for measuring the hash rate by counting low difficulty hashes. You seem to be suggesting that one drawback of tatcm's suggestion is that you need to compare the hash to the target. You imply that your method avoids this comparison and is superior on that basis. You furthermore imply that the comparison operation has a significant computational cost compared to the hash operation.

In order for you not to be wrong you need to show
1) how your method avoids comparing the hash against the target.
2) that the comparison is not of an insignificant cost compared to the hash operation.

I assert that you will be unable to do so and that therefore you should withdraw your "second issue" with tatcm's suggestion.

What's to stop a client from lying?  They simply generate a few hashes just to send and tell the server, "These are the best hashes I came up with, honest."

You seem to be under the impression that the rate calculation can be tricked by sending a few presumably low quality hashes which are the result of an insignificant computational effort. I assert that this is not possible. You need to show why you think the client can lie about its computational effort in this fashion.

Please let me know whether you will only be convinced by an explicit numerical demonstration of the superiority of the probablistic hash rate meaurement scheme and also your response to the two outstanding issues from your previous posts which remain unanswered which are outlined above. Also an explanation of how your metahash scheme can efficiently tolerate computational errors would be welcome.

ByteCoin
member
Activity: 103
Merit: 17
Your "metahash" method of measuring work, though more accurate than the probablisitic methods of gavinandresen and tcatm, is a major weakness of your method. In order to check a portion of a client's work, you have to duplicate it. This will not scale well to even tens of clients. An attacker submitting bad metahashes to earn coins without really doing the work will just shut down when found out and start a new client on a new IP.
You would have to exclude new clients until a certain proportion of their results have been checked and then it becomes a probability game where the attackers only falsify perhaps an increasing proportion of their results.

Obviously you don't compute every single metahash sent by the clients.  You hash them periodically, and when an erroneous one is found, or you have a suspicious client, you check more of them.

The second issue is that each miner would now need to check the calculated hash against the target after every hash.  This is another operation the client must perform and it will slow down generation.

Eh? This happens anyway! How else do you tell if you've got a winning hash?

You compare 1 byte of the hash and only if that byte is 0 do you fully check the hash.

What's to stop a client from lying?  They simply generate a few hashes just to send and tell the server, "These are the best hashes I came up with, honest."
Nothing. But then the inferred hash rate is very low. To clarify - the hash rate is calculated from the quality and/or number of hashes and NOTHING ELSE. The client doesn't say that it's done a certain amount of work - just the hashes matter.

That's exactly how it is now.

The server creates the block, which obviously doesn't include transactions utilizing the coins that are generated by that same block, mainly because there are no transactions derived from the block since no one knows about the block yet except the server. 
Ah. I thought you and I were thinking along the same lines of an elegant payment method that requires no trust and cannot be scammed which is unfortunately forbidden by a fairly inessential and inelegant Bitcoin rule.
You need to go into more detail about how payment works. The solutions I can think of require the client to trust the server and/or can be scammed.

ByteCoin

The server is capable of sending the block with all transactions to the client for verification.  It does not do so now, but the code would not be too difficult to add.  There is actually a comment in the code to add this feature later.  This way the client can verify they will get their share.  If you would like more detail, the code is the best source of detail you can get.

I bet you'd get a good approximation of hash rate if clients submitted their best (highest difficulty) hash every N minutes.  Over a period of a few hours the average of all of those best hashes should be proportional to the client's hash rate (unless a client were somehow repeatedly very lucky or unlucky, but that would be extremely unlikely).


What's to stop a client from lying?  They simply generate a few hashes just to send and tell the server, "These are the best hashes I came up with, honest."  In reality the client is spending the rest of the time trying to generate his own block.  There is absolutely no way to verify that the client is not lying this way.  With the metahash approach, you can verify every individual hash a client has reported solving.  If they are lying, even about 1 of those hashes, you will know because the metahash doesn't match.


Maybe you don't get what he's saying. Let's look at a target hash say 10 characters, the more 0's at the front the better.

A bitcoin block needs 9 zeros to get a 50 coin block.

You ask for their best result every 10 minutes. If they work for 1 minute on an average machine, they give you a result with 2 zeros. If they work for the whole 10 minutes, they give you a result with 4 zeros. All the results that they are looking for, would be based on the current hash, so there is NO wasted work. They just store their best in the client for the current hash, and send it when the server requests it.

Switch back to the way bitcoin works. Difficulty factor of 1398. Someone trying to cheat sends you back a has worth a difficulty of 1. A real client sends you back a difficulty of 7. Someone on a GPU machine sends you back a difficulty of 190.

The better way to do it, would be to request their best POW for each block. You could take the time each block took, the hash that they were able to find (which verify very quickly on the server side) and come up with a formula of what share they would get. The formula may include smoothing out the high peaks and lows.

It may be the most complicated method... but it sounds so far to me, like the most accurate way to get both a relatively honest answer, and have it be relatively hack proof. All the while generating an appropriate potential block.

How do they hack finding the best hash? If they are finding a stronger hash, they very well could be finding an actual block. Nothing is wasted in this scenario.


I understand what this suggestion is, but I don't see it as a reliable method.  It assumes you will generate a specific difficulty at a specific hash rate in a set amount of time.  We all know this doesn't happen.  How do you factor in unlucky clients?  What about the lucky ones?  The best you can do is take an average over an unacceptably long period of time to smooth out the hash rate.  I think this method is too imperfect, and has potential to penalize or over reward clients too easily.  I certainly wouldn't want to be the client penalized because I couldn't generate a good hash in a given amount of time.
sr. member
Activity: 416
Merit: 277
Your "metahash" method of measuring work, though more accurate than the probablisitic methods of gavinandresen and tcatm, is a major weakness of your method. In order to check a portion of a client's work, you have to duplicate it. This will not scale well to even tens of clients. An attacker submitting bad metahashes to earn coins without really doing the work will just shut down when found out and start a new client on a new IP.
You would have to exclude new clients until a certain proportion of their results have been checked and then it becomes a probability game where the attackers only falsify perhaps an increasing proportion of their results.

More seriously, you exclude clients (and servers) which are not 100% reliable. I know from Mersenne prime testing that some computers occasionally produce bad results. You can run normal software for years without noticing an error rate of one mistake in every 2^40 ops but your metahash would be very sensitive to such errors. This would result in people running genuine clients on slightly imperfect machines getting annoyed. Note that these imperfect machines are fine for normal hash generation or the probablistic hash rate calculation.

You seem to think that the probablistic hash rate measurement schemes are insufficiently accurate. You might wish to do some calculations to convince yourself otherwise. There's already lots of unavoidable unfair randomness in the amount of computation required to produce a new block.

The second issue is that each miner would now need to check the calculated hash against the target after every hash.  This is another operation the client must perform and it will slow down generation.

Eh? This happens anyway! How else do you tell if you've got a winning hash?

What's to stop a client from lying?  They simply generate a few hashes just to send and tell the server, "These are the best hashes I came up with, honest."
Nothing. But then the inferred hash rate is very low. To clarify - the hash rate is calculated from the quality and/or number of hashes and NOTHING ELSE. The client doesn't say that it's done a certain amount of work - just the hashes matter.

The server creates the block, which obviously doesn't include transactions utilizing the coins that are generated by that same block, mainly because there are no transactions derived from the block since no one knows about the block yet except the server. 
Ah. I thought you and I were thinking along the same lines of an elegant payment method that requires no trust and cannot be scammed which is unfortunately forbidden by a fairly inessential and inelegant Bitcoin rule.
You need to go into more detail about how payment works. The solutions I can think of require the client to trust the server and/or can be scammed.

ByteCoin
member
Activity: 103
Merit: 17
I was curious and read the source. Your metahash approach isn't using separate block as I first thought and it's actually pretty similar to my idea. Maybe you can get rid of the array and just sum all previous hashes to save memory. That might work better with GPUs with low memory (2^32 tried metahashes are ~17 Gbytes).

It's only 1 byte of the resulting hash, so 4GiB for 2^32 hashes, which a modern card can transfer to main memory faster than the hashes can be computed.  Memory on the card wouldn't factor into it anyway as you just send the results back in small blocks.

I see what you mean now.  Sending hashes for the target block that are a low difficulty.  Well that could work, but I see 2 issues immediately.  The first is that these hashes may or may not be produced in a regular manner, and there is not even a guarantee a client would produce one.  The calculated hash rate based on this metric would have to be taken from a large sample size.  The second issue is that each miner would now need to check the calculated hash against the target after every hash.  This is another operation the client must perform and it will slow down generation.

I bet you'd get a good approximation of hash rate if clients submitted their best (highest difficulty) hash every N minutes.  Over a period of a few hours the average of all of those best hashes should be proportional to the client's hash rate (unless a client were somehow repeatedly very lucky or unlucky, but that would be extremely unlikely).


What's to stop a client from lying?  They simply generate a few hashes just to send and tell the server, "These are the best hashes I came up with, honest."  In reality the client is spending the rest of the time trying to generate his own block.  There is absolutely no way to verify that the client is not lying this way.  With the metahash approach, you can verify every individual hash a client has reported solving.  If they are lying, even about 1 of those hashes, you will know because the metahash doesn't match.

The distribution of bitcoins are actually included in the output of the generated block, so everyone gets their share as soon as the block is generated and no fees are needed.

So just to clarify, you're saying that the coins created as a result of block generation can be spent in the transactions included in that same block. Won't the new block be rejected because the coinbase is not matured? See ConnectInputs. The lack of fees if fees are warranted could also be a problem as ConnectInputs checks for that too.

ByteCoin


The server creates the block, which obviously doesn't include transactions utilizing the coins that are generated by that same block, mainly because there are no transactions derived from the block since no one knows about the block yet except the server. 
newbie
Activity: 7
Merit: 0
This is definitely awesome. I wonder if medium businesses would run something like this on their corporate LAN once bitcoin takes off. Assuming it's worth the electricity costs, having company resources earn income while idle could be very attractive.

Of course, businesses tend to have terrible pcs, but quantity has a quality all of its own. ;-)

One thing is slightly worrying though. If, say, there was a very large global pooled mining server with a majority of CPU power, wouldn't it be able to produce double spending blocks?
sr. member
Activity: 416
Merit: 277
The distribution of bitcoins are actually included in the output of the generated block, so everyone gets their share as soon as the block is generated and no fees are needed.

So just to clarify, you're saying that the coins created as a result of block generation can be spent in the transactions included in that same block. Won't the new block be rejected because the coinbase is not matured? See ConnectInputs. The lack of fees if fees are warranted could also be a problem as ConnectInputs checks for that too.

ByteCoin
legendary
Activity: 1304
Merit: 1015
This is awesome, I hope to become a zombie to a botnet server soon (since I don't think I have the hardware and bandwidth to support a super server).
legendary
Activity: 1652
Merit: 2301
Chief Scientist
I see what you mean now.  Sending hashes for the target block that are a low difficulty.  Well that could work, but I see 2 issues immediately.  The first is that these hashes may or may not be produced in a regular manner, and there is not even a guarantee a client would produce one.  The calculated hash rate based on this metric would have to be taken from a large sample size.  The second issue is that each miner would now need to check the calculated hash against the target after every hash.  This is another operation the client must perform and it will slow down generation.

I bet you'd get a good approximation of hash rate if clients submitted their best (highest difficulty) hash every N minutes.  Over a period of a few hours the average of all of those best hashes should be proportional to the client's hash rate (unless a client were somehow repeatedly very lucky or unlucky, but that would be extremely unlikely).

sr. member
Activity: 337
Merit: 285
I was curious and read the source. Your metahash approach isn't using separate block as I first thought and it's actually pretty similar to my idea. Maybe you can get rid of the array and just sum all previous hashes to save memory. That might work better with GPUs with low memory (2^32 tried metahashes are ~17 Gbytes).
member
Activity: 103
Merit: 17
If you did that, the client would then only have to send low difficulty hashes as fast as possible.  There would be no incentive to hash the actual block trying to be solved.  In order to make sure the client is hashing what it is supposed to and know how fast the client is hashing, the metahashes must be derived from the actual block the client is attempting to solve.  There might be other ways to do this, but I think this is the easiest and most convenient.

By calculating low-difficulty hashes for a block it will eventually calculate a target difficulty hash. Why shouldn't it submit it? It'll make it win coins.

I see what you mean now.  Sending hashes for the target block that are a low difficulty.  Well that could work, but I see 2 issues immediately.  The first is that these hashes may or may not be produced in a regular manner, and there is not even a guarantee a client would produce one.  The calculated hash rate based on this metric would have to be taken from a large sample size.  The second issue is that each miner would now need to check the calculated hash against the target after every hash.  This is another operation the client must perform and it will slow down generation.
sr. member
Activity: 337
Merit: 285
If you did that, the client would then only have to send low difficulty hashes as fast as possible.  There would be no incentive to hash the actual block trying to be solved.  In order to make sure the client is hashing what it is supposed to and know how fast the client is hashing, the metahashes must be derived from the actual block the client is attempting to solve.  There might be other ways to do this, but I think this is the easiest and most convenient.

By calculating low-difficulty hashes for a block it will eventually calculate a target difficulty hash. Why shouldn't it submit it? It'll make it win coins.
Pages:
Jump to: