There is a lot of confusion over this, but the attack and solutions were covered a couple of years ago in section 6.2.1 of
Meni Rosenfeld's "Analysis of Bitcoin Pooled Mining Reward Systems". It has been discussed recently in other threads, this thread is simply to have the canonical information in one place on the forum.
Since I know many of you won't hit the link, I've copied an abridged except of the relevant sections here.
AOBPMRS should be considered a bible for miners; you should at least attempt to understand it. If you don't understand something about bitcoin mining it probably has already been explained in this paper.
Also, it should be noted that the most recent example of this attack was apparently not an attack, rather it was probably a software problem - although this should have already been detected during testing on the testnet, so I'm not sure how "unknown" it could have been.
6.2 Block withholding
The analysis so far rested on the assumption that the atomic action that can be carried out
by participants in a pool is to perform the work needed to find a share, and submit it to the
pool unconditionally. However, with the current protocol, miners can determine whether a share they have found is a valid block or not, and refrain from or delay submitting of blocks.
This can be used for two kinds of attack, sabotage and lie in wait.
6.2.1 Sabotage
The simpler attack is sabotage, where the attacker never submits any blocks. This has no
direct benefit for the attacker, only causing harm to the pool operator or participants. Using a reward system without operator risk like PPLNS, each participant (including the attacker) will lose a portion of his rewards equal to the attacker’s portion of the pool’s hashrate.
.......
6.2.3 Proposed solution – Oblivious shares
One workaround for block withholding attacks is the “pop quiz” – occasionally, the pool will provide miners with work for which a solution is known, and flag participants who do not submit it. However, this leaves much to be desired with both precision and recall. A true solution is to modify the Bitcoin protocol to allow oblivious shares – shares which, when found by miners, cannot be identified as a valid block with submitting to the pool for review.
A possible way to do this will be as follows:
• Every block will have 3 additional field associated with it – SecretSeed, ExtraHash and
SecretHash.
• ExtraHash must be the hash of SecretSeed.
• ExtraHash will be a part of the block header and will be one of the fields used in the
calculation of the block hash.
• SecretHash must be the hash of the concatenation of the block hash and SecretSeed.
• For the block to be valid: Instead of requiring that the block hash is less than 2^256/(2^32D), it is required that the block hash is less than 2^256/2^32 and that SecretHash is less than 2^256/D. The pool operator will choose SecretSeed and keep it secret. He will calculate ExtraHash and provide it to miners along with the other fields that go into the block hash. The miners can calculate the block hash and see if it is less than 2^256/2^32 (which happens with probability 2^−32) and in this case submit it as a share. They do not know if this is a valid block because they don’t know SecretSeed and cannot calculate SecretHash. The operator knows SecretSeed, calculates SecretHash, and if it is less than 2^256/D (with probability p) this is a valid block and it is broadcast to the network.