Now, I'm on a network of untrusted nodes. We're all working to factor a number. I find a solution, and then declare "The solution is X*Y=Z here's my public key, now give me my 50 BTC!" What is stopping the first node that receives this message from simply claiming the solution is his own, swapping his public key into the message, and then broadcasting it? Nodes can try to compare timestamps, but network latency is high variance. How does he know that the other guy isn't the true "winner" but his network packets had more hops?
The nice thing about the current hashing solution is that the coinbase tx to yourself is part of your hashed solution. If someone tries to swap in their own public key, they break the hash and it becomes invalid. The only way to "steal" it is just to do the work themself.
So my question is, given a problem that is otherwise ideal, but does not have ownership somehow tied into the proof-of-work, is there a way to avoid the problem I described above? If not, I think this severely limits the number of candidate problems.
The whole purpose of mining is to secure the transactions. Any algorithm that didn't secure the transactions, including the coinbase transaction, would be useless.
Factoring a number could be used for proof of work, but the number would have to be based on the previous block's hash and the hash of the transaction tree. So the point is, you could use factoring for your proof of work, but you couldn't choose what numbers you had to factor -- the transactions and the chain would have to do that.
In that case, the difficulty would be how big the smallest factor had to be -- otherwise, you could just work to get an even number and divide it by 2. In fact, you could drop in factoring as a replacement PoW algorithm. You'd hash just the same as you do now, but instead of looking for a low hash output, you'd look for a hash output you could factor but such that the smaller factor exceeded the difficulty. Verifying the proof of work would include making sure that neither factor was smaller than the difficulty and that the smaller factor was prime. (Otherwise, it's too easy.)