I have been considering the attacks you have described. I still want to move away from RSA, Adaptive chosen-ciphertext attacks (despite being expensive due to Bitmessage's POW requirement) must be more carefully guarded against, and separate keys can be used for encryption and signing after the upgrade as a matter of best-practices. But while the encrypt and decrypt_bigfile function is flawed, I don't believe the flaw you have described could be implemented as an attack against Bitmessage. If an attacker modifies an encrypted message, the receiver will decrypt it but then see that the message signature is invalid: the message signature algorithm is just a signed hash and makes no use of the flawed blocks. The receiver will reject the message as invalid and ignore it.
Right. It seems at a first glance that the signature verification would stop the ACK from being sent.
Still the attack can be executed using a timing attack. It's easy to detect if 100 RSA blocks are being decrypted or just only two. You send another message right after the Bleichenbacher message. If it takes one second to process, then 100 blocks have been decrypted. If it takes 100 msec, then only two blocks have been decrypted.
I'm sure there are still other ways to detect the correct/incorrect PKCS padding and carry the attack.
When a crypto protocol shows many vulnerabilities, then is better to make a pause and re-design carefully from scratch than to start patching.
I recommend you that you first write a technical paper on the crypto protocol, and send it to some researchers for them to analyze. If they say is ok, then go on and implement it. I offer my free advise to read the paper and give it some thought.
Last, there is still another vulnerability in the anonymization logic: If a node X receives a message and immediately sends an ACK then an attacker Z connected to X can detect that X is the receiver of the message.
To deter this attack nodes should continuously send messages at a fixed rate, creating new ones (with no destination) when no new message is received. Other solution is that nodes should be protected with Tor at all times. I will post about this in the bitmessage forum.
Best regards,
Sergio.
PS: I never meant to say that the idea is bad. Go ahead and improve it!