If all of your code were available as a downloadable client as an .html file, or a series of .html files (that could be saved to someone's hard drive, or burned to the LinuxCoin CD), and used ajax calls or something similar to query for transactions on the blockchain and submit signed transactions via your website, it could be deemed a whole lot more secure. The risk isn't in interacting with your website, it's the theoretical risk that you or an attacker could serve a different page once in a while. A third party can't really rule that out, because they can only validate the page you sent them.
Yes, this would interact with the internet, but somebody else could validate that it had no capability of "leaving the sandbox" at the request of the server, and then it would be good.