How would one know if the p2pool network is getting attacked? It seems at times, over 100GH,while being only a 200-300GH network currently, can come jump on the p2pool LTC network and global rejection rates seems to spike. Rejections 30-40%, sometimes even 50% on some of the bigger nodes it seems. It could just be going crazy watching the networks numbers and trying to figure out whats going on. I know it would take someone with 50% or so of our hashrate to preform an attack. They would just submit mass invalid shares purposefully, correct?
It's far less likely to be an intentional attack than it is to be plain performance problems. The litecoin network has seen higher transaction volume recently, and that results in higher p2pool CPU load. If your CPU is not fast enough, it will take several seconds to process a share and issue new work to miners. Any work returned from your miners during that time will be considered DOA (dead) shares. Once you find a share, you have to send it to all of your peers, and they have to send it to all of theirs, etc. While the share is being propagated, there's a chance that someone else will mine a share that will out-propagate yours, which can cause your shares to be orphaned. Furthermore, each time mainnet p2pool changes blocks, the most recent share gets orphaned unless the person who mines the next share is the same as the person who mined that share. (This third effect was removed on jtoomimnet.) These three effects make p2pool biased towards miners with a lot of hashrate, and that bias increases as transaction volume increases. The first two biases are substantially reduced if node operators use CPUs with high single-threaded performance (e.g. Core i7 4790k, which runs at 4.4 GHz) and if they use pypy.
I've put in a lot of work on p2pool's code on jtoomimnet to reduce the CPU load of p2pool and reduce these biases, but I have not yet managed to get jtoomimnet working for Litecoin. It might be ready now, actually -- the most recent issue I had was the port conflict with Bitcoin Cash, which I fixed, but I haven't tested it since. However, the improvements I've made so far do not eliminate the problem, and probably only reduce it by about 50% or so. In the long term, there are a few major design changes I can make to p2pool to nearly completely eliminate this issue, but I have not had enough time to implement one yet.
The most important metric is your node's efficiency rating. If your efficiency is above 100%, you're gaining extra revenue at someone else's expense due to the average miner having more DOA and orphan shares than you. If it's below 100%, you're losing revenue because you have more DOAs and orphans than average. If your efficiency is 90%, for example, then you're effectively donating 10% of your potential revenue and hashrate to other miners who have better node hardware and maybe more hashrate than you. If you're not okay with that, then you should either upgrade your node hardware or (ugh) switch to a centralized pool.
The data you've described suggests that the new large miners are poorly configured (e.g. running on slow CPUs using CPython), and are losing revenue because of that. If that's the case, other miners are probably benefiting from their incompetence.
No, the smartest attack would not be to submit mass invalid shares purposefully. Attacks would probably take the form of submitting valid shares, but ignoring valid shares submitted by users other than the selfish miner. If this is the case, then the node that is selfishly mining will show a low-normal orphan and DOA rate, and everyone else will see normal DOA rates but high orphan rates.