We never looked down on Paul Butler but I also don't see he has that strong punch to take down Inoue if he was able to connect some good shots.
Inoue is also used to receiving a different levels of punches and if we talk about Butler's punch, Inoue can surely handle it.
It's also hard to connect some blows on Inoue as this guy is very quick and versatile. On the other hand, Butler is not even known for some special skills related to speed, quickness, strong punches, absorbing heavy body blows, and so on.
Oftentimes, heavy punchers have high resistance to the same amount of punching power. So it is somehow scientific to say that Butler's punching power will almost not affect Inoue because Inoue having the heavier hitter possibly possessed high durability and resistance against weaker punching power opponents. We can testify to it in his fight against Donaire. Even though Inoue's face is somehow ruined in their first fight, Inoue has never been knockdown by the hard-hitting Donaire. The reason behind this is, Inoue is a harder puncher thus his durability would be better.
Let's see, just like what you said only few weeks and we will see them fighting inside the same ring, winner take all!
Yes, Inoue, being the heavier hitter between the two, and Inoue's durability is capable of absorbing Butler's punches. So it will be hard for Butler to knock down Inoue unless he is trained enough to acquire precision in punching where he can hit anyone on the weakest part of the face or of the body while the boxer isn't prepared to receive the blow.