For weakness of Inoue to unfold, he just needs to fight a better boxer than he is. He may be dominated the bantamweight division, but I do not think that he will do the same in the super bantamweight. Super Bantamweight is a different division and there are several known boxers that can possibly defeat Inoue in this division. The boxer in this division have heavier punches and more resistance to blows or punches, and since Inoue is taking a shortcut, he probably doesn't know the extent of the punching power of this division.
Again with this shortcut thing
Really? Why do people keep on insisting that Inoue did a much easier path that directs him to the championship fight when in-fact he earned it because he managed to unify all the belts at 118 and then there's this WBO who gave him a super-champion title which mandates that he can fight directly with Fulton and not Murodjon.
Did you honestly expect that Inoue and the other guys who did a successful run in their previous division to start at the bottom? That's a perk, not a shortcut. They earned it.