But the thing is that we don't know how Inoue is when face unfamiliar situations, maybe he had feel this before when he fights in the US and that's why he doesn't want to experience it again.
So far the toughest challenge Inoue experienced was his first fight against the legendary Nonito Donaire and it was also in Japan. Inoue passed that test with flying colors. So it shows he can adjust and handle very tough situations. And he just became better after that. But I expect his body to start diminishing slowly as he age.
All of Inoue's fights outside Japan were dominant victories and ended in knockouts. I think Inoue is willing to travel everywhere as shown in the past. But Bob Arum and ESPN has to give higher purse if they want Inoue to fight outside Japan because the country is also very rich and is capable of bringing big events at home.