It's all about preparation. I could go into alot of detail, but here are the main points.
Go back to the previous land and drop the following into the portal:
- As many villagers as you can
- As much wood as you can
- As much food as you can
- As many spiritual shields as you can. You can create your own miracle bubbles by taking a miracle from your altar and dropping it on an empty miracle dispenser (which you can build yourself if you like)
Once you get to the next land:
- Drop the majority of the food in the worship site
- Get about 75% of the villagers to worship there.
- Don't worry too much about building up your villager just yet. Just build a few houses built near the temple.
- Using the miracle shields you brought with you, build a wall of small shields between the big gate and your village. Your villagers at the whorship site will power these shields. It's more efficient to use a bunch of small shields rather than one big one as when a fireball hits the big one, it'd draw much more energy from the worhsip site.
As all the fireballs come from the direction of the gates, your wall of shields will stop them hitting your village. If/when one of the shields go down, put up a new one.
All that should buy you enough time to defeat the ogre and stop the fireballs (which can be done by impressing the Japanese village and doing the bell trial.
Your villager population will slowly go down, but you should've bought enouh time to get things done and then start building up your main village