The poet who wrote the phrase, "God's in his heaven — All's right with the world"! was Robert Browning. This particular quote is found in 'Pippa Passes' and is now a well known verse of literary drama. It was published in 1841 in the first volume of Browning's "Bells and Pomegranates" series.

Some literary scholars and/or critics have interpreted the quote to mean or imply that, at least in Evangelion's context, a god who interferes with the realm of man causes problems and sets the world off kilter. So, if a God remains in his' heaven, it will not cause problems for man's world. In other words, when God is in heaven, there's no problem on earth that he has to intervene or fix.

More Info: en.wikipedia.org