The wise men or magi did not arrive in Bethlehem until approximately 2 years after Jesus was born. They were never at the manger at all The Bible never states the number of wise men although there were three gifts given.

Matthew 2:11 And when they had come into the house, they saw the young Child with Mary His mother, and fell down and worshiped Him. And when they had opened their treasures, they presented gifts to Him: gold, frankincense, and myrrh.

There are several traditions that not only say there were 3 wise men, but even name them. None of this misinformation is in the Bible. Besides the above verse, we are given an estimate of their arrival by the age of the children King Herod kills when the wise men flee.

Matthew 2:16 Then Herod, when he saw that he was deceived by the wise men, was exceedingly angry; and he sent forth and put to death all the male children who were in Bethlehem and in all its districts, from two years old and under, according to the time which he had determined from the wise men.

Since Jesus is called a child, not an infant (there is a vast difference in the Greek language in which the Bible was written) and He was at a house where the family lived, the correct answer is there were no wise men at the manger at all but they came about 2 years later.

More Info: en.m.wikipedia.org