The theory is in contradiction to the belief that the Earth's moon formed some 4.5 billion years ago when a planet-size body collided with nascent Earth at high speed.
"I think part of the key to (understanding) the Moon may be that Venus has no moon, and we certainly have to study it (Venus) more," said Dave Stevenson, professor of planetary science at Caltech University, who proposed the Venus idea.
The "moon capture" theory assumes that Earth used its gravity to attract a pre-formed space body into its orbit, thus making a satellite of this object.
However, the geochemical composition of the Moon and Earth likely trips up this theory, SPACE.com reported.
Analyses of the lunar rocks brought back by NASA's Apollo Moon landing missions have shown that the satellite has an isotopic composition very similar to that of Earth.
If both Moon and Earth have very similar isotopes, it makes the capture theory difficult to maintain, said Alex Halliday, head of science at Oxford University.
Some aspects of the idea that the Moon may have come from Venus are however very intriguing, he said.
"The reason why it's interesting is that Earth and Venus are close to each other. They have similar mass, and people think they have probably formed in a similar way," he said.
The capture theory faces a challenge explaining the similar composition of the Moon and Earth, Stevenson said.
However, if scientists analyse rocks from Venus and they turn out to be very similar to those on Earth, that would argue in favour of the capture theory, the report said.
The giant impact idea also has trouble explaining why the Earth and the Moon are so peculiarly similar.