Photos and video on social media showed the crane crashing through the mosque roof and the aftermath, with bodies, blood and debris spread across the courtyard.
The crane fell 10 days before the start of the Hajj, the annual pilgrimage expected to bring 2 million people to Mecca. The Masjid al-Haram is the largest mosque in the world and surrounds Islam's holiest site, the Kaaba -- a cube-shaped shrine that worshippers circle.
One man said the bad weather contributed to the chaos after the crane toppled over.
"We just washed and were getting ready to head to the Masjid al-Haram for the Maghrib prayer (sunset prayer)," said Yahya Al Hashemi, 30, a CNN iReporter who shot iPhone video of the crane striking the roof of the mosque. "It was a sandstorm which turned to rainstorm and lot of the construction covering boards were flying around, and lots of cracking noises which unfortunately ended with this tragedy. ... Everybody were pushing trying to escape from inside towards the exits."
More than 50 rescue teams and 80 ambulances converged on the mosque as part of the rescue effort after the crane fell at 5:24 p.m. local time (10:23 a.m. ET), said Saudi Civil Defense Director Maj. Gen Suleiman al-Amro.
A strong thunderstorm developed over Mecca at about 4 p.m. local time Friday (9 a.m. ET), bringing gusty winds that shifted direction and caused the local temperature to drop from 42 to 25 degrees Celsius (107.6 to 77 degrees Fahrenheit), CNN meteorologists reported.
Khaled Al-Maeena, editor at large at the Saudi Gazette in Jeddah, said the storm was so strong it uprooted trees and broke window panes throughout Mecca. The crane fell between times the mosque is crowded with people, he said.
"Had it happened an hour later it would have been much worse," he said. "Had it happened five hours earlier or four hours earlier, I think the death toll would have been more than a thousand.
Construction cranes surround the Grand Mosque, which is being enlarged to make the pilgrimage more manageable, he said
"The irony is that all this expansion was being done to see to the welfare of the pilgrims," he said.
At least 12 Egyptian pilgrims were injured, the Egyptian health ministry said. Ministry spokesman Hossam Abdel-Ghaffar told Ahram Online that two were in critical condition. India's external affairs ministry received reports nine of its pilgrims were injured, a ministry spokesman said Friday via Twitter.
The Grand Mosque is no stranger to tragedies, often because of the crush of people in Mecca for Hajj.
In 2006, a stampede killed at least 363 people. That stampede, like others in the past, happened during a stone-throwing ritual in which the pilgrims stone a symbolic devil.
Hundreds were killed in stampedes in 2004 and 1998 and 1,426 died in 1990.
Islam requires every Muslim who is physically and financially able to make the journey to Mecca at least once in his/her life.
Hajj occurs two months and 10 days after Ramadan ends, during the Islamic month of Dhul-Hijjah.