Cossacks and pro-Moscow militias were seen patrolling at some polling stations and Russian flags were being flown everywhere from city buses to convoys of bikers roaming the streets.
Ukraine's new government and most of the international community except Russia have said they will not recognise a result expected to be overwhelmingly in favour of immediate secession.
"This is a historic moment, everyone will live happily," Sergiy Aksyonov, the local pro-Moscow prime minister, told reporters after casting his ballot in the regional capital Simferopol.
"We will celebrate this evening," Aksyonov said, after a man waving a blue-and-yellow Ukrainian flag and demanding it be put up inside the polling station was pushed away by security guards.
The Black Sea peninsula is inhabited mostly by ethnic Russians and has been seized by Russian forces over the past month after the ouster of Ukraine's pro-Kremlin leader in February, plunging US-Russia ties to their lowest point since the fall of the Berlin Wall in 1989.
Ukraine's interim President Oleksandr Turchynov, who last month replaced the ousted Viktor Yanukovych, called for a boycott of the ballot, accusing Moscow of preparing an invasion.
"The result has been pre-planned by the Kremlin as a formal justification to send in its troops and start a war that will destroy people's lives and the economic prospects for Crimea," he said.
Some Crimeans who requested anonymity said they were planning to spoil their ballots in protest and there was a call on social media for people to stay at home and cook vareniki -- Ukrainian dumplings -- instead of going out to vote.
Accredited journalists including AFP were prevented from entering some polling stations in the port city of Sevastopol and the regional capital Simferopol, and several people were seen voting in Sevastopol even before the polls opened.
Foreign observers were present but the Organisation for Security and Cooperation in Europe (OSCE) declined an invitation to monitor saying it was not official because it did not come from Ukraine's national government.
Voters can choose to become part of Russia or retain more autonomy but stay in Ukraine -- a vote for the status quo is not an option.
Preliminary results were expected to be announced soon after polls close at 8:00 pm (1800 GMT). The referendum committee said turnout was at 44 per cent a third of the way through voting.