The vote took place several weeks after Russian-led forces took control of Crimea, a predominantly ethnic Russian region.
Its residents say they fear the Ukrainian government that took over when pro-Russia President Viktor Yanukovych was ousted last month will oppress them.
On Saturday, Ukrainian officials said Russian forces backed by helicopter gunships and armored vehicles had advanced about 6 miles (10 kilometers) over the Crimean border into another Ukrainian region, where they took control of a village that holds a natural gas distribution facility.
If the referendum passes, Russia faces the prospect of sanctions from Western nations, but Moscow has vigorously resisted calls to pull back in Crimea.
Since Yanukovych fled to Russia, Crimea has come under control of local militia forces, as well as heavily armed troops under apparent command from Moscow.
Crimea's pro-Russia authorities say that if Ukrainian soldiers resolutely occupying their garrisons don't surrender after Sunday's vote, they will be considered "illegal."
But Ukraine's acting defense minister, Ihor Tenyuk, said in an interview published Sunday by the Interfax news agency that "this is our land and we're not going anywhere from this land."
In Sevastopol, the Crimean capital where the Russian Black Sea Fleet is based under a lease agreement with Ukraine, enthusiasm for the referendum was high, with voters lining up outside polling stations before they opened.
"Today is an important day for all Crimea, Ukraine and Russia," said voter Manita Meshchina.
"I think that people are expecting the majority of people will vote 'yes.' What it means is that people believe and think they need to be with Russia."
In Sevastopol, more than 70 people surged into a polling station within the first 15 minutes of voting.
"Today is a holiday," said one of them, 66-year-old Vera Sverkunova.
Asked how she voted, she broke into a patriotic war song: "I want to go home to Russia. It's been so long since I've seen my mama."
Speakers blared the city anthem up and down the streets, giving Sevastopol a feeling of a block party.
A Russian naval warship stood blocking the outlet leading from the port to the open Black Sea.
At a polling station 850097 set up inside a historic school building in downtown Sevastopol, Vladimir Lozovoy, a 75-year-old retired Soviet naval officer, began tearing up as he talked about his vote today.
Other voters cried out "Well Done! Hurrah"
"I want to cry. I have finally returned to my motherland. It is an incredible feeling. This is the thing I have been waiting for for 23 years and finally it has happened," he said.
Crimea's large Tatar Muslim minority opposes annexation to Russia.
The referendum "is a clown show, a circus," a leader of the Crimean community, Refat Chubarov, said on Crimea's Tatar television station Sunday.
"This is a tragedy, an illegitimate government, with armed forces from another country."
Blue-and-yellow Ukrainian flags were nowhere to seen around the streets of Simferopol; red, white and blue Russian and Crimean flags fluttered around the sidewalks, city buildings and on many cars.
Ethnic Ukrainians interviewed outside the Ukrainian Orthodox cathedral of Vladimir and Olga said they refused to take part in the referendum, calling it an illegal charade that they said was stage managed by Moscow.
Some said they were scared of the potential for ethnic cleansing in the coming weeks, like what happened in parts of the former Soviet republic of Georgia.
"We're just not going to play these separatist games," said Yevgen Sukhodolsky, a 41-year-old prosecutor from Saki, a town outside of Simferopol.
"Putin is the fascist. The Russian government is fascist."
Vasyl Ovcharuk, a retired gas pipe layer who also worked on the Chernobyl nuclear disaster in 1986, predicted dark days ahead for Crimea.
"This will end up in military action, in which peaceful people will suffer. And that means everybody. Shells and bullets are blind," he said.
At the United Nations, Russia vetoed a Security Council resolution declaring the referendum illegal, and China, its ally, abstained in a sign of Moscow's isolation on the issue.
Supporters of the U.S.-sponsored resolution knew ahead of time that Russia would use its veto on Saturday.
But they put the resolution to a vote to show the strength of opposition in the 15-member U.N. Security Council to Moscow's takeover of Crimea.
The final vote was 13 members in favor, China's abstention, and Russia as a permanent council member casting a veto.