The protest is said to be the largest since the 2004 Orange Revolution, which overturned a rigged presidential poll.
Police fired tear gas as protesters tried to break through a cordon around government buildings.
A pro-government rally a few miles away attracted about 10,000 people.
Kiev police said they had fired tear gas after protesters threw a smoke grenade at officers in an attempt to break into the Cabinet of Ministers building.
Police estimated the crowd at about 50,000, but BBC reporters there said a vast crowd thronged central Kiev and 100,000 was a better estimate.
Ukraine made the decision on the EU deal last week, saying it could not afford to break ties with Moscow.
Russia is trying to bring Kiev into its own customs union.
Russian President Vladimir Putin accused the EU of blackmailing Ukraine to sign the deal during a summit in Lithuania next week.
People arrived at the rally, on European Square, with families and children, many holding banners with slogans like "I want to live in Europe" or "Ukraine is part of Europe".
"We want to be together with Europe," Volodymyr Mnikh, a 62-year-old retired chemist, told the Associated Press news agency.
"We want our children to have a future and not to be pressured by Russia."
Opposition leaders also joined the protests.
Several rallies in Kiev and other cities have been held over the last few days, but Sunday's has been the largest so far.
World heavyweight boxing champion Vitali Klitschko, who leads the Udar movement, addressed the rally in Kiev saying Ukrainians would not tolerate what President Yanukovich had done to the country's prospects for integrating with the EU.
"We will fight in order for the association agreement to be signed," he said.
On Friday, Ukrainian Prime Minister Mykola Azarov said the decision not to sign the deal was motivated solely by economics and was "tactical".
He said it did not alter Ukraine's overall development strategy.
The Ukrainian government says it is now looking into setting up a joint commission to promote ties between Ukraine, Russia and the EU.
Ukraine depends on imports of Russian gas, but recently the supplier, Gazprom, complained that Ukraine had fallen behind in payments.
Pipelines transiting Ukraine pump Russian gas to many EU member states.
In 2009 Russia cut off gas supplies to Ukraine in a dispute about prices, causing fuel shortages across Europe in the middle of winter.
Russia has a customs union with two other former Soviet republics - Belarus and Kazakhstan - and has been urging Ukraine to join it.
A free trade agreement with the EU would mean Ukraine adopting different trade rules.
Kiev's decision on Thursday prompted EU Enlargement Commissioner Stefan Fuele to cancel a trip to Ukraine.
It had been planned for the run-up to the key "Eastern Partnership" summit between the EU and several ex-Soviet states, which will be held in Vilnius, Lithuania, on 28 and 29 November.
Source: BBC News