Russian troops have seized most of Ukraine's bases in the peninsula, including a naval base at Feodosia on Monday.
Russia annexed Crimea last week after a controversial referendum that Ukraine and the West say was illegal.
"The acting President of Ukraine has given an order to the Ukraine Defense Ministry for the withdrawal of all Ukrainian forces from Crimea," a spokeswoman for Turchynov told CNN.
The soldiers' families will be evacuated as well.
Earlier, Russian troops stormed and seized the Ukrainian naval base in Feodosia, a port in Crimea, a Ukrainian Defense Ministry spokesman said.
Witnesses said several Mi-24 and Mi-8 helicopters were involved in the raid.
Between 60 and 80 Ukrainian troops were captured and taken from the base, said Vladislav Seleznev, a Ukrainian Defense Ministry spokesman in Crimea.
Ukrainians had said they would leave the base if they were allowed to take their weapons with them.
Russian forces consolidated their control of the Black Sea peninsula over the weekend.
On Saturday, six Russian special forces armored personnel carriers broke through the gates of Belbek air base, firing warning shots into the air.
In a separate incident, pro-Russian self-defense forces stormed the Novofederoskoe military base, also in Crimea, taking control of it, a Ukraine Defense Ministry spokesman said.
Seleznev also said that Crimean self-defense forces and Russian special forces took a Ukrainian ship, the Slavutych.
The White House urged Russia to open talks with the Ukrainian government.
But in a statement, it also held the Russian military directly responsible for any casualties inflicted on Ukrainian military members -- whether from regular Russian troops or militias not wearing insignias.
NATO's top military commander expressed concern Sunday about the buildup of Russian forces on Ukraine's border.
Supreme Allied Commander Europe Gen. Philip Breedlove said that Russia had a large force on Ukraine's eastern border and that he was worried it could threaten Moldova's separatist Transnistria region.
"The (Russian) force that is at the Ukrainian border now to the east is very, very sizable and very, very ready," said Breedlove, a U.S. Air Force general.
Russia said its forces complied with international agreements.
Moscow annexed Crimea after a snap referendum in the autonomous region last week that produced an overwhelming majority of votes in favor of leaving Ukraine to join Russia.
"Russia is acting much more like an adversary than a partner," Breedlove said, speaking at an event held in Brussels, Belgium, by the German Marshall Fund think tank.
Claims on Crimea
Ukraine's new leaders, the United States and other Western powers say Crimea is still a part of Ukraine.
Russia insists its actions are legitimate.
Crimea had belonged to Russia until 1954, when it was given to Ukraine.
The region also has a majority ethnic Russian population and other long historic ties to Russia.
Russia's Black Sea Fleet is based in the Crimean city of Sevastopol, making it strategically important to Moscow.
Moscow has doggedly pursued its own course, even as Western leaders have denounced its actions as violations of Ukraine's sovereignty and a breach of international law.
U.S. President Barack Obama plans to meet with leaders of the G7 group of industrialized nations this week to discuss Ukraine.
Russia has been excluded from the talks on the sidelines of a nuclear summit in the Netherlands.
A planned EU-Russia summit has also been canceled, as the West seeks to increase Moscow's isolation over its actions in Ukraine.
EU leaders imposed a new round of sanctions against 12 people last week, bringing the total number of people facing EU asset freezes and travel bans to 33.
The United States announced its own new round of sanctions against 20 people and a bank that U.S. officials say is linked to Putin and senior Russian officials.
Washington had already announced sanctions on 11 people.