Government forces first raised the Ukraine flag on the building but later left - allowing the rebels who captured it last week to be back in control.
Rebels have occupied official buildings in dozens of towns in eastern Ukraine in recent weeks.
The Kiev government has sent troops to restore its authority.
Shortly after the Russian and "Donetsk Peoples' Republic" flags were hoisted over Mariupol's city hall, the scene of tensions moved to the police station where 16 arrested pro-Russian activists were being held.
Heavily-armed police fired warning shots into the air as an angry crowd of friends and relatives gathered at the police gates concerned that the detainees would be moved to another province, the BBC's Sarah Rainsford reports from the scene.
Earlier on Wednesday, Ukrainian Interior Minister Arsen Avakov said government troops had taken over Mariupol's city hall following a joint operation by ministry troops and the army.
Mr Avakov told the BBC the defence minister of the separatist Donetsk People's Republic, Igor Kakidzyanov, was among those detained.
There were also reports of clashes in the separatist stronghold of Slaviansk, as Ukrainian troops advanced on rebel positions.
The Kiev government says 14 Ukrainian soldiers have been killed and 66 wounded since the start of its operation in the east last month.
In related developments on Wednesday:
The government has asked Ukraine's football league to hold remaining matches in the season without spectators and for those in the Donetsk, Luhansk, Odessa and Kharkiv regions to be moved to other areas.
Ukraine's central bank has received the first $3.19bn (£1.9bn; 2.3bn euros) tranche of a loan previously agreed with the IMF - part of an overall package of $17bn.
The government has rejected the pro-Russian activists' demands for greater autonomy for eastern regions, fearing they could lead to the break-up of the country.
Kiev has called a presidential election on 25 May, and said it was ready to back new international talks in Geneva, as long as Moscow supported the poll.
UK Foreign Secretary William Hague, after meeting Ukrainian leaders in Kiev, accused Moscow of "trying to orchestrate conflict and provocations" in eastern and southern Ukraine.
He said Moscow's "immediate goal" was to disrupt the forthcoming elections, "although of course they might also be trying to provide a pretext for intervention".
Russian President Vladimir Putin, meeting the chair of the Organization for Security and Co-operation in Europe (OSCE) in Moscow, said he was ready to "seek ways out of this crisis".
On Tuesday, Russian Foreign Minister Sergei Lavrov ruled out holding another round of international talks unless pro-Russian opposition groups were involved.
He added that there was no point attending as an accord agreed in April between the US, EU and Russia had not been implemented.
The Geneva agreement called for all parties, including the separatists, to refrain from violence, disarm and vacate occupied buildings.
The separatists of the Donetsk People's Republic and are preparing to hold an independence referendum on Sunday.
US Secretary of State John Kerry has condemned the plan as an "illegal effort to further divide Ukraine".
Tensions have been high since Kremlin-backed forces seized control of the Crimean peninsula, which then voted to join Russia in a March referendum that Kiev and the West deemed to be illegal.
At the scene
The situation outside Mariupol police station has now calmed down.
Earlier, special force police fired volleys of warning shots into the air as an angry crowd gathered at the police gates.
They came to protest against the detention of 16 anti-government activists, seized when police temporarily regained control of the occupied city administration building.
The police were armed with AK47s.
The crowd had been calling the police "murderers" - claiming they were ultra nationalist "right sector" militants, loyal to Kiev.
As the mood turned, two minibuses and a truckload of several dozen additional, heavily-armed police arrived at the police station.
More warning shots were fired.
The special forces have now gone.
It seems they were sent to remove their colleagues and calm the mood.
Source: BBC News Europe