The Cambodian government has been accused of using "direct and subtle threats" to evict thousands of families living near the Angkor Wat Unesco World Heritage Site.
The report by human rights group Amnesty International concluded international law had been breached.
A government spokesperson said that was "not right" and insisted the relocations were voluntary.
Families are being moved to a new community 15 miles (25km) away.
Cambodian authorities claim squatters are setting up informal settlements which damage the environment.
Government spokesman Pen Bona said the relocation of 10,000 families was in line with rules set down by the United Nations' heritage body Unesco, which banned structures or people living on the site.
However Unesco said it "never requested, nor supported, nor was a party to this programme" and has asked the authorities to take "corrective measures" in response to the report.
"Unesco is deeply concerned about the population relocation programme in Angkor," the UN body said in a statement.
Amnesty International alleges the group which manages the temple complex, the Apsara National Authority, is using Unesco to justify the relocations.
One resident told the AFP news agency that Cambodian authorities told her explicitly that "Unesco wants you to leave" or the site's world heritage status would be at risk.
At least seven villagers who live around Angkor Wat have been sued by Apsara, for allegedly inciting and obstructing public work, according to court summons seen by AFP.
"Unless there is serious pushback from Unesco, conservation efforts may increasingly be weaponised by states to their own ends, at the expense of human rights," said Montse Ferrer of Amnesty.
The temple complex received Unesco World Heritage status in 1992 and has since become Cambodia's most popular tourist attraction with two million yearly visitors.
That in turn feeds a micro-economy of stallholders, food and souvenir vendors.
Located near Siem Reap in the north-west of the country, the site was built by Khmer King Suryavarman II in the 12th Century.
It is thought to be a mortuary temple and faces west to symbolise the setting sun and death.
French archaeologist Bernard Philippe Groslier dubbed it the "Hydraulic City" because of its complex water management network.