A city mayor says residents who were evacuated from their homes will not get an update for two weeks.
Around 400 people were asked to leave Barton House in Bristol on Tuesday because the council had concerns about the structure of the building,
When asked whether they would ever return to their homes, mayor Marvin Rees said it would be "a challenge".
Earlier today, around 80 tower block residents staged a protest at City Hall in Bristol, demanding an investigation.
They said Bristol City Council had not offered enough support since Barton House was evacuated earlier this week.
Mayor Marvin Rees told the BBC: "We are beginning to make plans for the possibility that people won't be able to return to Barton House.
"It is going to be a challenge, but nothing about this situation is easy."
More than 400 people are living with friends or in hotels until a decision on the building's future is made.
One of the protesters, Ann Fisher, has been sleeping at friend's house for the last three nights.
She told BBC West: "I've lived 30 years in that flat and all my belongings, all my memories, everything my (late) husband bought me is in that flat.
"What am I supposed to do? Let it go? I'm not going to let it go. All I've got of my husband at the moment is his wedding ring.
"It'll be two years in December since he died and I had to fight to get myself back to where I am, and now this - I have to fight again.
"If (the council) thinks we're just going to just roll over and say 'Ok, we'll do this, that and the other' without an explanation, they've got another thing coming."
Supported by the renters' union ACORN, protesters held placards, and called on individual council members to act.
Shaban Ali, a resident of Barton House and member of Acorn, said: "We have all been through hell in the last few days.
"We have brought everyone here today to make sure that our voices are heard.
"We are calling for an independent investigation into what happened at Barton House and the safety of tower blocks across the city."
Several carried placards asking why the Mayor of Bristol, who is at a conference in Rwanda, had not returned home to offer his support on the ground.
Marvin Rees apologised for being absent on his website earlier this week, reassuring residents: "I will be back in Bristol as soon as I can, but in the meantime I know housing officers, cabinet members, my office, senior council staff have all been on hand."
Two miles away - there was more anger at Barton House.
Three residents had agreed to put their belongings into storage so surveyors could inspect the flats, which led to fears the council was starting to clear peoples homes.
Labour councillor Yassin Mohamud told the BBC there was a lack of communication and warning from the council, but nobody's possessions were removed without permission.
It is not known how long residents will have to remain in temporary accommodation, but people staying in hotels have been told the council was ensuring rooms would be available until at least Monday.
Bristol City Council has also said it was trying to arrange more suitable accommodation where people could cook their own food.
It added urgent surveys are taking place at the moment to determine if Barton House would need to be condemned.