Grant Shapps is the latest cabinet minister to distance himself from the home secretary's choice of words, after she criticised police ahead of Saturday's pro-Palestinian rally.
In a Times article, Suella Braverman called protesters "hate marchers" and accused police of a "double standard".
Mr Shapps said it was "proper" for the home secretary to debate the issue, but he "wouldn't use that set of words".
Labour's Yvette Cooper said she "shouldn't carry on in her job".
She also suggested Ms Braverman's remarks had made disorder during Saturday's demonstrations worse.
On Sunday, the home secretary thanked police for "their professionalism in the face of violence and aggression from protesters and counter-protesters".
She criticised chants and placards from the march, saying: "This can't go on. Week by week, the streets of London are being polluted by hate, violence, and antisemitism... Jewish people in particular feel threatened - further action is necessary."
Her earlier claims that police were "biased" prompted widespread criticism and calls for the prime minister to sack her.
Pressure has increased on the home secretary after the Metropolitan Police made more than 100 arrests on Saturday and said officers faced "aggression" from counter-protesters.
On Saturday, Assistant Commissioner Matt Twist said the march had taken place against a backdrop of conflict in the Middle East, remembrance events and a "week of intense debate" about protest and policing, which "all combined to increase community tensions".
Speaking on Sunday, the defence secretary refused to say whether Ms Braverman would still be in her role in a week's time.
"A week is a long time in politics," Mr Shapps told Sky News's Trevor Phillips adding the make-up of the cabinet is "entirely a matter for the prime minister".
Downing Street is currently investigating how the article was published without edits they had wanted to be made - for now Prime Minister Rishi Sunak has backed his home secretary.
In her article in the Times, Ms Braverman claimed aggressive right-wing protesters were "rightly met with a stern response", while "pro-Palestinian mobs" were "largely ignored".
She went on to say police were applying "double standards" and "played favourites when it comes to demonstrators".
On Friday, while Downing Street gave its backing to Ms Braverman, Chancellor Jeremy Hunt said her comments were "not words that I myself would have used".
Mr Shapps told BBC's Sunday with Laura Kuenssberg he also would not endorse Mrs Braverman's choice of language.
He added: "I think there have been concerns sometimes that people have felt at liberty, perhaps because they haven't seen swift enough action to carry on going out carrying these banners, singing these chants and breaking laws which were in place to prevent racial hatred.
"On the other hand, I wouldn't put it in those particular set of words, because I recognise the police have a very difficult job to do in managing marches which contain large numbers of people - a lot of that work has to be done afterwards."
Scotland's First Minister Humza Yousaf has called for the home secretary to resign, as has the Welsh First Minister Mark Drakeford, who said Ms Braverman's comments "put other people in harms way".
Downing Street had, in effect, hit pause on deciding the home secretary's political future on Thursday as it turned its focus to the weekend's events.
With Remembrance commemorations passed and a clearer picture of what happened at protests and counter protests on Saturday, pressure will resume on Rishi Sunak to send a clear message on his home secretary.
He could, as the opposition are calling for, sack her.
That would almost certainly mean carrying out a bigger reshuffle, which has been rumoured to be on the cards for months.
The downside would be that it would anger her supporters, who are predominantly on the right of the party.
That could trigger letters of no confidence and other public displays of division.
One Tory MP told the BBC efforts were already under way to lobby to keep her in the job.
Mrs Braverman has also never hidden her ambitions to one day be the party leader and sacked ministers have been known to be a thorn in the side of their former bosses.
On the other hand, Rishi Sunak could decide keeping her on outweighs the controversies and he could choose to back her.
In order to avoid claims of weakness though, he'd have to find a convincing reason for why she went ahead and published a newspaper opinion piece that hadn't been cleared by Downing Street.
He would also frustrate those Tory MPs who have tired of her knack for attracting controversy and are starting to use phrases like "unhelpful" and "brand damage" in relation to the home secretary.
Her continued presence in the Cabinet would also provide an ongoing target for Labour.
Now, of course, Mrs Braverman has one big option of her own: she could choose to resign.
That would free her from the constraints of having to stick to the government line.
But it would also mean leaving a job that carries huge influence and gives her a platform to push her particular agenda on issues such as immigration.
Just to complicate matters, this all comes days before the Supreme Court gives a ruling on the government's Rwanda plan, with which Mrs Braverman has become closely associated.