getty (3) From L to R: Jeremy Renner, Robert Downey Jr., Chris Hemsworth
Jeremy Renner is getting candid about some playful behind-the-scenes friction on the Marvel movies.
The two-time Academy Award nominee, 51, reminisced about the time Robert Downey Jr. jokingly suggested breaking Chris Hemsworth's knee on the set of 2012's The Avengers, as he appeared Monday on Dax Shepard's Armchair Expert podcast.
"We're all kind of parading around in our costumes," Renner recalled. "So it looks like it's Halloween. We're excited and equally feeling ridiculous. I feel like we all kind of knew each other, somehow, some way. Except we didn't know this Hemsworth guy, because he comes from Australia."
"He's the tallest, he's the most good-looking. Downey [was like], 'We've got to break his knee. We've got to take him out. This guy's too good-looking. He's too tall. He's too charming. F--- this guy,' " Renner added.
Marvel Studios Poster for The Avengers (2012)
In all seriousness, Renner said that he and his Avengers costars have grown to see each other as "family," adding, "You can't replace that or quantify it."
"Everyone's a celebrity in their own right," the Hawkeye star explained. "But a Marvel celebrity, especially as the original six, has just been a different kind of journey."
Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human-interest stories.
"There's been marriages and divorces and kids being born and a lot of shifts and changes in our personal lives, as well as our acting lives, that we all share together in a very specific way," he also said, noting that the cast has gotten matching tattoos to symbolize their bond.
RELATED VIDEO: Paul Rudd Embraced His Ant-Man Training Until He Stood Next to Chris Hemsworth: "What's the Point?"
Hemsworth appears to have intimidated several of his Marvel costars with his physique, as PEOPLE's reigning Sexiest Man Alive Paul Rudd recounted a similar experience from his first encounter with the Thor: Love and Thunder actor, a fellow SMA alum.
Rudd added, "And I stood next to Chris Hemsworth, and I thought, 'What's the point?' Why even try, 'cause there's that."