Weeks after Neal died at age 41 following an ongoing battle with substance abuse, Cowell shared his thoughts on the tragedy with PEOPLE.
"Unfortunately, it happens too much over the years," Cowell, 62, says ahead of AGT's live shows on Tuesday. "I was thinking about this the other day. People [who] passed too soon. Every time it happens, because you've gotten to know them, it's horrible."
"At that moment, it's like, 'What can I say?'" he notes. "The unfortunate thing is it's happened too many times and every time it happens, it's hard."
Amy Sussman/Getty, Chris Haston/NBC/NBCU Photo Bank via Getty Simon Cowell (left) and Nolan Neal
Referencing the February death of fellow AGT contestant Nightbirde, who was also his Golden Buzzer selection in season 16, Cowell says: "After the last three years, it's been tough. And then, you think everything's okay and then something hits you like a story like that and it's hard."
"But all I can say is, what I've learned from this, is that you just take every day as it is and you try and stay positive, you try and stay healthy," he adds. "That's it. I mean, I was talking about this to someone earlier on, you're never prepared for these days."
Neal's death was announced last month. The local medical examiner's office confirmed to TMZ that the Nashville-based artist died on July 18, and Neal's cousin Dylan Seals also confirmed to the news outlet that he was found dead in his apartment.
The exact cause has not yet been determined by authorities, but Seals told PEOPLE that he "ultimately succumbed to his battle with substance abuse."
The late musician's cousin also called him "one of the most talented people" he's ever met while working in the entertainment industry. "He was always open and honest about [his substance abuse] struggle," Seals said. "He was a loving father and son. A light to all who knew him. My heart goes out to his two children and his mother Cathy."
Following the news, Neal's daughter, Caylin Cate, set up his GoFundMe page, and wrote that her father didn't have life insurance "or any money whatsoever."
"The expenses that come from this are extensive as we are trying to prepare for his funeral and fly in his son & loved ones," her statement continued. "If you want to help in anyway, you can donate here. Anything is appreciated and I am so grateful for the community rallying behind us."
Chris Haston/NBC/NBCU Photo Bank via Getty Images Nolan Neal on AGT
Two years before his death, Neal competed on America's Got Talent. He auditioned with an original song called "Lost" but was later eliminated in the Judges' Choice section of the Quarterfinals.
Prior to his time on AGT, Neal competed on The Voice's 15th season, airing in 2016. His audition performance of the Incubus song "Drive" allowed him to nab a spot on Adam Levine's team, but he was eventually eliminated from the competition.
Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.
Neal had previously opened up about his struggles with substance abuse.
"I remember I got clean in 2010; May 15, went to rehab. Stayed clean," Neal told WBIR in 2020. "I joined the rock band Hinder, they were all about drinking and partying. This is not their fault. I had decided I wanted to drink like a normal person. I remember trying to be normal and fitting in. I remember going to a bar and ordering a drink. I tried to hide it. I remember pretending to be normal. I was just lying to myself telling myself that I could control it."
Neal even admitted to having "lost my way on The Voice" as he continued drinking while competing on the show. But he eventually discovered a process that worked for him and allowed him to remain sober.
"I found happiness by helping others in recovery," he added. "I found a way to be happy without the things I thought made me happy before."
If you or someone you know is struggling with substance abuse, please contact the SAMHSA helpline at 1-800-662-HELP. Support is also available at American Addiction Centers.