Actor Jason Lee recently allayed concerns he was buying up land for the Church of Scientology in the small Texas town of Denton by telling a local website he no longer practiced the religion. Now, another ex-member says Scientology is bleeding members and that Lee’s defection is part of what former Scientology members say is a sign of the controversial church’s troubles.

Lee’s ex-wife and former Scientologist Carmen Llywelyn told FOX411 that Lee left the church three years ago, but didn’t make a big deal about it.

“I was asked in 2013 by a reporter what I thought about Jason leaving Scientology. At the time, I was definitely surprised but unsure,” she said. “I would have liked to think that by Jason also leaving that maybe certain aspects between us could've gotten cleared up. But it didn't.”

Llywelyn said Lee talking publicly about his decision was good news. “I was glad to hear that Jason came out and said he no longer practices Scientology,” she said.

Lee joins a list of Hollywood personalities who have left Scientology, including director Paul Haggis, actresses Nicole Kidman, Katie Holmes and Leah Remini, who wrote a tell-all book, “Troublemaker: Surviving Hollywood and Scientology,” denouncing the church, in 2015.

Llywelyn said the church “considers celebrities as opinion leaders and put a lot of value on their ability to influence people.” But that value may be diminishing.

Former high-ranking Scientologist Chris Shelton said the church has been skeptical of celebrity endorsements since Tom Cruise’s media whirlwind in 2005, when he infamously jumped on Oprah Winfrey’s couch, and had a painful interview with Matt Lauer.

“They made a tactical decision to not use a celebrity to endorse the church,” Shelton said. “They made a decision to put a muzzle on them. We haven’t seen a prominent celeb in at least five to seven years. Tom Cruise wanted to be the savior of Scientology, but instead convinced everybody he was nuts.”

Shelton said Lee had no choice but to publicly deny the rumors he was starting a Scientology facility in his new home state.

“I feel like Jason Lee left (the church) a while ago, and started a new life and finally felt safe to say he left,” Shelton said. “He had to say that so people knew he wasn’t a part of the religion anymore.”

Shelton, a Scientologist for for 27 years before also exiting in 2013, said the church membership is on the downslope.

“At most in 2009, 50,000 people were attending Scientology events, but now it’s about 20,000 people,” said Shelton, who blogs about Scientology and other subjects at mncriticalthinking.com. “But the church won’t provide hard numbers.”

Shelton said the availability of information about Scientology online led directly to the group’s decline. “The Internet was Scientology’s Vietnam,” Shelton said. “Getting new people to join Scientology is almost impossible due to the negative news about it on the Internet.”

Scientology was founded in 1954 by author L. Ron Hubbard. According to the official Scientology website, it is a “religion that offers a precise path leading to a complete and certain understanding of one’s true spiritual nature and one’s relationship to self, family, groups, Mankind, all life forms, the material universe, the spiritual universe and the Supreme Being.” The church is currently led by David Miscavige and counts Tom Cruise, John Travolta, Jenna Elfman and Kirstie Alley among its current members.

FOX411 reached out to Jason Lee and the Church of Scientology but did not receive comment.