When will Woody Allen’s reckoning come? Some say that’s up to the actors.

On Thursday, just a few days after a Golden Globes ceremony dedicated to celebrating women of all social strata and their declarations that “Time’s Up” on discrimination, harassment and power imbalances in the workplace, the actress Mira Sorvino published a letter. Addressed to Dylan Farrow, the estranged adoptive daughter of filmmaker Woody Allen, the note was a formal apology from Sorvino for starring in Allen’s 1995 comedy Mighty Aphrodite in light of 32-year-old Farrow’s repeated allegations that the esteemed director molested her when she was 7 years old. (Allen has never publicly wavered in his denial of any wrongdoing).

“I cannot begin to imagine how you have felt, all these years as you watched someone you called out as having hurt you as a child, a vulnerable little girl in his care, be lauded again and again, including by me and countless others in Hollywood who praised him and ignored you,” Sorvino wrote for the Huffington Post. “As a mother and a woman, this breaks my heart for you. I am so, so sorry!”

Sorvino is the latest in a long line of actors to distance themselves from Allen in the wake of a cultural reckoning that has made it harder and harder to sweep the longstanding accusations against the director under the rug. In a roundtable interview with the New York Times just two days before Sorvino’s letter was published, director Greta Gerwig said that she regretted working with Allen on his 2012 film To Rome With Love.

“I can only speak for myself and what I’ve come to is this: If I had known then what I know now, I would not have acted in the film,” Gerwig said. “I have not worked for him again, and I will not work for him again.”

In recent weeks, Ellen Page, Griffin Newman and David Krumholtz have also publicly lamented their decisions to work alongside Allen, with Page notably calling her part in To Rome With Love the “biggest regret of [her] career.” But despite the quickening pace of the anti-Allen drumbeat in Hollywood, the four-time Oscar winner continues to be the perennial exception to the fallout that so many of his contemporaries have faced as a result of the #MeToo movement.

Melissa Silverstein, the founder and publisher of the women’s advocacy group Women and Hollywood, said in an interview that Allen’s clout, in part, has enabled him to continue to navigate a system that so many other men have found no longer works in their favor.

“The reason why people always worked with Woody Allen was because they always put him in a separate category, of being a quote-unquote ‘auteur-director,’” she said. “I think that he gets people nominations in his movies, so it’s a great stepping stone for a lot of actors and agents and it’s just one of those things where they give him a pass.”

Silverstein’s point is hard to argue against. Throughout his career, Allen has amassed more Academy Award nominations in the original screenwriting category alone than any other writer, and the stars of his movies tend to be similarly blessed during award season. In addition to Sorvino, who took home the best supporting actress Oscar for Mighty Aphrodite, six other actresses have also netted Academy Awards for their performances in Allen’s films, including Diane Keaton and Cate Blanchett, who won best actress trophies for Annie Hall (1977) and Blue Jasmine (2014), respectively.

While Allen’s name alone might still be a draw for those seeking some shiny new hardware for their mantlepieces, Zing Tsjeng, the U.K. editor for Broadly, said that our collective cultural acknowledgment of the severity of sexual abuse allegations is enough to dull the sheen of working alongside the filmmaker.

“To be honest, Allen’s movies offer ever-diminishing artistic returns — he’s basically a one-trick pony and his new films have very little in the way of actual originality, not to mention his proclivity for inserting creepy May-December romances in his scripts,” she said. “Right now, the prestige factor of working with Allen seems to be the main reason actors flock to him. But that prestige factor won’t exist if his name becomes toxic in Hollywood.”

Allen’s name, in fact, has long carried an aura of toxicity with it, but that stigma has never risen to a level capable of derailing his prolific film career. In the early 1990s, in the aftermath of Allen and Mia Farrow’s acrimonious split and the custody battle and molestation accusations that followed it, Hollywood was roiled by the allegations, with famous friends of the couple forced to choose sides. But the scandal hasn’t hampered Allen’s ability to continue to produce films at a speedy clip of nearly one a year — proof that the existence of the controversy on its own is not enough to deter many of Hollywood’s biggest stars from collaborating with the famed director.

Tsjeng also acknowledged the role that the “cult of celebrity” surrounding Allen — the reverence with which some actors, especially those still early in their careers, refer to his body of work — plays in fueling his staying power in Hollywood.

“Actors like Jesse Eisenberg have talked about growing up with him as a movie-making idol, and that idol worship tends to make people turn a blind eye to the kind of alleged behavior they would never countenance anywhere else,” she said.

Again, Allen has maintained his innocence when it comes to the accusations that he molested Dylan. In a 2014 Times op-ed, he gave his most substantial response to the allegations to date, adding that the piece would be his final word on the matter because “enough people have been hurt.” But even in an age when it’s said that we should believe women, and when women are pursuing consequences for their alleged abusers with a renewed tenacity, Tsjeng said that if Allen is to have a true moment of reckoning, the onus will largely be on the actors that work with him to bring it about.

“I think it’s going to take more people in Hollywood to stand up and say that they cannot in good conscience continue to work with Allen, and that they regret working with him in the past,” she said. “It’s going to take more people like Jessica Chastain, Greta Gerwig, Griffin Newman, Mira Sorvino, Dave Krumholtz and Ellen Page.”

Silverstein said that there’s a chance that the cloud of accusations swirling around Allen won’t only damage his own reputation in Hollywood, but also those of the people who choose to work with him going forward. That looming threat, she said, might be the thing that finally brings about a sea change.

“I do think now, in this post-#MeToo world, it’s going to give people pause, and they are going to be the ones to pay the price,” Silverstein said. “I don’t know if Woody Allen is going to pay the price, but I think other people will for working with him, and maybe that’s what will stop this.”