President Donald Trump has reportedly told some advisers that he hopes Rob Porter—his former staff secretary who exited his job after domestic violence allegations surfaced—would eventually return to work in the White House.

The New York Times, citing three unnamed sources, reported that Trump has been in touch with Porter and has increased the frequency of phone calls over the last few weeks as more officials departed their positions.

A White House official told the Times that Trump misses the staff structure created and introduced by Porter, but realizes that bringing Porter back may be a bridge too far.

The White House did not immediately respond to Fox News for comment.

The president reportedly had discussions with Porter about trade and tariffs on certain goods – a measure Trump recently introduced and worked on together with the official before his departure.

Porter was forced to resign last month after two of his ex-wives came forward with abuse allegations. One of the ex-wives, Colbie Holderness, reportedly said Porter chocked and punched her during their marriage. A photo of her bruised face was released. Porter denied the allegations and said the photo did not reflect the full story of his marriages.

Trump stressed that Porter denied the allegations, saying, “He says he's innocent, and I think you have to remember that. He said very strongly yesterday that he's innocent. So you'll have to talk to him about that. But we absolutely wish him well.”

The abuse allegations against Porter also led to a controversy over the security clearance process in the administration. Porter had an interim security clearance despite the FBI concluding the investigation. One of his former wives reportedly told her abuse story to the FBI back in January when she was interviewed regarding Porter’s security clearance and provided the photos to the agency.

The Associated Press contributed to this report.