Updated

In a case of who's watching the watchdog, the inspector general's office for a top housing agency is facing tough questions from Congress over what ex-employees say are efforts to "gut" a key office and allegations of retaliation.

Since last October, Senate Judiciary Committee Chairman Charles Grassley has been scrutinizing the IG for the Federal Housing Finance Agency (FHFA), the watchdog responsible for overseeing the regulator of mortgage giants Fannie Mae and Freddie Mac.

Not getting very far, Grassley, R-Iowa, and Homeland Security Committee Chairman Ron Johnson, R-Wis., recently took the unusual step of seeking an early outside review.

Their focus is on changes made after Laura Wertheimer was sworn in as IG in September 2014 and slashed staffing in the Office of Audits, with the apparent goal of making it leaner and more efficient.

The office, however, is tasked with rooting out waste, fraud and abuse -- at the agency responsible for monitoring $5 trillion in housing investments following the 2008 federal takeover of Fannie and Freddie. One ex-employee described the downsizing as as a "systematic dismantling."

Buyouts were used to achieve the cuts, which brought the staff down from 36 full-time employees to just nine.

In a June 30 letter seeking an outside review to the Council of the Inspectors General on Integrity and Efficiency (CIGIE), Grassley and Johnson raised concerns about whether the office could do its job. They said they want to ensure the "office is properly fulfilling its role of providing accountability and transparency over government programs” and questioned the OIG's independence.

CIGIE is an independent entity that monitors the integrity of IG offices.

The letter also questioned the FHFA-OIG’s decision to hire outside attorneys and employees who allegedly lacked proper auditing and IG experience -- and allegations that threats of poor performance reviews and other retaliatory actions were used against staff following the reorganization.

In her written response, Wertheimer said the changes would enable the OIG to “deploy our resources strategically and realign our audit and evaluation plan to adapt to changing current risks.” She also claimed the Office of Audits was failing to produce when she entered the IG post.

Grassley, however, challenged the claims in a March 24 letter.

“According to your website, it appears that the lack of production from OA occurred after, not before, your tenure,” he wrote, adding that that “the precipitous decline in level of work product is troubling.”

Leonard DePasquale, chief counsel at the OIG, denied the office failed to produce, telling FoxNews.com that under Wertheimer’s tenure, OIG investigations have resulted in billions in fines and civil settlements.

“The record is there on the website. And to a person, she has gained the respect of her employees,” DePasquale said.

Former employees, however, tell a different story.

“What we saw starting to happen shortly after her arrival was a systematic dismantling of a cohesive and productive organization. There was cohesion in the office and no one was afraid to do their jobs,” said Randal Stewart, a former senior investigative evaluator with the IG’s office.

Stewart, who worked for more than 20 years in the inspector general community, chose to retire rather than contest an unfavorable performance review. Another former employee with 25 years of mortgage fraud auditing expertise backed Stewart’s account.

The employee, who asked for anonymity due to pending litigation, said it was clear within weeks of Wertheimer’s arrival that the goal was to “gut” the office. In short order, he said, career professionals were reassigned to less critical offices or were replaced by individuals who lacked sufficient background in government auditing and investigations.

“When any agency makes changes, there are going to be disgruntled employees. This is not a new story,” DePasquale countered.

FHFA-OIG spokeswoman Kristine Belisle maintained the committee was being misled by these former employees.

She pointed to claims that were made of nearly half a dozen audits being canceled after Wertheimer, a former law partner at Wilmer Cutler Pickering Hale and Dorr, arrived. She told FoxNews.com that career employees searched their official system and found Wertheimer canceled no audits.

“Her record stands in sharp contrast to that of her predecessor, who canceled nine audits, at a significant cost of time and money,” countered Belisle.

Grassley and Johnson wrote to the OIG in May raising additional concerns about the OIG’s hiring of at least two outside attorneys and why “the FHFA-OIG Office of Counsel [was] deemed insufficient to provide representation” in certain circumstances.

DePasquale told FoxNews.com that “two firms” had been retained for occasions when “specialized expertise” was needed on a limited basis. He declined to comment further, citing attorney-client privilege.

A source with knowledge of the inquiry confirmed the law firm of Zuckerman Spaeder had been retained.

Outside legal representation was the subject of a June 28 meeting between staff members at OIG and the respective committees, but FHFA-OIG did not come “prepared with approaches to satisfy” the senators’ questions, according to a committee source.