Defense Secretary Chuck Hagel on Wednesday sought to ease concerns about the controversial swap of five hardened Taliban leaders for Sgt. Bowe Bergdahl, but under pressure from lawmakers acknowledged that the administration mishandled the announcement.

"We didn't handle some of this right," Hagel admitted to the House Armed Services Committee, toward the end of the first public hearing on the prisoner exchange.

The hearing lasted more than five hours, as lawmakers from both sides of the aisle voiced concerns about the trade -- while some also accused Republicans of exaggerating the security threats.

Though White House and other officials have adamantly defended their handling of the exchange, Hagel did express some regret over the decision to keep Congress in the dark until after the swap had been executed. Republicans and Democrats alike vented to Hagel that they should have been trusted with that information.

Hagel acknowledged "we could have done a better job of keeping you informed."

At the same time, Hagel stressed that the Bergdahl trade was an "extraordinary situation" and claimed the administration "complied with the law."

Explaining why the administration moved so quickly and secretly, Hagel said Qatari intermediaries warned them "time was not on our side" and said risks to the American prisoner were likely growing.

"The bottom line is we don't leave people behind," Hagel said.

Lawmakers, though, were not forgiving over being shut out of those talks. "There's no reason why that 30-day notice couldn't have been given to the leadership in Congress," Rep. Adam Smith, D-Wash., said.

The wide-ranging hearing jumped from tense exchanges about security threats to in-the-weeds discussions about congressional notification, while all sides generally eased off criticism of Bergdahl himself.

In his opening remarks, Hagel also said both he and President Obama were on board with the decision -- amid some confusion in Washington over who technically approved the trade. "I want to make one fundamental point -- I would never sign any document or make any agreement ... that I did not feel was in the best interests of this country," Hagel said. "Nor would the president of the United States, who made the final decision with the full support of his national security team."

Earlier in the week, White House officials stressed that Hagel signed off on the decision without clearly saying who made the final call.

Hagel also tried to downplay the risk inherent in the exchange, claiming the former Guantanamo detainees were planners, and had not directly participated in attacks on Americans.

The chairman of the House Armed Services Committee, Rep. Buck McKeon, R-Calif., quickly pointed out that Usama bin Laden was also a planner. "Bin Laden didn't pull a trigger, but we went after him," McKeon said.

Hagel said he has no illusions about the five men traded for Bergdahl's freedom. He acknowledged the possibility that they could rejoin the fight, but said they would do so "at their own peril."

But in defending the trade, he said they had not been implicated in any attacks against the U.S. and there was no basis to prosecute them.

He said there was "no direct evidence of any direct involvement in their direct attacks on the United States or any of our troops," though they were combatants and "part of planning."

Rep. Mac Thornberry, R-Texas, asked him to clarify.

"So your point was they didn't pull the trigger, but they were senior commanders of the Taliban military who directed operations against the United States?" he asked.

"That's right," Hagel said.

The Defense secretary gave a detailed timeline on how the deal, which accelerated in a matter of weeks, came to fruition. He said that after U.S. officials received a "disturbing" video of Bergdahl in January that showed his appearance deteriorating, the Taliban signaled an interest in renewed "indirect talks" in April. On May 12, the U.S. signed an agreement with Qatar on security measures that would be taken if Taliban detainees were transferred there. All sides agreed to the mechanics of an exchange on May 27, after three days of "intensive talks."

Hagel said they were told by the Qataris that any leak could end negotiations; after the exchange was set in motion, he said, 96 hours passed before Bergdahl was in their hands. He said they did not know the precise location of the hand-off until one hour beforehand. After Bergdahl was recovered, on May 31, they released the five Guantanamo detainees.

McKeon, though, called the trade "deeply troubling" and said his committee has launched a full investigation into the "unprecedented decision."

"This transfer sets a dangerous precedent in negotiating with terrorists," McKeon said, adding that it could "incentivize" other terror groups including Al Qaeda to kidnap Americans.

Smith, the top Democrat on the panel, said those were "appropriate questions." But he also urged lawmakers to "take a step back" in terms of speculating on the circumstances of Bergdahl's capture -- some of his fellow soldiers have accused him of deserting.

"The way Mr. Bergdahl has been slandered has been scandalous," Smith said.

Hagel said the military will review the circumstances of Bergdahl's capture, but said no charges were brought or are pending against him.