Holder unsure whether he'll stay as AG, as election opens door to Cabinet reshuffle

June 11, 2012: Attorney General Eric Holder speaks at the League of Women Voters National Convention in Washington. (AP)

Attorney General Eric Holder, who endured what was arguably the most embattled term of any Obama Cabinet member, said Thursday he doesn't know whether he'll stick around for another four years.

"Do I have some gas left in the tank? It has been an interesting and tough four years. I really just don't know," Holder said, during a session at the University of Baltimore Law School.

The comments effectively kick off Washington's guessing game over what President Obama's inner circle will look like in a second term. The president could try to retain the current Cabinet to whatever extent possible -- or hit the reset with a broad reshuffling, perhaps sending a signal to Republicans in Congress that he wants a new start.

Click here for election results

The attorney general was a top target of House Republicans, who voted earlier this year to hold him in contempt for not turning over documents related to the botched Fast and Furious anti-gunrunning operation. Lawmakers also fought him over moves to bring the 9/11 terror suspects to trial in New York and to investigate the CIA over its interrogation program.

Steven Hess, with the Brookings Institution, said he doesn't expect Obama will push Holder out, but he anticipated Holder might be looking to make an exit anyway.

Holder touched on his dissatisfaction Thursday, recalling a conversation he had early on in which he said "they're taking the joy out of this."

Holder said he'll have to consider what he can "contribute" in a second term and talk to his family before making a decision.

For the last four years, Obama's Cabinet has been relatively stable. Only Commerce Secretary John Bryson resigned, earlier this year, after suffering a seizure and allegedly getting in a pair of hit-and-run crashes.

At least one other member of the Cabinet is on the way out in Obama's second term. Secretary of State Hillary Clinton has already said she'll leave as soon as a successor is chosen. Spokeswoman Victoria Nuland affirmed Wednesday that Clinton's plans have not changed.

"You've heard her say many times that she intends to see through a transition of a successor and then she will go back to private life and enjoy some rest and think and write and all those things," she said.

Speculation continues to stir about a Clinton 2016 presidential bid, though the secretary has downplayed that narrative.

Clinton, though, is still bogged down by an internal investigation into and congressional questions about the Libya terror attack. Addressing them could delay her departure from the department. And one of the potential candidates thought to be a contender for her job, U.S. Ambassador to the U.N. Susan Rice, could see her star fade over criticism of her initial claims the Libya attack was a "spontaneous" act. Sen. John Kerry, D-Mass., who gave a rousing speech in support of the president toward the end of the Democratic National Convention, is also seen as a potential contender for Clinton's slot.

Another Cabinet member, Treasury Secretary Tim Geithner, could also be held up in leaving by recent developments. He has indicated he will leave at some point, but the current stand-off in Congress over the so-called "fiscal cliff" may keep him in Washington a bit longer. Unless Congress acts, a wave of defense and domestic program cuts and tax hikes are set to go into effect in January, in a shock some economists say could trigger another recession.

Defense Secretary Leon Panetta is bound to Washington for the near term by the same crisis. He has argued forcefully that Congress must avert the defense cuts and is unlikely to leave until that is resolved. Still, Panetta has what is likely the most expensive and longest commute of anybody in the president's Cabinet -- going back and forth regularly between California and Washington.

Pentagon spokesman George Little, asked Thursday about Panetta's plans, said he would not "speculate on the secretary's future intentions."

"He is absolutely committed to his job," he said.

The tenures of the other Cabinet heads have ranged from quiet to laden with controversy. Both Health Secretary Kathleen Sebelius and Energy Secretary Steven Chu have been targets of Republicans on the Hill -- Sebelius for her role in implementing and promoting Obama's health care overhaul, Chu for his oversight of the controversial green-energy loan program that gave millions to Solyndra and other companies that later filed for bankruptcy.

Fox News' Shannon Bream contributed to this report.