Updated

Two weeks after the Obama administration announced charges against suspects in the Benghazi attack, a large portion of the U.S. team that hunted the suspects and trained Libyans to help capture or kill them is leaving Libya permanently.

Special operators in the region tell Fox News that while Benghazi targets have been identified for months, officials in Washington could "never pull the trigger." In fact, one source insists that much of the information on Benghazi suspects had been passed along to the White House after being vetted by the Department of Defense and the State Department -- and at least one recommendation for direct action on a Benghazi suspect was given to President Obama as recently as Aug. 7.

Meanwhile, months after video, photo and voice documentation on the Benghazi suspects was first presented to high-level military leaders, the State Department and ultimately the White House, prison breaks in the country have eroded security. U.S. special forces have now been relegated to a "villa," a stopover for the operators before they're shipped out of the country entirely.

"We put American special operations in harm's way to develop a picture of these suspects and to seek justice and instead of acting, we stalled. We just let it slip and pass us by and now it's going to be much more difficult," one source said, citing 1,200 prisoners escaping two weeks ago. "It's already blowing up. Daily assassinations, bi-weekly prison escapes, we waited way too long."

The latest development raises questions about when the attackers will be brought to justice in the murder of Ambassador Chris Stevens and three other Americans last September.

The special operators are starting to get frustrated at the lack of action, and Fox News has been told by multiple sources that one special forces leader "literally yelled" at former Libyan Chief of Mission William Roebuck "and told him, 'so you're willing to let these guys get away with murder?'"

The outburst was "met with crickets," the sources said.

Asked about what actions have been taken on the suspects, the U.S. Department of Justice declined to comment. However, a senior U.S. official with knowledge of the situation in Libya suggested there is always intelligence to be gained by simply watching and listening to the suspects.

In addition, Pentagon officials disputed what the operators in question are claiming, saying that group was not specifically tasked with finding the Libyan suspects responsible for the Benghazi attack. These officials said other forces out of Fort Bragg are tasked with that mission, and they are not leaving. Pentagon officials also say the trainers, which were authorized by Congress under part of the defense budget to facilitate training of Libyans for counterterrorism, were not there to track the Benghazi suspects. They insist congressional funding is very clear in its mission: for training locals in counterterrorism.

However, special operators in the region counter the claims and suggest the Pentagon and State Department are playing with words, saying those being pulled are in fact tasked with both training the Libyans and identifying Benghazi attack suspects. "The training is partly a cover and some of these guys ... provided the information on suspects directly to U.S. military commanders and the U.S. State Department last November and again in January. They are there and trained to find, fix and finish," one said.

Fox News reported earlier this year that American forces had identified suspects by the end of November 2012, and reported on their whereabouts to Roebuck last January, yet no action was taken. They returned again in January to identify and locate these same suspects after being requested to do so by military leaders. In the months since, the operators in the region have been sitting in de facto standby, despite the Justice Department charges being filed.

To make matters worse, the U.S. trainers have been sitting in their Libyan "villa" now for a number of days after a Libyan military leader kicked the Americans out of the camp where they had been standing-up a Libyan special forces team for nearly a year, backed by U.S. taxpayer dollars. The maneuver by the Libyan chief of defense has "left our guys high and dry," and this same chief has locked down Tripoli as well, Fox News is told. The sources say U.S. leaders have now let the Libyan government occupy the special forces camp -- and in turn undermine the effort to train a legitimate force capable of countering Al Qaeda, which was the initial assignment before the Sept. 11 attack in Benghazi.

The men tell Fox News their mission was to capture or kill the suspects in question, and they briefed the acting U.S. ambassador in Libya and the senior CIA representative in the country. The men were told both Secretary of State Hillary Clinton and an under secretary were briefed about their information and their ability to capture or kill those responsible. Still, they did not get approval from the Department of Defense or the White House.

Some of the individuals report seeing former AFRICOM commander General Carter Ham tell former Libya Chief of Mission Laurence Pope that he could easily submit a plan to kill or capture the suspects, but "politics and fallout kept us from acting. To do an operation we have to have (Chief of Mission) and state approval. We didn't get it. ... They sat on it."

Multiple sources in Libya tell Fox News that the politics in the lack of response to the Benghazi attacks involves the U.S. State Department position in Libya. "No career diplomat wants to be responsible for giving the green light or supporting an operation that if goes wrong, another Somalia, Blackhawk Down, turns into a political fragmentation grenade that puts any group, party, or element in the public scrutiny spotlight ... especially after the train wreck on September 11th."

Sources told Fox News, though, that the Obama administration will be under pressure to produce some result from its investigation with the one-year anniversary of the attack looming.

Meanwhile, some of the same suspects in the Benghazi attack are continuing to help the Muslim Brotherhood in eastern Libya and are directly aligned with the militant group Ansar al-Sharia, which has already begun operations to undermine the fledgling Libyan government.

The special operators tell Fox News that Libyan militia leader Ahmed Khattalah, among those charged by the DOJ, is a member of a prominent and influential group in the eastern part of Libya and directly tied to Ansar al-Sharia, the group believed to be behind the attack on Benghazi .

"It won't be long, they're already at war. We are just behind," said one operator after being asked about the intent of the Muslim Brotherhood to overtake the current Libyan government in Tripoli. "So the terrorism will continue to grow and the terrorists responsible for killing an ambassador are right now growing along with it."