Barack Obama is making his first presidential visit to a U.S. mosque on Wednesday, but the historic occasion is being overshadowed by criticism that the Baltimore-area center he chose has extremist ties.

The controversy centers around the Islamic Society of Baltimore's former imam, who has ties not only to the Muslim Brotherhood but the Northern Virginia mosque where the radical Anwar al-Awlaki used to preach.

“As a Muslim American I’m just insulted, this is disgraceful that this is one of the mosques -- or the mosque -- that he’s chosen to visit,” Zuhdi Jasser, of the American Islamic Forum for Democracy, told Fox News on Sunday. “This mosque is very concerning.”

The former imam in question is Mohamad Adam El-Sheikh, who served at the Baltimore mosque from 1983-1989 and 1994-2003. A member of the Muslim Brotherhood in Sudan, he had moved to the U.S. in 1978 and went on to receive several advanced law degrees as he became involved in the religious community.

During his time in Baltimore, El-Sheikh was a regional director for the Islamic American Relief Agency, the international parent organization of which has been cited by the U.S. Treasury Department for connections to Al Qaeda and the Taliban. After 2003, he was the imam for the Dar Al-Hijrah Islamic Center in Falls Church, Va., near Washington. It was there that Awlaki, just months earlier, gave his fiery sermons, before going on to be a top Al Qaeda affiliate operative in Yemen. Awlaki, a U.S. citizen, was killed in a U.S. drone strike in 2011.

The ties could pose a distraction for Obama at a time when the White House is trying to emphasize religious tolerance -- in part, as a counterpoint to what the White House says are intolerant statements being made on the Republican presidential campaign trail.

The Islamic Society of Baltimore serves thousands of area Muslims and includes a school and a housing complex. White House spokesman Josh Earnest said Monday that Obama is visiting to “celebrate the contributions [of] Muslim Americans.”

“Religious freedom and religious tolerance is central to the way of life in this country,” and this visit will show that “those values endure more than 240 years” after America’s founding, he added.

While the president has visited mosques overseas, this is the first visit in the U.S.

It will also be an opportunity, Earnest said, for the president “to talk about the role that faith plays in his own life.”

It's not just El-Sheikh's affiliations but his past comments that have stirred controversy.

He told The Washington Post in August 2003, when he was the imam in Falls Church, that Islamic law does not allow suicide bombings in most instances, but, "if certain Muslims are to be cornered where they cannot defend themselves, except through these kinds of means, and their local religious leaders issued fatwas to permit that, then it becomes acceptable as an exceptional rule, but should not be taken as a principle."

But he told the Post in that same article they had cut ties to the Muslim Brotherhood abroad and “we don't receive an order from any organization abroad, and [they] have no authority to tell us what to do."

He went on to work with the Fiqh Council of North America, which has issued several condemnations of terrorist ideology and attacks. He is also the founder and director of an Islamic law consultancy in Falls Church.

Maya Berry, executive director of the Arab American Institute says the criticism is the kind of guilt-by-association tactic that serves as a catalyst for intolerance in the first place. “I know of no other religious community that has this kind of litmus test applied to it,” she told FoxNews.com “I can’t speak to this individual mosque but what I can say is this is so incredibly illogical. It’s just not relevant.”

“Given the political discourse that’s been taking place in the 2016 election season, it is very important our president stand with our community at a time like this,” she added, noting presidential candidate Donald Trump's proposal to keep Muslims out of the country and Ben Carson's suggestion that Muslims would be unfit to be president.

"At the end of the day, young Muslim-American kids who have seen the onslaught of bigotry in their local news are going to look at this differently, they are going to see that 'President Obama came to be here with me.' That is an important message."

President George W. Bush took similar heat when, just six days after the 9/11 attacks, he visited a Washington, D.C., mosque and attempted to reach out to the Muslim-American population.

“These acts of violence against innocents violate the fundamental tenets of the Islamic faith,” Bush said at the time. “And it’s important for my fellow Americans to understand that.”

But Jasser says the Baltimore center is not the best representative of tolerance in the faith community. “[Obama] is visiting a mosque that [is] against our Muslim reform movement,” he charged. “Historically, as I said before, they are basically a radical, extreme mosque and is not representative of modern Muslims in America.”

Calls for comment from the Baltimore mosque were unreturned Monday. A programmed message on the phone line refers calls back to the White House.

A brief statement put out by the ISB on Monday described the complex as “one of the flagship communities in the nation,” but said representatives are not granting media interviews at this time.

The White House did not comment directly on the charges critics were making, but Earnest said any criticism will “serve to elevate the debate the president believes is worth having.”

FoxNews.com's Kelley Beaucar Vlahos contributed to this report.