The former head of the IRS visited the White House more times than any Cabinet member, according to an analysis by The Daily Caller, raising questions about the nature of those visits -- particularly around the time the agency was targeting conservative groups.

The Caller analysis of White House visitor logs showed former IRS Commissioner Douglas Shulman visited the White House at least 157 times under the Obama administration.

Even Attorney General Eric Holder, one of Obama's closest allies, visited only 62 times according to the records.

The records may not reflect every single visit, as some officials do not have to sign in every time they come to the White House.

But they could lend weight to concerns voiced by lawmakers at a hearing last week about the frequency of Shulman's White House contact. During the time period when the IRS was singling out Tea Party and other groups for extra vetting -- as they applied for tax-exempt status -- Shulman visited the White House 118 times.

Asked to explain the visits, Shulman gave lawmakers a list of possible reasons.

"The Easter Egg roll with my kids ... questions about the administratibility of tax policy ... our budget, us helping the Department of Education streamline application processes for financial aid," he said.

According to the Caller analysis, no other top official logged more than 100 visits.

The official with the next-highest number of visits -- close to 90 -- was Rebecca Blank, former deputy secretary and now acting secretary of the Commerce Department. Next in line was Thomas Perez, a top Justice Department official who has since been nominated to lead the Labor Department.

Health and Human Services Secretary Kathleen Sebelius and Treasury Secretary Tim Geithner each logged fewer than 50 visits.

Former IRS officials have testified that the scrutiny of conservative groups, while inappropriate, was not politically motivated.