Judge Brett Kavanaugh has recently been confirmed as a Supreme Court Justice. But to those who oppose him, that’s not the end of the story. Some are hoping—particularly if the Democrats retake Congress—that Kavanaugh can be forced out.

Federal judges are given lifetime appointments, but they can be removed. As Article III, Section 1 of the Constitution puts it, judges “shall hold their Offices during good Behavior.”

The Constitution’s method of removal is impeachment. As Article II, Section 4 of the Constitution states: “The President, Vice President and all civil Officers of the United States, shall be removed from Office on Impeachment for, and Conviction of, Treason, Bribery, or other high Crimes and Misdemeanors.”

Though Article II is generally concerned with the executive branch of government, the phrase “civil officers” in Section 4 is understood to include officers in both the executive and judicial branches, including federal judges.

The House of Representatives is given “the sole power of impeachment” under Article 1, Section 2, Clause 5 of the Constitution.

The process requires Articles of Impeachment to be drawn up, which would include a list of specific charges. What sort of charge fits the definition of a high crime or misdemeanor? No one is quite sure. Some say a proper charge is whatever the House says it is.

Those who want Kavanaugh impeached have listed various potential areas of wrongdoing. Some would like to charge him with sexual misconduct, and others question his testimony before the Senate during the recent hearings. Another charge some mention is the claim that during the Bush administration he lied under oath about receiving stolen memos.

Whatever the charges, the House would vote on them. To impeach, all that’s required is a simple majority. This is why some believe if the Senate changes hands in the upcoming election, impeachment is possible. But it’s hardly the end of the process.

Next, the charges would be brought before the Senate, which is given the sole power to try impeachments under Article 1, Section 3, Clause 6 of the Constitution.

After hearing the charges, and whatever evidence and witnesses are made available, the Senate would deliberate and vote to convict or acquit. However, a two-thirds majority is required to convict, which is where the hope to remove Kavanaugh would likely run aground, even under the best circumstances.

Historically, 15 federal judges have been impeached. Of those, eight were convicted, four acquitted and three resigned before the Senate could vote. Only one Supreme Court Justice has ever been impeached—Samuel Chase, in 1804. This being the early days of the republic, the role of the federal courts was not yet clear, and many at the time saw the impeachment as a political maneuver. Chase was easily acquitted.

It’s impossible to know what the political atmosphere will be a few months from now. Tempers may cool, or perhaps there will be momentum to go after Kavanaugh. But whatever happens, it’s doubtful that impeachment will be easy.

Even if the Democrats take the House, and assuming a vote mostly along party lines, it wouldn’t take too many Democrats switching to prevent impeachment. Kavanaugh is only suspected of committing crimes—crimes that would generally be hard to prove. And if a potential impeachment is seen as a political battle (a central reason federal judges are given lifetime tenure is to insulate them from political pressure), the public might not look too kindly on the House playing games with the courts.

And even if they succeeded, the chances of a conviction in the Senate seems highly unlikely. So while Kavanaugh supporters may think all the talk of impeachment is outrageous, they should recognize, so far, that’s all it is—talk.