Matt Gaetz (R-FL) leaves the U.S. Capitol after U.S. Speaker of the House Kevin McCarthy (R-CA) was ousted form his position
Several Republicans walked over to McCarthy and shook his hand.
Gaetz was joined by seven other GOP rebels, inckluding Arizona's Andy Biggs, Colorado's Ken Buck, Tennessee's Tim Burchett, Arizona's Eli Crane, Virginia's Bob Good, South Carolina's Nancy Mace and Montana's Matt Rosendale.
McCarthy's route to the Speaker's Chair in January was equally as chaotic.
The 58-year-old required 15 ballots to win his Speaker's election.
The number of votes marked the highest number of ballots needed to secure victory since 1859.
US Republican Representative from California Kevin McCarthy walks from the House Chamber after he was ousted as Speaker at the US Capitol
Gaetz took aim at McCarthy following the historic vote, labelling him a "creature of the swamp".
He added: "He has risen to power by collecting special interest money and redistributing that money in exchange for favours.
"We are breaking the fever and we should elect a speaker who is better."
However, McCarthy defended his decision to strike a deal with Democrats.
Ahead of the vote, he said: "Keeping the Government open and paying our troops was the right decision.
"I stand by that decision and at the end of the day if I have to lose my job over it, so be it."