Starmer's plan for an inner Cabinet of four people has been knocked back by Gordon Brown, who compared the idea to Communist China.
The Labour Party is weighing up plans to form an executive cabinet should it win power at the next election.
The smaller group would discuss ideas and make decisions before presenting them to the wider Cabinet for discussion.
But Brown said he had doubts the plan "could ever work". The former prime minister told an Institute for Government (IfG) event: "Triumvirates have been pretty difficult in the first place. To have a quadrumvirate though, is very difficult."
Starmer's plan for an inner Cabinet of four people has been knocked back by Gordon Brown, who compared the idea to Communist China
PA
Joking about the plans, he added: "The historical experience of that is very inauspicious, if I may say so.
"King Herod was part of a quadrumvirate when the four of them governed the Roman Empire, and you can take it right through to recent times and the Gang of Four, which if I remember right has not survived to tell much of the tale now."
The Gang of Four was a political faction under the Maoist regime in the 1960s and 70s. The group were kicked out of power after Mao Zedong died, being handed suspended death sentences and long prison sentenees.
The former prime minister added: "So I think the inner Cabinet idea may need some work.
"I doubt that the other 20 members of the Cabinet would be very happy if they were told that they were outside this inner circle."
It is likely that Starmer's planned inner Cabinet would be made up of Shadow Chancellor Rachel Reeves, Shadow Deputy Prime Minister Angela Rayner, and Pat McFadden - the Labour Party's national campaign co-ordinator.
The plans were based on a report into the functioning of Government from the IfG which argued a "full Cabinet is almost never a functional decision-making body".
It blamed the size of the Cabinet, which often sees the number of attendees surpass 30 people.
Last week, Labour grandee Peter Mandelson undermined Starmer's plans for reform of the House of Lords, warning the proposals are "not even half baked".
Brown said he had doubts the plan "could ever work"
PA
He said the proposals to reform the Lords were adopted by Starmer's party without a "substantive discussion" over their benefits.
Mandelson also warned that the plan - which would see an unelected Lords swapped for an elected second chamber representing the UK's nations and regions - could risk undermining proper scrutiny of legislation.
Starmer has endorsed ex-Labour PM Gordon Brown's plan for reform of the Lords.
LATEST DEVELOPMENTS:
In an interview with Lord McFall of Alcluith, the Lord Speaker, Mandelson said: "I think that we’ve got to have a far deeper conversation and analysis about this than has taken place to date.
"We haven’t had a substantive discussion about it in our own party, let alone a debate in the country.
"And yet we’re told six months away from a general election, all this is going to happen, abracadabra, in the first term of a Labour government. I mean there are real issues of principle."