In a wide-ranging interview with GB News, Mr Portillo hit out at Tories who he said were looking to “throw a grenade in the pond”.
Speaking to Gloria De Piero in an interview to be screened on Thursday, he said: “I’m not going to take a position of saying the Prime Minister should go, and I've been rather surprised to hear people who were leaders of the Conservative Party saying that.
“And I'll tell you the reason why I think that's quite important. Boris Johnson has many attributes and qualities and many, many flaws. But he has won four elections.
“I see people telling him that he should resign, that the Conservatives should get someone else. These are people who, to my knowledge and memory, haven't won anything.
“I mean, they may have held on to safe seats having been selected. But, you know, they have never won a national election…
“I do think it's interesting that lots of Conservative MPs seem to think they should change the leader without having a clue who would take over and why that would be any better.”
He said it seemed to be a habit among Tory MPs to plot against their leader in the absence of outstanding alternative candidates.
Mr Portillo added: “There are quite a lot of people in the Conservative Party today, and it happens again and again, who just want to throw a grenade in the pond,.
“They have no idea whether, having got rid of Boris, whether things will be better or not. And why does this happen? One of the reasons is because the party now consists largely of people who are disappointed or unappointed, or de-appointed.
“In other words, they didn't get into office, or they've been thrown out of office. And this becomes a very large group of people.
“And what they think is, whoever is the Prime Minister after Boris Johnson, that ‘I have a better chance of being a junior minister and having a ministerial car and red box’ under someone else. And that becomes a very powerful engine.”
He also criticised former Tory leader Michael Howard who has called for the Prime Minister to resign.
Mr Portillo said: "If it were someone who had won three general elections saying, ‘I think this person should go because they're not good enough’, I could just about understand it.
“But people who have won nothing, telling someone…that he should go, I find that very difficult to deal with.“
Mr Portillo also reflected on his own political career saying he felt he would have been “well suited” to the job of Prime Minister but acknowledged he lost a safe seat and did not feel he had the right temperament to go out and win elections.
He continued: “I could imagine myself doing the job of Prime Minister. I think I'm quite decisive, and I'm quite organised, things which actually Boris is not.
"So, in a way I can imagine that part of the job of Prime Minister as being to me, quite easy. But the bit that is not easy for me, clearly, is winning elections, I lost a safe seat.
"So, I wouldn't be comfortable with that part of it. And I don't think I had the temperament."
Commenting on that now infamous general election defeat, Mr Portillo said he actually felt a sense of relief.
He added: “You may think that odd, but what I really cared about was being in Government and I had no doubt whatsoever that the Tories were getting out of Government that night.
"Had I won my seat, I would have had to contest the leadership of the Conservative Party, which I was not keen to do because we had been reduced to a rump, and I knew the Tories wouldn't be back for 10 to 15 years as indeed turned out to be the case.
"So, there was a certain relief to not winning and a certain amount of graveyard humour.”
He also revealed to Gloria that he and Labour MP Diane Abbott attended drama classes together when they were at schools in Harrow in north-west London.
Mr Portillo said: “In fact, I cast her as Lady Macduff in a film version of Macbeth which never got made.
Mr Portillo ruled out a Strictly appearance GB News
“However, years later – I remember Diane as being very shy and self-effacing – and when years later a rather self-confident woman got elected to Parliament, who also had the name Diane Abbott, I couldn't quite believe it was the same person.”
Mr Portillo also brushed off suggestions his new role as a TV presenter had turned him into a national treasure.
He said ex-Labour MP Ed Balls was the only politician who could lay claim to such a title thanks mainly to his stint on Strictly Come Dancing.
Asked if he’d be tempted to compete on the BBC show himself he said: “No, I would not do it. No.
“Ed [Balls] did it very well. I mean Ed has amazing talent outside politics and he’s also done a terrific cooking thing hasn’t he?”
Michael Portillo speaking to GB News GB News
GB News' Gloria spoke to the former Defence Secretary GB News