Analysis of the latest HMRC figures showed fewer than four per cent of estates paid inheritance tax between 2020 and 2021.
The number is expected to rise to more than seven per cent across the next ten years, as a result of the growth in wealth of older adults.
WATCH NOW: Jacob Rees-Mogg says inheritance tax should be scrapped
Inheritance tax is paid on estates worth over £325,000.
The IFS report said this year, the cost of abolishing Inheritance Tax would be £7 billion, but this would rise to £15 billion per year by 2032.
The report warned: "The 90 per cent or so of estates not paying inheritance tax would not be directly affected by such a reform."
No 10 is trying to play down reports that the Government plans to slash the tax. But discussions are said to be taking place within the Government about changes to the tax at the next manifesto ahead of next year's election.
While Chancellor Jeremy Hunt has said there is no room for tax cuts, Defence Secretary Grant Shapps this week described inheritance tax as "deeply unfair" and "punitive".
He said: "I think it’s a question, for many people, of aspiration, and people know that there’s something deeply unfair about being taxed all their lives and then being taxed in death as well."
An influential backbencher told GB News that the UK is unlikely to see immediate inheritance tax cuts at the Autumn Statement, but they could be included in the manifesto ahead of the next general election - expected to take place next May.
They questioned the urgency of changes to inheritance tax, saying: "I would not myself choose to cut inheritance tax on either of the budgets taking place ahead of the general election.
"I think there are more urgent priorities affecting people in Britain."
Shapps echoed this, admitting that the Chancellor is in a “fiscal straitjacket”, meaning that any changes are unlikely to feature in November’s Autumn Statement will contain any immediate changes to the way the tax is administered.
He said: "You will certainly have to wait for a Budget, or another event, for the government to set out whatever the plans will be.
"Generically, I’m in favour of all taxes being lower, but we’ve got to be fiscally responsible.”
Defence Secretary Grant Shapps this week described inheritance tax as "deeply unfair" and "punitive"
Labour has criticised the plans, with Shadow Treasury Chief Secretary Darren Jones saying: "A year ago, Liz Truss trashed the economy with unfunded tax cuts.
"Now Rishi Sunak is doing what Liz Truss wants.
"Abolishing inheritance tax – which 96 per cent of people never pay – is an unfunded tax cut of £7.2bn per year.
"The biggest threat to the economy is the Conservative Party."
A spokesperson for the Treasury said: “More than 93 per cent of estates are forecast to have zero inheritance tax liability in the coming years – however, the tax raises more than £7bn a year to help fund public services millions of us rely on daily.”