The pub applied for retrospective planning permission
Curran used part of the pub's car park to make room for the five beach huts and decking area.
However, neighbours started complaining to the council that the pub's garden visitors would get too loud.
One nearby resident said they had to "shut windows because of loud live music and loud intoxicated customers shouting and swearing till very late at night".
Curran has now been told to tear the huts down, calling the decision a "slap in the face".
The inspector’s report added: "There is evidence that the use of the beer garden does disturb the nearby residential occupiers in their homes and gardens, particularly the rears of the closest Oliver Crescent properties.
"The noise levels rise when customers are inebriated and when music is played in the beer garden.
"There are times when the neighbours are unable to enjoy their gardens and they need to close their windows to mitigate the noise."
While the report noted that the pub’s expansion had received some local support, the Planning Inspectorate concluded that the proposal would harm the living conditions of nearby residents.