The Titans – who have now lost 16 out of 16 Premiership encounters – shocked their visitors by racing to a 16-0 lead at the break, with a try from No.8 Brad MacLeod-Henderson and 11 points off the boot of fly-half Jon Benson.
But fourth-placed Gloucester showed their immense experience by hitting back with two tries each from Samoan star Terry Fanolua and England flanker James Forrester.
South African winger Gisdard Pieters added a try for Rotherham in the second half, but Gloucester's fly-half Duncan McRae added 15 points to make the difference count.
Rotherham now face another week of waiting after making a brilliant start to this game.
They took the lead after four minutes with a Benson penalty and he was on target twice more in the 18th and 37th minutes.
Rotherham looked in control and made their presence felt when MacLeod-Henderson intercepted a Fanolua pass to score.
It was a happy moment for the South African back rower who played despite sustaining a nasty eye injury against Wasps last weekend.
Unfortunately for the Titans, Gloucester found the extra gear during the break and McRae kicked a penalty early in the second half before linked up with flanker Andy Hazell to send Fanolua in for his score.
With an advantage now established, Gloucester scored again – replacement Forrester finishing of trademark forwards' move down the short side.
Rotherham answered with Pieters' try, but Gloucester poured forward to devastating effect.
In the 63rd minute Fanolua grabbed his second try following more good work by McRae.
And Gloucester's recovery was complete – and Titan misery compounded – when Forrester was the quickest to react to a messy Rotherham scrum to pounce on the ball and secure the 'Cherry and Whites' their bonus point.
The defeat confirms Rotherham's relegation from the Zurich Premiership as they are now mathematically unable to save themselves from the drop.
For Rotherham Titans: Tries: MacLeod-Henderson, PietersCon: BensonPens: Benson 3
For Gloucester:Tries: Fanolua 2, Forrester 2Cons: McRae 3Pens: McRae 2DG: McRae