Delport's score in the seventh minute of added time means that the north-easterners remain bottom of the table, despite never trailing the home side until Delport's late heroics, and the subsequent conversion from French fly-half Ludovic Mercier.
While Gloucester's top-of-the-table status was emphatic even before this match, the defeat from Newcastle leaves them level on points with Bristol Shoguns and London Irish, with Bristol having two games in hand on both of those sides.
But fourth-bottom Bath are the team in everyone's sights, with the West Country side only one point ahead, but having played one more game than both Newcastle and London Irish. Crucially, Newcastle travel to face Bath on May 10 in a match with massive importance, while Bath also have neighbours Bristol as their other remaining fixture.
On the day at Kinsholm, former Northampton fullback Joe Shaw put Newcastle into an early lead with a try on 15 minutes, with England fly-half Jonny Wilkinson adding a penalty before the half hour.
There was time for another Newcastle try before the interval, this time former All Black centre Mark Mayerhofler the man to breach the Gloucester defence, the 'Cherry and Whites' leaking points only a week after their emphatic Powergen Cup Final win at Twickenham.
Wilkinson converted that try, and at 15-0 up, the win looked assured for the visitors. When ex-Leicester scrum-half James Grindal went over for their third try five minutes into the second half that lead was extended to 20 points.
But on 55 minutes the fightback started, Mercier striking a penalty to get their scoreboard ticking over, before a try from England 'A' back row Andy Hazell nine minutes later got them back in with a serious shout.
Wilkinson struck a penalty to partially restore Newcastle's lead, but England loosehead prop Trevor Woodman crashed over with five minutes of normal time left to get the Kingsholm crowd singing.
Mercier missed the conversion but added a penalty not long after, but with the clock running down, deep into injury-time, Delport sent the Shed into fits of delight with a dramatic try to level the scores at 25-25, Mercier holding his nerve to convert and keep coach Nigel Melville's side on winning ways, as they set their sights on the play-off crown after an emphatic regular season campaign.
On the other hand, despite their efforts, Newcastle remain rock bottom, with an enthralling relegation race lying in wait.
For Gloucester:Tries: Hazell, Woodman, DelportCons: Mercier 2Pens: Mercier 2
For Newcastle Falcons:Tries: Shaw, Mayerhofler, GrindalCon: WilkinsonPens: Wilkinson 2