Walder, who minutes earlier had missed a tackle to allow Michael Roberts to go over for what seemed to be a match-winning try, turned from villain to hero when he landed the 85th-minute kick to seal an epic contest.
In truth, the hosts, who scored tries through Mark Wilkinson and Phil Dowson, should have wrapped up the game by then.
But Wasps, who lost the chance to narrow the gap to Premiership leaders Leicester, put in a terrific second-half rally and looked like nicking it before Walder's intervention.
The clash began feistily, with Epi Taione and Phil Greening at each other's throats in the first minute.
After the dust had settled, makeshift kicker Tom May slotted over a penalty to give Falcons an early lead but Wasps full-back Mark van Gisbergen hit back with one of his own to level the scores.
The visitors made much of the early running, with rampaging centre Ayoola Erinle prominent.
Refereeing decisions were not going the Falcons' way, although they managed to retake the lead in the 19th minute through the boot of May to make it 6-3.
Moments later, they went further ahead following the game's first try.
Lively scrum-half James Grindal picked up from a ruck and fed the onrushing Wilkinson, who breezed through non-existent midfield tackling to dive over. May converted.
Debutant Toby Flood, a late replacement before kick-off for Falcons' Australian full-back Matthew Burke, was showing impressive lines of running as yawning gaps continued to appear in the Wasps defence.
But the London outfit closed the gap in the scoreline when van Gisbergen notched his second three-pointer and narrowed the deficit even further when Roberts went over for their opening try.
After the pack had made inroads into the Falcons' 22, the ball was spun out along the backs and although he received it from a standing start, the 22-year-old winger – an England Sevens regular – managed to break free from May's tackle and scurry over.
May made amends by maintaining his 100% penalty kick record to leave Newcastle 16-11 in front as the entertaining first half came to a close.
The second half began at breakneck pace and Newcastle extended their advantage with a contender for try of the season.
It had its origins with Michael Stephenson on his own five-metre line. He fired a long pass out to Mark Mayerhofler who in turn fed Taione.
After a string of deft passes, the superb Flood was hauled down five metres out – but popped the ball up for the buccaneering Dowson who touched down amid a frenzy of celebration from the Kingston Park faithful. May converted.
Wasps head coach Warren Gatland reacted straightaway by making four substitutions.
And they had the desired effect when Erinle powered through tired-looking tackles by Mayerhofler and Flood to score his seventh league try this season just before the hour-mark – making him the Premiership's leading try-scorer. Van Gisbergen converted.
Falcons should have replied immediately when, with a two-on-one overlap, May fumbled a sloppy Semo Sititi pass with the line beckoning.
Van Gisbergen made them pay by further cutting the deficit with a penalty to make it 23-21 with 15 minutes left.
Substitute Walder then kicked a penalty for the Falcons before disaster struck.
Voyce returned a kick and laid it on for Roberts, who somehow handed off prop Ian Peel and went through a poor Walder tackle to dive over for a 55-metre effort.
The scores were tied – and van Gisbergen knocked over the conversion to earn Wasps a 28-26 lead.
But Walder proved the matchwinner when he knocked over the last-gasp penalty to give Newcastle the points.
For Newcastle Falcons:Tries: Dowson, M WilkinsonCons: May 2Pens: May 3, Walder 2
For London Wasps:Tries: Roberts 2, ErinleCons: Van Gisbergen 2Pens: Van Gisbergen 3