Gloucester had led through Sione Kalamafoni's early score, though, and applied the pressure to devastating effect at the start of the second period.
First Rob Cook slotted another kick before Kvesic and Dan Murphy crossed for tries and though Dave Ward set home nerves jangling with a late Quins try the Cherry & Whites, down to 14 men, held on.
Gloucester had not won in the Aviva Premiership since beating London Irish at the end of 2013 and were behind only two minutes into the match to a Botica penalty.
However, the Cherry & Whites were given a chance to take command once Mark Lambert was sin-binned on seven minutes.
Quins actually added the next points, Botica again slotting a penalty, but James Hudson, patched up after a blood injury, produced a rampaging run to get Gloucester on the board.
The 32-year-old lock barraged through the Quins line and produced a superb off-load that allowed Sione Kalamafoni to cross the try line.
Cook converted to put Gloucester 7-6 ahead and Quins' saw injury added to insult when Tim Molenaar limped off on his Kingsholm return.
Yet on the half-hour mark, Harlequins retook the lead when scrum-half Karl Dickson took a quick penalty and off-loaded for full-back Ollie Lindsay-Hague to fly over.
Botica converted to extend the lead to 13-7 and the visitors held firm to take the lead into half-time despite Gloucester pressure on their line.
Gloucester continued to press at the start of the second half with Shane Monahan and Mike Tindall, on his 200th Premiership appearance, both breaking the line.
The pressure finally told when Harlequins were penalised and Cook converted a penalty to reduce the deficit to 13-10.
And Gloucester were back in front when they won a lineout just short of the Quins line and drove over with Kvesic emerging having scored his first try for the club.
Cook failed narrowly with the conversion attempt but the hosts now led 15-13 and from another set-piece on 56 minutes, the Cherry & Whites extended their lead.
From a five metre scrum, the ball was fed to Murphy who powered his way over and Cook made no mistake with the conversion.
Harlequins' woes were increased on the hour mark when Louis Grimoldby was sin-binned for a deliberate knock-on and Cook punished the visitors with the resultant penalty to put Gloucester 25-13 up.
Tindall then got a standing ovation as he was replaced on his 200th appearance by Freddie Burns.
With just over five minutes to go Gloucester's nerves were frayed when replacement Yann Thomas was sent to the bin and Quins pushed over the line, Ward taking the credit and Botica converting.
That reduced the home side's lead to 25-20 but they held on to secure a first Aviva Premiership victory of 2014.