Neither side were able to break the deadlock in normal time with Jonny Maddison producing a string of fine saves for the visitors, with Rovers converting all five of their penalties to claim two points and get their cup campaign off the mark.
The Glovers come away from Bristol with a single point, meaning they’re in second place in Southern Group A with four points, with one match left to play against Reading Under-23’s.
Darren Way made six changes from the side that started at Plymouth Argyle at the weekend, with Maddison, Bevis Mugabi, Ryan Hedges, Otis Khan, Tahvon Campbell and Ben Whitfield all given starting berths.
Youngsters Joe Lea and Kyle Copp also returned to the Glovers bench, as Way used the cup match as a chance to rotate his squad.
Town got off to a positive start and tested Kelle Roos in the home goal after just three minutes, with Ryan Hedges’ effort from just over 20 yards calling the Dutch stopper into action early on.
In a relatively quiet opening 45 minutes neither side created a meaningful opportunity, although Alex Lawless had a well-hit shot from 25 yards deflected just over Roos’ goal.
The Glovers had the ball in the net with 26 minutes on the clock, but Nathan Smith was adjudged to have fouled the Rovers ‘keeper in the process of heading the ball from six yards.
At the other end, Chrristian Montano failed to hit the target from inside the box after a deep free-kick fell to the winger’s feet, dragging an effort wide of Maddison’s goal.
A rather subdued first half was summed up by fourth official Simon Barrow seemingly forgetting to raise his board for additional time at the end of the first period, as Town looked to find a way to secure their progression from Southern Group A at the Memorial Ground.
Six minutes into the second half and Maddison had to make his first meaningful stop of the game, doing well to get everything behind Montano’s fierce effort from the angle.
Rovers came closer to scoring just before the hour mark when a deflected Montano cross almost crept in past Maddison off of Alex Lacey, but the young stopper was able to jump on the ball leaving the Pirates claiming it had crossed the line.
Way introduced Izale McLeod and Kevin Dawson for Whitfield and Khan as the Glovers switched to a 4-4-2 formation with McLeod partnering Campbell up front.
The visitors’ plan was disrupted with a little under 20 minutes remaining however, as Dawson received his marching orders in strange circumstances.
The Glovers midfielder was fouled by Montano but received two yellow cards in the space of seconds for his reaction to the foul and then dissent towards referee James Adcock.
Daryl Clarke’s side looked to capitalise on their numerical advantage and introduced top scorer Matty Taylor in search of an opening goal, and the Pirates frontman almost did just that in additional time only to be denied by a fine diving save from Maddison.
The 22-year-old, making just his second start for Town, produced another good save to keep out a long range Hiram Boateng effort at the death, as Way’s side dug deep in the closing stages to keep out the hosts and force a penalty shoot-out.
In the lottery of the shoot-out Rovers converted all five of their spot-kicks to see off Town 5-3 on penalties, with Ryan Hedges the unlucky player for the Glovers after having his effort from 12 yards saved by Roos.
Bristol Rovers: Roos; Leadbitter, Clarke, Hartley, Brown (Colkett 66’); Montano, Sinclair, Boateng, Moore (Bodin 77’); James, Gaffney (Taylor 66’).
Bookings: Clarke 88’, Colkett 90’.
Subs not used: Mildenhall, McChrystal, Mansell, Easter.
Yeovil Town: Maddison; Mugabi, Lacey, Smith, Dickson; Whitfield (McLeod 55’), Lawless, Dolan; Hedges, Campbell, Khan (Dawson 60’).
Bookings: Lawless 63′, Dawson 72’ (red), Mugabi 90’.
Subs not used: Krysiak, Ezewele, Copp, Lea, Sowunmi.