Despite the freezing conditions, it was a lively start at the Weston Homes Community Stadium, as Town stopper Stuart Nelson saved a Sammie Szmodics penalty with just seven minutes on the clock.
Spurning the chance to take the lead the U’s were made to pay midway through the first-half when Fisher slammed home from inside the box.
Both Tom James and Bevis Mugabi went close to doubling the lead either side of half-time, but a trio of smart Nelson saves ensured the visitors left with a first away win since Boxing Day.
After halting a run of three defeats at Exeter in midweek, Darren Way made one change from the side that returned from Devon with a point.
Making an impact from the bench at St James Park, Rhys Browne earned a starting place at the expense of Otis Khan who dropped to the bench as Corey Whelan missed out as the unlucky sixth loan.
Sticking with the experience of Nelson between the sticks, the 36-year-old justified his selection inside the opening 10 minutes.
After the U’s were awarded a penalty following Ryan Dickson’s foul on Kane Vincent-Young, the former Gillingham stopper guessed correctly to dive left and keep out Szmodics low shot from 12 yards.
Whilst the penalty miss was undoubtedly a blow for the hosts, they remained on the front foot in the opening quarter-of-an-hour.
Looking to make amends for his spot-kick failure, Szmodics continued to pester the Yeovil defence and teased an effort just wide of the target from the edge of the box following good work from Drey Wright.
Having demonstrated the same defensive resolve as in midweek, the visitors did what they couldn’t manage in Devon and took the lead.
Dickson’s long throw was flicked on by the dominant figure of Omar Sowunmi, allowing captain Francois Zoko to tee up strike partner Fisher who coolly picked out the far corner.
Stalling the U’s early momentum, Town were able to comfortably hold the hosts at arm’s length’s as the home crowd grew in frustration.
Kurtis Guthrie headed straight at Nelson as John McGreal’s side struggled to engineer any chances of worth, before the Glovers almost doubled their lead on the stroke of half-time.
After a spell of pressure Jared Bird laid the ball off for Tom James’ whose rasping first-time drive from 20 yards was palmed away at full-stretch by Sam Walker.
Looking to prove the old adage that attack is the best form of defence, Way’s side remained on the front foot after the restart enjoying a flurry of corners before Fisher fired just wide from range with 50 minutes on the clock.
Unable to call Nelson into action since his early penalty save, United had to wait until the hour-mark before they once again tested the veteran ‘keeper.
Liam Mandeville’s free-kick from the left of the box was destined for the top corner but for the fingertips of Nelson who continued to build on an impressive debut four days previous.
Making all three of their changes the U’s rolled the dice, and it was final substitute Olomide Shodipo who tested Nelson on 74 minutes when his free-kick – almost identical to Mandeville’s – saw the same result.
Seemingly unbeatable, Yeovil’s stopper produced one more notable save to keep out Frankie Kent’s well-hit shot as the Glovers rode out a final few minutes of pressure to record an invaluable victory on the road.
Taking four points from two away trips in a week, Town head into a fortnight’s break with an ever-increasing gap between themselves and the bottom two.
Colchester United: Walker; Eastman, Kent, Prosser; Wright, Lapslie, Comley (Senior 46′), Vincent-Young; Szmodics, Guthrie (Mandron 65′), Mandeville (Shodipo 71′).
Bookings: Comley 36′
Subs not used: Barnes, Murray, Stevenson, Rooney.
Yeovil Town: Nelson; James, Sowunmi, Donnellan, Dickson (Mugabi 40′); Green, Wing, Bird, Browne; Zoko (Barnes 82′), Fisher (Surridge 65′).
Subs not used: Maddison, Santos, Khan, Seager.