Former Royal Callum Harriott caused the home side some early issues, seeing a shot blocked after five minutes and then angling a low cross into the six yard box from the left shortly after.
That low cross was met by a clearing leg of Michael Morrison but it deflected off the centre half and headed towards goal only for an instinctive stop from Luke Southwood to palm it away from danger.
Noah Chilvers arrowed an effort over the bar as the League Two side began the tie on the front foot.
But Reading soon began to find their feet and good work from McNulty and then Aluko allowed the latter to pick out Baldock on the edge of the box. The striker shimmied to one side and drilled low at goal to force keeper Dean Gerken into a smart save low to his left.
A sliding block from Morrison prevented Southwood from having to do anything with a drilled effort from Kwame Poku midway through the half. Then at the other end, some tidy footwork from Aluko led to a cross-shot that skidded across the six yard box and behind for a goal kick.
On the counter attack, Joao and McNulty combined to give the latter a sight of goal - but the offside flag was raised against the Royals forward before he could pick his spot with a shot.
But with nine minutes of of the first half remaining the visitors broke the deadlock. Cohen Brammall got in behind the Royals defence down the left and cut a low cross back for Jevani Brown to turn home from close range past a helpless Southwood.
The Royals looked to respond swiftly, an Olise set piece was too deep for everyone but then Joao nearly reached a deep Baldock cross from the other flank - the striker couldn't hook his right boot around it enough to prevent it from spinning behind for a goal kick.
And in first half injury time, Lucas Joao scored our first goal of 2020-21 to draw the home side level. Aluko fed Araruna on the right side and his first time cross was attacked by the powerful Joao who muscled his way above his marker to head home from eight yards out.
Half-time: Reading 1-1 Colchester United
Andy Rinomhota came off the bench at the break and immediately cause problems with a driving run from midfield that earned the hosts a corner. Olise swung the ball to the back post where an unmarked Laurent volleyed the ball wide with his left foot from a good position inside the United box.
A Bristow ball from the left caused confusion between defender and goalkeeper and between them they bundled the ball behind for another corner. Once again Olise's corner found its way to Laurent but again the midfielder's stabbed effort sailed inches over the crossbar.
Reading were dominating their opponents and another Olise corner was met by the head of Morrison who couldn't direct his effort anywhere other than high over the bar.
The Royals took the lead just before the hour mark. Olise slid a delightful ball between defenders which both Baldock and Joao looked to scamper onto. Joao got there first and calmly sidefooted past Gerken for 2-1.
And with 15 minutes remaining, Joao added a third! And he saved the best for last. A quickly taken free kick was whipped in by Araruna and flicked on by Morrison into the striker who held of his man with his back to goal. He then outfoxed the defenders around him completely with a lovely roll back. That earned him the space to take a shot and he made no mistake, sweeping home with his left foot to make it 3-1 and complete the perfect hat-trick.
Colchester had to find a response and Harriott got in behind the Royals back line only to scuff his toed shot harmlessly wide under pressure from backtracking defenders.
Instead, the Royals nearly added a fourth when Olise delivered a tasty low cross that substitute Omar Richards connected with in the middle but couldn't direct on target.
Nahum Melvin-Lambert replaced the hat-trick hero with ten minutes left to earn his first team debut - the third Academy product to make his Royals bow this afternoon.
And he had a golden chance to open his account with a minute of the ninety remaining. Olise was again the architect, threading a ball into Melvin-Lambert who had the keeper to beat. But Gerken was equal to his placed finish and denied the graduate a debut goal with a parrying save.
Southwood needed to come off his line to deny Brown an injury time consolation and he did so well, blocking the forward's stabbed effort with his body. And moment later, the final whistle sounded to send the Royals through to the next round of the Carabao Cup.
Full-time: Reading 3-1 Colchester United
Reading: Southwood, Araruna, Bristow, Moore (c), Morrison, Laurent, Olise, Aluko, McNulty (Rinomhota HT), Baldock (Richards 67) , João (Melvin-Lambert 81).
Unusued subs: Andresson, Abbey, Samuels, Tetek, Melvin-Lambert.
Goals: Joao (45, 56, 75)
Colchester United:Gerken, Scarlett, Brammall, Eastman, Smith, Pell (c), Stevenson (Marshall-Miranda 88), Chilvers (Senior 71), Brown, Harriott, Poku.
Unused subs: Welch-Hayes, Sowunmi, George, Hutchinson, McLeod.
Goals: Brown (36)