<<...if I need to be adjusting for those movements on my controller?>>
Not unless you are going to hit something.
Why is the drone not fly in a straight line? Is the wind pushing it off track? Does it always fly off track in the same direction? Are you tilting the control stick inadvertently causing to move off track? Does it consistently move in one direction or the other when you hover?
Unlike fixed wing RC planes, the drone controller does not have trim tabs to adjust the flight path. The drone does not have a propeller pulling it straight forward and horizontal and vertical stabilizers to keep it on track. One of the beauties of the drone is that it will fly in all directions, facing in any direction.
I have tried to fly my HS720 like a fixed wing airplane, close to the ground, following a straight line and making 90 degree turns while flying a precise rectangular pattern. It is difficult to keep it pointed straight forward and flying straight ahead while trying fly a pattern. If the drone will hover in place, then all of the movements are the result of moving the sticks.
When I am flying close to the ground, less than 3 or 4 feet, then I have better control with the GPS turned off. Not sure why, but the GPS seems to fight the control inputs at very low altitudes. Maybe it does the same at higher altitudes but I can't see it.
Practice, practice, practice.