The pole as you call it is not infinitely stiff and if you pushed on one end it would not move the other end instantaneously as you appear to imagine. What would happen is that the movement at your end would be transmitted along the length of the pole in much the same way a wave travels or a worm moves. In fact the movement of the pole would be incredibly slow.
You also wouldn't be able to tell when the pole started moving as it would take two years for the light at the end of the pole to reach your eyes.
You'd be dead long before anything observable actually happened.
