This is a common auditory illusion. You don't hear the sound of the jet at first because you are concentrating on something else. Your brain hears the jet noise, but filters it out because it is a recognised non-threatening sound. Then the noise changes and your brain focuses on it. To your conscious mind it seems that a loud noise has suddenly appeared from nowhere. You then continue to hear the jet noise as the plane moves away because you are now concentrating on it.

The cause of the noise is, as you suspected, a deceleration of the engines. There are many reasons why a pilot might suddenly throttle back the engines, but the most common one is speed limits. Some areas of controlled airspace have speed limits and an aircraft entering that airspace will have to slow down to comply with the limit.