Whether it burst or not would depend on the internal pressure, the elasticity of the balloon’s envelope, and whereabouts in the Solar System (or, indeed, interstellar space, if you could get that far) it was actually inflated.
Balloons manufactured of Mylar-type materials have been inflated in Earth orbit since the early 1960’s; indeed, the first experimental communications satellite, Echo 1A, was just such a balloon, placed in orbit by rocket before inflation (and hence nicknamed a “rockoon” by NASA engineers). Although inflated with helium, not air, there is no reason why any suitable gas could not be employed, provided too great a mass was not used. If it were, it would, indeed, expand until the envelope tore, but because there would be no external atmospheric pressure, the balloon would not collapse immediately, but would deform over time due to the combination of thermally-induced stresses, gravitational forces, and the so-called “solar wind”.
If it did not burst and drifted off into deep space, possibly carried along by that same solar wind (the stream of charged particles from the Sun), then, eventually, the gas inside would cool. First of all, it would condense as a liquid on the inner skin of the envelope (this is one of the reasons why steam, otherwise a very good lifting agent, cannot be used for terrestrial balloons or airships), the additional mass causing the balloon to spin faster (the spin being initially induced by the differential heating of the gas by the heat from the Sun).
Then, sufficiently far from the Sun, beyond the orbit of Neptune, the air would freeze (helium, of course, could never do so, because even in interstellar space, the temperature would never reach absolute zero). The stresses caused by this internal icing combined with the balloon’s spin and the reduced elasticity of the envelope due to the extreme cold might well then result in the balloon’s structural failure, but it would not burst in the manner with which we are familiar from party balloon’s, but would tear itself to pieces, the fragments being spun off by centrifugal force.
Still, with modern material technology, balloons should make excellent deep-space probes, and one really wonders if the only reason stopping the world’s space agencies adopting the concept is the embarrassment factor of presenting such a proposal to scientifically-illiterate funding bodies.
