If you provide the source code with the modified program, but the
receiver loses it, he may ask for it again.
Under the GPL, if you distribute the source with the binaries, nobody
can insist on getting anything from you subsequently.
If you distribute just binaries, you must provide a written offer for
source code. However, that written offer only needs to be valid for
three years. So you only have to save the source code for three
years.