Rationale for removing binary data from debian source package

[Mathieu Malaterre <malat@debian.org>]

Dear Debian Med,

I am trying to remember the rationale for removing binary data from a
Debian source package.

Let's consider charls:

```
Files-Excluded: */test/conformance
                */test/jlsimage
                */test/*.raw
                */test/*.jls
                */test/*.dcm
                */test/*.ppm
                */test/MR2_UNC
```
Long story short, upstream is relying on some of these files to run
unit tests (please no debate). Would it be acceptable to add a portion
of these binary data back into the source package ? If so, what should
I pay attention to?

Thanks,

ref:
https://salsa.debian.org/med-team/charls/-/blob/master/debian/copyright#L5-13