You can do that, but you need to use the exact same debian/changelog.
Otherwise your SOURCE_DATE_EPOCH will differ and that'll likely change
the binary. Cross building will not help you, because the cross
toolchain presently imposes build-id differences. You may use a
porterbox to build the package twice (with and without the patch but
without changing d/changelog). Alternatively, debusine.debian.net (I'm
being compensated for working on that) also has native arm builders and
you might upload packages there for comparison.