This seems a bit heavy-handed on big-endian ppc64.
On this specific target, the only failing test is the AltiVec acceleration for lossless audio DSP.
The failure is annoyingly sporadic as well.
It might be more surgical to just remove libavcodec/ppc/lossless_audiodsp_altivec.c from the build (and it's correlating registration in libavcodec/lossless_audiodsp.c ). I'd be more than happy to prepare such a change and send a merge request if this is OK.
I can't speak for ppc64el as upstream currently lacks access to such equipment.