Re: libav - FPE in libswscale/utils.c (initFilter)

[Hugo Lefeuvre <hle@debian.org>]

Hi Diego,
 
> I discovered the following floating point exception in libswscale/utils.c:
> 
> $ ./avconv -i fpe.jpg out.jpg
> avconv version 0.8.21, Copyright (c) 2000-2014 the Libav developers
>   built on Dec 12 2017 14:57:35 with gcc 7.2.1 20171205
> Input #0, image2, from '../samples/output.jpg':
>   Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
>     Stream #0.0: Video: mjpeg, yuvj444p, 658x61951, 25 tbr, 25 tbn, 25 tbc
> Incompatible pixel format 'yuvj444p' for codec 'mjpeg', auto-selecting format 'yuvj420p'
> [buffer @ 0x60a0000002a0] w:658 h:61951 pixfmt:yuvj444p
> [avsink @ 0x60a0000003e0] auto-inserting filter 'auto-inserted scaler 0' between the filter 'src' and the filter 'out'
> [scale @ 0x60a000000520] w:658 h:61951 fmt:yuvj444p -> w:658 h:61951 fmt:yuvj420p flags:0x4
> ASAN:DEADLYSIGNAL
> =================================================================
> ==21916==ERROR: AddressSanitizer: FPE on unknown address 0x561e62fa19cc (pc 0x561e62fa19cc bp 0x7fff06ec4300 sp 0x7fff06ec4130 T0)
>     #0 0x561e62fa19cb in initFilter libswscale/utils.c:504
>     #1 0x561e62fa6b72 in sws_init_context libswscale/utils.c:986
>     #2 0x561e62fa91c8 in sws_getContext libswscale/utils.c:1131
>     #3 0x561e61f8a6f0 in config_props libavfilter/vf_scale.c:215
>     #4 0x561e61f62cbc in avfilter_config_links libavfilter/avfilter.c:191
>     #5 0x561e61f67e2c in ff_avfilter_graph_config_links libavfilter/avfiltergraph.c:120
>     #6 0x561e61f69074 in avfilter_graph_config libavfilter/avfiltergraph.c:245
>     #7 0x561e61f23cb3 in configure_video_filters /home/hle/Development/C/libav/libav-0.8.21/avconv.c:617
>     #8 0x561e61f38641 in transcode_init /home/hle/Development/C/libav/libav-0.8.21/avconv.c:2475
>     #9 0x561e61f3a800 in transcode /home/hle/Development/C/libav/libav-0.8.21/avconv.c:2680
>     #10 0x561e61f4c1a7 in main /home/hle/Development/C/libav/libav-0.8.21/avconv.c:4551
>     #11 0x7fdf26779560 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x20560)
>     #12 0x561e61f20ca9 in _start (/home/hle/Development/C/libav/libav-0.8.21/avconv+0x17eca9)

ffmpeg has the following check to avoid this situation:

if (!sum) {
    av_log(NULL, AV_LOG_WARNING, "SwScaler: zero vector in scaling\n");
    sum = 1;
}

This check is not present in any libav version (why ?). Even after an hour
digging into swscale, it is still unclear to me whether libav > 0.8.21 is
affected or not. It would be great if you could have a look at it.

Do you consider this check to be a valid fix for this FPE ?

Regards,
 Hugo

-- 
             Hugo Lefeuvre (hle)    |    www.owl.eu.com
4096/ 9C4F C8BF A4B0 8FC5 48EB 56B8 1962 765B B9A8 BACA