After checking, (I have removed all "weird" fonts dirs from my XF86Config-4 file), I have noticed X still crashes when applying the same process as described in my initial description. Therefore, I suppose it should be possible to reproduce this bug on a system with only debian packaged fonts. Vincent