I've been having the same issue when xmodmap is executed from .xinitrc; it works fine if I run xmodmap manually after X11 is started. This problem appears to be similar to a few previously closed reports: #444025, #443044, #443473.