Running xmodmap breaks keyboard layouts

[George <pinkisntwell@gmail.com>]

I added the following lines into my .xmodmaprc and I call xmodmap at
startup to configure my keys how I want them:

remove Lock = Caps_Lock
remove Mod4 = Super_L
remove Mod1 = Alt_L
remove Control = Control_L
add Lock = Super_L
add Lock = Control_L
add Control = Caps_Lock
add Mod4 = Alt_L

However now my keyboard layout change no longer works. It's configured
through those lines in /etc/X11/xorg.conf:

Section "InputDevice"
        Identifier      "Generic Keyboard"
        Driver          "kbd"
        Option          "XkbRules"      "xorg"
        Option          "XkbModel"      "pc104"
        Option          "XkbLayout"     "us,el"
        Option          "XkbOptions"
"grp:alt_shift_toggle,ctrl_shift_toggle,eurosign:e"
EndSection

How can I fix this?