Конфиги DRBD и Pacemaker во вложении.
Split brain:
version: 8.3.11 (api:88/proto:86-96)
srcversion: F937DCB2E5D83C6CCE4A6C9
0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-----
ns:2796 nr:1832 dw:4628 dr:124530 al:6 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
1: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-----
ns:0 nr:0 dw:0 dr:784 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
10: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-----
ns:104696 nr:1136 dw:105832 dr:186265 al:82 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1
wo:f oos:0
Периодически всё работает нормально:
0:system_data Connected Primary/Primary UpToDate/UpToDate C r-----
1:vm_volumes Connected Primary/Primary UpToDate/UpToDate C r-----
10:repository SyncTarget Primary/Primary Inconsistent/UpToDate C r-----
[=========>..........] sync'ed: 54.2% (432504/933888)K
...
0:system_data Connected Primary/Primary UpToDate/UpToDate C r-----
/mnt/system ocfs2 90G 968M 90G 2%
1:vm_volumes Connected Primary/Primary UpToDate/UpToDate C r-----
10:repository Connected Primary/Primary UpToDate/UpToDate C r----- /mnt/repo
ocfs2 250G 82G 169G 33%
...
Master/Slave Set: ms_drbd_repo [p_drbd_repo]
Masters: [ cluster-data-1 cluster-data-2 ]
Master/Slave Set: ms_drbd_system_data [p_drbd_system_data]
Masters: [ cluster-data-1 cluster-data-2 ]
Master/Slave Set: ms_drbd_vm_volumes [p_drbd_vm_volumes]
Masters: [ cluster-data-1 cluster-data-2 ]
Clone Set: ce_ocfs2mgmt [g_ocfs2mgmt]
Started: [ cluster-data-1 cluster-data-2 ]
Clone Set: ce_mysql [p_mysql]
Started: [ cluster-data-1 cluster-data-2 ]
Clone Set: ce_system_fs [p_system_fs]
Started: [ cluster-data-1 cluster-data-2 ]
Clone Set: ce_rabbitmq [p_rabbitmq]
Started: [ cluster-data-1 cluster-data-2 ]
Clone Set: ce_repo_fs [p_repo_fs]
Started: [ cluster-data-1 cluster-data-2 ]
Clone Set: ce_data_ip [p_data_ip]
Started: [ cluster-data-1 cluster-data-2 ]
Clone Set: ce_webserver [p_webserver]
Started: [ cluster-data-1 cluster-data-2 ]
В Primary/Primary заработало после перевода в Primay/Secondary и обратно:
master-max у ms_drbd_system_data был 1.
root@cluster-data-1:~# drbd-overview
0:system_data StandAlone Primary/Unknown UpToDate/DUnknown r-----
1:vm_volumes SyncTarget Primary/Primary Inconsistent/UpToDate C r-----
[>....................] sync'ed: 1.7% (729260/741688)Mfinish: 1:44:23
speed: 119,200 (112,632) want: 1,000,001 K/sec
10:repository Connected Primary/Primary UpToDate/UpToDate C r-----
root@cluster-data-1:~# drbd-overview
^C
root@cluster-data-1:~# crm resource stop ms_drbd_system_data
root@cluster-data-1:~# crm configure edit ms_drbd_system_data
root@cluster-data-1:~# drbd-overview
0:system_data StandAlone Primary/Unknown UpToDate/DUnknown r-----
1:vm_volumes SyncTarget Primary/Primary Inconsistent/UpToDate C r-----
[>....................] sync'ed: 3.0% (719600/741688)Mfinish: 1:46:58
speed: 114,784 (113,096) want: 1,000,001 K/sec
10:repository Connected Primary/Primary UpToDate/UpToDate C r-----
root@cluster-data-1:~# drbd-overview
0:system_data StandAlone Primary/Unknown UpToDate/DUnknown r-----
1:vm_volumes SyncTarget Primary/Primary Inconsistent/UpToDate C r-----
[>....................] sync'ed: 3.3% (717228/741688)Mfinish: 1:45:44
speed: 115,744 (113,340) want: 1,000,001 K/sec
10:repository Connected Primary/Primary UpToDate/UpToDate C r-----
root@cluster-data-1:~# crm resource manage ms_drbd_system_data
root@cluster-data-1:~# drbd-overview
0:system_data Unconfigured . . . .
1:vm_volumes SyncTarget Primary/Primary Inconsistent/UpToDate C r-----
[>....................] sync'ed: 4.2% (710848/741688)Mfinish: 1:45:17
speed: 115,212 (113,600) want: 1,000,001 K/sec
10:repository Connected Primary/Primary UpToDate/UpToDate C r-----
root@cluster-data-1:~# crm resource start ms_drbd_system_data
root@cluster-data-1:~# drbd-overview
0:system_data WFBitMapS Primary/Secondary UpToDate/Consistent C r-----
1:vm_volumes SyncTarget Primary/Primary Inconsistent/UpToDate C r-----
[>....................] sync'ed: 4.8% (706760/741688)Mfinish: 1:41:44
speed: 118,540 (113,908) want: 1,000,001 K/sec
10:repository Connected Primary/Primary UpToDate/UpToDate C r-----
root@cluster-data-1:~# drbd-overview
0:system_data Connected Primary/Secondary UpToDate/UpToDate C r-----
1:vm_volumes SyncTarget Primary/Primary Inconsistent/UpToDate C r-----
[>...................] sync'ed: 5.2% (703360/741688)Mfinish: 1:44:21
speed: 115,008 (113,764) want: 1,000,001 K/sec
10:repository Connected Primary/Primary UpToDate/UpToDate C r-----
root@cluster-data-1:~# crm configure edit ms_drbd_system_data
root@cluster-data-1:~# crm configure show ms_drbd_system_data
ms ms_drbd_system_data p_drbd_system_data \
meta notify="true" clone-max="2" master-max="2" target-role="Started"
is-managed="true"
root@cluster-data-1:~# drbd-overview
0:system_data Connected Primary/Primary UpToDate/UpToDate C r-----
1:vm_volumes SyncTarget Primary/Primary Inconsistent/UpToDate C r-----
[>...................] sync'ed: 7.0% (690308/741688)Mfinish: 1:40:21
speed: 117,388 (114,128) want: 1,000,001 K/sec
10:repository Connected Primary/Primary UpToDate/UpToDate C r-----
Иногда получается так, что вообще не останавливается ресурс:
root@cluster-data-1:~# cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: F937DCB2E5D83C6CCE4A6C9
0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-----
ns:0 nr:0 dw:600 dr:263205 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:696
1: cs:SyncTarget ro:Primary/Primary ds:Inconsistent/UpToDate C r-----
ns:0 nr:5327932 dw:5327616 dr:32 al:0 bm:325 lo:2 pe:7494 ua:2 ap:1 ep:1
wo:f oos:853695940
[>....................] sync'ed: 0.7% (833684/838888)Mfinish: 2:03:44
speed: 114,984 (108,724) want: 1,000,001 K/sec
10: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
ns:84 nr:12 dw:44 dr:263073 al:0 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
root@cluster-data-1:~# drbd-overview
0:system_data StandAlone Primary/Unknown UpToDate/DUnknown r-----
/mnt/system ocfs2 90G 862M 90G 1%
1:vm_volumes SyncTarget Primary/Primary Inconsistent/UpToDate C r-----
[>....................] sync'ed: 0.7% (833252/838888)Mfinish: 1:58:12
speed: 120,292 (108,924) want: 1,000,001 K/sec
10:repository Connected Primary/Primary UpToDate/UpToDate C r-----
/mnt/repo ocfs2 250G 82G 169G 33%
root@cluster-data-1:~# drbd-overview
0:system_data StandAlone Primary/Unknown UpToDate/DUnknown r-----
/mnt/system ocfs2 90G 862M 90G 1%
1:vm_volumes SyncTarget Primary/Primary Inconsistent/UpToDate C r-----
[>....................] sync'ed: 1.0% (830752/838888)Mfinish: 2:01:35
speed: 116,584 (109,628) want: 1,000,001 K/sec
10:repository Connected Primary/Primary UpToDate/UpToDate C r-----
/mnt/repo ocfs2 250G 82G 169G 33%
root@cluster-data-1:~# crm resource restart ms_drbd_system_data
INFO: ordering ms_drbd_system_data to stop
waiting for stop to finish
......................................................................................................................................................................................................................................................................................................................................................................................
done
INFO: ordering ms_drbd_system_data to start
root@cluster-data-1:~# drbd-overview
0:system_data StandAlone Primary/Unknown UpToDate/DUnknown r-----
/mnt/system ocfs2 90G 862M 90G 1%
1:vm_volumes SyncTarget Primary/Primary Inconsistent/UpToDate C r-----
[>....................] sync'ed: 4.3% (803584/838888)Mfinish: 1:58:20
speed: 115,884 (112,964) want: 1,000,001 K/sec
10:repository Connected Primary/Primary UpToDate/UpToDate C r-----
/mnt/repo ocfs2 250G 82G 169G 33%
root@cluster-data-1:~# crm configure show ms_drbd_system_data
ms ms_drbd_system_data p_drbd_system_data \
meta notify="true" clone-max="2" master-max="2" target-role="Started"
root@cluster-data-1:~# crm configure edit ms_drbd_system_data
root@cluster-data-1:~# crm configure show ms_drbd_system_data
ms ms_drbd_system_data p_drbd_system_data \
meta notify="true" clone-max="2" master-max="1" target-role="Started"
root@cluster-data-1:~# crm resource restart ms_drbd_system_data
INFO: ordering ms_drbd_system_data to stop
waiting for stop to finish
.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^CCtrl-C,
leaving
root@cluster-data-1:~# drbd-overview
0:system_data StandAlone Primary/Unknown UpToDate/DUnknown r-----
/mnt/system ocfs2 90G 862M 90G 1%
1:vm_volumes SyncTarget Primary/Primary Inconsistent/UpToDate C r-----
[=>..................] sync'ed: 10.2% (754088/838888)Mfinish: 1:52:40
speed: 114,200 (113,804) want: 1,000,001 K/sec
10:repository Connected Primary/Primary UpToDate/UpToDate C r-----
/mnt/repo ocfs2 250G 82G 169G 33%
root@cluster-data-1:~# crm resource stop ms_drbd_system_data
root@cluster-data-1:~# drbd-overview
0:system_data StandAlone Primary/Unknown UpToDate/DUnknown r-----
/mnt/system ocfs2 90G 862M 90G 1%
1:vm_volumes SyncTarget Primary/Primary Inconsistent/UpToDate C r-----
[=>..................] sync'ed: 10.5% (750956/838888)Mfinish: 1:51:34
speed: 114,840 (113,688) want: 1,000,001 K/sec
10:repository Connected Primary/Primary UpToDate/UpToDate C r-----
/mnt/repo ocfs2 250G 82G 169G 33%
root@cluster-data-1:~# crm resource stop ms_drbd_system_data
root@cluster-data-1:~# drbd-overview
0:system_data StandAlone Primary/Unknown UpToDate/DUnknown r-----
/mnt/system ocfs2 90G 862M 90G 1%
1:vm_volumes SyncTarget Primary/Primary Inconsistent/UpToDate C r-----
[=>..................] sync'ed: 10.9% (748088/838888)Mfinish: 1:46:49
speed: 119,500 (113,804) want: 1,000,001 K/sec
10:repository Connected Primary/Primary UpToDate/UpToDate C r-----
/mnt/repo ocfs2 250G 82G 169G 33%