Cluster mit Pacemaker und OCFS2
Hallo allerseits,
ich hänge z.Zt. an einer Probeimplementierung eines Clusters mit Shared Storage und Clusterfilesystem.
Hier meine Konfiguration:
Drei virtuelle Nodes: cln07, cln08, cln09
jeweils:
- ein Netzwerkinterface (eth0)
- ein lokales Volume als Rootdevice (/dev/vda)
- ein an alle drei Nodes gemaptes Volume (/dev/vdb)
- Debian 6.0.2 mit aktuellen Patchlevel
- passwortfreier SSH-Zugriff auf die anderen Nodes ist möglich
- Clusterspezifisch installierte Software:
* pacemaker
* corosync
* openais
* dlm-pcmk
* ocfs2-tools
* ocfs2-tools-pacemaker
Clusterconfig:
node cln07
node cln08
node cln09 \
attributes standby="off"
primitive resDLM ocf:pacemaker:controld \
operations $id="resDLM-operations" \
op start interval="0" timeout="90" \
op stop interval="0" timeout="100" \
op monitor interval="120" \
meta target-role="started"
primitive resO2CB ocf:pacemaker:o2cb \
meta target-role="started"
primitive stonith stonith:external/ssh \
params hostlist="cln07 cln08 cln09" \
operations $id="stonith-operations" \
op monitor interval="15" timeout="15" start-delay="0"
clone cloneDLM resDLM \
meta clone-max="3" interleave="true"
clone cloneO2CB resO2CB \
meta clone-max="3" interleave="true"
clone stonith_clone stonith \
meta clone-max="3" target-role="started"
colocation colO2CBDLM inf: cloneO2CB cloneDLM
order ordDLMO2CB 0: cloneDLM cloneO2CB
property $id="cib-bootstrap-options" \
dc-version="1.0.9-74392a28b7f31d7ddc86689598bd23114f58978b" \
cluster-infrastructure="openais" \
expected-quorum-votes="3" \
no-quorum-policy="ignore" \
stonith-enabled="true" \
default-resource-stickiness="1000" \
last-lrm-refresh="1314276966" \
stonith-timeout="30s"
Ich habe mich überwiegend an das Testszenario von Ubuntu-HA gehalten:
https://wiki.ubuntu.com/ClusterStack/LucidTesting#Pacemaker.2C_drbd8_and_OCFS2_or_GFS2
Das Cluster erzeugt immer beim Start der pacemaker:o2cb Ressource gibt es immer einen Fehler und
geht auf Störung
Aug 25 14:09:00 cln07 o2cb[5362]: ERROR: ocfs2_controld.pcmk did not come up
Hier nochmal ein kompletter daemon.log Mitschnitt beim Starten der Ressource:
Aug 25 15:47:56 cln07 crm_resource: [7754]: info: Invoked: /usr/sbin/crm_resource -C -r cloneO2CB -H
cln07
Aug 25 15:47:56 cln07 attrd: [1659]: info: attrd_trigger_update: Sending flush op to all hosts for:
fail-count-resO2CB:0 (<null>)
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_lrm_invoke: Removing resource resO2CB:0 from the LRM
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_lrm_invoke: Resource 'resO2CB:0' deleted for
7754_crm_resource on cln07
Aug 25 15:47:56 cln07 crmd: [1661]: info: notify_deleted: Notifying 7754_crm_resource on cln07 that
resO2CB:0 was deleted
Aug 25 15:47:56 cln07 attrd: [1659]: info: attrd_perform_update: Sent delete 40: node=cln07,
attr=fail-count-resO2CB:0, id=<n/a>, set=(null), section=status
Aug 25 15:47:56 cln07 crmd: [1661]: info: send_direct_ack: ACK'ing resource op
resO2CB:0_delete_60000 from 0:0:crm-resource-7754: lrm_invoke-lrmd-1314280076-76
Aug 25 15:47:56 cln07 cib: [1657]: info: cib_process_request: Operation complete: op cib_delete for
section //node_state[@uname='cln07']//lrm_resource[@id='resO2CB:0'] (origin=local/crmd/68,
version=0.127.18): ok (rc=0)
Aug 25 15:47:56 cln07 crmd: [1661]: info: abort_transition_graph: te_update_diff:267 - Triggered
transition abort (complete=1, tag=lrm_rsc_op, id=resO2CB:0_monitor_0,
magic=0:7;10:6:7:6221d8f2-6b8e-484b-8765-cfeddc1b5ae7, cib=0.127.18) : Resource op removal
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_state_transition: State transition S_IDLE ->
S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_state_transition: All 3 cluster nodes are eligible to
run resources.
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_pe_invoke: Query 71: Requesting the current CIB:
S_POLICY_ENGINE
Aug 25 15:47:56 cln07 crmd: [1661]: notice: do_lrm_invoke: Not creating resource for a delete event:
(null)
Aug 25 15:47:56 cln07 crmd: [1661]: info: send_direct_ack: ACK'ing resource op
resO2CB:1_delete_60000 from 0:0:crm-resource-7754: lrm_invoke-lrmd-1314280076-77
Aug 25 15:47:56 cln07 crmd: [1661]: notice: do_lrm_invoke: Not creating resource for a delete event:
(null)
Aug 25 15:47:56 cln07 crmd: [1661]: info: send_direct_ack: ACK'ing resource op
resO2CB:2_delete_60000 from 0:0:crm-resource-7754: lrm_invoke-lrmd-1314280076-78
Aug 25 15:47:56 cln07 crmd: [1661]: info: abort_transition_graph: te_update_diff:157 - Triggered
transition abort (complete=1, tag=transient_attributes, id=cln07, magic=NA, cib=0.127.19) :
Transient attribute: removal
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_pe_invoke: Query 72: Requesting the current CIB:
S_POLICY_ENGINE
Aug 25 15:47:56 cln07 attrd: [1659]: info: attrd_perform_update: Sent delete -22: node=cln07,
attr=fail-count-resO2CB:0, id=<n/a>, set=(null), section=status
Aug 25 15:47:56 cln07 cib: [1657]: info: log_data_element: cib:diff: - <cib admin_epoch="0"
epoch="127" num_updates="19" >
Aug 25 15:47:56 cln07 cib: [1657]: info: log_data_element: cib:diff: - <configuration >
Aug 25 15:47:56 cln07 cib: [1657]: info: log_data_element: cib:diff: - <crm_config >
Aug 25 15:47:56 cln07 cib: [1657]: info: log_data_element: cib:diff: - <cluster_property_set
id="cib-bootstrap-options" >
Aug 25 15:47:56 cln07 cib: [1657]: info: log_data_element: cib:diff: - <nvpair value="1314280014"
id="cib-bootstrap-options-last-lrm-refresh" />
Aug 25 15:47:56 cln07 cib: [1657]: info: log_data_element: cib:diff: - </cluster_property_set>
Aug 25 15:47:56 cln07 cib: [1657]: info: log_data_element: cib:diff: - </crm_config>
Aug 25 15:47:56 cln07 cib: [1657]: info: log_data_element: cib:diff: - </configuration>
Aug 25 15:47:56 cln07 cib: [1657]: info: log_data_element: cib:diff: - </cib>
Aug 25 15:47:56 cln07 cib: [1657]: info: log_data_element: cib:diff: + <cib admin_epoch="0"
epoch="128" num_updates="1" >
Aug 25 15:47:56 cln07 cib: [1657]: info: log_data_element: cib:diff: + <configuration >
Aug 25 15:47:56 cln07 cib: [1657]: info: log_data_element: cib:diff: + <crm_config >
Aug 25 15:47:56 cln07 crmd: [1661]: info: abort_transition_graph: need_abort:59 - Triggered
transition abort (complete=1) : Non-status change
Aug 25 15:47:56 cln07 crmd: [1661]: info: need_abort: Aborting on change to admin_epoch
Aug 25 15:47:56 cln07 cib: [1657]: info: log_data_element: cib:diff: + <cluster_property_set
id="cib-bootstrap-options" >
Aug 25 15:47:56 cln07 cib: [1657]: info: log_data_element: cib:diff: + <nvpair value="1314280076"
id="cib-bootstrap-options-last-lrm-refresh" />
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_pe_invoke: Query 73: Requesting the current CIB:
S_POLICY_ENGINE
Aug 25 15:47:56 cln07 cib: [1657]: info: log_data_element: cib:diff: + </cluster_property_set>
Aug 25 15:47:56 cln07 cib: [1657]: info: log_data_element: cib:diff: + </crm_config>
Aug 25 15:47:56 cln07 cib: [1657]: info: log_data_element: cib:diff: + </configuration>
Aug 25 15:47:56 cln07 cib: [1657]: info: log_data_element: cib:diff: + </cib>
Aug 25 15:47:56 cln07 cib: [1657]: info: cib_process_request: Operation complete: op cib_modify for
section crm_config (origin=local/crmd/70, version=0.128.1): ok (rc=0)
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_pe_invoke_callback: Invoking the PE: query=73,
ref=pe_calc-dc-1314280076-79, seq=172, quorate=1
Aug 25 15:47:56 cln07 crmd: [1661]: info: config_query_callback: Checking for expired actions every
900000ms
Aug 25 15:47:56 cln07 crmd: [1661]: info: config_query_callback: Sending expected-votes=3 to corosync
Aug 25 15:47:56 cln07 crmd: [1661]: info: ais_dispatch: Membership 172: quorum retained
Aug 25 15:47:56 cln07 crmd: [1661]: info: crm_ais_dispatch: Setting expected votes to 3
Aug 25 15:47:56 cln07 pengine: [1660]: notice: unpack_config: On loss of CCM Quorum: Ignore
Aug 25 15:47:56 cln07 pengine: [1660]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow'
= 0, 'green' = 0
Aug 25 15:47:56 cln07 pengine: [1660]: info: determine_online_status: Node cln08 is online
Aug 25 15:47:56 cln07 pengine: [1660]: WARN: unpack_rsc_op: Processing failed op resO2CB:1_start_0
on cln08: unknown error (1)
Aug 25 15:47:56 cln07 pengine: [1660]: info: determine_online_status: Node cln09 is online
Aug 25 15:47:56 cln07 pengine: [1660]: WARN: unpack_rsc_op: Processing failed op resO2CB:2_start_0
on cln09: unknown error (1)
Aug 25 15:47:56 cln07 pengine: [1660]: info: determine_online_status: Node cln07 is online
Aug 25 15:47:56 cln07 pengine: [1660]: notice: clone_print: Clone Set: cloneDLM
Aug 25 15:47:56 cln07 pengine: [1660]: notice: short_print: Started: [ cln07 cln08 cln09 ]
Aug 25 15:47:56 cln07 pengine: [1660]: notice: clone_print: Clone Set: cloneO2CB
Aug 25 15:47:56 cln07 pengine: [1660]: notice: short_print: Stopped: [ resO2CB:0 resO2CB:1
resO2CB:2 ]
Aug 25 15:47:56 cln07 pengine: [1660]: notice: clone_print: Clone Set: stonith_clone
Aug 25 15:47:56 cln07 pengine: [1660]: notice: short_print: Started: [ cln07 cln08 cln09 ]
Aug 25 15:47:56 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln08
Aug 25 15:47:56 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln08 after 1000000 failures (max=1000000)
Aug 25 15:47:56 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln08
Aug 25 15:47:56 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln08 after 1000000 failures (max=1000000)
Aug 25 15:47:56 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln08
Aug 25 15:47:56 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln08 after 1000000 failures (max=1000000)
Aug 25 15:47:56 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln09
Aug 25 15:47:56 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln09 after 1000000 failures (max=1000000)
Aug 25 15:47:56 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln09
Aug 25 15:47:56 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln09 after 1000000 failures (max=1000000)
Aug 25 15:47:56 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln09
Aug 25 15:47:56 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln09 after 1000000 failures (max=1000000)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: clone_rsc_colocation_rh: Cannot pair resO2CB:1 with
instance of cloneDLM
Aug 25 15:47:56 cln07 pengine: [1660]: info: native_color: Resource resO2CB:1 cannot run anywhere
Aug 25 15:47:56 cln07 pengine: [1660]: notice: clone_rsc_colocation_rh: Cannot pair resO2CB:2 with
instance of cloneDLM
Aug 25 15:47:56 cln07 pengine: [1660]: info: native_color: Resource resO2CB:2 cannot run anywhere
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource resDLM:0#011(Started cln07)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource resDLM:1#011(Started cln08)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource resDLM:2#011(Started cln09)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Start resO2CB:0#011(cln07)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource resO2CB:1#011(Stopped)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource resO2CB:2#011(Stopped)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource stonith:0#011(Started cln07)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource stonith:1#011(Started cln08)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource stonith:2#011(Started cln09)
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_state_transition: State transition S_POLICY_ENGINE ->
S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Aug 25 15:47:56 cln07 crmd: [1661]: info: unpack_graph: Unpacked transition 10: 7 actions in 7 synapses
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_te_invoke: Processing graph 10
(ref=pe_calc-dc-1314280076-79) derived from /var/lib/pengine/pe-input-386.bz2
Aug 25 15:47:56 cln07 crmd: [1661]: info: te_rsc_command: Initiating action 10: monitor
resO2CB:0_monitor_0 on cln07 (local)
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_lrm_rsc_op: Performing
key=10:10:7:6221d8f2-6b8e-484b-8765-cfeddc1b5ae7 op=resO2CB:0_monitor_0 )
Aug 25 15:47:56 cln07 cib: [7756]: info: write_cib_contents: Archived previous version as
/var/lib/heartbeat/crm/cib-28.raw
Aug 25 15:47:56 cln07 lrmd: [1658]: info: rsc:resO2CB:0:14: probe
Aug 25 15:47:56 cln07 pengine: [1660]: info: process_pe_message: Transition 10: PEngine Input stored
in: /var/lib/pengine/pe-input-386.bz2
Aug 25 15:47:56 cln07 cib: [1657]: info: cib_process_request: Operation complete: op cib_modify for
section crm_config (origin=local/crmd/76, version=0.128.1): ok (rc=0)
Aug 25 15:47:56 cln07 cib: [7756]: info: write_cib_contents: Wrote version 0.128.0 of the CIB to
disk (digest: eedbb62e444e487068fe924f186d92a3)
Aug 25 15:47:56 cln07 cib: [7756]: info: retrieveCib: Reading cluster configuration from:
/var/lib/heartbeat/crm/cib.rKtwa0 (digest: /var/lib/heartbeat/crm/cib.PHd8sv)
Aug 25 15:47:56 cln07 crm_resource: [7761]: info: Invoked: /usr/sbin/crm_resource -C -r cloneO2CB -H
cln08
Aug 25 15:47:56 cln07 cib: [1657]: info: cib_process_request: Operation complete: op cib_delete for
section //node_state[@uname='cln08']//lrm_resource[@id='resO2CB:1'] (origin=cln08/crmd/26,
version=0.128.2): ok (rc=0)
Aug 25 15:47:56 cln07 crmd: [1661]: info: abort_transition_graph: te_update_diff:267 - Triggered
transition abort (complete=0, tag=lrm_rsc_op, id=resO2CB:1_monitor_0,
magic=0:7;11:7:7:6221d8f2-6b8e-484b-8765-cfeddc1b5ae7, cib=0.128.2) : Resource op removal
Aug 25 15:47:56 cln07 crmd: [1661]: info: update_abort_priority: Abort priority upgraded from 0 to
1000000
Aug 25 15:47:56 cln07 crmd: [1661]: info: update_abort_priority: Abort action done superceeded by
restart
Aug 25 15:47:56 cln07 crmd: [1661]: info: abort_transition_graph: te_update_diff:157 - Triggered
transition abort (complete=0, tag=transient_attributes, id=cln08, magic=NA, cib=0.128.3) : Transient
attribute: removal
Aug 25 15:47:56 cln07 cib: [1657]: info: cib_process_request: Operation complete: op cib_modify for
section crm_config (origin=cln08/crmd/28, version=0.128.3): ok (rc=0)
Aug 25 15:47:56 cln07 cibadmin: [7766]: info: Invoked: /usr/sbin/cibadmin -Ql
Aug 25 15:47:56 cln07 crm_resource: [7782]: info: Invoked: /usr/sbin/crm_resource -C -r cloneO2CB -H
cln09
Aug 25 15:47:56 cln07 cibadmin: [7784]: info: Invoked: /usr/sbin/cibadmin -Ql
Aug 25 15:47:56 cln07 crmd: [1661]: info: process_lrm_event: LRM operation resO2CB:0_monitor_0
(call=14, rc=7, cib-update=77, confirmed=true) not running
Aug 25 15:47:56 cln07 crmd: [1661]: info: match_graph_event: Action resO2CB:0_monitor_0 (10)
confirmed on cln07 (rc=0)
Aug 25 15:47:56 cln07 crmd: [1661]: info: te_rsc_command: Initiating action 9: probe_complete
probe_complete on cln07 (local) - no waiting
Aug 25 15:47:56 cln07 crmd: [1661]: info: run_graph:
====================================================
Aug 25 15:47:56 cln07 crmd: [1661]: notice: run_graph: Transition 10 (Complete=2, Pending=0,
Fired=0, Skipped=4, Incomplete=1, Source=/var/lib/pengine/pe-input-386.bz2): Stopped
Aug 25 15:47:56 cln07 crmd: [1661]: info: te_graph_trigger: Transition 10 is now complete
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_state_transition: State transition S_TRANSITION_ENGINE
-> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_state_transition: All 3 cluster nodes are eligible to
run resources.
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_pe_invoke: Query 78: Requesting the current CIB:
S_POLICY_ENGINE
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_pe_invoke_callback: Invoking the PE: query=78,
ref=pe_calc-dc-1314280076-83, seq=172, quorate=1
Aug 25 15:47:56 cln07 pengine: [1660]: notice: unpack_config: On loss of CCM Quorum: Ignore
Aug 25 15:47:56 cln07 pengine: [1660]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow'
= 0, 'green' = 0
Aug 25 15:47:56 cln07 pengine: [1660]: info: determine_online_status: Node cln08 is online
Aug 25 15:47:56 cln07 pengine: [1660]: info: determine_online_status: Node cln09 is online
Aug 25 15:47:56 cln07 pengine: [1660]: WARN: unpack_rsc_op: Processing failed op resO2CB:2_start_0
on cln09: unknown error (1)
Aug 25 15:47:56 cln07 pengine: [1660]: info: determine_online_status: Node cln07 is online
Aug 25 15:47:56 cln07 pengine: [1660]: notice: clone_print: Clone Set: cloneDLM
Aug 25 15:47:56 cln07 pengine: [1660]: notice: short_print: Started: [ cln07 cln08 cln09 ]
Aug 25 15:47:56 cln07 pengine: [1660]: notice: clone_print: Clone Set: cloneO2CB
Aug 25 15:47:56 cln07 pengine: [1660]: notice: short_print: Stopped: [ resO2CB:0 resO2CB:1
resO2CB:2 ]
Aug 25 15:47:56 cln07 pengine: [1660]: notice: clone_print: Clone Set: stonith_clone
Aug 25 15:47:56 cln07 crmd: [1661]: info: abort_transition_graph: te_update_diff:157 - Triggered
transition abort (complete=1, tag=transient_attributes, id=cln09, magic=NA, cib=0.128.5) : Transient
attribute: removal
Aug 25 15:47:56 cln07 pengine: [1660]: notice: short_print: Started: [ cln07 cln08 cln09 ]
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_pe_invoke: Query 79: Requesting the current CIB:
S_POLICY_ENGINE
Aug 25 15:47:56 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln09
Aug 25 15:47:56 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln09 after 1000000 failures (max=1000000)
Aug 25 15:47:56 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln09
Aug 25 15:47:56 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln09 after 1000000 failures (max=1000000)
Aug 25 15:47:56 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln09
Aug 25 15:47:56 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln09 after 1000000 failures (max=1000000)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: clone_rsc_colocation_rh: Cannot pair resO2CB:2 with
instance of cloneDLM
Aug 25 15:47:56 cln07 pengine: [1660]: info: native_color: Resource resO2CB:2 cannot run anywhere
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource resDLM:0#011(Started cln07)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource resDLM:1#011(Started cln08)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource resDLM:2#011(Started cln09)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Start resO2CB:0#011(cln07)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Start resO2CB:1#011(cln08)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource resO2CB:2#011(Stopped)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource stonith:0#011(Started cln07)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource stonith:1#011(Started cln08)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource stonith:2#011(Started cln09)
Aug 25 15:47:56 cln07 crmd: [1661]: info: handle_response: pe_calc calculation
pe_calc-dc-1314280076-83 is obsolete
Aug 25 15:47:56 cln07 cib: [1657]: info: cib_process_request: Operation complete: op cib_delete for
section //node_state[@uname='cln09']//lrm_resource[@id='resO2CB:2'] (origin=cln09/crmd/26,
version=0.128.6): ok (rc=0)
Aug 25 15:47:56 cln07 crmd: [1661]: info: abort_transition_graph: te_update_diff:267 - Triggered
transition abort (complete=1, tag=lrm_rsc_op, id=resO2CB:2_monitor_0,
magic=0:7;13:7:7:6221d8f2-6b8e-484b-8765-cfeddc1b5ae7, cib=0.128.6) : Resource op removal
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_pe_invoke: Query 80: Requesting the current CIB:
S_POLICY_ENGINE
Aug 25 15:47:56 cln07 pengine: [1660]: info: process_pe_message: Transition 11: PEngine Input stored
in: /var/lib/pengine/pe-input-387.bz2
Aug 25 15:47:56 cln07 cib: [1657]: info: cib_process_request: Operation complete: op cib_modify for
section crm_config (origin=cln09/crmd/28, version=0.128.6): ok (rc=0)
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_pe_invoke_callback: Invoking the PE: query=80,
ref=pe_calc-dc-1314280076-85, seq=172, quorate=1
Aug 25 15:47:56 cln07 pengine: [1660]: notice: unpack_config: On loss of CCM Quorum: Ignore
Aug 25 15:47:56 cln07 pengine: [1660]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow'
= 0, 'green' = 0
Aug 25 15:47:56 cln07 pengine: [1660]: info: determine_online_status: Node cln08 is online
Aug 25 15:47:56 cln07 pengine: [1660]: info: determine_online_status: Node cln09 is online
Aug 25 15:47:56 cln07 pengine: [1660]: info: determine_online_status: Node cln07 is online
Aug 25 15:47:56 cln07 pengine: [1660]: notice: clone_print: Clone Set: cloneDLM
Aug 25 15:47:56 cln07 pengine: [1660]: notice: short_print: Started: [ cln07 cln08 cln09 ]
Aug 25 15:47:56 cln07 pengine: [1660]: notice: clone_print: Clone Set: cloneO2CB
Aug 25 15:47:56 cln07 pengine: [1660]: notice: short_print: Stopped: [ resO2CB:0 resO2CB:1
resO2CB:2 ]
Aug 25 15:47:56 cln07 pengine: [1660]: notice: clone_print: Clone Set: stonith_clone
Aug 25 15:47:56 cln07 pengine: [1660]: notice: short_print: Started: [ cln07 cln08 cln09 ]
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource resDLM:0#011(Started cln07)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource resDLM:1#011(Started cln08)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource resDLM:2#011(Started cln09)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Start resO2CB:0#011(cln07)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Start resO2CB:1#011(cln08)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Start resO2CB:2#011(cln09)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource stonith:0#011(Started cln07)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource stonith:1#011(Started cln08)
Aug 25 15:47:56 cln07 pengine: [1660]: notice: LogActions: Leave resource stonith:2#011(Started cln09)
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_state_transition: State transition S_POLICY_ENGINE ->
S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Aug 25 15:47:56 cln07 crmd: [1661]: info: unpack_graph: Unpacked transition 12: 11 actions in 11
synapses
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_te_invoke: Processing graph 12
(ref=pe_calc-dc-1314280076-85) derived from /var/lib/pengine/pe-input-388.bz2
Aug 25 15:47:56 cln07 crmd: [1661]: info: te_rsc_command: Initiating action 11: monitor
resO2CB:1_monitor_0 on cln08
Aug 25 15:47:56 cln07 crmd: [1661]: info: te_rsc_command: Initiating action 13: monitor
resO2CB:2_monitor_0 on cln09
Aug 25 15:47:56 cln07 pengine: [1660]: info: process_pe_message: Transition 12: PEngine Input stored
in: /var/lib/pengine/pe-input-388.bz2
Aug 25 15:47:56 cln07 cibadmin: [7787]: info: Invoked: /usr/sbin/cibadmin -Ql
Aug 25 15:47:56 cln07 crmd: [1661]: info: match_graph_event: Action resO2CB:1_monitor_0 (11)
confirmed on cln08 (rc=0)
Aug 25 15:47:56 cln07 crmd: [1661]: info: te_rsc_command: Initiating action 10: probe_complete
probe_complete on cln08 - no waiting
Aug 25 15:47:56 cln07 crmd: [1661]: info: match_graph_event: Action resO2CB:2_monitor_0 (13)
confirmed on cln09 (rc=0)
Aug 25 15:47:56 cln07 crmd: [1661]: info: te_rsc_command: Initiating action 12: probe_complete
probe_complete on cln09 - no waiting
Aug 25 15:47:56 cln07 crmd: [1661]: info: te_pseudo_action: Pseudo action 8 fired and confirmed
Aug 25 15:47:56 cln07 crmd: [1661]: info: te_pseudo_action: Pseudo action 29 fired and confirmed
Aug 25 15:47:56 cln07 crmd: [1661]: info: te_pseudo_action: Pseudo action 27 fired and confirmed
Aug 25 15:47:56 cln07 crmd: [1661]: info: te_rsc_command: Initiating action 24: start
resO2CB:0_start_0 on cln07 (local)
Aug 25 15:47:56 cln07 crmd: [1661]: info: do_lrm_rsc_op: Performing
key=24:12:0:6221d8f2-6b8e-484b-8765-cfeddc1b5ae7 op=resO2CB:0_start_0 )
Aug 25 15:47:56 cln07 lrmd: [1658]: info: rsc:resO2CB:0:15: start
Aug 25 15:47:57 cln07 crmd: [1661]: info: te_rsc_command: Initiating action 25: start
resO2CB:1_start_0 on cln08
Aug 25 15:47:57 cln07 crmd: [1661]: info: te_rsc_command: Initiating action 26: start
resO2CB:2_start_0 on cln09
Aug 25 15:47:57 cln07 cibadmin: [7796]: info: Invoked: /usr/sbin/cibadmin -Ql
Aug 25 15:47:57 cln07 cibadmin: [7812]: info: Invoked: /usr/sbin/cibadmin -Ql
Aug 25 15:47:57 cln07 o2cb[7789]: INFO: Starting resO2CB:0
Aug 25 15:47:57 cln07 corosync[1633]: [pcmk ] info: pcmk_notify: Enabling node notifications for
child 7824 (0x9dfd730)
Aug 25 15:47:57 cln07 ocfs2_controld[7824]: Unable to connect to CKPT: Object does not exist
Aug 25 15:48:09 cln07 crmd: [1661]: WARN: status_from_rc: Action 26 (resO2CB:2_start_0) on cln09
failed (target: 0 vs. rc: 1): Error
Aug 25 15:48:09 cln07 crmd: [1661]: WARN: update_failcount: Updating failcount for resO2CB:2 on
cln09 after failed start: rc=1 (update=INFINITY, time=1314280089)
Aug 25 15:48:09 cln07 crmd: [1661]: info: abort_transition_graph: match_graph_event:272 - Triggered
transition abort (complete=0, tag=lrm_rsc_op, id=resO2CB:2_start_0,
magic=0:1;26:12:0:6221d8f2-6b8e-484b-8765-cfeddc1b5ae7, cib=0.128.9) : Event failed
Aug 25 15:48:09 cln07 crmd: [1661]: info: update_abort_priority: Abort priority upgraded from 0 to 1
Aug 25 15:48:09 cln07 crmd: [1661]: info: update_abort_priority: Abort action done superceeded by
restart
Aug 25 15:48:09 cln07 crmd: [1661]: info: match_graph_event: Action resO2CB:2_start_0 (26) confirmed
on cln09 (rc=4)
Aug 25 15:48:09 cln07 crmd: [1661]: info: abort_transition_graph: te_update_diff:146 - Triggered
transition abort (complete=0, tag=transient_attributes, id=cln09, magic=NA, cib=0.128.10) :
Transient attribute: update
Aug 25 15:48:09 cln07 crmd: [1661]: info: update_abort_priority: Abort priority upgraded from 1 to
1000000
Aug 25 15:48:09 cln07 crmd: [1661]: info: update_abort_priority: 'Event failed' abort superceeded
Aug 25 15:48:09 cln07 crmd: [1661]: info: abort_transition_graph: te_update_diff:146 - Triggered
transition abort (complete=0, tag=transient_attributes, id=cln09, magic=NA, cib=0.128.11) :
Transient attribute: update
Aug 25 15:48:09 cln07 o2cb[7789]: ERROR: ocfs2_controld.pcmk did not come up
Aug 25 15:48:09 cln07 crmd: [1661]: WARN: status_from_rc: Action 25 (resO2CB:1_start_0) on cln08
failed (target: 0 vs. rc: 1): Error
Aug 25 15:48:09 cln07 crmd: [1661]: WARN: update_failcount: Updating failcount for resO2CB:1 on
cln08 after failed start: rc=1 (update=INFINITY, time=1314280089)
Aug 25 15:48:09 cln07 crmd: [1661]: info: abort_transition_graph: match_graph_event:272 - Triggered
transition abort (complete=0, tag=lrm_rsc_op, id=resO2CB:1_start_0,
magic=0:1;25:12:0:6221d8f2-6b8e-484b-8765-cfeddc1b5ae7, cib=0.128.12) : Event failed
Aug 25 15:48:09 cln07 crmd: [1661]: info: match_graph_event: Action resO2CB:1_start_0 (25) confirmed
on cln08 (rc=4)
Aug 25 15:48:09 cln07 crmd: [1661]: info: abort_transition_graph: te_update_diff:146 - Triggered
transition abort (complete=0, tag=transient_attributes, id=cln08, magic=NA, cib=0.128.13) :
Transient attribute: update
Aug 25 15:48:09 cln07 crmd: [1661]: info: abort_transition_graph: te_update_diff:146 - Triggered
transition abort (complete=0, tag=transient_attributes, id=cln08, magic=NA, cib=0.128.14) :
Transient attribute: update
Aug 25 15:48:09 cln07 cibadmin: [8044]: info: Invoked: /usr/sbin/cibadmin -Ql
Aug 25 15:48:09 cln07 crmd: [1661]: info: process_lrm_event: LRM operation resO2CB:0_start_0
(call=15, rc=1, cib-update=81, confirmed=true) unknown error
Aug 25 15:48:09 cln07 crmd: [1661]: WARN: status_from_rc: Action 24 (resO2CB:0_start_0) on cln07
failed (target: 0 vs. rc: 1): Error
Aug 25 15:48:09 cln07 crmd: [1661]: WARN: update_failcount: Updating failcount for resO2CB:0 on
cln07 after failed start: rc=1 (update=INFINITY, time=1314280089)
Aug 25 15:48:09 cln07 crmd: [1661]: info: abort_transition_graph: match_graph_event:272 - Triggered
transition abort (complete=0, tag=lrm_rsc_op, id=resO2CB:0_start_0,
magic=0:1;24:12:0:6221d8f2-6b8e-484b-8765-cfeddc1b5ae7, cib=0.128.15) : Event failed
Aug 25 15:48:09 cln07 crmd: [1661]: info: match_graph_event: Action resO2CB:0_start_0 (24) confirmed
on cln07 (rc=4)
Aug 25 15:48:09 cln07 crmd: [1661]: info: te_pseudo_action: Pseudo action 28 fired and confirmed
Aug 25 15:48:09 cln07 crmd: [1661]: info: run_graph:
====================================================
Aug 25 15:48:09 cln07 crmd: [1661]: notice: run_graph: Transition 12 (Complete=11, Pending=0,
Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-388.bz2): Complete
Aug 25 15:48:09 cln07 crmd: [1661]: info: te_graph_trigger: Transition 12 is now complete
Aug 25 15:48:09 cln07 crmd: [1661]: info: do_state_transition: State transition S_TRANSITION_ENGINE
-> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Aug 25 15:48:09 cln07 crmd: [1661]: info: do_state_transition: All 3 cluster nodes are eligible to
run resources.
Aug 25 15:48:09 cln07 crmd: [1661]: info: do_pe_invoke: Query 82: Requesting the current CIB:
S_POLICY_ENGINE
Aug 25 15:48:09 cln07 attrd: [1659]: info: attrd_trigger_update: Sending flush op to all hosts for:
fail-count-resO2CB:0 (INFINITY)
Aug 25 15:48:09 cln07 attrd: [1659]: info: attrd_perform_update: Sent update 50:
fail-count-resO2CB:0=INFINITY
Aug 25 15:48:09 cln07 attrd: [1659]: info: attrd_trigger_update: Sending flush op to all hosts for:
last-failure-resO2CB:0 (1314280089)
Aug 25 15:48:09 cln07 attrd: [1659]: info: attrd_perform_update: Sent update 52:
last-failure-resO2CB:0=1314280089
Aug 25 15:48:09 cln07 crmd: [1661]: info: do_pe_invoke_callback: Invoking the PE: query=82,
ref=pe_calc-dc-1314280089-93, seq=172, quorate=1
Aug 25 15:48:09 cln07 crmd: [1661]: info: abort_transition_graph: te_update_diff:146 - Triggered
transition abort (complete=1, tag=transient_attributes, id=cln07, magic=NA, cib=0.128.16) :
Transient attribute: update
Aug 25 15:48:09 cln07 crmd: [1661]: info: abort_transition_graph: te_update_diff:146 - Triggered
transition abort (complete=1, tag=transient_attributes, id=cln07, magic=NA, cib=0.128.17) :
Transient attribute: update
Aug 25 15:48:09 cln07 crmd: [1661]: info: do_pe_invoke: Query 83: Requesting the current CIB:
S_POLICY_ENGINE
Aug 25 15:48:09 cln07 crmd: [1661]: info: do_pe_invoke: Query 84: Requesting the current CIB:
S_POLICY_ENGINE
Aug 25 15:48:09 cln07 pengine: [1660]: notice: unpack_config: On loss of CCM Quorum: Ignore
Aug 25 15:48:09 cln07 pengine: [1660]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow'
= 0, 'green' = 0
Aug 25 15:48:09 cln07 pengine: [1660]: info: determine_online_status: Node cln08 is online
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: unpack_rsc_op: Processing failed op resO2CB:1_start_0
on cln08: unknown error (1)
Aug 25 15:48:09 cln07 pengine: [1660]: info: determine_online_status: Node cln09 is online
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: unpack_rsc_op: Processing failed op resO2CB:2_start_0
on cln09: unknown error (1)
Aug 25 15:48:09 cln07 pengine: [1660]: info: determine_online_status: Node cln07 is online
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: unpack_rsc_op: Processing failed op resO2CB:0_start_0
on cln07: unknown error (1)
Aug 25 15:48:09 cln07 pengine: [1660]: notice: clone_print: Clone Set: cloneDLM
Aug 25 15:48:09 cln07 pengine: [1660]: notice: short_print: Started: [ cln07 cln08 cln09 ]
Aug 25 15:48:09 cln07 pengine: [1660]: notice: clone_print: Clone Set: cloneO2CB
Aug 25 15:48:09 cln07 pengine: [1660]: notice: native_print:
resO2CB:0#011(ocf::pacemaker:o2cb):#011Started cln07 FAILED
Aug 25 15:48:09 cln07 pengine: [1660]: notice: native_print:
resO2CB:1#011(ocf::pacemaker:o2cb):#011Started cln08 FAILED
Aug 25 15:48:09 cln07 pengine: [1660]: notice: native_print:
resO2CB:2#011(ocf::pacemaker:o2cb):#011Started cln09 FAILED
Aug 25 15:48:09 cln07 pengine: [1660]: notice: clone_print: Clone Set: stonith_clone
Aug 25 15:48:09 cln07 pengine: [1660]: notice: short_print: Started: [ cln07 cln08 cln09 ]
Aug 25 15:48:09 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln08
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln08 after 1000000 failures (max=1000000)
Aug 25 15:48:09 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln08
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln08 after 1000000 failures (max=1000000)
Aug 25 15:48:09 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln08
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln08 after 1000000 failures (max=1000000)
Aug 25 15:48:09 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln09
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln09 after 1000000 failures (max=1000000)
Aug 25 15:48:09 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln09
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln09 after 1000000 failures (max=1000000)
Aug 25 15:48:09 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln09
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln09 after 1000000 failures (max=1000000)
Aug 25 15:48:09 cln07 pengine: [1660]: notice: clone_rsc_colocation_rh: Cannot pair resO2CB:1 with
instance of cloneDLM
Aug 25 15:48:09 cln07 pengine: [1660]: info: native_color: Resource resO2CB:1 cannot run anywhere
Aug 25 15:48:09 cln07 pengine: [1660]: notice: clone_rsc_colocation_rh: Cannot pair resO2CB:2 with
instance of cloneDLM
Aug 25 15:48:09 cln07 pengine: [1660]: info: native_color: Resource resO2CB:2 cannot run anywhere
Aug 25 15:48:09 cln07 pengine: [1660]: notice: LogActions: Leave resource resDLM:0#011(Started cln07)
Aug 25 15:48:09 cln07 pengine: [1660]: notice: LogActions: Leave resource resDLM:1#011(Started cln08)
Aug 25 15:48:09 cln07 pengine: [1660]: notice: LogActions: Leave resource resDLM:2#011(Started cln09)
Aug 25 15:48:09 cln07 pengine: [1660]: notice: LogActions: Recover resource resO2CB:0#011(Started cln07)
Aug 25 15:48:09 cln07 pengine: [1660]: notice: LogActions: Stop resource resO2CB:1#011(cln08)
Aug 25 15:48:09 cln07 pengine: [1660]: notice: LogActions: Stop resource resO2CB:2#011(cln09)
Aug 25 15:48:09 cln07 pengine: [1660]: notice: LogActions: Leave resource stonith:0#011(Started cln07)
Aug 25 15:48:09 cln07 pengine: [1660]: notice: LogActions: Leave resource stonith:1#011(Started cln08)
Aug 25 15:48:09 cln07 pengine: [1660]: notice: LogActions: Leave resource stonith:2#011(Started cln09)
Aug 25 15:48:09 cln07 crmd: [1661]: info: do_pe_invoke_callback: Invoking the PE: query=84,
ref=pe_calc-dc-1314280089-94, seq=172, quorate=1
Aug 25 15:48:09 cln07 crmd: [1661]: info: handle_response: pe_calc calculation
pe_calc-dc-1314280089-93 is obsolete
Aug 25 15:48:09 cln07 pengine: [1660]: info: process_pe_message: Transition 13: PEngine Input stored
in: /var/lib/pengine/pe-input-389.bz2
Aug 25 15:48:09 cln07 pengine: [1660]: notice: unpack_config: On loss of CCM Quorum: Ignore
Aug 25 15:48:09 cln07 pengine: [1660]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow'
= 0, 'green' = 0
Aug 25 15:48:09 cln07 pengine: [1660]: info: determine_online_status: Node cln08 is online
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: unpack_rsc_op: Processing failed op resO2CB:1_start_0
on cln08: unknown error (1)
Aug 25 15:48:09 cln07 pengine: [1660]: info: determine_online_status: Node cln09 is online
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: unpack_rsc_op: Processing failed op resO2CB:2_start_0
on cln09: unknown error (1)
Aug 25 15:48:09 cln07 pengine: [1660]: info: determine_online_status: Node cln07 is online
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: unpack_rsc_op: Processing failed op resO2CB:0_start_0
on cln07: unknown error (1)
Aug 25 15:48:09 cln07 pengine: [1660]: notice: clone_print: Clone Set: cloneDLM
Aug 25 15:48:09 cln07 pengine: [1660]: notice: short_print: Started: [ cln07 cln08 cln09 ]
Aug 25 15:48:09 cln07 pengine: [1660]: notice: clone_print: Clone Set: cloneO2CB
Aug 25 15:48:09 cln07 pengine: [1660]: notice: native_print:
resO2CB:0#011(ocf::pacemaker:o2cb):#011Started cln07 FAILED
Aug 25 15:48:09 cln07 pengine: [1660]: notice: native_print:
resO2CB:1#011(ocf::pacemaker:o2cb):#011Started cln08 FAILED
Aug 25 15:48:09 cln07 pengine: [1660]: notice: native_print:
resO2CB:2#011(ocf::pacemaker:o2cb):#011Started cln09 FAILED
Aug 25 15:48:09 cln07 pengine: [1660]: notice: clone_print: Clone Set: stonith_clone
Aug 25 15:48:09 cln07 pengine: [1660]: notice: short_print: Started: [ cln07 cln08 cln09 ]
Aug 25 15:48:09 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln07
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln07 after 1000000 failures (max=1000000)
Aug 25 15:48:09 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln07
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln07 after 1000000 failures (max=1000000)
Aug 25 15:48:09 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln07
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln07 after 1000000 failures (max=1000000)
Aug 25 15:48:09 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln08
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln08 after 1000000 failures (max=1000000)
Aug 25 15:48:09 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln08
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln08 after 1000000 failures (max=1000000)
Aug 25 15:48:09 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln08
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln08 after 1000000 failures (max=1000000)
Aug 25 15:48:09 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln09
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln09 after 1000000 failures (max=1000000)
Aug 25 15:48:09 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln09
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln09 after 1000000 failures (max=1000000)
Aug 25 15:48:09 cln07 pengine: [1660]: info: get_failcount: cloneO2CB has failed INFINITY times on cln09
Aug 25 15:48:09 cln07 pengine: [1660]: WARN: common_apply_stickiness: Forcing cloneO2CB away from
cln09 after 1000000 failures (max=1000000)
Aug 25 15:48:09 cln07 pengine: [1660]: info: native_merge_weights: cloneDLM: Rolling back scores
from cloneO2CB
Aug 25 15:48:09 cln07 pengine: [1660]: notice: clone_rsc_colocation_rh: Cannot pair resO2CB:0 with
instance of cloneDLM
Aug 25 15:48:09 cln07 pengine: [1660]: info: native_color: Resource resO2CB:0 cannot run anywhere
Aug 25 15:48:09 cln07 pengine: [1660]: notice: clone_rsc_colocation_rh: Cannot pair resO2CB:1 with
instance of cloneDLM
Aug 25 15:48:09 cln07 pengine: [1660]: info: native_color: Resource resO2CB:1 cannot run anywhere
Aug 25 15:48:09 cln07 pengine: [1660]: notice: clone_rsc_colocation_rh: Cannot pair resO2CB:2 with
instance of cloneDLM
Aug 25 15:48:09 cln07 pengine: [1660]: info: native_color: Resource resO2CB:2 cannot run anywhere
Aug 25 15:48:09 cln07 pengine: [1660]: notice: LogActions: Leave resource resDLM:0#011(Started cln07)
Aug 25 15:48:09 cln07 pengine: [1660]: notice: LogActions: Leave resource resDLM:1#011(Started cln08)
Aug 25 15:48:09 cln07 pengine: [1660]: notice: LogActions: Leave resource resDLM:2#011(Started cln09)
Aug 25 15:48:09 cln07 pengine: [1660]: notice: LogActions: Stop resource resO2CB:0#011(cln07)
Aug 25 15:48:09 cln07 pengine: [1660]: notice: LogActions: Stop resource resO2CB:1#011(cln08)
Aug 25 15:48:09 cln07 pengine: [1660]: notice: LogActions: Stop resource resO2CB:2#011(cln09)
Aug 25 15:48:09 cln07 pengine: [1660]: notice: LogActions: Leave resource stonith:0#011(Started cln07)
Aug 25 15:48:09 cln07 pengine: [1660]: notice: LogActions: Leave resource stonith:1#011(Started cln08)
Aug 25 15:48:09 cln07 pengine: [1660]: notice: LogActions: Leave resource stonith:2#011(Started cln09)
Aug 25 15:48:09 cln07 crmd: [1661]: info: do_state_transition: State transition S_POLICY_ENGINE ->
S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Aug 25 15:48:09 cln07 crmd: [1661]: info: unpack_graph: Unpacked transition 14: 6 actions in 6 synapses
Aug 25 15:48:09 cln07 crmd: [1661]: info: do_te_invoke: Processing graph 14
(ref=pe_calc-dc-1314280089-94) derived from /var/lib/pengine/pe-input-390.bz2
Aug 25 15:48:09 cln07 crmd: [1661]: info: te_pseudo_action: Pseudo action 27 fired and confirmed
Aug 25 15:48:09 cln07 crmd: [1661]: info: te_rsc_command: Initiating action 9: stop resO2CB:0_stop_0
on cln07 (local)
Aug 25 15:48:09 cln07 crmd: [1661]: info: do_lrm_rsc_op: Performing
key=9:14:0:6221d8f2-6b8e-484b-8765-cfeddc1b5ae7 op=resO2CB:0_stop_0 )
Aug 25 15:48:09 cln07 lrmd: [1658]: info: rsc:resO2CB:0:16: stop
Aug 25 15:48:09 cln07 pengine: [1660]: info: process_pe_message: Transition 14: PEngine Input stored
in: /var/lib/pengine/pe-input-390.bz2
Aug 25 15:48:09 cln07 crmd: [1661]: info: te_rsc_command: Initiating action 3: stop resO2CB:1_stop_0
on cln08
Aug 25 15:48:09 cln07 crmd: [1661]: info: te_rsc_command: Initiating action 6: stop resO2CB:2_stop_0
on cln09
Aug 25 15:48:09 cln07 cibadmin: [8059]: info: Invoked: /usr/sbin/cibadmin -Ql
Aug 25 15:48:09 cln07 crmd: [1661]: info: match_graph_event: Action resO2CB:1_stop_0 (3) confirmed
on cln08 (rc=0)
Aug 25 15:48:09 cln07 crmd: [1661]: info: match_graph_event: Action resO2CB:2_stop_0 (6) confirmed
on cln09 (rc=0)
Aug 25 15:48:09 cln07 crmd: [1661]: info: process_lrm_event: LRM operation resO2CB:0_stop_0
(call=16, rc=0, cib-update=85, confirmed=true) ok
Aug 25 15:48:09 cln07 crmd: [1661]: info: match_graph_event: Action resO2CB:0_stop_0 (9) confirmed
on cln07 (rc=0)
Aug 25 15:48:09 cln07 crmd: [1661]: info: te_pseudo_action: Pseudo action 28 fired and confirmed
Aug 25 15:48:09 cln07 crmd: [1661]: info: te_pseudo_action: Pseudo action 10 fired and confirmed
Aug 25 15:48:09 cln07 crmd: [1661]: info: run_graph:
====================================================
Aug 25 15:48:09 cln07 crmd: [1661]: notice: run_graph: Transition 14 (Complete=6, Pending=0,
Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-390.bz2): Complete
Aug 25 15:48:09 cln07 crmd: [1661]: info: te_graph_trigger: Transition 14 is now complete
Aug 25 15:48:09 cln07 crmd: [1661]: info: notify_crmd: Transition 14 status: done - <null>
Aug 25 15:48:09 cln07 crmd: [1661]: info: do_state_transition: State transition S_TRANSITION_ENGINE
-> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Aug 25 15:48:09 cln07 crmd: [1661]: info: do_state_transition: Starting PEngine Recheck Timer
Aug 25 15:48:09 cln07 cibadmin: [8078]: info: Invoked: /usr/sbin/cibadmin -Ql
Aug 25 15:48:09 cln07 cibadmin: [8081]: info: Invoked: /usr/sbin/cibadmin -Ql
Ich komme hier einfach nicht weiter, hat von Euch jemand eine Idee?
--
Mit freundlichen Grüßen
Vadim Bulst
Systemadministrator BBZ
Biotechnologisch-Biomedizinisches Zentrum
Universität Leipzig
Deutscher Platz 5, 04103 Leipzig
Tel.: 0341 97 - 31 307
Fax : 0341 97 - 31 309
Reply to: