7. Forced stop resource details

This chapter provides the detailed information on forced stop resources.

7.1. What is the forced stop function?

When a server crash is recognized due to a heartbeat loss, the forced stop function makes the remaining servers (operating properly) forcibly stop the down server.
Suppose the recognized server crash is actually a temporary inability to operate due to the server's stall. In this case, the forced stop function surely stops the down server before its application is failed over to a healthy server. This reduces the risk of the corruption of data in the same resource accessed from multiple servers.

The method for forcibly stopping the failing server varies depending on the type of environment where the cluster was created: physical machines, virtual machines, or the cloud. When using the forced stop function, configure a forced stop resource corresponding to the environment type.

Moreover, you can execute a script in which the procedure for stopping the failing server is written. For details, refer to "7.7. Understanding forced stop with script" in this guide.

A forced stop resource operates in two ways: performing a forced stop and periodically checking if the target can be forcibly stopped. The following explains what these are and when they are done:

  • Performing a forced stop

    • Forcibly stops a down server by using a device or infrastructure system as a server status manager.

    • Done by recognizing the server crash. For more information on the conditions, see "7.1.1. Conditions for performing forced stop".

  • Periodically checking if the target can be forcibly stopped

    • Checks whether a forced stop can be performed, by communicating with a device or infrastructure system for forcibly stopping a server. Depending on the result, the forced stop resource shows whether the server can be forcibly stopped: "Normal" (yes) or "Error" (no).

    • Done on a regular basis while the cluster service is running.

7.1.1. Conditions for performing forced stop

  • Forced stop is not performed when:

    • The failover group successfully stops before the server fails

    • The server is shut down by the clpdown command or Cluster WebUI and the failover group successfully stops

    • The cluster is stopped by the clpcl command or Cluster WebUI and the failover group successfully stops

    • The server fails and there is no failover group to perform failover from the failing server to another server
      (including when the failover group is not activated in the failing server)

  • Forced stop is performed when:

    • The server is failing and there is a failover group to perform failover from the failing server to another server

7.2. Understanding forced stop on physical environment

7.2.1. Settings of BMC forced stop resource

For the BMC forced stop resource, you can use two methods: IPMI command and Redfish APIs.

Setting parameters of BMC forced stop resource

For the BMC forced stop resource, set parameters listed below.
With the IPMI, these parameters are specified for command options.
With vSphere Redfish APIs, these parameters are specified for requests.

Parameter

item to be set in Server Properties -> BMC tab

ip_address

IP address

username

User name

password

Password

IPMI commands for BMC forced stop resource

To use the IPMI command for the BMC forced stop resource, install ipmiutil on each cluster server.

For information on how to get ipmiutil and how to install it, refer to "Setup of BMC and ipmiutil" in "Settings after configuring hardware" in "Determining a system configuration" in the "Installation and Configuration Guide".

For periodically checking the status of the forced stop resource and for checking whether a forced stop succeeds, use the [ipmiutil health] command.

The following table shows commands to be executed for forced stop actions by the BMC forced stop resource:

Forced Stop Action

Parameters

BMC Power Off

ipmiutil reset -d -N ip_address -U username -P password

BMC Reset

ipmiutil reset -r -N ip_address -U username -P password

BMC Power Cycle

ipmiutil reset -c -N ip_address -U username -P password

BMC NMI

ipmiutil reset -n -N ip_address -U username -P password

If the above commands fail to be executed, perform the following commands:

Forced Stop Action

Parameters

BMC Power Off

ipmiutil reset -d -N ip_address -U username -P password -J 3

BMC Reset

ipmiutil reset -r -N ip_address -U username -P password -J 3

BMC Power Cycle

ipmiutil reset -c -N ip_address -U username -P password -J 3

BMC NMI

ipmiutil reset -n -N ip_address -U username -P password -J 3

The following table shows commands to be executed for periodically checking the status of the BMC forced stop resource and for checking whether a forced stop succeeds:

Command to periodically check the status
Command to check a forced stop

ipmiutil health -N ip_address -U username -P password

If the above command fails to be executed, perform the following command:

Command to periodically check the status
Command to check a forced stop

ipmiutil health -N ip_address -U username -P password -J 3

APIs for BMC forced stop resource

For a forced stop by the BMC forced stop resource with Redfish APIs, the following APIs are used:

Forced Stop Action

API

BMC Power Off
POST https://{ip_address}/redfish/v1/Systems/1/Actions/ComputerSystem.Reset
Content-Type: application/json
Body :
{
"ResetType" : "ForceOff"
}

BMC Reset
POST https://{ip_address}/redfish/v1/Systems/1/Actions/ComputerSystem.Reset
Content-Type: application/json
Body :
{
"ResetType" : "ForceRestart"
}

BMC Power Cycle
POST https://{ip_address}/redfish/v1/Systems/1/Actions/ComputerSystem.Reset
Content-Type: application/json
Body :
{
"ResetType" : "PowerCycle"
}

BMC NMI
POST https://{ip_address}/redfish/v1/Systems/1/Actions/ComputerSystem.Reset
Content-Type: application/json
Body :
{
"ResetType" : "Nmi"
}

For the BMC forced stop resource to be periodically checked and to be checked after a forced stop, the following APIs are used:

For a periodical check
For a stop check

GET https://{ip_address}/redfish/v1/Systems/1

For more information on the settings of the BMC forced stop resource,see "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced Stop" -> "BMC Forced Stop Properties", and "2. Parameter details" -> "Server Properties" -> "BMC tab".

7.2.2. Notes on BMC forced stop resource

  • Impacts on forced stop
    When you use the forced stop function, the following functions are influenced because power off, reset, power cycle or NMI is forcibly performed regardless of the OS or server status.
    • Dump collection
      Because it is not recognized that dump files are being collected, power off, reset or power cycle is performed even though dump collection is being performed, so dump collection does not complete.
    • Power on within heartbeat timeout
      When the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, power off, reset, power cycle or NMI may occur after heartbeat timeout has elapsed.
  • BMC network settings
    When using the BMC forced stop resource, disable the iLO shared network port.
    Configure the settings so that the IP address of the LAN port for BMC management and the IP address which OS uses can communicate with each other. This function cannot be used when BMC is not installed in the server, or in the environment where the network for the BMC management is blocked.
    Navigate BMC Forced-Stop Properties -> Server List -> Edit -> Enter BMC, then enter the IP address assigned to the LAN port for BMC management.
    See the server's manuals etc. for information on how to configure the IP address of the LAN port for the BMC management etc.
  • Power Options settings of the OS
    Conducting a power-off or power cycle by BMC may perform an action selected in Power Options of the OS, such as Sleep, Hibernate, or Shutdown.
    The settings can be referred to and configured by the following instruction:
    Open Power Options in Control Panel and select Choose what the power button does, Power button settings and When I press the power button:.
    When Forced stop is used in EXPRESSCLUSTER, it is recommended that this setting is configured as No Operation.
  • Forced-stop actions
    Not all Redfish API-based actions are supported by some server models.
    Check forced-stop actions available on your servers.

7.3. Understanding forced stop on vCenter environment

7.3.1. Settings of vCenter forced stop resource

The vCenter forced stop resource can be used with vSphere Automation APIs or with the VMware vSphere Command Line Interface (vCLI).

In an environment with VMware vSphere 7.0 Update 3 or higher, the vCLI cannot be used. In an environment with VMware vSphere earlier than 7.0 Update 2, vSphere Automation APIs cannot be used.

Setting parameters of vCenter forced stop resource

For the vCenter forced stop resource, set parameters listed below.
With vSphere Automation APIs, these parameters are specified for requests.
With the vCLI, these parameters are specified for command options.

Parameter

Item to be set in vCenter Forced-Stop Properties -> the vCenter tab

Item to be set in vCenter Forced-Stop Properties -> Server List -> Edit -> Input for Virtual Machine name

ip_address

IP address

-

username

User name

-

password

Password

-

virtualmachine

-

Virtual machine name

datacenter

-

Datacenter name

APIs for vCenter forced stop resource

When the vCenter forced stop resource is used with vSphere Automation APIs, the following APIs are executed:

Creating a session to obtain the session ID (api_session_id)

Creating a session

curl -k -X POST -u {username}:{password} https://{ip_address}/api/session
Retrieving data center information to obtain the data center ID (datacenter_id)

Retrieving data center information

curl -k -X GET https://{ip_address}/api/vcenter/datacenter?names={datacenter} -H "vmware-api-session-id: {api_session_id}"
Retrieving virtual-machine information to check the power-supply state and obtain the virtual-machine ID (vm_id)
The data on the power-supply state is used for periodical checks and for stop checks after forced-stop executions, by the vCenter forced stop resource.

Retrieving virtual-machine information

curl -k -X GET https://{ip_address}/api/vcenter/vm?names={virtualmachine}&datacenters={datacenter_id} -H "vmware-api-session-id: {api_session_id}"
Forcibly stopping the guest OS on the virtual machine

Forced stop

poweroff

curl -k -X POST https://{ip_address}/api/vcenter/vm/{vm_id}/power?action=stop -H "vmware-api-session-id: {api_session_id}"

reset

curl -k -X POST https://{ip_address}/api/vcenter/vm/{vm_id}/power?action=reset -H "vmware-api-session-id: {api_session_id}"
Deleting the created session

Deleting the session

curl -k -X DELETE https://{ip_address}/api/session -H "vmware-api-session-id: {api_session_id}"

vCLI commands for vCenter forced stop resource

For forcibly stopping the guest OS on a virtual machine, use the [vmcontrol] command of the vCLI. For periodically checking the status of the forced stop resource and for checking whether a forced stop succeeds, use the [vminfo] command.

Using the vCLI requires installing the package.

Note

If the version of vCLI is 6.5 or later, Perl execution environment is required to be installed. For the information on the versions of Perl necessary to execute vCLI, refer to the website of Broadcom Inc.
Perform the following procedure after installing the Perl execution environment.

  • Set the Perl path

    Select Cluster Properties -> Fencing tab -> Forced stop -> vCenter Forced-Stop Properties -> vCenter tab, specify the path to the Perl execution module for Perl Path. This is common to all the servers in the cluster.
    For more information about the Perl path, refer to "Extension Tab" in "Cluster properties" in "2.2.3. Fencing tab" in "Forced stop" in "vCenter Forced-Stop Properties" in this guide.

  • Add the system environment variable

    Add the following variable for the system environment variable. Then restart the OS.
    Variable name: PERL5LIB
    Variable value: vCLI Perl module path (Example: C:\Program Files (x86)\VMware\VMware vSphere CLI\Perl\lib
The following table shows commands to be executed for forced stop actions by the vCenter forced stop resource:

Forced-stop commands

poweroff

vmcontrol.pl --server ip_address --username username --password password --vmname virtualmachine --datacenter datacenter --operation poweroff

reset

vmcontrol.pl --server ip_address --username username --password password --vmname virtualmachine --datacenter datacenter --operation reset
The following table shows commands to be executed for periodically checking the status of the vCenter forced stop resource:

Command to periodically check the status

vminfo.pl --server ip_address --username username --password password --vmname virtualmachine --datacenter datacenter
The following table shows commands to be executed for checking whether a forced stop succeeds:

Command to check a forced stop

vminfo.pl --server ip_address --username username --password password --vmname virtualmachine --datacenter datacenter --powerstatus "poweredOff"

For more information on configuring the vCenter forced stop resource, see this guide: "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced stop" -> "vCenter Forced-Stop Properties".

7.3.2. Notes on vCenter forced stop resource

  • Forcibly stopping the guest OS on a virtual machine
    Only power off operation can be performed. Moreover, this function cannot be used in the following cases:
    - vSphere infrastructure: Communication with VMware vCenter Server is not possible.
  • Impacts on forced stop
    When you use the forced stop function, the following functions are influenced because power off, reset is forcibly performed regardless of the OS or server status.
    • Dump collection
      Because it is not recognized that dump files are being collected, power off, reset is performed even though dump collection is being performed, so dump collection does not complete.
    • Power on within heartbeat timeout
      When the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, power off, reset may occur after heartbeat timeout has elapsed.
  • Power Options settings of the OS
    Conducting a power-off of the guest OS on a virtual machine with the vCLI may perform an action selected in Power Options of the OS, such as Sleep, Hibernate, or Shutdown.
    The settings can be referred to and configured by the following instruction:
    Open Power Options in Control Panel and select Choose what the power button does, Power button settings and When I press the power button:.
    When Forced stop is used in EXPRESSCLUSTER, it is recommended that this setting is configured as No Operation.

7.4. Understanding forced stop on AWS environment

7.4.1. Settings of AWS forced stop resource

Using the AWS forced stop resource requires installing the AWS Command Line Interface (AWS CLI).

Regarding specific settings and precautions in the cloud environment, see "Started Guide" -> "Notes and Restrictions" -> "Before installing EXPRESSCLUSTER" -> "Time synchronization in the AWS environment" and "IAM settings in the AWS environment".

Commands for AWS forced stop resource

For forcibly stopping an AWS instance, for periodically checking the status of the forced stop resource, and for checking whether a forced stop succeeds, use the command of the AWS CLI.

When executing either of the commands, specify the following option values:

command option

item to be set in AWS Forced-Stop Properties -> Server List -> Edit -> Input of Instance

--instance-ids instance-ids

InstanceID

The following table shows commands to be executed for forced stop actions by the AWS forced stop resource:

Forced Stop Action

Parameters

stop

aws ec2 stop-instances --instance-ids instance-ids --force

reboot

aws ec2 reboot-instances --instance-ids instance-ids

The following table shows commands to be executed for periodically checking the status of the AWS forced stop resource:

Forced Stop Action

Command to periodically check the status

stop
aws ec2 stop-instances --instance-ids instance-ids --dry-run
aws ec2 describe-instances --instance-ids instance-ids
aws ec2 describe-instance-attribute --instance-ids instance-ids --attribute disableApiStop

reboot
aws ec2 reboot-instances --instance-ids instance-ids --dry-run
aws ec2 describe-instances --instance-ids instance-ids

The following table shows commands to be executed for checking whether a forced stop succeeds:

Command to periodically check the status

aws ec2 describe-instances --instance-ids instance-ids --filters \"Name=instance-state-name,Values=stopped\"

7.4.2. Applying command line options to AWS CLI run from AWS forced stop resource

7.4.3. Applying environment variables to AWS CLI run from the AWS forced stop resource

7.4.4. Notes on AWS forced stop resource

  • For forcibly stopping instance
    You can perform only the following actions: stop and reboot.
  • Impacts on forced stop
    When you use the forced stop function, the following functions are influenced because stop, reboot is forcibly performed regardless of the OS or server status.
    • Dump collection
      Because it is not recognized that dump files are being collected, stop, reboot is performed even though dump collection is being performed, so dump collection does not complete.
    • Power on within heartbeat timeout
      When the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, stop, reboot may occur after heartbeat timeout has elapsed.
  • Power Options settings of the OS
    Conducting the action (stop or reboot) with the AWS CLI may perform an action selected in Power Options of the OS, such as Sleep, Hibernate, or Shutdown.
    The settings can be referred to and configured by the following instruction:
    Open Power Options in Control Panel and select Choose what the power button does, Power button settings and When I press the power button:.
    When Forced stop is used in EXPRESSCLUSTER, it is recommended that this setting is configured as No Operation.
  • stop protection setting for AWS
    With stop protection of Instance settings enabled, setting Forced Stop Action to stop leads to a failure in forced stopping and a periodical checking.
    Avoid enabling stop protection if you use the forced-stop function of EXPRESSCLUSTER and set Forced Stop Action to stop.
  • Cluster configuration
    For a cluster configured in a multi-region environment, you cannot use the forced-stop function.

7.5. Understanding forced stop on Azure environment

7.5.1. Settings of Azure forced stop resource

Using the Azure forced stop resource requires installing the Azure CLI.

Regarding specific settings and precautions in the cloud environment, see "Getting Started Guide" -> "Notes and Restrictions" -> "Before installing EXPRESSCLUSTER" -> "Settings in the Azure environment" and "IAM settings in the Azure environment".

Commands for Azure forced stop resource

For forcibly stopping an Azure instance, for periodically checking the status of the forced stop resource, and for checking whether a forced stop succeeds, use the command of the Azure CLI.

When executing either of the commands, specify the following option values:

command option

item to be set in Azure Forced-Stop Properties -> the Azure tab

item to be set in Azure Forced-Stop Properties -> Server List -> Edit -> Input for Virtual Machine name

--name/-n

-

Virtual Machine Name

--username/-u

Application ID

-

--tenant/-t

Tenant ID

-

--password/-p

File Path of Service Principal

-

--resource-group/-g

Resource Group Name

-

For the use of Azure CLI commands, the following command is executed:

Logging in to the Azure CLI

az login --service-principal -u MyApplicationId --certificate MyCertfile.pem --tenant MyTenantId

Note

If the Azure CLI version is 2.66.0 or earlier, kindly substitute the --certificate option with the -p option.

The following table shows commands to be executed for forced stop actions by the Azure forced stop resource:

Forced Stop Action

Parameters

stop and deallocate [1]

az vm deallocate -g MyResourceGroup -n MyVm --no-wait

stop only [2]

az vm stop -g MyResourceGroup -n MyVm --no-wait --skip-shutdown

reboot

az vm restart -g MyResourceGroup -n MyVm --no-wait

The following table shows commands to be executed for periodically checking the status of the Azure forced stop resource:

Command to periodically check the status

az vm update -g MyResourceGroup -n MyVm --no-wait

az vm get-instance-view -g MyResourceGroup -n MyVm

The following table shows commands to be executed for checking whether a forced stop succeeds:

Command to periodically check the status

az vm get-instance-view -g MyResourceGroup -n MyVm

7.5.2. Notes on Azure forced stop resource

  • For forcibly stopping instance
    Any of the following actions can be performed: stop and deallocate, stop only, or reboot.
  • Impacts on forced stop
    When you use the forced stop function, the following functions are influenced because stop, reboot is forcibly performed regardless of the OS or server status.
    • Dump collection
      Because it is not recognized that dump files are being collected, stop, reboot is performed even though dump collection is being performed, so dump collection does not complete.
    • Power on within heartbeat timeout
      When the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, stop, reboot may occur after heartbeat timeout has elapsed.
  • Power Options settings of the OS
    Conducting the action (stop or reboot) with the OCI CLI may perform an action selected in Power Options of the OS, such as Sleep, Hibernate, or Shutdown.
    The settings can be referred to and configured by the following instruction:
    Open Power Options in Control Panel and select Choose what the power button does, Power button settings and When I press the power button:.
    When Forced stop is used in EXPRESSCLUSTER, it is recommended that this setting is configured as No Operation.
  • Cluster configuration
    For a cluster configured in a multi-region environment, you cannot use the forced-stop function.

7.6. Understanding forced stop on OCI environment

7.6.1. Settings of OCI forced stop resource

Using the OCI forced stop resource requires installing the Oracle Cloud Infrastructure CLI (OCI CLI).

Regarding specific settings and precautions in the cloud environment, see "Getting Started Guide" -> "Notes and Restrictions" -> "Before installing EXPRESSCLUSTER" -> "CLI settings in the OCI environment" and "Policy setting in the OCI environment".

Commands for OCI forced stop resource

For forcibly stopping an OCI instance, for periodically checking the status of the forced stop resource, and for checking whether a forced stop succeeds, use the command of the OCI CLI.

When executing either of the commands, specify the following option values:

command option

item to be set in OCI Forced-Stop Properties -> Server List -> Edit -> Input of Instance

--instance-ids instance-ids

InstanceID

The following table shows commands to be executed for forced stop actions by the OCI forced stop resource:

Forced Stop Action

Parameters

stop

oci compute instance action --action STOP --instance-id instance-ids

reboot

oci compute instance action --action RESET --instance-id instance-ids

The following table shows commands to be executed for periodically checking the status of the OCI forced stop resource:

Command to periodically check the status

oci compute instance update --instance-id instance-ids

oci compute instance get --instance-id instance-ids

The following table shows commands to be executed for checking whether a forced stop succeeds:

Command to periodically check the status

oci compute instance get --instance-id instance-ids

7.6.2. Notes on OCI forced stop resource

  • For forcibly stopping instance
    You can perform only the following actions: stop and reboot.
  • Impacts on forced stop
    When you use the forced stop function, the following functions are influenced because stop, reboot is forcibly performed regardless of the OS or server status.
    • Dump collection
      Because it is not recognized that dump files are being collected, stop, reboot is performed even though dump collection is being performed, so dump collection does not complete.
    • Power on within heartbeat timeout
      When the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, stop, reboot may occur after heartbeat timeout has elapsed.
  • Power Options settings of the OS
    Conducting the action (stop or reboot) with the OCI CLI may perform an action selected in Power Options of the OS, such as Sleep, Hibernate, or Shutdown.
    The settings can be referred to and configured by the following instruction:
    Open Power Options in Control Panel and select Choose what the power button does, Power button settings and When I press the power button:.
    When Forced stop is used in EXPRESSCLUSTER, it is recommended that this setting is configured as No Operation.
  • Cluster configuration
    For a cluster configured in a multi-region environment, you cannot use the forced-stop function.

7.7. Understanding forced stop with script

7.7.1. Settings of custom forced stop resource

You can create a script for a forced stop. When a server crash is recognized, using the script on the remaining servers (operating properly) allows you to forcibly stop the down server.

The script is executed in both of the following modes: performing a forced stop and periodically checking if the target can be forcibly stopped. For appropriate processing based on each of the modes, write conditional branches by using environment variables described later.

Environment variables for script

When executing the script, EXPRESSCLUSTER sets environment variable values such as which mode (a periodical status check or a forced stop) to be performed and what server has crashed.

In the script, you can use the following environment variables:

Environment variable

Setting value

Description
CLP_FORCESTOP_MODE
...Mode
0 : When periodically checking the status
1 : When performing a forced stop
Means a mode to be performed.
Can be used for process branches for each of the modes.
CLP_SERVER_DOWN
...Down server name

Server name
Means the name of a down server.
For periodically checking the status, "" is set.
CLP_SERVER_LOCAL
...Local server name

Server name

Means the name of a server to execute the script.

Returned value of script

Return 0 when the script terminates normally.

For more information on configuring the custom forced stop resource, see this guide: "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced stop" -> "Custom Forced-Stop Properties".

7.7.2. Notes on custom forced stop resource

  • Describe the customer-defined process in the script to stop the server.

  • If there is nothing to be done as periodically checking the status, write the process as such (so that the value 0 can be returned).

7.8. Notes on settings of forced stop resource

  • You can configure only one forced stop resource for one cluster.

  • If you want to configure a forced stop resource, it is recommended to configure a network partition resolution resource as well.

  • In configuring a forced stop resource, all the cluster servers must be set to use the forced stop resource.

  • To prevent a split-brain syndrome in a failover group with a forced-stop resource operating, set the service startup delay time as follows:

    • When setting up DISK network partition resolution resources
      Service startup delay time >= forced-stop timeout of forced-stop resource + time to wait for stop to be completed of forced-stop resource + heartbeat timeout + heartbeat interval + IO Wait Time of DISK network partition resolution resources + 10 seconds
    • When not setting up DISK network partition resolution resources
      Service startup delay time >= forced-stop timeout of forced-stop resource + time to wait for stop to be completed of forced-stop resource + heartbeat timeout + heartbeat interval

    For more information on the service startup delay time, see "Adjustment of time for EXPRESSCLUSTER services to start up (Required)".