7. Forced stop resource details

This chapter provides the detailed information on forced stop resources.

7.1. What is the forced stop function?

When a server crash is recognized due to a heartbeat loss, the forced stop function makes the remaining servers (operating properly) forcibly stop the down server.
Suppose the recognized server crash is actually a temporary inability to operate due to the server's stall. In this case, the forced stop function surely stops the down server before its application is failed over to a healthy server. This reduces the risk of the corruption of data in the same resource accessed from multiple servers.

The method for forcibly stopping the failing server varies depending on the type of environment where the cluster was created: physical machines, virtual machines, or the cloud. When using the forced stop function, configure a forced stop resource corresponding to the environment type.

Moreover, you can execute a script in which the procedure for stopping the failing server is written. For details, refer to "7.7. Understanding forced stop with script" in this guide.

A forced stop resource operates in two ways: performing a forced stop and periodically checking if the target can be forcibly stopped. The following explains what these are and when they are done:

  • Performing a forced stop

    • Forcibly stops a down server by using a device or infrastructure system as a server status manager.

    • Done by recognizing the server crash. For more information on the conditions, see "7.1.1. Conditions for performing forced stop".

  • Periodically checking if the target can be forcibly stopped

    • Checks whether a forced stop can be performed, by communicating with a device or infrastructure system for forcibly stopping a server. Depending on the result, the forced stop resource shows whether the server can be forcibly stopped: "Normal" (yes) or "Error" (no).

    • Done on a regular basis while the cluster service is running.

7.1.1. Conditions for performing forced stop

  • Forced stop is not performed when:

    • The failover group successfully stops before the server fails

    • The server is shut down by the clpdown command, the OS shutdown command or Cluster WebUI and the failover group successfully stops

    • The cluster is stopped by the clpcl command or Cluster WebUI and the failover group successfully stops

    • The server fails and there is no failover group to perform failover from the failing server to another server
      (including when the failover group is not activated in the failing server)

  • Forced stop is performed when:

    • The server is failing and there is a failover group to perform failover from the failing server to another server

7.2. Understanding forced stop on physical environment

7.2.1. Settings of BMC forced stop resource

Using the BMC forced stop resource requires installing ipmitool on each cluster server.

Commands for BMC forced stop resource

For forcibly stopping a physical machine, for periodically checking the status of the forced stop resource, and for checking whether a forced stop succeeds, use the [ipmitool] command.

When executing either of the commands, specify the following option values:

[ipmitool] command option

item to be set in BMC Forced-Stop Properties -> Server List -> Edit -> Enter BMC

-H ip_address

IP address

-U username

User name

-P password

Password

The following table shows commands to be executed for forced stop actions by the BMC forced stop resource:

Forced Stop Action

Parameters

BMC Power Off

ipmitool -H ip_address -U username -P password power off

BMC Reset

ipmitool -H ip_address -U username -P password power reset

BMC Power Cycle

ipmitool -H ip_address -U username -P password power cycle

BMC NMI

ipmitool -H ip_address -U username -P password power diag

If the above commands fail to be executed, perform the following commands:

Forced Stop Action

Parameters

BMC Power Off

ipmitool -H ip_address -I lanplus -U username -P password power off

BMC Reset

ipmitool -H ip_address -I lanplus -U username -P password power reset

BMC Power Cycle

ipmitool -H ip_address -I lanplus -U username -P password power cycle

BMC NMI

ipmitool -H ip_address -I lanplus -U username -P password power diag

The following table shows commands to be executed for periodically checking the status of the BMC forced stop resource and for checking whether a forced stop succeeds:

Command to periodically check the status
Command to check a forced stop

ipmitool -H ip_address -U username -P password power status

If the above command fails to be executed, perform the following command:

Command to periodically check the status
Command to check a forced stop

ipmitool -H ip_address -I lanplus -U username -P password power status

Editing commands for BMC forced stop resource

For forced stop resources, you can also specify a command line for a forced stop by editing a script file for the command execution.

The following file can be edited:

<EXPRESSCLUSTER_install_path>\bin\clpbmcforcestop.sh

The BMC forced stop resource sets environment variable values necessary for commands to be executed with the script.

The following table shows environment variables written in the script:

Environment variable

Setting value

Description
CLP_BMC_ACTION
...Forced Stop Action
power off : BMC Power Off
power reset : BMC Reset
power cycle : BMC Power Cycle
power diag : BMC NMI

Specifies the Forced Stop Action set in the BMC Forced-Stop Properties.
CLP_BMC_HOST
...IP address for BMC

IP Address

Specifies the IP address set in the BMC Forced-Stop Properties.
CLP_BMC_USER
...User name for BMC

User name

Specifies the user name set in the BMC Forced-Stop Properties.
CLP_BMC_PASSWORD
...Password for BMC

Password

Specifies the password set in the BMC Forced-Stop Properties.

For more information on configuring the BMC forced stop resource, see this guide: "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced stop" -> "BMC Forced-Stop Properties".

7.2.2. Notes on BMC forced stop resource

  • Notes on ipmitool
    See "IPMI command" in "4. Monitor resource details" of "Monitor resource".
  • Impacts of forced stop
    When you use the forced stop function, the following functions are influenced because power off, reset, power cycle or NMI is forcibly performed regardless of the OS or server status.
    • Dump collection
      Since it is not recognized that dump files are being collected, power off, reset or power cycle is performed even though dump collection is being performed, so dump collection does not complete.
    • Power on within the heartbeat timeout
      When the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, power off, reset, power cycle or NMI may occur after heartbeat timeout has elapsed.
  • BMC network settings
    When using the BMC forced stop resource, disable the iLO shared network port.
    Configure the settings so that the IP address of the LAN port for BMC management and the IP address which OS uses can communicate with each other. This function cannot be used in the environment where the network for the BMC management is blocked.
    Navigate BMC Forced-Stop Properties -> Server List -> Edit -> Enter BMC, then enter the IP address assigned to the LAN port for BMC management.
    See the server's manuals etc. for information on how to configure the IP address of the LAN port for the BMC management etc.

  • When using forced stop function, settings of BMC IP address, user name and password of each server are necessary. Use definitely the user name to which the password is set.

7.3. Understanding forced stop on vCenter environment

7.3.1. Settings of vCenter forced stop resource

The vCenter forced stop resource can be used with vSphere Automation APIs or with the VMware vSphere Command Line Interface (vCLI).

In an environment with VMware vSphere 7.0 Update 3 or higher, the vCLI cannot be used. Instead, use vSphere Automation APIs.

Setting parameters of vCenter forced stop resource

For the vCenter forced stop resource, set parameters listed below.
With vSphere Automation APIs, these parameters are specified for requests.
With the vCLI, these parameters are specified for command options.

Parameter

Item to be set in vCenter Forced-Stop Properties -> the vCenter tab

Item to be set in vCenter Forced-Stop Properties -> Server List -> Edit -> Input for Virtual Machine name

ip_address

IP address

-

username

User name

-

password

Password

-

virtualmachine

-

Virtual machine name

datacenter

-

Datacenter name

APIs for vCenter forced stop resource

When the vCenter forced stop resource is used with vSphere Automation APIs, the following APIs are executed:

Creating a session to obtain the session ID (api_session_id)

Creating a session

curl -k -X POST -u {username}:{password} https://{ip_address}/api/session
Retrieving data center information to obtain the data center ID (datacenter_id)

Retrieving data center information

curl -k -X GET https://{ip_address}/api/vcenter/datacenter?names={datacenter} -H "vmware-api-session-id: {api_session_id}"
Retrieving virtual-machine information to check the power-supply state and obtain the virtual-machine ID (vm_id)
The data on the power-supply state is used for periodical checks and for stop checks after forced-stop executions, by the vCenter forced stop resource.

Retrieving virtual-machine information

curl -k -X GET https://{ip_address}/api/vcenter/vm?names={virtualmachine}&datacenters={datacenter_id} -H "vmware-api-session-id: {api_session_id}"
Forcibly stopping the guest OS on the virtual machine

Forced stop

poweroff

curl -k -X POST https://{ip_address}/api/vcenter/vm/{vm_id}/power?action=stop -H "vmware-api-session-id: {api_session_id}"

reset

curl -k -X POST https://{ip_address}/api/vcenter/vm/{vm_id}/power?action=reset -H "vmware-api-session-id: {api_session_id}"
Deleting the created session

Deleting the session

curl -k -X DELETE https://{ip_address}/api/session -H "vmware-api-session-id: {api_session_id}"

vCLI commands for vCenter forced stop resource

For forcibly stopping the guest OS on a virtual machine, use the [vmcontrol] command of the vCLI. For periodically checking the status of the forced stop resource and for checking whether a forced stop succeeds, use the [vminfo] command.

The following table shows commands to be executed for forced stop actions by the vCenter forced stop resource:

Forced-stop commands

poweroff

vmcontrol.pl --server ip_address --username username --password password --vmname virtualmachine --datacenter datacenter --operation poweroff

reset

vmcontrol.pl --server ip_address --username username --password password --vmname virtualmachine --datacenter datacenter --operation reset
The following table shows commands to be executed for periodically checking the status of the vCenter forced stop resource:

Command to periodically check the status

vminfo.pl --server ip_address --username username --password password --vmname virtualmachine --datacenter datacenter --powerstatus "poweredOn"
The following table shows commands to be executed for checking whether a forced stop succeeds:

Command to check a forced stop

vminfo.pl --server ip_address --username username --password password --vmname virtualmachine --datacenter datacenter --powerstatus "poweredOff"

Editing commands for vCenter forced stop resource

For forced stop resources, you can also specify a command line for a forced stop by editing a script file for the command execution.

When using the vCenter forced stop resource with vSphere Automation APIs, you can edit the following file:

<EXPRESSCLUSTER_installation_path>\bin\clpvcenterrestapiforcestop.sh

When using the vCenter forced stop resource with the vCLI, you can edit the following file:

<EXPRESSCLUSTER_installation_path>\bin\clpvcentercliforcestop.sh

The vCenter forced stop resource sets environment variable values necessary for commands to be executed with the script.

The following table shows environment variables written in the script:

Environment variable

Setting value

Description
CLP_VCLI_PATH
...vCLI install path

Install path

Specifies the VMware vSphere CLI install path set in the vCenter Forced-Stop Properties.
CLP_VCENTER_ACTION
...Forced Stop Action
poweroff : power off
reset : reset

Specifies the Forced Stop Action set in the vCenter Forced-Stop Properties.
CLP_VCENTER_HOST
...Host name for vCenter

Host name

Specifies the host name set in the vCenter Forced-Stop Properties.
CLP_VCENTER_USER
...User name for vCenter

User name

Specifies the user name set in the vCenter Forced-Stop Properties.
CLP_VCENTER_PASSWORD
...Password for vCenter

Password

Specifies the password set in the vCenter Forced-Stop Properties.
CLP_VMNAME
...Virtual machine name

Virtual machine name

Specifies the virtual machine name set in the vCenter Forced-Stop Properties.
CLP_DATACENTER_NAME
...Data center name

Data center name

Specifies the data center name set in the vCenter Forced-Stop Properties.

For more information on configuring the vCenter forced stop resource, see this guide: "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced stop" -> "vCenter Forced-Stop Properties".

7.3.2. Notes on vCenter forced stop resource

  • Forcibly stopping the guest OS on a virtual machine
    Only power off operation can be performed. This function cannot be used if communication with VMWare vCenter Server cannot be performed.
    - vSphere infrastructure: Communication with VMWare vCenter Server is not possible.
  • Impacts on forced stop
    When you use the forced stop function, the following functions are influenced because power off, reset is forcibly performed regardless of the OS or server status.
    • Dump collection
      Because it is not recognized that dump files are being collected, power off, reset is performed even though dump collection is being performed, so dump collection does not complete.
    • Power on within heartbeat timeout
      When the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, power off, reset may occur after heartbeat timeout has elapsed.
  • Power Options settings of the OS
    Conducting a power-off of the guest OS on a virtual machine with the vCLI may perform an action selected in Power Options of the OS, such as Sleep, Hibernate, or Shutdown.
    The settings can be referred to and configured by the following instruction:
    Open Power Options in Control Panel and select Choose what the power button does, Power button settings and When I press the power button:.
    When Forced stop is used in EXPRESSCLUSTER, it is recommended that this setting is configured as No Operation.

7.4. Understanding forced stop on AWS environment

7.4.1. Settings of AWS forced stop resource

Using the AWS forced stop resource requires installing the AWS Command Line Interface (AWS CLI).

For information on how to obtain and install the AWS CLI, see "Started Guide" -> "Notes and Restrictions" -> "Before installing EXPRESSCLUSTER" -> "Time synchronization in the AWS environment" and "IAM settings in the AWS environment".

Commands for AWS forced stop resource

For forcibly stopping an AWS instance, for periodically checking the status of the forced stop resource, and for checking whether a forced stop succeeds, use the command of the AWS CLI.

When executing either of the commands, specify the following option values:

command option

item to be set in AWS Forced-Stop Properties -> Server List -> Edit -> Input of Instance

--instance-ids instance-ids

InstanceID

The following table shows commands to be executed for forced stop actions by the AWS forced stop resource:

Forced Stop Action

Parameters

stop

aws ec2 stop-instances --instance-ids instance-ids --force

reboot

aws ec2 reboot-instances --instance-ids instance-ids

The following table shows commands to be executed for periodically checking the status of the AWS forced stop resource:

Forced Stop Action

Command to periodically check the status

stop

aws ec2 stop-instances --instance-ids instance-ids --dry-run

reboot

aws ec2 reboot-instances --instance-ids instance-ids --dry-run

The following table shows commands to be executed for checking whether a forced stop succeeds:

Command to periodically check the status

aws ec2 describe-instances --instance-ids instance-ids --filters \"Name=instance-state-name,Values=stopped\"

Editing commands for AWS forced stop resource

For forced stop resources, you can also specify a command line for a forced stop by editing a script file for the command execution.

The following file can be edited:

<EXPRESSCLUSTER_install_path>\cloud\aws\clpawsforcestop.sh

For more information on configuring the AWS forced stop resource, see this guide: "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced stop" -> "AWS Forced-Stop Properties".

7.4.2. Applying command line options to AWS CLI run from AWS forced stop resource

7.4.3. Applying environment variables to AWS CLI run from the AWS forced stop resource

7.4.4. Notes on AWS forced stop resource

  • For forcibly stopping instance
    You can perform only the following actions: stop and reboot.
  • Impacts on forced stop
    When you use the forced stop function, the following functions are influenced because stop, reboot is forcibly performed regardless of the OS or server status.
    • Dump collection
      Because it is not recognized that dump files are being collected, stop, reboot is performed even though dump collection is being performed, so dump collection does not complete.
    • Power on within heartbeat timeout
      When the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, stop, reboot may occur after heartbeat timeout has elapsed.

7.5. Understanding forced stop on Azure environment

7.5.1. Settings of Azure forced stop resource

Using the Azure forced stop resource requires installing the Azure CLI.

Commands for Azure forced stop resource

For forcibly stopping an Azure instance, for periodically checking the status of the forced stop resource, and for checking whether a forced stop succeeds, use the command of the Azure CLI.

When executing either of the commands, specify the following option values:

command option

item to be set in Azure Forced-Stop Properties -> the Azure tab

item to be set in Azure Forced-Stop Properties -> Server List -> Edit -> Input for Virtual Machine name

--name/-n

-

Virtual Machine Name

--username/-u

User URI

-

--tenant/-t

Tenant ID

-

--password/-p

File Path of Service Principal

-

--resource-group/-g

Resource Group Name

-

For the use of Azure CLI commands, the following command is executed:

Logging in to the Azure CLI

az login --service-principal -u MyUserUri -p MyCertfile.pem --tenant MyTenantId

The following table shows commands to be executed for forced stop actions by the Azure forced stop resource:

Forced Stop Action

Parameters

stop

az vm deallocate -g MyResourceGroup -n MyVm

reboot

az vm restart -g MyResourceGroup -n MyVm --force

The following table shows commands to be executed for periodically checking the status of the Azure forced stop resource:

Command to periodically check the status

az vm update -g MyResourceGroup -n MyVm --no-wait

The following table shows commands to be executed for checking whether a forced stop succeeds:

Command to periodically check the status

az vm get-instance-view -g MyResourceGroup -n MyVm --query instanceView.statuses[1].displayStatus --output tsv

Editing commands Azure forced stop resource

For forced stop resources, you can also specify a command line for a forced stop by editing a script file for the command execution.

The following file can be edited:

<EXPRESSCLUSTER_install_path>\cloud\azure\clpazureforcestop.sh

For more information on configuring the Azure forced stop resource, see this guide: "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced stop" -> "Azure Forced-Stop Properties".

7.5.2. Notes on Azure forced stop resource

  • For forcibly stopping instance
    You can perform only the following actions: stop and reboot.
  • Impacts on forced stop
    When you use the forced stop function, the following functions are influenced because stop, reboot is forcibly performed regardless of the OS or server status.
    • Dump collection
      Because it is not recognized that dump files are being collected, stop, reboot is performed even though dump collection is being performed, so dump collection does not complete.
    • Power on within heartbeat timeout
      When the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, stop, reboot may occur after heartbeat timeout has elapsed.

7.6. Understanding forced stop on OCI environment

7.6.1. Settings of OCI forced stop resource

Using the OCI forced stop resource requires installing the Oracle Cloud Infrastructure CLI (OCI CLI).

For information on how to obtain and install the OCI CLI, see "Started Guide" -> "Notes and Restrictions" -> "Before installing EXPRESSCLUSTER" -> "CLI settings in the OCI environment" and "Configuring OCI forced-stop resource".

Commands for OCI forced stop resource

For forcibly stopping an OCI instance, for periodically checking the status of the forced stop resource, and for checking whether a forced stop succeeds, use the command of the OCI CLI.

When executing either of the commands, specify the following option values:

command option

item to be set in OCI Forced-Stop Properties -> Server List -> Edit -> Input of Instance

--instance-ids instance-ids

InstanceID

The following table shows commands to be executed for forced stop actions by the OCI forced stop resource:

Forced Stop Action

Parameters

stop

oci compute instance action --action STOP --instance-id instance-ids

reboot

oci compute instance action --action RESET --instance-id instance-ids

The following table shows commands to be executed for periodically checking the status of the OCI forced stop resource:

Command to periodically check the status

oci compute instance update --instance-id instance-ids --wait-for-state RUNNING --max-wait-seconds 1

The following table shows commands to be executed for checking whether a forced stop succeeds:

Command to periodically check the status

oci compute instance get --instance-id instance-ids | grep lifecycle-state | awk -F" '{print $4}'

Editing commands OCI forced stop resource

For forced stop resources, you can also specify a command line for a forced stop by editing a script file for the command execution.

The following file can be edited:

<EXPRESSCLUSTER_install_path>\cloud\oci\clpociforcestop.sh

For more information on configuring the OCI forced stop resource, see this guide: "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced stop" -> "OCI Forced-Stop Properties".

7.6.2. Notes on OCI forced stop resource

  • For forcibly stopping instance
    You can perform only the following actions: stop and reboot.
  • Impacts on forced stop
    When you use the forced stop function, the following functions are influenced because stop, reboot is forcibly performed regardless of the OS or server status.
    • Dump collection
      Because it is not recognized that dump files are being collected, stop, reboot is performed even though dump collection is being performed, so dump collection does not complete.
    • Power on within heartbeat timeout
      When the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, stop, reboot may occur after heartbeat timeout has elapsed.

7.7. Understanding forced stop with script

7.7.1. Settings of custom forced stop resource

You can create a script for a forced stop. When a server crash is recognized, using the script on the remaining servers (operating properly) allows you to forcibly stop the down server.

The script is executed in both of the following modes: performing a forced stop and periodically checking if the target can be forcibly stopped. For appropriate processing based on each of the modes, write conditional branches by using environment variables described later.

Environment variables for script

When executing the script, EXPRESSCLUSTER sets environment variable values such as which mode (a periodical status check or a forced stop) to be performed and what server has crashed.

In the script, you can use the following environment variables:

Environment variable

Setting value

Description
CLP_FORCESTOP_MODE
...Mode
0 : When periodically checking the status
1 : When performing a forced stop
Means a mode to be performed.
Can be used for process branches for each of the modes.
CLP_SERVER_DOWN
...Down server name

Server name
Means the name of a down server.
For periodically checking the status, "" is set.
CLP_SERVER_LOCAL
...Local server name

Server name

Means the name of a server to execute the script.

Returned value of script

Return 0 when the script terminates normally.

For more information on configuring the custom forced stop resource, see this guide: "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced stop" -> "Custom Forced-Stop Properties".

7.7.2. Notes on custom forced stop resource

  • Describe the customer-defined process in the script to stop the server.

  • If there is nothing to be done as periodically checking the status, write the process as such (so that the value 0 can be returned).

7.8. Notes on settings of forced stop resource

  • You can configure only one forced stop resource for one cluster.

  • If you want to configure a forced stop resource, it is recommended to configure a network partition resolution resource as well.

  • In configuring a forced stop resource, all the cluster servers must be set to use the forced stop resource.