7. Forced stop resource details¶
This chapter provides the detailed information on forced stop resources.
7.1. What is the forced stop function?¶
The method for forcibly stopping the failing server varies depending on the type of environment where the cluster was created: physical machines, virtual machines, or the cloud. When using the forced stop function, configure a forced stop resource corresponding to the environment type.
Moreover, you can execute a script in which the procedure for stopping the failing server is written. For details, refer to "7.7. Understanding forced stop with script" in this guide.
A forced stop resource operates in two ways: performing a forced stop and periodically checking if the target can be forcibly stopped. The following explains what these are and when they are done:
Performing a forced stop
Forcibly stops a down server by using a device or infrastructure system as a server status manager.
Done by recognizing the server crash. For more information on the conditions, see "7.1.1. Conditions for performing forced stop".
Periodically checking if the target can be forcibly stopped
Checks whether a forced stop can be performed, by communicating with a device or infrastructure system for forcibly stopping a server. Depending on the result, the forced stop resource shows whether the server can be forcibly stopped: "Normal" (yes) or "Error" (no).
Done on a regular basis while the cluster service is running.
7.1.1. Conditions for performing forced stop¶
Forced stop is not performed when:
The failover group successfully stops before the server fails
The server is shut down by the clpdown command or Cluster WebUI and the failover group successfully stops
The cluster is stopped by the clpcl command or Cluster WebUI and the failover group successfully stops
- The server fails and there is no failover group to perform failover from the failing server to another server(including when the failover group is not activated in the failing server)
Forced stop is performed when:
The server is failing and there is a failover group to perform failover from the failing server to another server
7.2. Understanding forced stop on physical environment¶
7.2.1. Settings of BMC forced stop resource¶
When you use the ipmiutil command, it is necessary to install ipmiutil in each cluster server. For information on how to get ipmiutil and how to install it, refer to "Setup of BMC and ipmiutil" in "Settings after configuring hardware" in "Determining a system configuration" in the "Installation and Configuration Guide".
Commands for BMC forced stop resource
For forcibly stopping a physical machine, use the [ipmiutil reset] command of IPMI Management Utilities.
For periodically checking the status of the forced stop resource and for checking whether a forced stop succeeds, use the [ipmiutil health] command.
When executing either of the commands, specify the following option values:
command option |
item to be set in BMC Forced-Stop Properties -> Server List -> Edit -> Enter BMC |
---|---|
-N ip_address |
IP address |
-U username |
User name |
-P password |
Password |
The following table shows commands to be executed for forced stop actions by the BMC forced stop resource:
Forced Stop Action |
Parameters |
---|---|
BMC Power Off |
ipmiutil reset -d -N ip_address -U username -P password |
BMC Reset |
ipmiutil reset -r -N ip_address -U username -P password |
BMC Power Cycle |
ipmiutil reset -c -N ip_address -U username -P password |
BMC NMI |
ipmiutil reset -n -N ip_address -U username -P password |
If the above commands fail to be executed, perform the following commands:
Forced Stop Action |
Parameters |
---|---|
BMC Power Off |
ipmiutil reset -d -N ip_address -U username -P password -J 3 |
BMC Reset |
ipmiutil reset -r -N ip_address -U username -P password -J 3 |
BMC Power Cycle |
ipmiutil reset -c -N ip_address -U username -P password -J 3 |
BMC NMI |
ipmiutil reset -n -N ip_address -U username -P password -J 3 |
The following table shows commands to be executed for periodically checking the status of the BMC forced stop resource and for checking whether a forced stop succeeds:
Command to periodically check the status
Command to check a forced stop
|
---|
ipmiutil health -N ip_address -U username -P password |
If the above command fails to be executed, perform the following command:
Command to periodically check the status
Command to check a forced stop
|
---|
ipmiutil health -N ip_address -U username -P password -J 3 |
Editing commands for BMC forced stop resource
For forced stop resources, you can also specify a command line for a forced stop by editing a script file for the command execution.
The following file can be edited:
<EXPRESSCLUSTER_install_path>\bin\clpbmcforcestop.bat
For the BMC forced stop resource, set environment variable values necessary for commands to be executed with the script.
The following table shows environment variables written in the script:
Environment variable |
Setting value |
Description |
---|---|---|
CLP_BMC_ACTION
...Forced Stop Action
|
-d : BMC Power Off
-r : BMC Reset
-c : BMC Power Cycle
-n : BMC NMI
|
Specifies the Forced Stop Action set in the BMC Forced-Stop Properties. |
CLP_BMC_HOST
...IP address for BMC
|
IP Address |
Specifies the IP address set in the BMC Forced-Stop Properties. |
CLP_BMC_USER
...User name for BMC
|
User name |
Specifies the user name set in the BMC Forced-Stop Properties. |
CLP_BMC_PASSWORD
...Password for BMC
|
Password |
Specifies the password set in the BMC Forced-Stop Properties. |
For more information on configuring the BMC forced stop resource, see this guide: "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced stop" -> "BMC Forced-Stop Properties".
7.2.2. Notes on BMC forced stop resource¶
- Impacts on forced stopWhen you use the forced stop function, the following functions are influenced because power off, reset, power cycle or NMI is forcibly performed regardless of the OS or server status.
- Dump collectionBecause it is not recognized that dump files are being collected, power off, reset or power cycle is performed even though dump collection is being performed, so dump collection does not complete.
- Power on within heartbeat timeoutWhen the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, power off, reset, power cycle or NMI may occur after heartbeat timeout has elapsed.
- BMC network settingsWhen using the BMC forced stop resource, disable the iLO shared network port.Configure the settings so that the IP address of the LAN port for BMC management and the IP address which OS uses can communicate with each other. This function cannot be used when BMC is not installed in the server, or in the environment where the network for the BMC management is blocked.Navigate BMC Forced-Stop Properties -> Server List -> Edit -> Enter BMC, then enter the IP address assigned to the LAN port for BMC management.See the server's manuals etc. for information on how to configure the IP address of the LAN port for the BMC management etc.
- Power Options settings of the OSConducting a power-off or power cycle by BMC may perform an action selected in Power Options of the OS, such as Sleep, Hibernate, or Shutdown.The settings can be referred to and configured by the following instruction:Open Power Options in Control Panel and select Choose what the power button does, Power button settings and When I press the power button:.When Forced stop is used in EXPRESSCLUSTER, it is recommended that this setting is configured as No Operation.
7.3. Understanding forced stop on vCenter environment¶
7.3.1. Settings of vCenter forced stop resource¶
The vCenter forced stop resource can be used with vSphere Automation APIs or with the VMware vSphere Command Line Interface (vCLI).
In an environment with VMware vSphere 7.0 Update 3 or higher, the vCLI cannot be used. Instead, use vSphere Automation APIs.
Setting parameters of vCenter forced stop resource
Parameter |
Item to be set in vCenter Forced-Stop Properties -> the vCenter tab |
Item to be set in vCenter Forced-Stop Properties -> Server List -> Edit -> Input for Virtual Machine name |
---|---|---|
ip_address |
IP address |
- |
username |
User name |
- |
password |
Password |
- |
virtualmachine |
- |
Virtual machine name |
datacenter |
- |
Datacenter name |
APIs for vCenter forced stop resource
When the vCenter forced stop resource is used with vSphere Automation APIs, the following APIs are executed:
Creating a session |
---|
curl -k -X POST -u {username}:{password} https://{ip_address}/api/session |
Retrieving data center information |
---|
curl -k -X GET https://{ip_address}/api/vcenter/datacenter?names={datacenter} -H "vmware-api-session-id: {api_session_id}" |
Retrieving virtual-machine information |
---|
curl -k -X GET https://{ip_address}/api/vcenter/vm?names={virtualmachine}&datacenters={datacenter_id} -H "vmware-api-session-id: {api_session_id}" |
Forced stop |
|
---|---|
poweroff |
curl -k -X POST https://{ip_address}/api/vcenter/vm/{vm_id}/power?action=stop -H "vmware-api-session-id: {api_session_id}" |
reset |
curl -k -X POST https://{ip_address}/api/vcenter/vm/{vm_id}/power?action=reset -H "vmware-api-session-id: {api_session_id}" |
Deleting the session |
---|
curl -k -X DELETE https://{ip_address}/api/session -H "vmware-api-session-id: {api_session_id}" |
vCLI commands for vCenter forced stop resource
For forcibly stopping the guest OS on a virtual machine, use the [vmcontrol] command of the vCLI. For periodically checking the status of the forced stop resource and for checking whether a forced stop succeeds, use the [vminfo] command.
Using the vCLI requires installing the package.
Note
Set the Perl path
Select Cluster Properties -> Fencing tab -> Forced stop -> vCenter Forced-Stop Properties -> vCenter tab, specify the path to the Perl execution module for Perl Path. This is common to all the servers in the cluster.For more information about the Perl path, refer to "Extension Tab" in "Cluster properties" in "2.2.3. Fencing tab" in "Forced stop" in "vCenter Forced-Stop Properties" in this guide.Add the system environment variable
Add the following variable for the system environment variable. Then restart the OS.Variable name: PERL5LIBVariable value: vCLI Perl module path (Example: C:\Program Files (x86)\VMware\VMware vSphere CLI\Perl\lib
Forced-stop commands |
|
---|---|
poweroff |
vmcontrol.pl --server ip_address --username username --password password --vmname virtualmachine --datacenter datacenter --operation poweroff |
reset |
vmcontrol.pl --server ip_address --username username --password password --vmname virtualmachine --datacenter datacenter --operation reset |
Command to periodically check the status |
---|
vminfo.pl --server ip_address --username username --password password --vmname virtualmachine --datacenter datacenter |
Command to check a forced stop |
---|
vminfo.pl --server ip_address --username username --password password --vmname virtualmachine --datacenter datacenter --powerstatus "poweredOff" |
Editing commands for vCenter forced stop resource
For forced stop resources, you can also specify a command line for a forced stop by editing a script file for the command execution.
When using the vCenter forced stop resource with vSphere Automation APIs, you can edit the following file:
<EXPRESSCLUSTER_installation_path>\bin\clpvcenterrestapiforcestop.bat
When using the vCenter forced stop resource with the vCLI, you can edit the following file:
<EXPRESSCLUSTER_installation_path>\bin\clpvcentercliforcestop.bat
For the vCenter forced stop resource, set environment variable values necessary for commands to be executed with the script.
The following table shows environment variables written in the script:
Environment variable |
Setting value |
Description |
---|---|---|
CLP_VCLI_PATH
...vCLI install path
|
Install path |
Specifies the VMware vSphere CLI install path set in the vCenter Forced-Stop Properties. |
CLP_VCENTER_ACTION
...Forced Stop Action
|
poweroff : power off
reset : reset
|
Specifies the Forced Stop Action set in the vCenter Forced-Stop Properties. |
CLP_VCENTER_HOST
...Host name for vCenter
|
Host name |
Specifies the host name set in the vCenter Forced-Stop Properties. |
CLP_VCENTER_USER
...User name for vCenter
|
User name |
Specifies the user name set in the vCenter Forced-Stop Properties. |
CLP_VCENTER_PASSWORD
...Password for vCenter
|
Password |
Specifies the password set in the vCenter Forced-Stop Properties. |
CLP_VMNAME
...Virtual machine name
|
Virtual machine name |
Specifies the virtual machine name set in the vCenter Forced-Stop Properties. |
CLP_DATACENTER_NAME
...Data center name
|
Data center name |
Specifies the data center name set in the vCenter Forced-Stop Properties. |
CLP_PERL_PATH
...Perl path
|
Perl path |
Specifies the perl path set in the vCenter Forced-Stop Properties. |
For more information on configuring the vCenter forced stop resource, see this guide: "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced stop" -> "vCenter Forced-Stop Properties".
7.3.2. Notes on vCenter forced stop resource¶
- Forcibly stopping the guest OS on a virtual machineOnly power off operation can be performed. Moreover, this function cannot be used in the following cases:- vSphere infrastructure: Communication with VMWare vCenter Server is not possible.
- Impacts on forced stopWhen you use the forced stop function, the following functions are influenced because power off, reset is forcibly performed regardless of the OS or server status.
- Dump collectionBecause it is not recognized that dump files are being collected, power off, reset is performed even though dump collection is being performed, so dump collection does not complete.
- Power on within heartbeat timeoutWhen the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, power off, reset may occur after heartbeat timeout has elapsed.
- Power Options settings of the OSConducting a power-off of the guest OS on a virtual machine with the vCLI may perform an action selected in Power Options of the OS, such as Sleep, Hibernate, or Shutdown.The settings can be referred to and configured by the following instruction:Open Power Options in Control Panel and select Choose what the power button does, Power button settings and When I press the power button:.When Forced stop is used in EXPRESSCLUSTER, it is recommended that this setting is configured as No Operation.
7.4. Understanding forced stop on AWS environment¶
7.4.1. Settings of AWS forced stop resource¶
Using the AWS forced stop resource requires installing the AWS Command Line Interface (AWS CLI).
For information on how to obtain and install the AWS CLI, see "Started Guide" -> "Notes and Restrictions" -> "Before installing EXPRESSCLUSTER" -> "Time synchronization in the AWS environment" and "IAM settings in the AWS environment".
Commands for AWS forced stop resource
For forcibly stopping an AWS instance, for periodically checking the status of the forced stop resource, and for checking whether a forced stop succeeds, use the command of the AWS CLI.
When executing either of the commands, specify the following option values:
command option |
item to be set in AWS Forced-Stop Properties -> Server List -> Edit -> Input of Instance |
---|---|
--instance-ids instance-ids |
InstanceID |
The following table shows commands to be executed for forced stop actions by the AWS forced stop resource:
Forced Stop Action |
Parameters |
---|---|
stop |
aws ec2 stop-instances --instance-ids instance-ids --force |
reboot |
aws ec2 reboot-instances --instance-ids instance-ids |
The following table shows commands to be executed for periodically checking the status of the AWS forced stop resource:
Forced Stop Action |
Command to periodically check the status |
---|---|
stop |
aws ec2 stop-instances --instance-ids instance-ids --dry-run |
reboot |
aws ec2 reboot-instances --instance-ids instance-ids --dry-run |
The following table shows commands to be executed for checking whether a forced stop succeeds:
Command to periodically check the status |
---|
aws ec2 describe-instances --instance-ids instance-ids --filters \"Name=instance-state-name,Values=stopped\" |
Editing commands for AWS forced stop resource
For forced stop resources, you can also specify a command line for a forced stop by editing a script file for the command execution.
The following file can be edited:
<EXPRESSCLUSTER_install_path>\cloud\aws\clpawsforcestop.ps1
For more information on configuring the AWS forced stop resource, see this guide: "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced stop" -> "AWS Forced-Stop Properties".
7.4.2. Applying command line options to AWS CLI run from AWS forced stop resource¶
7.4.3. Applying environment variables to AWS CLI run from the AWS forced stop resource¶
7.4.4. Notes on AWS forced stop resource¶
- For forcibly stopping instanceYou can perform only the following actions: stop and reboot.
- Impacts on forced stopWhen you use the forced stop function, the following functions are influenced because stop, reboot is forcibly performed regardless of the OS or server status.
- Dump collectionBecause it is not recognized that dump files are being collected, stop, reboot is performed even though dump collection is being performed, so dump collection does not complete.
- Power on within heartbeat timeoutWhen the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, stop, reboot may occur after heartbeat timeout has elapsed.
- Power Options settings of the OSConducting the action (stop or reboot) with the AWS CLI may perform an action selected in Power Options of the OS, such as Sleep, Hibernate, or Shutdown.The settings can be referred to and configured by the following instruction:Open Power Options in Control Panel and select Choose what the power button does, Power button settings and When I press the power button:.When Forced stop is used in EXPRESSCLUSTER, it is recommended that this setting is configured as No Operation.
7.5. Understanding forced stop on Azure environment¶
7.5.1. Settings of Azure forced stop resource¶
Using the Azure forced stop resource requires installing the Azure CLI.
Commands for Azure forced stop resource
For forcibly stopping an Azure instance, for periodically checking the status of the forced stop resource, and for checking whether a forced stop succeeds, use the command of the Azure CLI.
When executing either of the commands, specify the following option values:
command option |
item to be set in Azure Forced-Stop Properties -> the Azure tab |
item to be set in Azure Forced-Stop Properties -> Server List -> Edit -> Input for Virtual Machine name |
---|---|---|
--name/-n |
- |
Virtual Machine Name |
--username/-u |
User URI |
- |
--tenant/-t |
Tenant ID |
- |
--password/-p |
File Path of Service Principal |
- |
--resource-group/-g |
Resource Group Name |
- |
For the use of Azure CLI commands, the following command is executed:
Logging in to the Azure CLI |
---|
az login --service-principal -u MyUserUri -p MyCertfile.pem --tenant MyTenantId |
The following table shows commands to be executed for forced stop actions by the Azure forced stop resource:
Forced Stop Action |
Parameters |
---|---|
stop |
az vm deallocate -g MyResourceGroup -n MyVm |
reboot |
az vm restart -g MyResourceGroup -n MyVm --force |
The following table shows commands to be executed for periodically checking the status of the Azure forced stop resource:
Command to periodically check the status |
---|
az vm update -g MyResourceGroup -n MyVm --no-wait |
The following table shows commands to be executed for checking whether a forced stop succeeds:
Command to periodically check the status |
---|
az vm get-instance-view -g MyResourceGroup -n MyVm --query instanceView.statuses[1].displayStatus --output tsv |
Editing commands Azure forced stop resource
For forced stop resources, you can also specify a command line for a forced stop by editing a script file for the command execution.
The following file can be edited:
<EXPRESSCLUSTER_install_path>\cloud\azure\clpazureforcestop.ps1
For more information on configuring the Azure forced stop resource, see this guide: "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced stop" -> "Azure Forced-Stop Properties".
7.5.2. Notes on Azure forced stop resource¶
- For forcibly stopping instanceYou can perform only the following actions: stop and reboot.
- Impacts on forced stopWhen you use the forced stop function, the following functions are influenced because stop, reboot is forcibly performed regardless of the OS or server status.
- Dump collectionBecause it is not recognized that dump files are being collected, stop, reboot is performed even though dump collection is being performed, so dump collection does not complete.
- Power on within heartbeat timeoutWhen the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, stop, reboot may occur after heartbeat timeout has elapsed.
- Power Options settings of the OSConducting the action (stop or reboot) with the OCI CLI may perform an action selected in Power Options of the OS, such as Sleep, Hibernate, or Shutdown.The settings can be referred to and configured by the following instruction:Open Power Options in Control Panel and select Choose what the power button does, Power button settings and When I press the power button:.When Forced stop is used in EXPRESSCLUSTER, it is recommended that this setting is configured as No Operation.
7.6. Understanding forced stop on OCI environment¶
7.6.1. Settings of OCI forced stop resource¶
Using the OCI forced stop resource requires installing the Oracle Cloud Infrastructure CLI (OCI CLI).
For information on how to obtain and install the OCI CLI, see "Started Guide" -> "Notes and Restrictions" -> "Before installing EXPRESSCLUSTER" -> "CLI settings in the OCI environment" and "Configuring OCI forced-stop resource".
Commands for OCI forced stop resource
For forcibly stopping an OCI instance, for periodically checking the status of the forced stop resource, and for checking whether a forced stop succeeds, use the command of the OCI CLI.
When executing either of the commands, specify the following option values:
command option |
item to be set in OCI Forced-Stop Properties -> Server List -> Edit -> Input of Instance |
---|---|
--instance-ids instance-ids |
InstanceID |
The following table shows commands to be executed for forced stop actions by the OCI forced stop resource:
Forced Stop Action |
Parameters |
---|---|
stop |
oci compute instance action --action STOP --instance-id instance-ids |
reboot |
oci compute instance action --action RESET --instance-id instance-ids |
The following table shows commands to be executed for periodically checking the status of the OCI forced stop resource:
Command to periodically check the status |
---|
oci compute instance update --instance-id instance-ids --wait-for-state RUNNING --max-wait-seconds 1 |
The following table shows commands to be executed for checking whether a forced stop succeeds:
Command to periodically check the status |
---|
oci compute instance get --instance-id instance-ids | grep lifecycle-state | awk -F" '{print $4}' |
Editing commands OCI forced stop resource
For forced stop resources, you can also specify a command line for a forced stop by editing a script file for the command execution.
The following file can be edited:
<EXPRESSCLUSTER_install_path>\cloud\oci\clpociforcestop.ps1
For more information on configuring the OCI forced stop resource, see this guide: "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced stop" -> "OCI Forced-Stop Properties".
7.6.2. Notes on OCI forced stop resource¶
- For forcibly stopping instanceYou can perform only the following actions: stop and reboot.
- Impacts on forced stopWhen you use the forced stop function, the following functions are influenced because stop, reboot is forcibly performed regardless of the OS or server status.
- Dump collectionBecause it is not recognized that dump files are being collected, stop, reboot is performed even though dump collection is being performed, so dump collection does not complete.
- Power on within heartbeat timeoutWhen the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, stop, reboot may occur after heartbeat timeout has elapsed.
- Power Options settings of the OSConducting the action (stop or reboot) with the OCI CLI may perform an action selected in Power Options of the OS, such as Sleep, Hibernate, or Shutdown.The settings can be referred to and configured by the following instruction:Open Power Options in Control Panel and select Choose what the power button does, Power button settings and When I press the power button:.When Forced stop is used in EXPRESSCLUSTER, it is recommended that this setting is configured as No Operation.
7.7. Understanding forced stop with script¶
7.7.1. Settings of custom forced stop resource¶
You can create a script for a forced stop. When a server crash is recognized, using the script on the remaining servers (operating properly) allows you to forcibly stop the down server.
The script is executed in both of the following modes: performing a forced stop and periodically checking if the target can be forcibly stopped. For appropriate processing based on each of the modes, write conditional branches by using environment variables described later.
Environment variables for script
When executing the script, EXPRESSCLUSTER sets environment variable values such as which mode (a periodical status check or a forced stop) to be performed and what server has crashed.
In the script, you can use the following environment variables:
Environment variable |
Setting value |
Description |
---|---|---|
CLP_FORCESTOP_MODE
...Mode
|
0 : When periodically checking the status
1 : When performing a forced stop
|
Means a mode to be performed.
Can be used for process branches for each of the modes.
|
CLP_SERVER_DOWN
...Down server name
|
Server name |
Means the name of a down server.
For periodically checking the status, "" is set.
|
CLP_SERVER_LOCAL
...Local server name
|
Server name |
Means the name of a server to execute the script. |
Returned value of script
Return 0 when the script terminates normally.
For more information on configuring the custom forced stop resource, see this guide: "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced stop" -> "Custom Forced-Stop Properties".
7.7.2. Notes on custom forced stop resource¶
Describe the customer-defined process in the script to stop the server.
If there is nothing to be done as periodically checking the status, write the process as such (so that the value 0 can be returned).
7.8. Notes on settings of forced stop resource¶
You can configure only one forced stop resource for one cluster.
If you want to configure a forced stop resource, it is recommended to configure a network partition resolution resource as well.
In configuring a forced stop resource, all the cluster servers must be set to use the forced stop resource.
To prevent a split-brain syndrome in a failover group with a forced-stop resource operating, set the service startup delay time as follows:
- When setting up DISK network partition resolution resourcesService startup delay time >= forced-stop timeout of forced-stop resource + time to wait for stop to be completed of forced-stop resource + heartbeat timeout + heartbeat interval + IO Wait Time of DISK network partition resolution resources + 10 seconds
- When not setting up DISK network partition resolution resourcesService startup delay time >= forced-stop timeout of forced-stop resource + time to wait for stop to be completed of forced-stop resource + heartbeat timeout + heartbeat interval
For more information on the service startup delay time, see "Adjustment of time for EXPRESSCLUSTER services to start up (Required)".