7. Forced stop resource details¶
This chapter provides the detailed information on forced stop resources.
7.1. What is the forced stop function?¶
The method for forcibly stopping the failing server varies depending on the type of environment where the cluster was created: physical machines, virtual machines, or the cloud. When using the forced stop function, configure a forced stop resource corresponding to the environment type.
Moreover, you can execute a script in which the procedure for stopping the failing server is written. For details, refer to "7.7. Understanding forced stop with script" in this guide.
A forced stop resource operates in two ways: performing a forced stop and periodically checking if the target can be forcibly stopped. The following explains what these are and when they are done:
Performing a forced stop
Forcibly stops a down server by using a device or infrastructure system as a server status manager.
Done by recognizing the server crash. For more information on the conditions, see "7.1.1. Conditions for performing forced stop".
Periodically checking if the target can be forcibly stopped
Checks whether a forced stop can be performed, by communicating with a device or infrastructure system for forcibly stopping a server. Depending on the result, the forced stop resource shows whether the server can be forcibly stopped: "Normal" (yes) or "Error" (no).
Done on a regular basis while the cluster service is running.
7.1.1. Conditions for performing forced stop¶
Forced stop is not performed when:
The failover group successfully stops before the server fails
The server is shut down by the clpdown command, the OS shutdown command or Cluster WebUI and the failover group successfully stops
The cluster is stopped by the clpcl command or Cluster WebUI and the failover group successfully stops
- The server fails and there is no failover group to perform failover from the failing server to another server(including when the failover group is not activated in the failing server)
Forced stop is performed when:
The server is failing and there is a failover group to perform failover from the failing server to another server
7.2. Understanding forced stop on physical environment¶
7.2.1. Settings of BMC forced stop resource¶
Using the BMC forced stop resource requires installing ipmitool on each cluster server.
Commands for BMC forced stop resource
For forcibly stopping a physical machine, for periodically checking the status of the forced stop resource, and for checking whether a forced stop succeeds, use the [ipmitool] command.
When executing either of the commands, specify the following option values:
[ipmitool] command option |
item to be set in BMC Forced-Stop Properties -> Server List -> Edit -> Enter BMC |
---|---|
-H ip_address |
IP address |
-U username |
User name |
-P password |
Password |
The following table shows commands to be executed for forced stop actions by the BMC forced stop resource:
Forced Stop Action |
Parameters |
---|---|
BMC Power Off |
ipmitool -H ip_address -U username -P password power off |
BMC Reset |
ipmitool -H ip_address -U username -P password power reset |
BMC Power Cycle |
ipmitool -H ip_address -U username -P password power cycle |
BMC NMI |
ipmitool -H ip_address -U username -P password power diag |
If the above commands fail to be executed, perform the following commands:
Forced Stop Action |
Parameters |
---|---|
BMC Power Off |
ipmitool -H ip_address -I lanplus -U username -P password power off |
BMC Reset |
ipmitool -H ip_address -I lanplus -U username -P password power reset |
BMC Power Cycle |
ipmitool -H ip_address -I lanplus -U username -P password power cycle |
BMC NMI |
ipmitool -H ip_address -I lanplus -U username -P password power diag |
The following table shows commands to be executed for periodically checking the status of the BMC forced stop resource and for checking whether a forced stop succeeds:
Command to periodically check the status
Command to check a forced stop
|
---|
ipmitool -H ip_address -U username -P password power status |
If the above command fails to be executed, perform the following command:
Command to periodically check the status
Command to check a forced stop
|
---|
ipmitool -H ip_address -I lanplus -U username -P password power status |
7.2.2. Notes on BMC forced stop resource¶
- Notes on ipmitool
- Impacts of forced stopWhen you use the forced stop function, the following functions are influenced because power off, reset, power cycle or NMI is forcibly performed regardless of the OS or server status.
- Dump collectionSince it is not recognized that dump files are being collected, power off, reset or power cycle is performed even though dump collection is being performed, so dump collection does not complete.
- Power on within the heartbeat timeoutWhen the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, power off, reset, power cycle or NMI may occur after heartbeat timeout has elapsed.
- BMC network settingsWhen using the BMC forced stop resource, disable the iLO shared network port.Configure the settings so that the IP address of the LAN port for BMC management and the IP address which OS uses can communicate with each other. This function cannot be used in the environment where the network for the BMC management is blocked.Navigate BMC Forced-Stop Properties -> Server List -> Edit -> Enter BMC, then enter the IP address assigned to the LAN port for BMC management.See the server's manuals etc. for information on how to configure the IP address of the LAN port for the BMC management etc.
When using forced stop function, settings of BMC IP address, user name and password of each server are necessary. Use definitely the user name to which the password is set.
7.3. Understanding forced stop on vCenter environment¶
7.3.1. Settings of vCenter forced stop resource¶
The vCenter forced stop resource can be used with vSphere Automation APIs or with the VMware vSphere Command Line Interface (vCLI).
In an environment with VMware vSphere 7.0 Update 3 or higher, the vCLI cannot be used. In an environment with VMware vSphere earlier than 7.0 Update 2, vSphere Automation APIs cannot be used.
Setting parameters of vCenter forced stop resource
Parameter |
Item to be set in vCenter Forced-Stop Properties -> the vCenter tab |
Item to be set in vCenter Forced-Stop Properties -> Server List -> Edit -> Input for Virtual Machine name |
---|---|---|
ip_address |
IP address |
- |
username |
User name |
- |
password |
Password |
- |
virtualmachine |
- |
Virtual machine name |
datacenter |
- |
Datacenter name |
APIs for vCenter forced stop resource
When the vCenter forced stop resource is used with vSphere Automation APIs, the following APIs are executed:
Creating a session |
---|
curl -k -X POST -u {username}:{password} https://{ip_address}/api/session |
Retrieving data center information |
---|
curl -k -X GET https://{ip_address}/api/vcenter/datacenter?names={datacenter} -H "vmware-api-session-id: {api_session_id}" |
Retrieving virtual-machine information |
---|
curl -k -X GET https://{ip_address}/api/vcenter/vm?names={virtualmachine}&datacenters={datacenter_id} -H "vmware-api-session-id: {api_session_id}" |
Forced stop |
|
---|---|
poweroff |
curl -k -X POST https://{ip_address}/api/vcenter/vm/{vm_id}/power?action=stop -H "vmware-api-session-id: {api_session_id}" |
reset |
curl -k -X POST https://{ip_address}/api/vcenter/vm/{vm_id}/power?action=reset -H "vmware-api-session-id: {api_session_id}" |
Deleting the session |
---|
curl -k -X DELETE https://{ip_address}/api/session -H "vmware-api-session-id: {api_session_id}" |
vCLI commands for vCenter forced stop resource
For forcibly stopping the guest OS on a virtual machine, use the [vmcontrol] command of the vCLI. For periodically checking the status of the forced stop resource and for checking whether a forced stop succeeds, use the [vminfo] command.
Forced-stop commands |
|
---|---|
poweroff |
vmcontrol.pl --server ip_address --username username --password password --vmname virtualmachine --datacenter datacenter --operation poweroff |
reset |
vmcontrol.pl --server ip_address --username username --password password --vmname virtualmachine --datacenter datacenter --operation reset |
Command to periodically check the status |
---|
vminfo.pl --server ip_address --username username --password password --vmname virtualmachine --datacenter datacenter --powerstatus "poweredOn" |
Command to check a forced stop |
---|
vminfo.pl --server ip_address --username username --password password --vmname virtualmachine --datacenter datacenter --powerstatus "poweredOff" |
7.3.2. Notes on vCenter forced stop resource¶
- Forcibly stopping the guest OS on a virtual machineOnly power off operation can be performed. This function cannot be used if communication with VMware vCenter Server cannot be performed.- vSphere infrastructure: Communication with VMware vCenter Server is not possible.
- Impacts on forced stopWhen you use the forced stop function, the following functions are influenced because power off, reset is forcibly performed regardless of the OS or server status.
- Dump collectionBecause it is not recognized that dump files are being collected, power off, reset is performed even though dump collection is being performed, so dump collection does not complete.
- Power on within heartbeat timeoutWhen the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, power off, reset may occur after heartbeat timeout has elapsed.
- Power Options settings of the OSConducting a power-off of the guest OS on a virtual machine with the vCLI may perform an action selected in Power Options of the OS, such as Sleep, Hibernate, or Shutdown.The settings can be referred to and configured by the following instruction:Open Power Options in Control Panel and select Choose what the power button does, Power button settings and When I press the power button:.When Forced stop is used in EXPRESSCLUSTER, it is recommended that this setting is configured as No Operation.
7.4. Understanding forced stop on AWS environment¶
7.4.1. Settings of AWS forced stop resource¶
Using the AWS forced stop resource requires installing the AWS Command Line Interface (AWS CLI).
For information on how to obtain and install the AWS CLI, see "Started Guide" -> "Notes and Restrictions" -> "Before installing EXPRESSCLUSTER" -> "Time synchronization in the AWS environment" and "IAM settings in the AWS environment".
Commands for AWS forced stop resource
For forcibly stopping an AWS instance, for periodically checking the status of the forced stop resource, and for checking whether a forced stop succeeds, use the command of the AWS CLI.
When executing either of the commands, specify the following option values:
command option |
item to be set in AWS Forced-Stop Properties -> Server List -> Edit -> Input of Instance |
---|---|
--instance-ids instance-ids |
InstanceID |
The following table shows commands to be executed for forced stop actions by the AWS forced stop resource:
Forced Stop Action |
Parameters |
---|---|
stop |
aws ec2 stop-instances --instance-ids instance-ids --force |
reboot |
aws ec2 reboot-instances --instance-ids instance-ids |
The following table shows commands to be executed for periodically checking the status of the AWS forced stop resource:
Forced Stop Action |
Command to periodically check the status |
---|---|
stop |
aws ec2 stop-instances --instance-ids instance-ids --dry-run
aws ec2 describe-instances --instance-ids instance-ids
aws ec2 describe-instance-attribute --instance-ids instance-ids --attribute disableApiStop
|
reboot |
aws ec2 reboot-instances --instance-ids instance-ids --dry-run
aws ec2 describe-instances --instance-ids instance-ids
|
The following table shows commands to be executed for checking whether a forced stop succeeds:
Command to periodically check the status |
---|
aws ec2 describe-instances --instance-ids instance-ids --filters \"Name=instance-state-name,Values=stopped\" |
7.4.2. Applying command line options to AWS CLI run from AWS forced stop resource¶
7.4.3. Applying environment variables to AWS CLI run from the AWS forced stop resource¶
7.4.4. Notes on AWS forced stop resource¶
- For forcibly stopping instanceYou can perform only the following actions: stop and reboot.
- Impacts on forced stopWhen you use the forced stop function, the following functions are influenced because stop, reboot is forcibly performed regardless of the OS or server status.
- Dump collectionBecause it is not recognized that dump files are being collected, stop, reboot is performed even though dump collection is being performed, so dump collection does not complete.
- Power on within heartbeat timeoutWhen the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, stop, reboot may occur after heartbeat timeout has elapsed.
- stop protection setting for AWSWith stop protection of Instance settings enabled, setting Forced Stop Action to stop leads to a failure in forced stopping and a periodical checking.Avoid enabling stop protection if you use the forced-stop function of EXPRESSCLUSTER and set Forced Stop Action to stop.
- Cluster configurationFor a cluster configured in a multi-region environment, you cannot use the forced-stop function.
7.5. Understanding forced stop on Azure environment¶
7.5.1. Settings of Azure forced stop resource¶
Using the Azure forced stop resource requires installing the Azure CLI.
Commands for Azure forced stop resource
For forcibly stopping an Azure instance, for periodically checking the status of the forced stop resource, and for checking whether a forced stop succeeds, use the command of the Azure CLI.
When executing either of the commands, specify the following option values:
command option |
item to be set in Azure Forced-Stop Properties -> the Azure tab |
item to be set in Azure Forced-Stop Properties -> Server List -> Edit -> Input for Virtual Machine name |
---|---|---|
--name/-n |
- |
Virtual Machine Name |
--username/-u |
User URI |
- |
--tenant/-t |
Tenant ID |
- |
--password/-p |
File Path of Service Principal |
- |
--resource-group/-g |
Resource Group Name |
- |
For the use of Azure CLI commands, the following command is executed:
Logging in to the Azure CLI |
---|
az login --service-principal -u MyUserUri -p MyCertfile.pem --tenant MyTenantId |
The following table shows commands to be executed for forced stop actions by the Azure forced stop resource:
Forced Stop Action |
Parameters |
---|---|
stop and deallocate 1 |
az vm deallocate -g MyResourceGroup -n MyVm --no-wait |
stop only 2 |
az vm stop -g MyResourceGroup -n MyVm --no-wait --skip-shutdown |
reboot |
az vm restart -g MyResourceGroup -n MyVm --no-wait |
- 1
Stops the instance through the shutdown sequence. Since allocated resources (e.g., public IP address) are also released, no charging occurs.
- 2
Stops the instance not through the shutdown sequence. Since allocated resources are maintained, the charging continues.
The following table shows commands to be executed for periodically checking the status of the Azure forced stop resource:
Command to periodically check the status |
---|
az vm update -g MyResourceGroup -n MyVm --no-wait |
az vm get-instance-view -g MyResourceGroup -n MyVm |
The following table shows commands to be executed for checking whether a forced stop succeeds:
Command to periodically check the status |
---|
az vm get-instance-view -g MyResourceGroup -n MyVm --query instanceView.statuses[1].displayStatus --output tsv |
7.5.2. Notes on Azure forced stop resource¶
- For forcibly stopping instanceAny of the following actions can be performed: stop and deallocate, stop only, or reboot.
- Impacts on forced stopWhen you use the forced stop function, the following functions are influenced because stop, reboot is forcibly performed regardless of the OS or server status.
- Dump collectionBecause it is not recognized that dump files are being collected, stop, reboot is performed even though dump collection is being performed, so dump collection does not complete.
- Power on within heartbeat timeoutWhen the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, stop, reboot may occur after heartbeat timeout has elapsed.
- Cluster configurationFor a cluster configured in a multi-region environment, you cannot use the forced-stop function.
7.6. Understanding forced stop on OCI environment¶
7.6.1. Settings of OCI forced stop resource¶
Using the OCI forced stop resource requires installing the Oracle Cloud Infrastructure CLI (OCI CLI).
For information on how to obtain and install the OCI CLI, see "Started Guide" -> "Notes and Restrictions" -> "Before installing EXPRESSCLUSTER" -> "CLI settings in the OCI environment".
Commands for OCI forced stop resource
For forcibly stopping an OCI instance, for periodically checking the status of the forced stop resource, and for checking whether a forced stop succeeds, use the command of the OCI CLI.
When executing either of the commands, specify the following option values:
command option |
item to be set in OCI Forced-Stop Properties -> Server List -> Edit -> Input of Instance |
---|---|
--instance-ids instance-ids |
InstanceID |
The following table shows commands to be executed for forced stop actions by the OCI forced stop resource:
Forced Stop Action |
Parameters |
---|---|
stop |
oci compute instance action --action STOP --instance-id instance-ids |
reboot |
oci compute instance action --action RESET --instance-id instance-ids |
The following table shows commands to be executed for periodically checking the status of the OCI forced stop resource:
Command to periodically check the status |
---|
oci compute instance update --instance-id instance-ids --wait-for-state RUNNING --max-wait-seconds 1 |
The following table shows commands to be executed for checking whether a forced stop succeeds:
Command to periodically check the status |
---|
oci compute instance get --instance-id instance-ids | grep lifecycle-state | awk -F" '{print $4}' |
7.6.2. Notes on OCI forced stop resource¶
- For forcibly stopping instanceYou can perform only the following actions: stop and reboot.
- Impacts on forced stopWhen you use the forced stop function, the following functions are influenced because stop, reboot is forcibly performed regardless of the OS or server status.
- Dump collectionBecause it is not recognized that dump files are being collected, stop, reboot is performed even though dump collection is being performed, so dump collection does not complete.
- Power on within heartbeat timeoutWhen the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, stop, reboot may occur after heartbeat timeout has elapsed.
- Cluster configurationFor a cluster configured in a multi-region environment, you cannot use the forced-stop function.
7.7. Understanding forced stop with script¶
7.7.1. Settings of custom forced stop resource¶
You can create a script for a forced stop. When a server crash is recognized, using the script on the remaining servers (operating properly) allows you to forcibly stop the down server.
The script is executed in both of the following modes: performing a forced stop and periodically checking if the target can be forcibly stopped. For appropriate processing based on each of the modes, write conditional branches by using environment variables described later.
Environment variables for script
When executing the script, EXPRESSCLUSTER sets environment variable values such as which mode (a periodical status check or a forced stop) to be performed and what server has crashed.
In the script, you can use the following environment variables:
Environment variable |
Setting value |
Description |
---|---|---|
CLP_FORCESTOP_MODE
...Mode
|
0 : When periodically checking the status
1 : When performing a forced stop
|
Means a mode to be performed.
Can be used for process branches for each of the modes.
|
CLP_SERVER_DOWN
...Down server name
|
Server name |
Means the name of a down server.
For periodically checking the status, "" is set.
|
CLP_SERVER_LOCAL
...Local server name
|
Server name |
Means the name of a server to execute the script. |
Returned value of script
Return 0 when the script terminates normally.
For more information on configuring the custom forced stop resource, see this guide: "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced stop" -> "Custom Forced-Stop Properties".
7.7.2. Notes on custom forced stop resource¶
Describe the customer-defined process in the script to stop the server.
If there is nothing to be done as periodically checking the status, write the process as such (so that the value 0 can be returned).
7.8. Notes on settings of forced stop resource¶
You can configure only one forced stop resource for one cluster.
If you want to configure a forced stop resource, it is recommended to configure a network partition resolution resource as well.
In configuring a forced stop resource, all the cluster servers must be set to use the forced stop resource.
To prevent a split-brain syndrome in a failover group with a forced-stop resource operating, set the service startup delay time as follows:
Service startup delay time >= forced-stop timeout of forced-stop resource + time to wait for stop to be completed of forced-stop resource + heartbeat timeout + heartbeat interval
For more information on the service startup delay time, see "Adjustment of time for EXPRESSCLUSTER services to start up (Required)".