7. Forced stop resource details

This chapter provides the detailed information on forced stop resources.

7.1. What is the forced stop function?

When a server crash is recognized due to a heartbeat loss, the forced stop function makes the remaining servers (operating properly) forcibly stop the down server.
Suppose the recognized server crash is actually a temporary inability to operate due to the server's stall. In this case, the forced stop function surely stops the down server before its application is failed over to a healthy server. This reduces the risk of the corruption of data in the same resource accessed from multiple servers.

The method for forcibly stopping the failing server varies depending on the type of environment where the cluster was created: physical machines, virtual machines, or the cloud. When using the forced stop function, configure a forced stop resource corresponding to the environment type.

Moreover, you can execute a script in which the procedure for stopping the failing server is written. For details, refer to "7.6. Understanding forced stop with script" in this guide.

A forced stop resource operates in two ways: performing a forced stop and periodically checking if the target can be forcibly stopped. The following explains what these are and when they are done:

  • Performing a forced stop

    • Forcibly stops a down server by using a device or infrastructure system as a server status manager.

    • Done by recognizing the server crash. For more information on the conditions, see "7.1.1. Conditions for performing forced stop".

  • Periodically checking if the target can be forcibly stopped

    • Checks whether a forced stop can be performed, by communicating with a device or infrastructure system for forcibly stopping a server. Depending on the result, the forced stop resource shows whether the server can be forcibly stopped: "Normal" (yes) or "Error" (no).

    • Done on a regular basis while the cluster service is running.

7.1.1. Conditions for performing forced stop

  • Forced stop is not performed when:

    • The failover group successfully stops before the server fails

    • The server is shut down by the clpdown command, the OS shutdown command or Cluster WebUI and the failover group successfully stops

    • The cluster is stopped by the clpcl command or Cluster WebUI and the failover group successfully stops

    • The server fails and there is no failover group to perform failover from the failing server to another server
      (including when the failover group is not activated in the failing server)

  • Forced stop is performed when:

    • The server is failing and there is a failover group to perform failover from the failing server to another server

7.2. Understanding forced stop on physical environment

7.2.1. Settings of BMC forced stop resource

Using the BMC forced stop resource requires installing ipmitool on each cluster server.

Commands for BMC forced stop resource

For forcibly stopping a physical machine, for periodically checking the status of the forced stop resource, and for checking whether a forced stop succeeds, use the [ipmitool] command.

When executing either of the commands, specify the following option values:

[ipmitool] command option

item to be set in BMC Forced-Stop Properties -> Server List -> Edit -> Enter BMC

-H ip_address

IP address

-U username

User name

-P password

Password

The following table shows commands to be executed for forced stop actions by the BMC forced stop resource:

Forced Stop Action

Parameters

BMC Power Off

ipmitool -H ip_address -U username -P password power off

BMC Reset

ipmitool -H ip_address -U username -P password power reset

BMC Power Cycle

ipmitool -H ip_address -U username -P password power cycle

BMC NMI

ipmitool -H ip_address -U username -P password power diag

If the above commands fail to be executed, perform the following commands:

Forced Stop Action

Parameters

BMC Power Off

ipmitool -H ip_address -I lanplus -U username -P password power off

BMC Reset

ipmitool -H ip_address -I lanplus -U username -P password power reset

BMC Power Cycle

ipmitool -H ip_address -I lanplus -U username -P password power cycle

BMC NMI

ipmitool -H ip_address -I lanplus -U username -P password power diag

The following table shows commands to be executed for periodically checking the status of the BMC forced stop resource and for checking whether a forced stop succeeds:

Command to periodically check the status
Command to check a forced stop

ipmitool -H ip_address -U username -P password power status

If the above command fails to be executed, perform the following command:

Command to periodically check the status
Command to check a forced stop

ipmitool -H ip_address -I lanplus -U username -P password power status

Editing commands for BMC forced stop resource

For forced stop resources, you can also specify a command line for a forced stop by editing a script file for the command execution.

The following file can be edited:

<EXPRESSCLUSTER_install_path>\bin\clpbmcforcestop.sh

The BMC forced stop resource sets environment variable values necessary for commands to be executed with the script.

The following table shows environment variables written in the script:

Environment variable

Setting value

Description
CLP_BMC_ACTION
...Forced Stop Action
power off : BMC Power Off
power reset : BMC Reset
power cycle : BMC Power Cycle
power diag : BMC NMI

Specifies the Forced Stop Action set in the BMC Forced-Stop Properties.
CLP_BMC_HOST
...IP address for BMC

IP Address

Specifies the IP address set in the BMC Forced-Stop Properties.
CLP_BMC_USER
...User name for BMC

User name

Specifies the user name set in the BMC Forced-Stop Properties.
CLP_BMC_PASSWORD
...Password for BMC

Password

Specifies the password set in the BMC Forced-Stop Properties.

For more information on configuring the BMC forced stop resource, see this guide: "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced stop" -> "BMC Forced-Stop Properties".

7.2.2. Notes on BMC forced stop resource

  • Notes on ipmitool
    See "IPMI command" in "4. Monitor resource details" of "Monitor resource".
  • Impacts of forced stop
    When you use the forced stop function, the following functions are influenced because power off, reset, power cycle or NMI is forcibly performed regardless of the OS or server status.
    • Dump collection
      Since it is not recognized that dump files are being collected, power off, reset or power cycle is performed even though dump collection is being performed, so dump collection does not complete.
    • Power on within the heartbeat timeout
      When the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, power off, reset, power cycle or NMI may occur after heartbeat timeout has elapsed.
  • BMC network settings
    Configure the settings so that the IP address of the LAN port for BMC management and the IP address which OS uses can communicate with each other. This function cannot be used in the environment where the network for the BMC management is blocked.
    Navigate BMC Forced-Stop Properties -> Server List -> Edit -> Enter BMC, then enter the IP address assigned to the LAN port for BMC management.
    See the server's manuals etc. for information on how to configure the IP address of the LAN port for the BMC management etc.

  • When using forced stop function, settings of BMC IP address, user name and password of each server are necessary. Use definitely the user name to which the password is set.

7.3. Understanding forced stop on vCenter environment

7.3.1. Settings of vCenter forced stop resource

Using the vCenter forced stop resource requires installing the VMware vSphere Command Line Interface (vCLI).

Commands for vCenter forced stop resource

For forcibly stopping the guest OS on a virtual machine, use the [vmcontrol] command of the vCLI. For periodically checking the status of the forced stop resource and for checking whether a forced stop succeeds, use the [vminfo] command.

When executing either of the commands, specify the following option values:

vmcontrol command option

item to be set in vCenter Forced-Stop Properties -> the vCenter tab

item to be set in vCenter Forced-Stop Properties -> Server List -> Edit -> Input for Virtual Machine name

--server ip_address

IP address

-

--username username

User name

-

--password password

Password

-

--vmname virtualmachine

-

Virtual machine name

--datacenter datacenter

-

Datacenter name

The following table shows commands to be executed for forced stop actions by the vCenter forced stop resource:

Forced Stop Action

Parameters

poweroff

vmcontrol.pl --server ip_address --username username --password password --vmname virtualmachine --datacenter datacenter --operation poweroff

reset

vmcontrol.pl --server ip_address --username username --password password --vmname virtualmachine --datacenter datacenter --operation reset

The following table shows commands to be executed for periodically checking the status of the vCenter forced stop resource:

Command to periodically check the status

vminfo.pl --server ip_address --username username --password password --vmname virtualmachine --datacenter datacenter --powerstatus "poweredOn"

The following table shows commands to be executed for checking whether a forced stop succeeds:

Command to check a forced stop

vminfo.pl --server ip_address --username username --password password --vmname virtualmachine --datacenter datacenter --powerstatus "poweredOff"

Editing commands for vCenter forced stop resource

For forced stop resources, you can also specify a command line for a forced stop by editing a script file for the command execution.

The following file can be edited:

<EXPRESSCLUSTER_install_path>\bin\clpvcenterforcestop.sh

The vCenter forced stop resource sets environment variable values necessary for commands to be executed with the script.

The following table shows environment variables written in the script:

Environment variable

Setting value

Description
CLP_VCLI_PATH
...vCLI install path

Install path

Specifies the VMware vSphere CLI install path set in the vCenter Forced-Stop Properties.
CLP_VCENTER_ACTION
...Forced Stop Action
poweroff : power off
reset : reset

Specifies the Forced Stop Action set in the vCenter Forced-Stop Properties.
CLP_VCENTER_HOST
...Host name for vCenter

Host name

Specifies the host name set in the vCenter Forced-Stop Properties.
CLP_VCENTER_USER
...User name for vCenter

User name

Specifies the user name set in the vCenter Forced-Stop Properties.
CLP_VCENTER_PASSWORD
...Password for vCenter

Password

Specifies the password set in the vCenter Forced-Stop Properties.
CLP_VMNAME
...Virtual machine name

Virtual machine name

Specifies the virtual machine name set in the vCenter Forced-Stop Properties.
CLP_DATACENTER_NAME
...Data center name

Data center name

Specifies the data center name set in the vCenter Forced-Stop Properties.

For more information on configuring the vCenter forced stop resource, see this guide: "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced stop" -> "vCenter Forced-Stop Properties".

7.3.2. Notes on vCenter forced stop resource

  • Forcibly stopping the guest OS on a virtual machine
    Only power off operation can be performed. This function cannot be used if communication with VMWare vCenter Server cannot be performed.
    - vSphere infrastructure: Communication with VMWare vCenter Server is not possible.
  • Impacts on forced stop
    When you use the forced stop function, the following functions are influenced because power off, reset is forcibly performed regardless of the OS or server status.
    • Dump collection
      Because it is not recognized that dump files are being collected, power off, reset is performed even though dump collection is being performed, so dump collection does not complete.
    • Power on within heartbeat timeout
      When the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, power off, reset may occur after heartbeat timeout has elapsed.
  • Power Options settings of the OS
    Conducting a power-off of the guest OS on a virtual machine with the vCLI may perform an action selected in Power Options of the OS, such as Sleep, Hibernate, or Shutdown.
    The settings can be referred to and configured by the following instruction:
    Open Power Options in Control Panel and select Choose what the power button does, Power button settings and When I press the power button:.
    When Forced stop is used in EXPRESSCLUSTER, it is recommended that this setting is configured as No Operation.

7.4. Understanding forced stop on AWS environment

7.4.1. Settings of AWS forced stop resource

Using the AWS forced stop resource requires installing the AWS Command Line Interface (AWS CLI).

For information on how to obtain and install the AWS CLI, see "Started Guide" -> "Notes and Restrictions" -> "Before installing EXPRESSCLUSTER" -> "Time synchronization in the AWS environtment" and "IAM settings in the AWS environtment".

Commands for AWS forced stop resource

For forcibly stopping an AWS instance, for periodically checking the status of the forced stop resource, and for checking whether a forced stop succeeds, use the command of the AWS CLI.

When executing either of the commands, specify the following option values:

command option

item to be set in AWS Forced-Stop Properties -> Server List -> Edit -> Input of Instance

--instance-ids instance-ids

InstanceID

The following table shows commands to be executed for forced stop actions by the AWS forced stop resource:

Forced Stop Action

Parameters

stop

aws ec2 stop-instances --instance-ids instance-ids --force

reboot

aws ec2 reboot-instances --instance-ids instance-ids

The following table shows commands to be executed for periodically checking the status of the AWS forced stop resource:

Forced Stop Action

Command to periodically check the status

stop

aws ec2 stop-instances --instance-ids instance-ids --dry-run

reboot

aws ec2 reboot-instances --instance-ids instance-ids --dry-run

The following table shows commands to be executed for checking whether a forced stop succeeds:

Command to periodically check the status

aws ec2 describe-instances --instance-ids instance-ids --filters \"Name=instance-state-name,Values=stopped\"

Editing commands for AWS forced stop resource

For forced stop resources, you can also specify a command line for a forced stop by editing a script file for the command execution.

The following file can be edited:

<EXPRESSCLUSTER_install_path>\cloud\aws\clpawsforcestop.sh

For more information on configuring the AWS forced stop resource, see this guide: "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced stop" -> "AWS Forced-Stop Properties".

7.4.2. Applying environment variables to AWS CLI run from the AWS forced stop resource

Specifying environment variables in their configuration file allows you to apply them to the AWS CLI that can be executed from the following AWS-related resources:

  • AWS Elastic IP resource

  • AWS Virtual IP resource

  • AWS Secondary IP resource

  • AWS DNS resource

  • AWS Elastic IP monitor resource

  • AWS Virtual IP monitor resource

  • AWS Secondary IP monitor resource

  • AWS AZ monitor resource

  • AWS DNS monitor resource

  • AWS Forced stop resource

This feature is useful when using a proxy server in an AWS environment.

The envirionment variable configuration file is stored in the following location.

<EXPRESSCLUSTER Installation path>/cloud/aws/clpaws_setting.conf

The format of the environment variable configuration file is as follows:

Envirionment variable name = Value

(Example)

[ENVIRONMENT]
HTTP_PROXY = http://10.0.0.1:3128
HTTPS_PROXY = http://10.0.0.1:3128

To specify multiple values for a parameter, enter them in comma-delimited format. The following shows an example of specifying more than one non-destination for the environment variable NO_PROXY:

(Example)

NO_PROXY = 169.254.169.254,ec2.ap-northeast-1.amazonaws.com

The specifications of the environment variable configuration file are as follows:

  • Write [ENVIRONMENT] on the first line. If this is not set, the environment variables will not be set.

  • If the environment variable configuration file does not exist or you do not have read permission for the file, the variables are ignored. This does not cause an activation failure or a monitor error.

  • If the same environment variables already exist in the file, the values are overwritten.

  • More than one environment variable can be set. Set one environment variable on each line.

  • The settings are valid regardless of whether there are spaces before and after "=" or not.

  • The settings are invalid if there is a space or tab in front of the environment variable name or if there are tabs before and after "=".

  • Environment variable names are case sensitive.

  • Even if a value contains spaces, you do not have to enclose the value in "" (double quotation marks).

  • Any line including "#", regardless of its position in the line, falls outside the environment variable configuration.

  • Environment variables specified in their configuration file are applied to the AWS CLI that can be executed from the above AWS-related resources, but not applied to other scripts such as one before final action, one before and after activation/deactivation, and one to be run from a script resource. To execute the AWS CLI with any of these scripts, specify necessary environment variables in the corresponding script.

7.4.3. Notes on AWS forced stop resource

  • For forcibly stopping instance
    You can perform only the following actions: stop and reboot.
  • Impacts on forced stop
    When you use the forced stop function, the following functions are influenced because stop, reboot is forcibly performed regardless of the OS or server status.
    • Dump collection
      Because it is not recognized that dump files are being collected, stop, reboot is performed even though dump collection is being performed, so dump collection does not complete.
    • Power on within heartbeat timeout
      When the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, stop, reboot may occur after heartbeat timeout has elapsed.

7.5. Understanding forced stop on OCI environment

7.5.1. Settings of OCI forced stop resource

Using the OCI forced stop resource requires installing the Oracle Cloud Infrastructure CLI (OCI CLI).

ToDo:For information on how to obtain and install the OCI CLI, see "Started Guide" -> "Notes and Restrictions" -> "Before installing EXPRESSCLUSTER" -> "CLI settings in the OCI environtment" and "Configuring OCI forced-stop resource".

Commands for OCI forced stop resource

For forcibly stopping an OCI instance, for periodically checking the status of the forced stop resource, and for checking whether a forced stop succeeds, use the command of the OCI CLI.

When executing either of the commands, specify the following option values:

command option

item to be set in OCI Forced-Stop Properties -> Server List -> Edit -> Input of Instance

--instance-ids instance-ids

InstanceID

The following table shows commands to be executed for forced stop actions by the OCI forced stop resource:

Forced Stop Action

Parameters

stop

oci compute instance action --action STOP --instance-id instance-ids

reboot

oci compute instance action --action RESET --instance-id instance-ids

The following table shows commands to be executed for periodically checking the status of the OCI forced stop resource:

Command to periodically check the status

oci compute instance update --instance-id instance-ids --wait-for-state RUNNING --max-wait-seconds 1

The following table shows commands to be executed for checking whether a forced stop succeeds:

Command to periodically check the status

oci compute instance get --instance-id instance-ids | grep lifecycle-state | awk -F" '{print $4}'

Editing commands OCI forced stop resource

For forced stop resources, you can also specify a command line for a forced stop by editing a script file for the command execution.

The following file can be edited:

<EXPRESSCLUSTER_install_path>\cloud\oci\clpociforcestop.sh

For more information on configuring the OCI forced stop resource, see this guide: "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced stop" -> "OCI Forced-Stop Properties".

7.5.2. Notes on OCI forced stop resource

  • For forcibly stopping instance
    You can perform only the following actions: stop and reboot.
  • Impacts on forced stop
    When you use the forced stop function, the following functions are influenced because stop, reboot is forcibly performed regardless of the OS or server status.
    • Dump collection
      Because it is not recognized that dump files are being collected, stop, reboot is performed even though dump collection is being performed, so dump collection does not complete.
    • Power on within heartbeat timeout
      When the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, stop, reboot may occur after heartbeat timeout has elapsed.

7.6. Understanding forced stop with script

7.6.1. Settings of custom forced stop resource

You can create a script for a forced stop. When a server crash is recognized, using the script on the remaining servers (operating properly) allows you to forcibly stop the down server.

The script is executed in both of the following modes: performing a forced stop and periodically checking if the target can be forcibly stopped. For appropriate processing based on each of the modes, write conditional branches by using environment variables described later.

Environment variables for script

When executing the script, EXPRESSCLUSTER sets environment variable values such as which mode (a periodical status check or a forced stop) to be performed and what server has crashed.

In the script, you can use the following environment variables:

Environment variable

Setting value

Description
CLP_FORCESTOP_MODE
...Mode
0 : When periodically checking the status
1 : When performing a forced stop
Means a mode to be performed.
Can be used for process branches for each of the modes.
CLP_SERVER_DOWN
...Down server name

Server name
Means the name of a down server.
For periodically checking the status, "" is set.
CLP_SERVER_LOCAL
...Local server name

Server name

Means the name of a server to execute the script.

Returned value of script

Return 0 when the script terminates normally.

For more information on configuring the custom forced stop resource, see this guide: "2. Parameter details" -> "Cluster properties" -> "Fencing tab" -> "Forced stop" -> "Custom Forced-Stop Properties".

7.6.2. Notes on custom forced stop resource

  • Describe the customer-defined process in the script to stop the server.

  • If there is nothing to be done as periodically checking the status, write the process as such (so that the value 0 can be returned).

7.7. Notes on settings of forced stop resource

  • You can configure only one forced stop resource for one cluster.

  • If you want to configure a forced stop resource, it is recommended to configure a network partition resolution resource as well.

  • In configuring a forced stop resource, all the cluster servers must be set to use the forced stop resource.

  • To prevent a split-brain syndrome in a failover group with a forced-stop resource operating, set the service startup delay time as follows:

    Service startup delay time >= forced-stop timeout of forced-stop resource + time to wait for stop to be completed of forced-stop resource + heartbeat timeout + heartbeat interval

    For more information on the service startup delay time, see "Adjustment of the operating system startup time (Required)".