7. Information on other settings¶

This chapter provides the information on the other monitor or notification settings.

This chapter covers:

7.1. The forced stop function
7.2. Script for forced stop
7.3. Chassis Identify
7.4. Alert Service
7.5. SNMP linkage
7.6. Grace period dependence at the automatic failover between server groups
7.7. Witness server service

7.1. The forced stop function¶

7.1.1. What is the forced stop function?¶

The forced stop function forcibly stops the failing server from one of the rest of servers working normally when it is recognized that the server is failing.

This function stops a physical machine by using the IPMI function.

It stops the guest OS on a virtual machine by using the VMware vCenter Server or System Center Virtual Machine Manager (SCVMM).

In addition to the functions above, you can execute a script in which the procedure for stopping the failing server is written. For details, refer to "Script for forced stop" in "7. Information on other settings" in this guide.

7.1.2. Conditions for performing forced stop¶

Forced stop is not performed when:
- The failover group successfully stops before the server fails
- The server is shut down by the clpdown command, the OS shutdown command or Cluster WebUI and the failover group successfully stops
- The cluster is stopped by the clpcl command or Cluster WebUI and the failover group successfully stops
- The server fails and there is no failover group to perform failover from the failing server to another server
  
  (including when the failover group is not activated in the failing server)
Forced stop is performed when the server is failing and there is a failover group to perform failover from the failing server to another server

7.1.3. Commands to be used for forced stop¶

The hwreset or ireset command in IPMI Management Utilities (ipmiutil) is used to forcibly stop a physical machine server. When the command cannot be used, this function cannot be used either.

Specify the following option values for the command execution in the BMC tab of Server Properties.

The hwreset or ireset command option	Configured in the BMC tab of the server properties
-N ip_address	IP address
-U username	User name
-P password	Password

When a command line is not specified for Forced Stop Action in the BMC tab of the server properties, the following commands are executed.

In case of hwreset

Forced Stop Action	Parameters
BMC Power Off	hwreset.exe -d -N ip_address -U username -P password
BMC Reset	hwreset.exe -r -N ip_address -U username -P password
BMC Power Cycle	hwreset.exe -c -N ip_address -U username -P password
BMC NMI	hwreset.exe -n -N ip_address -U username -P password

In case of ireset

Forced Stop Action	Parameters
BMC Power Off	ireset.cmd -d -N ip_address -U username -P password
BMC Reset	ireset.cmd -r -N ip_address -U username -P password
BMC Power Cycle	ireset.cmd -c -N ip_address -U username -P password
BMC NMI	ireset.cmd -n -N ip_address -U username -P password

The vmcontrol command of the VMware vSphere Command Line Interface (vCLI) is used to forcibly stop the guest OS on a virtual machine. This function cannot be used if VMware vSphere Command Line Interface (vCLI) is not installed.

Note

If the version of vCLI is 6.5 or later, Perl execution environment is required to be installed. For the information on the versions of Perl necessary to execute vCLI, refer to the website of VMware, Inc.

Perform the following procedure after installing the Perl execution environment.

Set the Perl path

Select Cluster Properties -> Extension tab -> Virtual Machine Forced Stop Setting, specify the path to the Perl execution module for Perl Path. This is common to all the servers in the cluster.

For more information about the Perl path, refer to "Extension Tab" in "Cluster properties" in "2. Parameter details" in this guide.
Add the system environment variable

Add the following variable for the system environment variable. Then restart the OS.

Variable name: PERL5LIB

Variable value: vCLI Perl module path (Example: C:\Program Files (x86)\VMware\VMware vSphere CLI\Perl\lib

Specify the following option values for the command execution.

vmcontrol command option	Configured in Virtual Machine Forcestop Setting on the Extension tab of Cluster Properties	Configured in Input for Virtual Machine name on the Info tab of Server Properties
--server ip_address	IP address	-
--username username	User name	-
--password password	Password	-
--vmname virtualmachine	-	Virtual machine name

The following option is used for action.

Command	Option	Overview
vmcontrol	--operation poweroff	Powers off the guest OS on a virtual machine.

7.1.4. Specifying the command to be used for forced stop¶

It is also possible to forcibly stop a physical machine server by specifying an arbitrary command line to be used for the forced stop in Forced Stop Action in the BMC tab of the server properties.

To specify the command line, use the following replacement strings so that the setting values of the server properties are applied on the command line.

Replacement string name	Replacement target (Setting item in the BMC tab of the server properties)	Replacement target (Setting item in the forced stop action in the extension tab of the cluster properties)
CLP_BMC_HOST	IP address	-
CLP_BMC_USER	User name	-
CLP_BMC_PASSWORD	Password	-
CLP_BMC_ACTION	-	Forced Stop Action

Characters to be replaced by the replacement string (CLP_BMC_ACTION) for the forced stop action are as follows.

Forced Stop Action	Characters to be replaced by replacement string
BMC Power Off	-d
BMC Reset	-r
BMC Power Cycle	-c
BMC NMI	-n

Note

In the forced stop action, the action to be executed differs depending on whether the replacement string, CLP_BMC_ACTION is specified or not.

When CLP_BMC_ACTION is included in the command line:

The action selected in the forced stop action of the cluster properties is executed.

When CLP_BMC_ACTION is not included in the command line:

The action selected in the forced stop action of the cluster properties is not applied.

Example of the command specified for the forced stop action by using the replacement strings:

ireset.cmd CLP_BMC_ACTION -N CLP_BMC_HOST -U CLP_BMC_USER -P CLP_BMC_PASSWORD

7.1.5. Displaying and changing the details of forced stop¶

For the forced stop settings, refer to "Cluster properties - Extension Tab", "Cluster properties - Extension Tab", and "Server Properties - Info tab" in "2. Parameter details" in this guide.

7.1.6. Notes on the forced stop¶

Forcibly stopping the guest OS on a virtual machine

Only power off operation can be performed. Moreover, this function cannot be used in the following cases:

- vSphere infrastructure: Communication with VMWare vCenter Server is not possible.
About ipmiutil

When you use the hwreset or irset command, it is necessary to install ipmiutil 2.0.0 or later in each cluster server. For information on how to get ipmiutil and how to install it, refer to "9. Setup of BMC and ipmiutil" in "Settings after configuring hardware" in "Determining a system configuration" in the "Installation and Configuration Guide".
Impacts on forced stop

When you use the forced stop function, the following functions are influenced because power off, reset, power cycle or NMI is forcibly performed regardless of the OS or server status.
Dump collection

Because it is not recognized that dump files are being collected, power off, reset or power cycle is performed even though dump collection is being performed, so dump collection does not complete.
Power on within heartbeat timeout

When the server is powered on again for the purpose of maintenance etc. within heartbeat timeout, power off, reset, power cycle or NMI may occur after heartbeat timeout has elapsed.
BMC network settings

Configure the settings so that the IP address of the LAN port for BMC management and the IP address which OS uses can communicate with each other. This function cannot be used when BMC is not installed in the server, or in the environment where the network for the BMC management is blocked.

Configure the same IP address that is configured for the LAN port for the BMC management to the BMC tab of the server properties.

See the server's manuals etc. for information on how to configure the IP address of the LAN port for the BMC management etc.
Power Options settings of the OS

When power off or power cycle is executed by BMC or power off of the guest OS on a virtual machine is executed by VMware vSphere, operation specified in Power Options of the OS (e.g. sleep, hibernation and shutdown) may be executed.

The settings can be referred to and configured by the following instruction:

Open Power Options in Control Panel and select Choose what the power button does, Power button settings and When I press the power button:.

When Forced stop is used in EXPRESSCLUSTER, it is recommended that this setting is configured as No Operation.

7.2. Script for forced stop¶

7.2.1. What is the script for forced stop?¶

When it is recognized that the server is failing, any script created by the user can be executed on one of the rest of servers working normally.
The failing server can be stopped forcibly by using the script.
Moreover, using the script enables to check whether the forced stop is successful or unsuccessful and to control whether to execute the failover or not.

7.2.2. Conditions for executing the script for forced stop¶

The script for forced stop is not executed when:
- The failover group successfully stops before the server fails
- The server is shut down by the clpdown command, the OS shutdown command or Cluster WebUI and the failover group successfully stops
- The cluster is stopped by the clpcl command or Cluster WebUI and the failover group successfully stops
- The server fails and there is no failover group to perform failover from the failing server to another server
  
  (including when the failover group is not activated in the failing server)
The script for forced stop is executed when the server is failing and there is a failover group to perform failover from the failing server to another server.

7.2.3. Features of the script for forced stop¶

Environment variables used in the script for forced stop

EXPRESSCLUSTER stores the data such as the information of a failing server to environment variables.

You can use the following environment variables for branch conditions in the script to describe the procedure tailored to the operations of your system.

Environment variable	Setting value	Description
CLP_SERVER_DOWN ...Down server name	Server name	Specifies the name of the failing server
CLP_SERVER_LOCAL ...Local server name	Server name	Specifies the name of the server where the script is executed.
CLP_VMNAME ...Virtual machine name	Virtual machine name	Specifies the virtual machine name set in the server properties.
CLP_DATACENTER_NAME ...Data center name	Data center name	Specifies the data center name set in the server properties.
CLP_VCENTER_HOST ...Host name for vCenter	Host name	Specifies the host name set in the virtual machine forced stop setting.
CLP_VCENTER_USER ...User name for vCenter	User name	Specifies the user name set in the virtual machine forced stop setting.
CLP_VCENTER_PASSWORD ...Password for vCenter	Password	Specifies the password set in the virtual machine forced stop setting.
CLP_SCVMM_HOST ...Host name for SCVMM	Host name	Specifies the host name set in the virtual machine forced stop setting.
CLP_SCVMM_USER ...User name for SCVMM	User name	Specifies the user name set in the virtual machine forced stop setting.
CLP_SCVMM_PASSWORD ...Password for SCVMM	Password	Specifies the password set in the virtual machine forced stop setting.
CLP_BMC_HOST ...IP address for BMC	IP Address	Specifies the IP address set in the server properties.
CLP_BMC_USER ...User name for BMC	User name	Specifies the user name set in the server properties.
CLP_BMC_PASSWORD ...Password for BMC	Password	Specifies the password set in the server properties.

Return value of the script for forced stop

Return 0 when the script terminates normally.

7.2.4. Displaying and changing the details of the script for forced stop¶

For the settings of the script for forced stop, refer to "Extension Tab" in "Cluster properties" in "2. Parameter details" in this guide.

7.2.5. Notes on the script for forced stop¶

Describe the customer-defined process in the script to stop the server.
When using the script for forced stop, refer to "Impacts on forced stop" of "Notes on the forced stop" in "The forced stop function" in "7. Information on other settings" in this guide.
When the forced stop function and the script for forced stop is used together, they are executed in the following order.
1. The forced stop function
2. The script for forced stop

7.3. Chassis Identify¶

7.3.1. What is chassis identify?¶

This function allows another normal server to report the server failure by blinking the chassis ID lamp using the IPMI function when it recognizes that the server is failing

7.3.2. Conditions for chassis ID lamp to blink¶

The chassis ID lamp does not blink when:
- Statuses other than server status becomes abnormal
- The cluster shuts down
- All the servers in the cluster fail
  
  If the servers do not fail simultaneously, they blink for 250 seconds at the maximum, and eventually the chassis ID lamps of all servers go off.
- BMC of the failing server cannot communicate with a normal server
- There is a normal server in the cluster but EXPRESSCLUSTER is stopped
The chassis ID lamp blinks when (the above conditions for not blinking are given priority over these conditions when they overlap):
- When some servers in the cluster fail due to some abnormality
- When some servers in the cluster are shut down by the shutdown command of the OS.
- When some servers in the cluster are made to shut down by the clpdown command or Cluster WebUI
- When EXPRESSCLUSTER is stopped by the clpcl command or Cluster WebUI in some servers in the cluster
- When some servers in the cluster are started while EXPRESSCLUSTER Server service is configured as manual start
Chassis ID lamp stops blinking and goes off when there are normal servers in the cluster, and the server status of the failing server returns to normal

7.3.3. Behavior of the chassis ID lamp blinking when the cluster stops¶

If the chassis ID lamp of a server in the cluster is in blinking when the cluster stops, the chassis ID lamp may keep blinking for 250 seconds at the maximum.

7.3.4. Commands to be used for chassis identify¶

The alarms or ialarms command of IPMI Management Utilities (ipmiutil) is used to control the chassis ID lamp. When the command cannot be executed, this function cannot be used.

Specify the following option values for the command execution in the BMC tab of Server Properties.

The alarms/ialarms command option	Configured in the BMC tab of the server properties
-N ip_address	IP address
-U username	Use name
-P password	Password

When the command lines are not specified for Flash and Turn off of the chassis identify lamp in the BMC tab of the server properties, the following command is executed.

In case of alarms:

Chassis Identify	Parameters
Flash	alarms.exe -i250 -N ip_address -U username -P password
Turn off	alarms.exe -i0 -N ip_address -U username -P password

In case of ialarms

Chassis Identify	Parameters
Flash	ialarms.cmd -i250 -N ip_address -U username -P password
Turn off	ialarms.cmd -i0 -N ip_address -U username -P password

7.3.5. Specifying the command to be used for the chassis identify function¶

It is also possible to execute the chassis identify function by specifying an arbitrary command line used for the chassis identify function in Flash and Turn off of the chassis identify lamp in the BMC tab of the server properties.

To specify the command line, use the following replacement strings so that the setting values of the server properties are applied to the command line.

Replacement string name	Replacement target (Setting item in the BMC tab of the server properties)
CLP_BMC_HOST	IP address
CLP_BMC_USER	Use name
CLP_BMC_PASSWORD	Password

Example of the chassis identify command specified by using the replacement strings:

ialarms.cmd 250 CLP_BMC_HOST -U CLP_BMC_USER -P CLP_BMC_PASSWORD

7.3.6. Displaying and changing the chassis identify details¶

For the chassis identify settings, refer to "Cluster properties - Alert Service tab" and "Server Properties - BMC tab"in "2. Parameter details" in this guide.

7.3.7. Notes on chassis identify¶

About ipmiutil

To use this function, it is necessary to install ipmiutil 2.0.0 or later in each cluster server. For how to obtain ipmiutil and how to install it, see "9. Setup of BMC and ipmiutil" in "Settings after configuring hardware" in "Determining a system configuration" in the "Installation and Configuration Guide".
BMC network settings

Configure the settings so that the IP address of the LAN port for BMC management and the IP address which OS uses can communicate with each other. This function cannot be used when BMC is not installed in the server, or in the environment where the network for the BMC management is blocked.

Configure the same IP address that is configured for the LAN port for the BMC management to the BMC tab of the server properties.

See the server's manuals etc. for how to configure the IP address of the LAN port for the BMC management etc.

7.4. Alert Service¶

7.4.1. Alert Service¶

EXPRESSCLUSTER Alert Service is a function to report failures found in operations on EXPRESSCLUSTER to system administrators in remote locations.

Failures are reported in three ways, each serving a different purpose.

E-mail report

Alert messages in the Cluster WebUI are sent by e-mail to administrators.
Warning light

The warning light is a visual display of the status of the server. When the server shuts down successfully, the warning light goes off.

The e-mail report and the warning light function work independently of each other.
SNMP trap sending

When a Cluster WebUI alert message is displayed, the contents of the alert are sent with an SNMP trap.

Fig. 7.1 Alert service¶

Alert Service allows you to:

Receive information about failures while not physically located in the place where management PC is. This is achieved via e-mail reporting function.
Receive e-mail messages on your mobile phone.
Visually be alerted of failures by viewing a light.
Recognize a failure audibly by reproducing the audio file for the network warning light.
Notify the servers that are configured as the destination of the details of errors by SNMP trap sending.

Mail Report notifies the content of the alert in the following format by e-mail.

Subject:
    EXPRESSCLUSTER
Body:
    Message: Server [down server] has been stopped.
    Type: nm
    ID: 2
    Host: [mail sending source server name]
    Date: [send time stamp]

7.4.2. Notes on Alert Service¶

To use the mail report and warning light function, the EXPRESSCLUSTER X Alert Service 4.3 license must be applied to the system.
The task of Alert Service is to send the first report of failure but not to examine or find the cause of failure. When a failure occurs, instead of using the Alert Service, try other methods, such as viewing EXPRESSCLUSTER logs or syslog, to find out the cause of the error.
When the warning light function is used, it is necessary to set up the command such as rsh that is supported by the warning light manufacturer.

7.4.3. Mail report actions¶

Alert Service sends the same messages as the Cluster WebUI. For the alert messages to be reported by e-mail, see "Messages reported by event log and alert" in "10. Error messages" in this guide.
You can change the alerts that are reported by e-mail. For more information, see "Alert Service tab" in "Cluster properties" in "2. Parameter details" in this guide.

7.4.4. Warning Light status¶

The network warning light performs the following operations.

When the server is started

When the server starts up successfully, warning light changes to green.
When the server shuts down

When the server shuts down successfully, warning light goes off.
When the server fails

When the server fails, its warning light flashes in red. If all servers in the cluster fail, the warning light of the server that failed last will not work because the warning light is controlled by a normal server that monitors other servers.

Once a network warning light is lit or starts flashing, it will not go off until the cluster shuts down. Run the clplamp command introduced in the following section to put the light out. For more information on the clplamp command, see "Switching off network warning light (clplamp command)" in "8. EXPRESSCLUSTER command reference" in this guide.

For a network warning light (specified by NEC) that suppors playback of an audio file, the setting also enables audio file reproduction to link to On/Off.

7.4.5. Operations of SNMP trap sending¶

The contents of Cluster WebUI alert messages are sent with an SNMP trap. For alert messages subject to SNMP trap sending, see "Messages reported by event log and alert" in "10. Error messages" in this guide.
The alerts subject to SNMP trap sending can be changed. For more information, see "Alert Service tab" in "Cluster properties" in "2. Parameter details" in this guide.
For details on the SNMP trap, see "SNMP trap sending".

7.5. SNMP linkage¶

7.5.1. SNMP linkage¶

SNMP linkage enables SNMP trap sending from EXPRESSCLUSTER and information acquisition by SNMP from an SNMP manager according to the EXPRESSCLUSTER MIB definitions.

Fig. 7.2 SNMP linkage¶

7.5.2. EXPRESSCLUSTER MIB definitions¶

The information sent/acquired with SNMP linkage is configured by the MIB definition files.

To use the functions of SNMP trap sending and information acquisition by SNMP, described later, MIB definition files are required.

To receive SNMP traps from EXPRESSCLUSTER by using an SNMP manager, or to acquire cluster statuses from an SNMP manager, set the EXPRESSCLUSTER MIB definition files in the SNMP manager.

For how to set the MIB definition files in an SNMP manager, refer to the manual for the SNMP manager.

The EXPRESSCLUSTER MIB definition files are placed in the following directory on the EXPRESSCLUSTER X CD-ROM.

<EXPRESSCLUSTER_X_CD-ROM>\Common\<version number>\common\mib

The MIB definition files provide the functions described below.

No.	MIB definition file	Description
	NEC-CLUSTER-SMI.mib	Configures the EXPRESSCLUSTER MIB tree root path.
	NEC-CLUSTER-EVENT-MIB.mib	Configures the trap and MIB definitions for the EXPRESSCLUSTER SNMP trap sending function.
	NEC-CLUSTER-MANAGEMENT-MIB.mib	Configures MIB definitions for the following EXPRESSCLUSTER information: - Cluster information - Server information - Group information

The available functions depend on the files set in the SNMP manager.

To receive SNMP traps from EXPRESSCLUSTER:

NEC-CLUSTER-SMI.mib

NEC-CLUSTER-EVENT-MIB.mib

To get information by SNMP:

1. NEC-CLUSTER-SMI.mib 3. NEC-CLUSTER-MANAGEMENT-MIB.mib

7.5.3. SNMP trap sending¶

SNMP trap sending serves to send the contents of Cluster WebUI alert messages to the SNMP manager.

To send a trap, the SNMP trap sending destination is required to be configured. Configure it by referring to Destination Settings of SNMP Trap in "Alert Service tab" in "Cluster properties" in "2. Parameter details" in this guide.

The traps to be sent are defined by NEC-CLUSTER-EVENT-MIB.

NEC-CLUSTER-EVENT-MIB defines the following MIB objects.

clusterEventNotifications group

This group defines the traps to be sent. The MIB objects defined for the group function as described below.

No.	SNMP TRAP OID	Description
	clusterEventInformation	Trap for information level alerts. A clusterEvent group MIB object is attached.
	clusterEventWarning	Trap for warning level alerts. A clusterEvent group MIB object is attached.
	clusterEventError	Trap for error level alerts. A clusterEvent group MIB object is attached.

clusterEvent group

This group defines the information appended to the traps. The MIB objects defined for the group function as described below.

No.	SNMP OID	Description
	clusterEventMessage	Indicates the alert message.
	clusterEventID	Indicates the event ID.
	clusterEventDateTime	Indicates the time at which the alert originated.
	clusterEventServerName	Indicates the server from which the alert originated.
	clusterEventModuleName	Indicates the module from which the alert originated.

7.5.4. Information acquisition by SNMP¶

By using the SNMP protocol, some information about the EXPRESSCLUSTER configuration and status can be acquired. However, EXPRESSCLUSTER does not include SNMP agent functions. For an SNMP agent, Windows SNMP Service needs to be implemented separately.

SNMP agent

The SNMP agent serves to return a response about the configuration information or status information (GetResponse) to information acquisition requests (GetRequest, GetNextRequest) from an SNMP manager (network management software).

Note

If Windows SNMP Service has been installed when EXPRESSCLUSTER Server is installed, the SNMP linkage function is automatically registered. Otherwise, it is not automatically registered.

It needs to be manually registered; for details on how to manually register it, refer to "Setting up the SNMP linkage function manually" in "Installing the EXPRESSCLUSTER Server" in "Installing EXPRESSCLUSTER" in the "Installation and Configuration Guide".

7.5.5. MIB objects acquirable with SNMP linkage¶

The MIB objects that can be acquired with the SNMP linkage function are defined by NEC-CLUSTER-MANAGEMENT-MIB.

NEC-CLUSTER-MANAGEMENT-MIB defines the following MIB objects.

clusterGeneral group

This group is used to acquire cluster information. The MIB objects defined for the group function as described below.

No.

SNMP OID

Description

clusterName

Indicates the name of the cluster.

clusterComment

Indicates the comment of the cluster.

clusterStatus

Indicates the current status of the cluster.

The correspondence between the MIB value and the Cluster WebUI status is as described below.

MIB value   status
---------------------
normal       Normal
caution      Caution
error        Error
unknown       -

clusterServer group

This group is used to acquire server information. Indexes on acquisition of clusterServerTable are sorted by server priority. The MIB objects defined for the group function as described below.

No.	SNMP OID	Description
	clusterServerLocalServerIndex	Indicates the index of the server receiving the present SNMP information acquisition request (clusterServerIndex).
	clusterServerTable	Indicates the information table for the server.
	clusterServerEntry	Indicates the server information list. The index for the list is clusterServerIndex.
	clusterServerIndex	Indicates the index for uniquely identifying the server.
	clusterServerName	Indicates the name of the server.
	clusterServerComment	Indicates a comment for the server.
	clusterServerStatus	Indicates the current status of the server. The correspondence between the MIB value and the Cluster WebUI status is as described below. MIB value status ------------------------------------------------ online Online caution Suspension (Network Partition Unsolved) isolated Suspension (Isolated) offline Offline unknown Unknown
	clusterServerPriority	Indicates the priority of the server.
	clusterServerProductName	Indicates the name of the EXPRESSCLUSTER product installed on the server.
	clusterServerProductVersion	Indicates the version of the EXPRESSCLUSTER product installed on the server.
	clusterServerProductInstallPath	Indicates the installation path of EXPRESSCLUSTER on the server. If the return value is other than an ASCII character, the data might be corrupt.
	clusterServerPlatformName	Indicates the name of the platform on the server.

clusterGroup group

This group is used to acquire group information. The MIB objects defined for the group function as described below.

No.	SNMP OID	Description
	clusterGroupTable	Indicates the information table for the group.
	clusterGroupEntry	Indicates the group information list. The index for the list is clusterGroupIndex.
	clusterGroupIndex	Indicates the index for uniquely identifying the group.
	clusterGroupName	Indicates the name of the group.
	clusterGroupComment	Indicates a comment for the group.
	clusterGroupType	Indicates the type of the group. The correspondence between the MIB value and the group type is as described below. MIB value Group type -------------------------------------- failover Failover group cluster Management group virtualMachine Virtual machine group
	clusterGroupStatus	Indicates the current status of the group. The correspondence between the MIB value and the Cluster WebUI status is as described below. MIB value status ----------------------------------- online Online onlineFailure Online Failure offlineFailure Offline Failure offline Offline unknown Unknown onlinePending Online Pending offlinePending Offline Pending
	clusterGroupCurrentServerIndex	Indicates the index of the server on which the group is currently active (clusterServerIndex). If the group has been deactivated, the return value is -1

7.6. Grace period dependence at the automatic failover between server groups¶

7.6.1. What is the grace period dependence?¶

One server group waits specified time for the other server group to start failover when the automatic failover is executed between server groups. When the grace period elapsed after the server down was detected, the failover is executed.

7.6.2. Condition for the grace period dependence¶

One server group waits for the other server group with any of the following configurations to start the failover.
- Use Server Group settings in the Info tab is selected.
- Multiple server groups are specified for Server Groups that can run the Group in the Startup Server tab
- Prioritize failover policy in the server group is selected and Enable only manual failover among the server groups is not selected for Automatic Failover of Failover Attribute in the Attribute tab.
In the following cases, one server group does not wait specified time for the other server group to start failover:
- One server executes the failover to another server within the same server group.
- The server down is detected by the server down notification.
- The script for forced stop is successfully executed while Execute Script for Forced Stop is selected, or the condition not to execute the script for forced stop is met.
- The forced stop is successfully executed while Execute Script for Forced Stop is not selected and Use Forced Stop is selected, or the condition not to execute the forced stop is met.
- The NP resolution resource is configured.

7.6.3. Displaying and changing the grace period dependence¶

Specify the waiting time for Grace period of server group failover policy.

If 0 is specified, one server group does not wait for the other server group to start failover

7.6.4. Notes on the grace period dependence¶

If any operation is done for the failover target group while the other server group waits during the grace period, the settings to wait during the grace period is canceled and the other server group does not failover.

If the once-failed server is detected to be alive while the other server waits during the grace period, the settings to wait during the grace period is canceled and the failover is not executed.

If the failover target server goes down, the failover may start later than when the grace period ends.

7.7. Witness server service¶

7.7.1. What is Witness server service?¶

Witness service is the service to receive Witness heartbeat from each server in the cluster and send back the status information of receiving the heartbeat from each server as a response. It is installed in a server outside of the cluster.

Fig. 7.3 Witness server service¶

7.7.2. Notes on Witness server service¶

Witness server service operates in Node.js environment. Therefore, Node.js needs to be installed before the installation of the Witness server service.

7.7.3. Operation verified environment for Witness server service¶

Its operation has been verified in the following environments.

OS	Requirement	Version
Windows Server 2012 R2	Node.js 10.13.0	4.1.0
Windows Server 2019	Node.js 12.10.0	4.2.0
Red Hat Enterprise Linux 7.4	Node.js 8.12.0	4.1.0
Red Hat Enterprise Linux 8.0	Node.js 12.10.0	4.2.0

7.7.4. How to install Witness server service¶

Install the Witness server service by using npm command for Node.js environment. Store the Witness server service module in an arbitrary folder, and execute the following command.

> npm install --global clpwitnessd-<version>.tgz

7.7.5. How to configure Witness server service¶

To change the settings of Witness server service, edit the configuration file directly. Open the folder indicated in the first row of the execution results of the command below.

> npm list --global clpwitnessd

Example of execution results:

C:\Users\Administrator\AppData\Roaming\npm

`-- clpwitnessd@4.1.0

Edit clpwitnessd.conf.js that is stored in node_modules\clpwitnessd under the opened folder, with a text editor such as notepad.

Setting items are as follows.

Item	Default	Description
http.enable	True	Specify whether to execute HTTP server or not. true: execute false: not execute
http.port	80	Specify the wait port number for HTTP server.
http.keepalive	10000	Specify the keep alive time for HTTP server in milliseconds.
https.enable	False	Specify whether to execute HTTPS server or not. true: execute false: not execute
https.port	443	Specify the wait port number for HTTPS server.
https.keepalive	10000	Specify the keep alive time for HTTPS server in milliseconds.
https.ssl.key	server_key.pem	Specify a secret key file to be used for HTTPS server.
https.ssl.crt	server_crt.pem	Specify a certification file to be used for HTTPS server.
log.directory	.	Specify the log output destination folder.
log.level	info	Specify the log output level. error: Only error logs are output. warn: Error logs and warning logs are output. info: Warning logs and information logs are output. debug: Information logs and detailed logs are output.
log.size	1024 * 1024 * 512	Specify the log rotation size in bytes.
data.available	10000	Specify the default time limit for the communication status information of the cluster server in milliseconds.

7.7.6. How to execute Witness server service¶

Excute the following command to start up Witness server service in the fore ground. For how to execute the Witness server service as Windows service or Linux daemon, refer to the following section, "Using Witness server service as the OS service".

> clpwitnessd

7.7.7. Using Witness server service as the OS service¶

If you want to start Witness server service at the OS startup, the Witness server service requires to be registered as the OS service.

The following exemplifies how to register Witness server service as the OS service (in case of Windows service control manager and Linux systemd). The method of registration for the OS service differs depending on the environment. Configure the registration to suit your environment by referring to the explanation below.

Registration for Windows service control manager

The following exemplifies the procedure to register by using npm package winser.

Install winser by npm command. Use the following command so that winser package is downloaded from npm repository and then installed.

> npm install --global winser

Create a folder to execute the service in any location. By default, this folder stores log files, SSL secret key file and SSL certificate file.
Create package json file for the service registration with winser, under the folder created in the above step 2. Enter "\\" to separate the characters of the path. The path specified for "start" is line-fed for the convenience of character numbers but actually is in one row.
{
  "name": "clpwitnessd-service",
  "version": "1.0.0",
  "license": "UNLICENSED",
  "private": true,
  "scripts": {
    "start": "C:\\Users\\Administrator\\AppData\\Roaming\\npm\\clpwitnessd.cmd"
  }
}
Execute winser command to register and start the Witness server service.

> winser -i -a

Select Control Panel -> Administration Tools -> Service, and confirm that the service (ex. clpwitnessd-service) with the name specified for "name" of pacage.json has been registered..

Registration for Linux systemd

The following exemplifies the procedure to register by creating the unit file of systemd.

Create a directory to execute the service in any location. By default, this folder stores log files, SSL secret key file and SSL certificate file.

(ex. /opt/clpwitnessd)
Create the unit file of the Witness server service in /etc/systemd/system.

(ex. clpwitnessd.service)
[Unit]
Description=EXPRESSCLUSTER Witness Server
After=syslog.target network.target

[Service]
Type=simple
ExecStart=/usr/bin/clpwitnessd
WorkingDirectory=/opt/clpwitnessd
KillMode=process
Restart= always

[Install]
WantedBy=multi-user.target
Execute systemctl command to register and start the Witness server service.
# systemctl enable clpwitnessd
# systemctl start clpwitnessd