10. Troubleshooting¶
This chapter provides instructions for troubleshooting problems with EXPRESSCLUSTER.
This chapter covers:
10.1. Troubleshooting¶
The following provides instructions for troubleshooting problems you experience in operating the EXPRESSCLUSTER system.
10.1.1. When the EXPRESSCLUSTER system does not start or end¶
A cluster system starts working by restarting servers after installing EXPRESSCLUSTER. If your cluster system does not behave properly, check the following:
- Registration of cluster configuration dataThe cluster configuration data should be registered with all servers (which will form a cluster system) when you cluster them. Make sure that the cluster configuration data is uploaded on all the servers.For details, see "Creating the cluster configuration data" in the "Installation and Configuration Guide" for registering the data.
- Server names and IP addresses in the cluster configuration dataCheck the server names and IP addresses are valid.(>hostname,>ipconfig....)
- License registrationThe license may not be registered yet. Run the license manager on all servers in the cluster to confirm that the license is registered:If you are using the trial version license or fixed term license, confirm if it is not expired yet.To run the license manager, select EXPRESSCLUSTER Server from the Start menu, and then License Manager.
- EXPRESSCLUSTER serviceStart the service control manager of the OS, and make sure that the following EXPRESSCLUSTER services have been started. If all of them have been started, EXPRESSCLUSTER is running normally. To run the service control manager, from Control Panel, select Administrative Tools and select Services.- EXPRESSCLUSTER- EXPRESSCLUSTER API- EXPRESSCLUSTER Disk Agent- EXPRESSCLUSTER Event- EXPRESSCLUSTER Information Base- EXPRESSCLUSTER Manager- EXPRESSCLUSTER Node Manager- EXPRESSCLUSTER Server- EXPRESSCLUSTER Transaction- EXPRESSCLUSTER Web Alert
- Free disk spaceRun [Disk Management] to check the size of the free disk space in the drive that contains <EXPRESSCLUSTER_installation_path>. For details on the disk space to be used by EXPRESSCLUSTER, see "Installation requirements for EXPRESSCLUSTER" in "Getting Started Guide". To run Disk Management, select Control Panel, select Administrative Tools, and select Computer Management. Then from the icon tree, select Services under Services and Applications.
- Usage of memory or OS resourceRun Task Manager of the OS to check the OS memory usage and CPU usage rate.
10.1.2. When activating or deactivating network partition resolution resource fails¶
- Majority methodMemory or OS resources may not be sufficient. Check them.
- PING methodMemory or OS resources may not be sufficient. Check them.
- DISK methodThe settings of the Cluster WebUI are invalid. Check that the disk heartbeat partition is set to be filtered on the HBA tab of Server Properties of the server that failed to be activated or deactivated. Check that the disk heartbeat partition is not used by other resource (disk resource, mirror disk resource).
10.1.3. When a network partition resolution resource error is detected¶
- Majority methodMemory or OS resources may not be sufficient. Check them.
- PING methodThere is no PING command response from the PING destination device. Check that there is no problem in the communication path from the cluster server to the PING destination device.
- DISK methodTimeout occurred in accessing the disk heartbeat partition, or disconnection of the cable to the shared disk was detected.If a timeout has occurred, select Cluster Properties -> disk network partition resolution resource in which the error occurred from the NP Resolution tab, and then open Properties. Adjust IO Wait Time in the Disk NP Properties dialog.If cable disconnection is detected, check the cable connection status.
10.1.4. When activating or deactivating group resources fails¶
If any error is detected in activation of a group resource, detailed error information is logged in the alert and event log. See " Detailed information in activating and deactivating group resources " and examine the logs to find the cause of the error and take appropriate action for it.
10.1.5. When a monitor resource error occurs¶
If a monitor resource detects any error, detailed information on error is logged in the alert and event logs. From the information, see " Detailed information of monitor resource errors " and examine the logs to find the cause of the error and take appropriate action for it.
10.1.6. When a heartbeat timeout occurs¶
Possible causes of heartbeat timeout between servers are listed below:
Cause |
Solution |
---|---|
Disconnection of LAN cables |
Check that you can send packets with the ping command. |
10.1.7. Recovering from failure of one server¶
If the automatic recovery mode is not set in Cluster Properties, the server that you have removed errors and restarted is in the Suspension (Isolated) status. To recover the server to the normally functioning cluster from this status, use the Cluster WebUI or the clpcl command.
If the Replicator is used, data between the disks that form a mirror set becomes inconsistent. However, by recovering the server, the mirror will be automatically rebuilt, and the data will become consistent.
To recover the server by using the Cluster WebUI, see the online manual.
To recover the server by using the clpcl command, see "Operating the cluster (clpcl command)" in "9. EXPRESSCLUSTER command reference" in this guide.
10.1.8. Recovering from failure of both servers¶
When Off is selected for Auto Return on the Extension tab in Cluster Properties, when all the servers shut down by failures such as a hardware failure, after starting them up, they are removed from the cluster. Recover all the servers by using the Cluster WebUI or the clpcl command.
Right after recovering the servers, all the groups are stopped. Start all the groups. If the Replicator is used, the mirror will be automatically built again by starting the groups, and the data will become consistent.
10.1.9. When network partitioning occurs¶
Network partitioning indicates that all communication routes are blocked between servers. This section describes how you can check whether or not the network is partitioned and what you should do about it when the network partition resolution resource is not registered. The following examples assume that you have registered kernel mode LAN heartbeat resources for heartbeat resources in a 2-node cluster configuration.
When all heartbeat resources are normal (the network is not partitioned), the result of executing the clpstat command is:
When you run the command on server1
# clpstat -n ==================== HEARTBEAT RESOURCE STATUS ================= Cluster : cluster *server0 : server1 server1 : server2 HB0 : lankhb1 HB1 : lankhb2 [on server0 : Online] HB 0 1 ----------------------------------------------------------------- server0 : o o server1 : o o [on server1 : Online] HB 0 1 ----------------------------------------------------------------- server0 : o o server1 : o o =================================================================
When you run the command on server2
# clpstat -n ==================== HEARTBEAT RESOURCE STATUS ================= Cluster : cluster server0 : server1 *server1 : server2 HB0 : lankhb1 HB1 : lankhb2 [on server0 : Online] HB 0 1 ----------------------------------------------------------------- server0 : o o server1 : o o [on server1 : Online] HB 0 1 ----------------------------------------------------------------- server0 : o o server1 : o o ================================================================
When the network is partitioned, the result of executing the clpstat command is what is described below. Both servers recognize each other that the counterpart is down.
When you run the command on server1
# clpstat -n ==================== HEARTBEAT RESOURCE STATUS ================= Cluster : cluster *server0 : server1 server1 : server2 HB0 : lankhb1 HB1 : lankhb2 [on server0 : Caution] HB 0 1 ----------------------------------------------------------------- server0 : o o server1 : x x [on server1 : Offline] HB 0 1 ----------------------------------------------------------------- server0 : - - server1 : - - =================================================================
When you run the command on server2
# clpstat -n ==================== HEARTBEAT RESOURCE STATUS ================= Cluster : cluster server0 : server1 *server1 : server2 HB0 : lankhb1 HB1 : lankhb2 [on server0 : Offline] HB 0 1 ----------------------------------------------------------------- server0 : - - server1 : - - [on server1 : Caution] HB 0 1 2 ----------------------------------------------------------------- server0 : x x server1 : o o =================================================================
Shut down both servers immediately if the network is partitioned. Check the following for heartbeat resources.
- Kernel mode LAN heartbeat resource- LAN cable status- Network interface status
If interconnection LAN is recovered from the network partitioning, EXPRESSCLUSTER causes the servers to shut down.
If EXPRESSCLUSTER detects that the same group is active on multiple servers, it causes the servers to shut down.
For the Replicator, depending on the server shutdown timing, the statuses of mirror disk resources may not be the same after rebooting the server.
Depending on the timing of server shutdown, the status of mirror disk resources may be the one requiring forced mirror recovery, mirror recovery, or normal.
10.2. Connecting mirror disks/hybrid disks manually¶
This section describes how to cancel the access restriction for the data partition of mirror disk resource or hybrid disk resource when you cannot start EXPRESSCLUSTER due to some sort of failure.
10.2.1. Normally connecting mirror disk when mirroring is available¶
When the EXPRESSCLUSTER Server service can be activated while the EXPRESSCLUSTER X Disk Agent service cannot be, access restriction can be canceled by following the steps below.
Run the following command on the server where you want to connect disks.
For mirror disks:
clpmdctrl --active <mirror_disk_resource_name (Example: md1)>For hybrid disks:
clphdctrl --active <hybrid_disk_resource_name (Example: hd1)>The mirror disk resource or hybrid disk resource becomes accessible. Written data is mirrored to the other server.
10.2.2. Forcibly connecting mirror disk when mirroring is not available¶
Follow the steps below to save data on mirror disks when both the EXPRESSCLUSTER Server service and the EXPRESSCLUSTER X Disk Agent service cannot be activated.
However, the mirroring status up to the moment just before both the EXPRESSCLUSTER Server service and the EXPRESSCLUSTER X Disk Agent service became unable to be activated must be normal, or you must know which server has the latest data.
The EXPRESSCLUSTER service cannot be started on Server 1 or Server 2. Server 1 has the latest data. Uninstall EXPRESSCLUSTER on the server which has the latest data and restart the server.
Fig. 10.1 Saving the data on the mirror disk (1)¶
Connect the backup device to Server 1, and back up the data in the data partition by using the backup command.
Fig. 10.2 Saving the data on the mirror disk (2)¶
As for hybrid disk resources, if the above is performed while another server in the same server group is using the shared disk, the data on the shared disk may be destroyed. Make sure to stop the other server or disconnect the disk cable of the other server when you perform the above.
10.3. Recovering from mirror breaks¶
When the auto-mirror recovery is enabled, no special operation is required. Mirroring is automatically recovered. However, if mirroring needs to be recovered forcibly, execution of a command or operations for forcible mirror recovery using the Cluster WebUI are required.
If the auto-mirror recovery is disabled, you have to recover mirroring by executing a command or using the Cluster WebUI.
The difference mirror recovery function is disabled and full copy is performed in the following cases:
- When the partition configuration for mirror disk resource or hybrid disk has been changed due to disk replacement etc.
- When both servers fail at the same time while mirror disk resource is normally activated
- When the current servers (servers updating and managing the disk in the server group) in the both server groups fail at the same time when the hybrid disk resource is normally activated
- When difference information could not be written successfully due to disk failure etc.
10.3.1. Automatically recovering from mirroring¶
When the auto-mirror recovery is enabled, mirroring is automatically recovered under the following conditions:
Mirror disk resource or hybrid disk resource is active. The server where mirror disk resource or hybrid disk resource is activated contains the latest data. Servers in the cluster are in the normal status, and you can verify their mirroring statuses. The data among the servers is not the same. In case of mirror disk, both the mirror disk monitor resource must operate normally on all the registered servers.In case of hybrid disk, hybrid disk monitor resource must operate normally on a server that monitor target resource is activated. Resource is not activated on a server/server group that does not store the latest data. Auto Mirror Recovery Setting in the cluster's properties must be checked.However, if the initial mirror configuration is yet to be set, the initial auto mirror configuration must be checked in accordance with the above indicated. Failures such as disk errors on the target disk of the mirror disk or hybrid disk are not occurring
The auto-mirror recovery is not performed if any of the following applies.
One of the servers (for hybrid disk resource, all servers in one of the server groups) is not started. You cannot confirm the mirroring status of the other server. There is no server whose mirror status is normal. Mirror disk monitor resource or hybrid disk monitor resource is not registered. Monitor resource is suspended or stopped in a server or server group that stores the latest data. Resource is forcibly activated in other server or server group, including when snap shot backup is being executed.
For information on how to verify the progress of recovering mirroring, see "Checking the mirror recovery progress with a command" and "Checking the mirror recovery progress from the Cluster WebUI".
10.3.2. Checking the mirror break status with a command¶
For mirror disk resources, run the following command to view the mirror break statuses.
clpmdstat --mirror <mirror_disk_resource_name (Example: md1)>
You can view the mirror disk resource statuses by running the clpmdstat command.
When normal:
Status: Normal md1 server1 server2 --------------------------------------------------------------- Mirror Color GREEN GREEN Fast Copy -- -- Lastupdate Time -- -- Break Time -- -- Needed Copy Percent 0% 0% Volume Used Percent 64% 64% Volume Size 10240MB 10240MB Server Name DP Error CP Error --------------------------------------------------------------- server1 NO ERROR NO ERROR server2 NO ERROR NO ERROR
When the mirror recovery is required:
Status: Abnormal md1 server1 server2 --------------------------------------------------------------- Mirror Color GREEN RED Fast Copy OK OK Lastupdate Time 2021/08/16 18:24:10 -- Break Time 2021/08/16 18:24:01 -- Needed Copy Percent 1% 0% Volume Used Percent 64% --% Volume Size 10240MB 10240MB Server Name DP Error CP Error --------------------------------------------------------------- server1 NO ERROR NO ERROR server2 NO ERROR NO ERROR
When the forcible mirror recovery is required:
Status: Abnormal md1 server1 server2 --------------------------------------------------------------- Mirror Color RED RED Fast Copy NG NG Lastupdate Time 2021/08/16 18:24:10 2021/08/16 18:50:33 Break Time 2021/08/16 18:24:01 2021/08/16 18:24:01 Needed Copy Percent 1% 1% Volume Used Percent 64% --% Volume Size 10240MB 10240MB Server Name DP Error CP Error --------------------------------------------------------------- server1 NO ERROR NO ERROR server2 NO ERROR NO ERROR
While the mirroring is being recovered:
See " Checking the mirror recovery progress with a command ".
For hybrid disk, execute the following command to check the mirror break status.
clphdstat --mirror <hybrid_disk_resource_name (Example: hd1)>
For details, see "Displaying the hybrid disk status (clphdstat command)" in "9. EXPRESSCLUSTER command reference" in this guide.
10.3.3. Checking the mirror recovery progress with a command¶
For mirror disk resources, run the following command to view the progress of recovering mirroring.
clpmdstat --mirror <mirror_disk_resource_name (Example: md1)>
You will see the following data while mirroring is being recovered.
Status: Recovering
md1 server1 server2
---------------------------------------------------------------
Mirror Color YELLOW -> YELLOW
15%
Recovery Status
---------------------------------------------------------------
Used Time 00:00:21
Remain Time 00:01:59
You will see the following information when the mirror recovery is successfully completed.
Status: Normal
md1 server1 server2
---------------------------------------------------------------
Mirror Color GREEN GREEN
Fast Copy -- --
Lastupdate Time -- --
Break Time -- --
Needed Copy Percent 0% 0%
Volume Used Percent 64% 64%
Volume Size 10240MB 10240MB
Server Name DP Error CP Error
---------------------------------------------------------------
server1 NO ERROR NO ERROR
server2 NO ERROR NO ERROR
For hybrid disks, execute the following command to check the mirror break status.
clphdstat --mirror <hybrid_disk_resource_name (Example: hd1)>
For details, see "Displaying the hybrid disk status (clphdstat command)" in "9. EXPRESSCLUSTER command reference" in this guide.
10.3.4. Recovering mirror with a command¶
Run the following command to start the mirror recovery.
For mirror disk:
clpmdctrl --recovery <mirror_disk_resource_name (Example: md1)>
For hybrid disk:
clphdctrl --recovery <hybrid_disk_resource_name (Example: md1)>
When the difference mirror recovery can be performed, the difference data is used to recover the mirror (FastSync technology).
This command immediately returns the control once the mirror recovery starts. For information on how to verify the mirror recovery progress, see "Checking the mirror recovery progress with a command" and "Checking the mirror recovery progress from the Cluster WebUI".
10.3.5. Running the forcible mirror recovery with a command¶
If EXPRESSCLUSTER cannot automatically determine which server contains the latest data, you have to run the forcible mirror recovery.
In this case, you have to manually identify the server that holds the latest data, and perform the forcible mirror recovery.
Note
The difference mirror recovery function is disabled in the forcible mirror recover, and the data may be fully copied.
Identify the server that holds the latest data by any of the following means:
Using Mirror disks of the Cluster WebUI
1. In the mirror disks of Cluster WebUI, click the mirror disk resource or hybrid disk resource you want to check.2. Click the Details icon.3. See the last update time stamp (Last data updated time) to identify the server which has the latest data. However, this Last data updated time depends on the operating system's clock.
Using the clpmdstat / clphdstat command
You can identify the server which has the latest data by using the following commands.
Run the following command.
For mirror disks:
clpmdstat --mirror <mirror_disk_resource_name (Example: md1)>For hybrid disks:
clphdstat --mirror <hybrid_disk_resource_name(Example: hd1)>See the last update time stamp (Last data updated time) to identify the server which has the latest data. However, this Last data updated time depends on the operating system's clock.
Using data on disks
Note
This method is not recommended because the data may be damaged if anything goes wrong in the procedure. Use the procedure described in "Using Mirror disks of the Cluster WebUI" or "Using the clpmdstat/clphdstat command" above when possible.
For mirror disks:
Confirm all groups are stopped.
Run the following command to connect the mirror disk resource.
clpmdctrl --active <mirror_disk_resource_name (Example: md1)> -fLogically examine the data on the connection destination.
Run the following command to disconnect the mirror disk resource.
clpmdctrl --deactive <mirror_disk_resource_name (Example: md1)>
For hybrid disks:
Confirm all groups are stopped.
Run the following command to connect the hybrid disk resource.
clphdctrl --active <hybrid_disk_resource_name (Example: hd1)> -fLogically examine the data on the connection destination.
Run the following command to disconnect the hybrid disk resource.
clphdctrl --deactive <hybrid_disk_resource_name (Example: hd1)>
When you have identified the server holding the latest data, run the following command to start the forcible mirror recovery.
For mirror disks (conducted on the server having the latest data):
clpmdctrl --force <mirror_disk_resource_name (Example: md1)>
For hybrid disks (conducted on the server having the latest data):
clphdctrl --force <hybrid_disk_resource_name (Example: hd1)>
Note
The clpmdctrl --force command and the clphdctrl --force command update data in the server where they are executed. If automatic mirror recovery does not work through this step, perform manual mirror recovery.
The clpmdctrl / clphdctrl command immediately returns the control once the forcible mirror recovery starts. For information on how to check the forcible mirror recovery progress, see "Checking the mirror recovery progress with a command" and "Checking the mirror recovery progress from the Cluster WebUI".
When the forcible mirror recovery is successfully completed, activate the groups. The mirror disks become available.
10.3.6. Running the forcible mirror recovery with a command only on one server¶
In some cases, you cannot start one of the servers due to a hardware or OS failure, and the server that can be started may not have the latest data. If you want to start applications at least on the server that can be started, you can perform the forcible mirror recovery on that server.
However, remember that if you do this, the data on the server where you run this command becomes the latest data no matter which server actually has it. Therefore, even if you are able to start the other server later, you cannot handle the data in that server as the latest one. Make sure you understand the consequence before running the following command.
Execute the following command on the target server to start forcible mirror recovery.
For mirror disk resources:
clpmdctrl --force <mirror_disk_resource_name (Example: md1)>For mirror disk resources:
clphdctrl --force <hybrid_disk_resource_name (Example: hd1)>
After executing the command, it becomes possible to start the group to use the resource.
10.3.7. Checking the mirror break status from the Cluster WebUI¶
You can see the mirror break status by starting Mirror disks from the Cluster WebUI.
When normal:
When mirror recovery is required:
When forcible mirror recovery is required:
While mirror recovery is in progress:
See " Checking the mirror recovery progress from the Cluster WebUI ".
10.3.8. Checking the mirror recovery progress from the Cluster WebUI¶
From the mirror disks of Cluster WebUI to view the mirror recovery progress.
You will see the following screen during the mirror recovery.
You will see the following screen when the mirror recovery is successfully completed.
10.3.9. Recovering mirror using the Cluster WebUI¶
From the mirrors disks of Cluster WebUI, click the name of the mirror disk that needs to be recovered. The window below will be displayed.
Click Difference Copy or Full Copy of the server that needs to be recovered. Click Execute to start the mirror recovery processing.
When the difference mirror recovery can be performed, the recovery is done using the difference data. (FastSync technology)The difference mirror recovery takes less time than the forcible mirror recovery.
For information on how to check the mirror recovery progress, see " Checking the mirror recovery progress with a command" and "Checking the mirror recovery progress from the Cluster WebUI".
10.3.10. Running the forcible mirror recovery using the Cluster WebUI¶
When EXPRESSCLUSTER cannot determine which server has the latest data, you have to perform the forcible mirror recovery. In this case, you have to manually identify the server which holds the latest data, and perform the forcible mirror recovery.
The difference mirror recovery function is disabled in the forcible mirror recovery and the data may be fully copied.
Identify the server that has the latest data by any of the following methods:
Using Mirror disks of the Cluster WebUI
1. From the mirror disks of Cluster WebUI, display the detailed data of the mirror disk resources you want to see.2. Click the Details icon.3. See the last update time stamp to identify the server which has the latest data. However, this Last Data Updated Time depends on the operating system's clock.
Click Mirror Recovery of the server containing the latest data to display the following window. Click Execute to start the mirror recovery processing.
For information on how to check the forcible mirror recovery progress, see "Checking the mirror recovery progress with a command" and "Checking the mirror recovery progress from the Cluster WebUI".
When the forcible mirror recovery is successfully completed, you can activate the groups and use the mirror disks.
10.3.11. Running the forcible mirror recovery from the Cluster WebUI only on one server¶
In some cases, you cannot start one of the servers due to a hardware or OS failure, and the server that can be started may not have the latest data. If you want to start applications at least on the server that can be started, you can perform the forcible mirror recovery on that server.
However, remember that if you do this, the data on the server where you run this command becomes the latest data no matter which server actually has it. Therefore, even if the other server becomes available later, you cannot handle the data in that server as the latest one. Make sure you understand the consequence before running the following command.
From the mirror disks of Cluster WebUI, execute the forcible mirror recovery. Click Mirror Recovery of the server to which you want to perform the forcible mirror recovery, and then the following window appears. Click Execute to perform the forcible mirror recovery processing.
When the forcible mirror recovery is successfully completed, you can activate the groups and use the mirror disks.
10.4. Media sense function becomes invalid¶
Media sense function is the OS function that detects disconnection of network cable. When disconnection is detected, TCP/IP receives a notification from the media sense function and renders the information such as an IP address assigned to the disconnected network card unavailable while it is disconnected. EXPRESSCLUSTER cannot be operated properly if the information such as IP address becomes invalid during its operation, so the media sense function is rendered invalid when installing EXPRESSCLUSTER.