1. Preface

1.1. Who Should Use This Guide

The HA Cluster Configuration Guide for Google Cloud Platform (Windows) is intended for administrators who want to build a cluster system, and for system engineers and maintenance personnel who provide user support.

The software and setup examples introduced in this guide are for reference only, and the software is not guaranteed to run.

1.2. Scope of application

For information on the system requirements, see "Getting Started Guide" -> "Installation requirements for EXPRESSCLUSTER".

This guide contains product- and service-related information (e.g., screenshots) collected at the time of writing this guide. For the latest information, which may be different from the content in this guide, refer to corresponding websites and manuals.

1.3. How This Guide is Organized

• "2. Overview ": Describes the functional overview.

• "3. Operating Environments ": Describes the tested operating environment of this function.

• "4. Cluster Creation Procedure (for an HA Cluster with an Internal TCP Load Balancer)": Describes how to create an HA cluster involving an internal TCP load balancer.

• "5. Error Messages": Describes the error messages and solutions.

• "6. Notes and Restrictions ": Describes the notes and restrictions on creating and operating a cluster.

1.4. EXPRESSCLUSTER X Documentation Set

The EXPRESSCLUSTER X manuals consist of the following four guides. The title and purpose of each guide is described below:

EXPRESSCLUSTER X Getting Started Guide

This guide is intended for all users. The guide covers topics such as product overview, system requirements, and known problems.

EXPRESSCLUSTER X Installation and Configuration Guide

This guide is intended for system engineers and administrators who want to build, operate, and maintain a cluster system. Instructions for designing, installing, and configuring a cluster system with EXPRESSCLUSTER are covered in this guide.

EXPRESSCLUSTER X Reference Guide

This guide is intended for system administrators. The guide covers topics such as how to operate EXPRESSCLUSTER, function of each module and troubleshooting. The guide is supplement to the Installation and Configuration Guide.

EXPRESSCLUSTER X Maintenance Guide

This guide is intended for administrators and for system administrators who want to build, operate, and maintain EXPRESSCLUSTER-based cluster systems. The guide describes maintenance-related topics for EXPRESSCLUSTER.

1.5. Conventions

In this guide, Note, Important, See also are used as follows:

Note

Used when the information given is important, but not related to the data loss and damage to the system and machine.

Important

Used when the information given is necessary to avoid the data loss and damage to the system and machine.

See also

Used to describe the location of the information given at the reference destination.

The following conventions are used in this guide.

Convention	Usage	Example
Bold	Indicates graphical objects, such as text boxes, list boxes, menu selections, buttons, labels, icons, etc.	Click Start. Properties dialog box
Angled bracket within the command line	Indicates that the value specified inside of the angled bracket can be omitted.	clpstat -s [-h host_name]
>	Prompt to indicate that a Windows user has logged on as root user.	> clpstat
Monospace	Indicates path names, commands, system output (message, prompt, etc.), directory, file names, functions and parameters.	C:\Program Files
bold	Indicates the value that a user actually enters from a command line.	Enter the following: > clpcl -s -a
italic	Indicates that users should replace italicized part with values that they are actually working with.	> ping <IP address>

In the figures of this guide, this icon represents EXPRESSCLUSTER.

1.6. Contacting NEC

For the latest product information, visit our website below:

https://www.nec.com/en/global/prod/expresscluster/

2. Overview

2.1. Functional overview

This guide describes how to create an HA cluster based on EXPRESSCLUSTER X (EXPRESSCLUSTER) on the cloud service of Google Cloud Platform (GCP).

GCP allows you to use regions and zones to create an HA cluster with virtual machines, increasing the business availability.

• Region

  On GCP, a region is a division: a physical and logical unit (like New York and London).

  It is possible to build all nodes in a single region. However, a network failure or a natural disaster may make all of them crash to prevent the business from continuing.

  To increase availability, distribute nodes to multiple regions.

  A region is a group of zones.

• Zone

  On GCP, a zone is a logical group to which each node can be distributed.

  By distributing each node to a different zone, you can minimize the effects of planned GCP maintenance and those of unplanned maintenance due to a physical hardware failure.

For more information on regions and zones, refer to the following:

Regions and zones:

https://cloud.google.com/compute/docs/regions-zones/

2.2. Basic configuration

This guide assumes an HA cluster (uni-directional standby cluster configuration) with a load balancer.

For the HA cluster, the following EXPRESSCLUSTER resource and GCP service are to be used:

Purpose	EXPRESSCLUSTER resource to be chosen	Necessary GCP service
Accessing the cluster at a virtual/internal IP address from a client	Google Cloud virtual IP resource	Internal TCP load balancing

HA cluster with a load balancer

For virtual machines in a GCP environment, client applications can use a virtual IP (VIP) address to access nodes that constitute a cluster. Using the VIP address eliminates the need for the clients to be aware of switching between the virtual machines even after a failover or a group migration occurs.

As Fig. 2.1 HA cluster with an internal TCP load balancer shows, the cluster in the GCP environment can be accessed by specifying the VIP address (front-end IP address for Cloud Load Balancing) of the GCP load balancer (for Cloud Load Balancing).

The GCP load balancer switches between the active server and the standby server, with its health check. The health check is performed through a port provided by the Google Cloud virtual IP resource.

Fig. 2.1 HA cluster with an internal TCP load balancer

SubNetwork-1	10.0.1.0/24
IP Address (Client)	10.0.1.200
Virtual IP Address (VIP)	10.0.1.100
SubNetwork-11	10.0.11.0/24
IP Address (Server1)	10.0.11.101
IP Address (Server2)	10.0.11.102
Health Check Port	12345

For more information on Cloud Load Balancing, refer to the following:

Cloud Load Balancing:

https://cloud.google.com/load-balancing/

The following is an example of an HA cluster with a load balancer:

Purpose	Load balancer to be used	Creation procedure
Sharing business within the GCP network	Internal TCP load balancer	See "4. Cluster Creation Procedure (for an HA Cluster with an Internal TCP Load Balancer)" of this guide.

An HA cluster configuration with a load balancer requires the following resources and monitor resources.

For a network partition resolution resource, which is not set in this guide, see "2.3. Network partition resolution" to determine how it should be depending on your system configuration.

Resource/monitor resource	Description	Setting
Google Cloud virtual IP resource	Provides a mechanism for awaiting access from the load balancer to a specific port for the alive monitoring (health check)--over the port, an application works on a node. At activation, this resource starts up a control process to await access from the GCP load balancer for its alive monitoring. At deactivation, the resource stops the control process.	Required
Google Cloud virtual IP monitor resource	Performs alive monitoring of a control process to be started in activating a Google Cloud virtual IP resource, for the node where the Google Cloud virtual IP resource is started.	Required
Google Cloud load balance monitor resource	Checks whether the same port as that for the health check is opened, for a node where the Google Cloud virtual IP resource is not started.	Required
Other resources and monitor resources	Depends on the configuration of the application (such as a mirror disk) to be used in the HA cluster.	Optional

2.3. Network partition resolution

Virtual machines constituting an HA cluster mutually perform alive monitoring with heartbeat.

If heartbeat ceases with each virtual machine existing on a different subnet, an undesirable event (such as double-launching a service) occurs.

Its prevention requires determining whether any of the virtual machines has been isolated from the network, in a network partition (NP) state, or another virtual machine has crashed.

The NP resolution feature pings a device which always operates and is expected to respond. If there is no reply, the feature considers the device to be in an NP state and takes a specified measure (such as warning, recovery, and server shutdown).

The target and method of NP resolution needs to be individually considered, in accordance with the locations of clients accessing the cluster system and with the conditions for connecting to an on-premise environment (e.g. using a leased line).

There is no recommended target or method of NP resolution.

For more information on NP resolution, refer to the following:

• "Installation and Configuration Guide" -> "Understanding network partition resolution resources"

• "Reference Guide" -> "Heartbeat resources"

• "Reference Guide" -> "Details on network partition resolution resources"

3. Operating Environments

3.1. HA cluster with a load balancer

Refer to the following manual:

• "Getting Started Guide" -> "Installation requirements for EXPRESSCLUSTER" -> "Operation environments for Google Cloud virtual IP resource, Google Cloud virtual IP monitor resource, and Google Cloud load balance monitor resource"

4. Cluster Creation Procedure (for an HA Cluster with an Internal TCP Load Balancer)

4.1. Creation example

This guide describes how to create a two-node, uni-directional standby cluster with EXPRESSCLUSTER.

Through this procedure, you are to create an HA cluster accessible from clients within the same VPC network on GCP.

This procedure is intended to create a mirror-disk type configuration in which server1 is used as the active server.

The following tables list parameters which do not have their default values, and parameters whose values are changed from their default values.

Of the firewall rules, IP range is necessary for allowing communication from the GCP health check system (130.211.0.0/22, 35.191.0.0/16).

• GCP settings (common to server1 and server2)

  Item	Value
  Configuration of the VPC network
  Name	test-vpc
  New subnet (name)	subnetwork-1, subnetwork-11
  New subnet (region)	asia-northeast1
  New subnet (IP address range)	10.0.1.0/24, 10.0.11.0/24
  Configuration of the firewall rules
  Name	test-allow-health-check
  Network	test-vpc
  Traffic direction	Upstream
  Action in response to agreement	Allow
  Target	Specified target tag
  Target tag	test-allow-health-check
  Source filter	IP range
  Source IP range	130.211.0.0/22, 35.191.0.0/16
  Specified protocol and port	Allow all
  Configuration of the load balancer
  Type	TCP load balancing
  For internet connection or private use	Only between VMs
  Multi- or mono-region	Only mono-region
  Name	test-lb
  Configuration of the load balancer (back end)
  Region	asia-northeast1
  Network	test-vpc
  Instance group	test-ig-a, test-ig-b
  Health check (name)	test-health-check
  Health check (protocol)	TCP
  Health check (port)	12345
  Health check (proxy protocol)	None
  Session affinity	None
  Configuration of the load balancer (front end)
  Name	test-frontend
  Subnetwork	subnetwork-1
  Internal IP address	10.0.1.100
  Port	80 (number of the port through which the application is available)

• GCP settings (to be set separately on server1 and server2)

  Item	Value
  	server1	server2
  Configuration of the instances
  Region	asia-northeast1	asia-northeast1
  Zone	asia-northeast1-a	asia-northeast1-b
  New disk	server1-datadisk-0	server2-datadisk-0
  Configuration of the instance groups
  Name	test-ig-a	test-ig-b
  Group type	Unmanaged instance group	Unmanaged instance group
  Region	asia-northeast1	asia-northeast1
  Zone	asia-northeast1-a	asia-northeast1-b
  Network	test-vpc	test-vpc
  Subnetwork	subnetwork-11	subnetwork-11
  VM instance	server1	server2
  Network configuration
  Network	test-vpc	test-vpc
  Subnetwork	subnetwork-11	subnetwork-11
  Internal IP address	10.0.11.101	10.0.11.102

• EXPRESSCLUSTER settings (cluster properties)

  Item	Value
  	server1	server2
  Cluster name	Cluster1	Cluster1
  Server name	server1	server2

• EXPRESSCLUSTER settings (failover group)

  Resource name	Item	Value
  Mirror disk resource	Resource name	md
  Mirror disk resource	Details tab - drive letter of the data partition	G:
  Mirror disk resource	Details tab - drive letter of the cluster partition	F:
  Google Cloud virtual IP resource	Resource name	gcvip1
  Google Cloud virtual IP resource	Port number	12345: as specified for Health check (port)

• EXPRESSCLUSTER settings (monitor resource)

  Monitor resource name	Item	Value
  Mirror disk monitor resource	Monitor resource name	mdw1
  Google Cloud virtual IP monitor resource	Monitor resource name	gcvipw1
  Google Cloud virtual IP monitor resource	Recovery target	gcvip1
  Google Cloud load balance monitor resource	Monitor resource name	gclbw1
  Google Cloud load balance monitor resource	Recovery target	gcvip1

4.2. Configuring GCP

1. Creating the VPC network

Access GCP Console (https://console.cloud.google.com/).

Create the VPC network and subnets.

For more information on the procedure, refer to the following:

Using VPC networks:

https://cloud.google.com/vpc/docs/using-vpc/

2. Creating the instances

Create each of the instances based on a public image.

During this creation, add a secondary disk for the mirror disk (cluster partition and data partition).

Create as many instances as the number of virtual machines constituting the cluster.

For more information on the procedure, refer to the following:

How-to guides:

https://cloud.google.com/compute/docs/how-to/

3. Configuring the instances

Access and log in to the created instances (server1 and server2).

For more information on the procedure, refer to the following:

Connecting to VMs:

https://cloud.google.com/compute/docs/instances/connecting-to-instance/

Next, set the partitions for the mirror disk resource.

Create the cluster partition and data partition on the secondary disk added to the instance.

For more information on setting the partitions for the mirror disk resource, refer to the following:

- "Installation and Configuration Guide" -> "Determining a system configuration" -> "Settings after configuring hardware" -> "Mirror partition settings (Required for mirror disks)"

4. Creating the firewall rules

Create the firewall rules for allowing communication from the GCP health check system (130.211.0.0/22, 35.191.0.0/16), where the health check is to be performed by the load balancer to the instance.

In addition, add the target tag to the network tag of the instances (server1 and server2).

For more information on the procedure, refer to the following:

Using firewall rules:

https://cloud.google.com/vpc/docs/using-firewalls/

Creating health checks:

https://cloud.google.com/load-balancing/docs/health-checks/

5. Creating the instance groups

Create the instance groups to be specified as the back ends of Cloud Load Balancing. To the group, add the instances (server1 and server2).

For more information on the procedure, refer to the following:

Creating groups of unmanaged instances:

https://cloud.google.com/compute/docs/instance-groups/creating-groups-of-unmanaged-instances

6. Creating the load balancer

Create the load balancer. Select TCP Load Balancing.

For more information on the procedure, refer to the following:

Setting up a network load balancer with a target pool:

https://cloud.google.com/load-balancing/docs/network/setting-up-network/

Next, configure the back end and the front end.

For Ports of the front end, specify the number of the port through which the application is available.

For more information on the procedure, refer to the following:

Setting up Internal TCP/UDP Load Balancing:

https://cloud.google.com/load-balancing/docs/internal/setting-up-internal

7. Adjusting the OS startup time, verifying the network settings, verifying the firewall settings, synchronizing the server clock, and turning off the power-saving function

For information on each of the procedures, refer to the following:

- "Installation and Configuration Guide" -> "Determining a system configuration" -> "Settings after configuring hardware"

8. Installing EXPRESSCLUSTER

For information on the procedure, refer to the following document. After completing the installation, reboot the OS.

- "Installation and Configuration Guide"

9. Registering the EXPRESSCLUSTER license

For information on the procedure, refer to the following document:

- "Installation and Configuration Guide"

4.3. Configuring EXPRESSCLUSTER

For information on how to set up and access Cluster WebUI, refer to the following:

• "Installation and Configuration Guide" -> "Creating the cluster configuration data"

This section describes how to add the following resources and monitor resources:

• Mirror disk resource

• Google Cloud virtual IP resource

• Google Cloud virtual IP monitor resource

• Google Cloud load balance monitor resource

For information on other settings, refer to the following:

• "Installation and Configuration Guide"

• "Reference Guide"

1. Creating a cluster

To create a cluster, start the cluster generation wizard first.

• Creating a cluster

  1. Access Cluster WebUI, and click Cluster generation wizard.
  2. Of Cluster generation wizard, Cluster appears.

     In Cluster Name, enter a cluster name.

     From Language, select an appropriate language. Click Next.
  3. Basic Settings appears.

     The instance connected to Cluster WebUI appears as a registered master server.

     Click Add to add the remaining instances (by specifying their internal IP addresses). Click Next.
  4. The Interconnect screen appears.

     Specify the instance's internal IP address to be used for the interconnect. For MDC, select mdc1 as the communication path of a mirror disk resource to be created later. Click Next.
  5. The NP Resolution screen appears.

     With nothing specified, click Next.

2. Adding group resources

• Defining a group

  Create a failover group.

  1. The Group List screen appears.

     Click Add.
  2. The Group Definition screen appears.

     In Name, enter failover1 as a failover group name. Click Next.
  3. The Startup Servers screen appears.

     With nothing specified, click Next.
  4. The Group Attributes screen appears.

     With nothing specified, click Next.
  5. The Group Resource List screen appears.

     Here, add group resources as below.

• Mirror disk resource

  Create a mirror disk resource.

  For more information, refer to the following:

  - "Reference Guide" -> "Understanding Mirror disk resources"
  1. In Group Resource List, click Add.
  2. The Resource Definition of Group | failover1 screen appears.

     From the Type box, select Mirror disk resource as a group resource type. In the Name box, enter the resource name. Click Next.
  3. The Dependency screen appears.

     With nothing specified, click Next.
  4. The Recovery Operation screen appears.

     Click Next.
  5. The Details screen appears.

     In each of Data Partition Drive Letter and Cluster Partition Drive Letter, enter the drive letter of the partition created through "3. Configuring the instances".

• Google Cloud virtual IP resource

  With EXPRESSCLUSTER used on GCP, this resource provides a mechanism for awaiting access from the load balancer to a specific port for the alive monitoring--over the port, an application works on a node.

  For more information on the Google Cloud virtual IP resource, refer to the following:

  - "Reference Guide" -> "Understanding Google Cloud virtual IP resources"
  1. In Group Resource List, click Add.
  2. The Resource Definition of Group | failover1 screen appears.

     From the Type box, select Google Cloud Virtual IP resource as a group resource type. In the Name box, enter the resource name. Click Next.
  3. The Dependency screen appears. With nothing specified, click Next.
  4. The Recovery Operation screen appears. Click Next.
  5. In Port Number, enter the same value as specified in Health check (port) during the configuration of the load balancer (back end).
  6. Click Finish.

3. Adding monitor resources

• Google Cloud virtual IP monitor resource

  This monitor resource provides a mechanism for monitoring the alive-monitoring port, for the node where the Google Cloud virtual IP resource is started.

  Adding one Google Cloud virtual IP resource automatically creates one Google Cloud virtual IP monitor resource.

  For more information, refer to the following:

  - "Reference Guide" -> "Understanding Google Cloud virtual IP resources"

• Google Cloud load balance monitor resource

  This monitor resource provides a mechanism for checking whether the same port as that for the health check is opened, for a node where the Google Cloud virtual IP resource is not started.

  Adding one Google Cloud virtual IP resource automatically creates one Google Cloud load balance monitor resource.

  For more information, refer to the following:

  - "Reference Guide" -> "Understanding Google Cloud load balance monitor resources"

4. Applying the settings and starting the cluster

   Refer to the following:

   - "Installation and Configuration Guide" -> "How to create a cluster"

4.4. Verifying the created environment

Verify whether the created environment works properly, by producing a monitoring error for a failover of the failover group.

With the cluster running normally, the verification procedure is as follows:

1. On the active node (server1), start the failover group (failover1).

   In the Status tab of Cluster WebUI, make sure that the status of failover1 is Online on server1.

   From a client, access the front-end IP address to make sure of being able to connect to the active node.
2. In the pull-down menu of Cluster WebUI, change the mode option from Operation mode to Verification mode.
3. In the Status tab of Cluster WebUI, select the Enable dummy failure icon of gcvipw1.
4. The Google Cloud virtual IP resource (gcvip1) is reactivated three times. Then the failover group (failover1) fails, being failed over to the corresponding node (server2).

   In the Status tab of Cluster WebUI, make sure that the status of failover1 is Online on server2.

   Also make sure that, after the failover, the front-end IP address of the load balancer can be normally accessed.

That is all for testing the failover through a dummy failure. If necessary, perform operation checks for other failures as well.

5. Error Messages

For information on error messages of the resources/monitor resources, refer to the following:

• "Reference Guide" -> "Error messages"

6. Notes and Restrictions

6.1. HA cluster with a load balancer

6.1.1. Notes on GCP

• In designing a performance-oriented system, keep this in mind: GCP tends to increase its performance deterioration rate in multi-tenant cloud environments, compared with that in physical environments or general and virtualized (non-cloud) environments.

6.1.2. Notes on EXPRESSCLUSTER

• GCP's specification requires HTTP-protocol-based legacy health checks for external TCP network load balancing.

  The Google Cloud virtual IP resource, which supports only TCP-protocol-based health checks, cannot respond to health checks by an external TCP network load balancer.

  Therefore, use an internal TCP load balancer instead of an external TCP network load balancer, with which and the Google Cloud virtual IP resource the HA cluster cannot be used.

  Refer to the following:

  Health checks overview:

  https://cloud.google.com/load-balancing/docs/health-check-concepts/
• For an HA cluster configuration with an internal TCP load balancer, the HA cluster cannot be accessed from any client which belongs to a region different from that of the HA cluster--due to GCP's specification.

  Refer to the following:

  Internal TCP/UDP Load Balancing overview:

  https://cloud.google.com/load-balancing/docs/internal/#architecture
• Make the OS startup time longer than the time of Heartbeat Timeout.

  Refer to the following:

  - "Reference Guide" -> "Cluster properties" -> "Timeout tab"

  - "Getting Started Guide" -> "Notes and Restriction" -> "Adjusting OS startup time"

See also:

• "Getting Started Guide" -> "Notes and Restriction" -> "Communication port number"

• "Getting Started Guide" -> "Notes and Restriction" -> "Setting up Google Cloud virtual IP resources"

• "Getting Started Guide" -> "Notes and Restriction" -> "Setting up Google Cloud load balance monitor resources"

• "Reference Guide" -> "Notes on Google Cloud Virtual IP resources"

• "Reference Guide" -> "Notes on Google Cloud Virtual IP monitor resources"

• "Reference Guide" -> "Notes on Google Cloud load balance monitor resources"