1. Preface¶
This document "EXPRESSCLUSTER X for Windows SAP NetWeaver System Configuration Guide" describes how to create and start a cluster for SAP NetWeaver.
1.1. Who Should Use This Guide¶
1.2. How This Guide is organized¶
This Guide consists of the following two sections.
"EXPRESSCLUSTER X for Windows SAP NetWeaver System Configuration Guide"
"EXPRESSCLUSTER X for Windows SAP NetWeaver Configuration Example"
1.3. Conventions¶
In this guide, Note, Important, See also are used as follows:
Note
Used when the information given is important, but not related to the data loss and damage to the system and machine
Important
Used when the information given is necessary to avoid the data loss and damage to the system and machine.
See also
Used to describe the location of the information given at the reference destination.
The following conventions are used in this guide.
Convention
Usage
Example
Bold Indicates graphical objects, such as fields, list boxes, menu selections, buttons, labels, icons, etc. In User Name, type your name.On the File menu, click Open Database. Angled bracket within the command line Indicates that the value specified inside of the angled bracket can be omitted.clpstat -s [-h host_name]
Monospace Indicates path names, commands, system output (message, prompt, etc), directory, file names, functions and parameters.c:\Program files\EXPRESSCLUSTER
bold Indicates the value that a user actually enters from a command line. Enter the following:# clpcl -s -aitalic
Indicates that users should replace italicized part with values that they are actually working with.clpstat -s [-h host_name]
In the figures of this guide, this icon represents EXPRESSCLUSTER.
1.4. EXPRESSCLUSTER X Documentation Set (for Internal Version 11.3x/12.0x)¶
EXPRESSCLUSTER X Getting Started Guide
This guide is intended for all users. The guide covers topics such as product overview, system requirements, and known problems.
EXPRESSCLUSTER X Installation and Configuration Guide
This guide is intended for system engineers and administrators who want to build, operate, and maintain a cluster system. Instructions for designing, installing, and configuring a cluster system with EXPRESSCLUSTER are covered in this guide.
EXPRESSCLUSTER X Reference Guide
This guide is intended for system administrators. The guide covers topics such as how to operateEXPRESSCLUSTER, function of each module, maintenance-related information, and troubleshooting. The guide is supplement to the Installation and Configuration Guide.
EXPRESSCLUSTER X Integrated WebManager Administrator's Guide
This guide is intended for system administrators who manage cluster systems using EXPRESSCLUSTER with Integrated WebManager, and also intended for system engineers who introduce Integrated WebManager. This guide describes detailed issues necessary for introducing Integrated WebManager in the actual procedures.
1.5. EXPRESSCLUSTER X Documentation Set (for Internal Version 12.1xor later)¶
The EXPRESSCLUSTER X manuals consist of the following four guides. The title and purpose of each guide is described below:
This guide is intended for all users. The guide covers topics such as product overview, system requirements, and known problems.
Installation and Configuration Guide
This guide is intended for system engineers and administrators who want to build, operate, and maintain a cluster system. Instructions for designing, installing, and configuring a cluster system with EXPRESSCLUSTER are covered in this guide.
This guide is intended for system administrators. The guide covers topics such as how to operate EXPRESSCLUSTER, function of each module and troubleshooting. The guide is supplement to the Installation and Configuration Guide.
This guide is intended for administrators and for system administrators who want to build, operate, and maintain EXPRESSCLUSTER-based cluster systems. The guide describes maintenance-related topics for EXPRESSCLUSTER.
1.7. Terminology in this guide¶
Provides information of terminology used in this guide.
- This product
EXPRESSCLUSTER X for Windows SAP NetWeaver
- Configuration Guide
EXPRESSCLUSTER X for Windows SAP NetWeaver System Configuration Guide
- Configuration Example
EXPRESSCLUSTER X for Windows SAP NetWeaver Configuration Example
- Connecter for SAP
The connecter which links with SAP included in this product.
- SAP NW
SAP NetWeaver
- ASCS
ABAP SAP Central Services Instance
- ERS
Enqueue Replication Server
- PAS
Primary Application Server
- AAS
Additional Application Server
- DA
Diagnostics Agent
- Exclusive
Failover group for exclusive control of ASCS/ERS instance
- ENSA
Standalone Enqueue Server
- ENSA2
Standalone Enqueue Server 2
2. Overview of the cluster system¶
2.1. Overview of SAP NW Cluster¶
2.1.1. Functional Overview¶
A cluster with the following configuration can be built by combining SAP NW and EXPRESSCLUSTER.
2.1.1.1. SAP NW cluster configuration using EXPRESSCLUSTER¶
ABAP SAP Central Services Instance (hereafter, ASCS)
(With ENSA2 used) ERS
Set up the following components as failover groups for a single server configuration in which failover groups operate on each node.
(With ENSA used) ERS
Primary Application Server Instance (hereafter, PAS)
Additional Application Server Instance (hereafter, AAS)
Diagnostics Agent (hereafter, DA)
saphostexec
The diagram below shows the configuration with ENSA used.
The diagram below shows the configuration with ENSA2 used.
2.1.1.2. Dependency between failover groups¶
2.1.1.3. SAP NW monitoring using EXPRESSCLUSTER¶
In addition to the monitoring function provided by EXPRESSCLUSTER, the SAP NW cluster system uses a monitoring package that supports the SAP system and an SAP NW-specific monitoring command to monitor the SAP NW components for response errors and hang-ups.
2.1.1.4. Illustration of linkage between SAP NW and EXPRESSCLUSTER¶
User requests to SAP NW are sent to EXPRESSCLUSTER via the Connecter for SAP (clp_shi_connector). The EXPRESSCLUSTER cluster is operated by SAP NW.
2.1.1.5. Illustration of exclusive control of ASCS/ERS instance by EXPRESSCLUSTER¶
EXPRESSCLUSTER handles the exclusive control of the ASCS/ERS instance that is required for SAP NW as follows.
Exclusive in the figure below indicates a failover group for exclusive control.
Start both ASCS and ERS instances on different nodes. Start ERS instance on only one node. If ENSA is used, start the failover group for exclusive control on all nodes except the node which ERS instance starts.
EXPRESSCLUSTER handles failover process of ASCS instance as follows.
If ENSA is used, fail over the ASCS instance to the node where ERS instance was started before. If ENSA2 is used, fail over the ASCS instance to the node determined by the startup priority set in the failover group for ASCS.
If ENSA is used, ERS instance stops automatically by ASCS instance, after the failover of it is executed. If ENSA2 is used and the ERS instance has already been started on the failover target node of ASCS instance, the custom monitor resource of EXPRESSCLUSTER will execute the failover of ERS instance to another node.
- If ENSA is used, EXPRESSCLUSTER works as follows once the ERS instance got stopped.Start of the failover group for exclusive control on the node where ASCS instance was moved to.Start the ERS instance on the node that the ASCS instance is currently not working.Stop the failover group for exclusive control on the node where the ERS instance just started.
The above mechanism of exclusive control of ASCS/ERS instances by EXPRESSCLUSTER works similarly in the case of more than 3 nodes.
2.1.1.6. Note on manual operation of the ERS instance¶
The ERS instance is used for the replication of the lock table from the ASCS instance. To ensure its redundancy the ERS instance must work on the node where the ASCS instance is not running. The ERS instance should not even manually be launched on the node where the ASCS instance is running. Additionally the ERS instance should not be launched on more than two nodes at same time.
The failover group of the ERS instance is not restarted automatically, when the node where the ERS instance was working recovers from a failure. After validating the health of the node a manual restart of the ERS instance failover group is required.
2.1.1.7. ENSA2 (for Internal Version 12.10 or later)¶
Differences in failover groups for the ERS instance are as follows:
Item |
ENSA |
ENSA2 |
---|---|---|
Number of failover groups |
One for each node |
One for each cluster |
Active node in a failover group |
Only the nodes belonging to the failover group |
Possible to start for all nodes. However, set the startup priority in reverse order of the startup priority of the failover group for ASCS. |
Startup attribute |
Manual startup |
Automatic startup |
Resources composing the failover group |
The number in <> is the depth of dependency.
<1> EXEC resource for ERS instance service
<1> EXEC resource for ERS instance
|
The number in <> is the depth of dependency.
<0> Floating IP resource for ERS instance
<1> EXEC resource for ASCS startup check
<2> EXEC resource for ERS instance service
<2> EXEC resource for ERS instance
|
2.1.2. Operating Environment¶
This section describes the OS and SAP NW versions on which the operation of the Connector for SAP has been verified.
x86_64
NW Version / ABAP Platform
|
SAP Kernel Version
|
EXPRESSCLUSTER
Version
|
OS
|
Cluster Configuration
|
---|---|---|---|---|
7.4 |
742
745
|
11.32 or later
|
Microsoft Windows Server 2012
Microsoft Windows Server 2012 R2
|
SAN connection, shared disk type, mirror disk type
|
7.5 |
745
749
|
11.33 or later
|
Microsoft Windows Server 2012
Microsoft Windows Server 2012 R2
|
SAN connection, shared disk type, mirror disk type
|
745
749
|
11.35 or later |
Microsoft Windows Server 2016 |
SAN connection, shared disk type, mirror disk type |
NW Version / ABAP Platform
|
SAP Kernel Version
|
EXPRESSCLUSTER
Version
|
OS
|
Cluster Configuration
|
---|---|---|---|---|
7.4 |
745
749
753
|
12.01 or later
|
Microsoft Windows Server 2012
Microsoft Windows Server 2012 R2
Microsoft Windows Server 2016
|
SAN connection, shared disk type, mirror disk type
|
7.5 |
745
749
753
|
12.00 or later
|
Microsoft Windows Server 2012
Microsoft Windows Server 2012 R2
Microsoft Windows Server 2016
|
SAN connection, shared disk type, mirror disk type
|
7.52 |
753
|
12.10 or later
|
Microsoft Windows Server 2012
Microsoft Windows Server 2012 R2
Microsoft Windows Server 2016
|
SAN connection, shared disk type, mirror disk type
|
7.52 |
753
|
12.20 or later
|
Microsoft Windows Server 2012
Microsoft Windows Server 2012 R2
Microsoft Windows Server 2016
Microsoft Windows Server 2019
|
SAN connection, shared disk type, mirror disk type
|
1809 |
773
|
12.10 or later
|
Microsoft Windows Server 2016
|
SAN connection, shared disk type, mirror disk type
|
2020 |
781
|
12.30 or later
|
Microsoft Windows Server 2019
|
SAN connection, shared disk type, mirror disk type
|
NW Version / ABAP Platform
|
SAP Kernel Version
|
EXPRESSCLUSTER
Version
|
OS
|
Cluster Configuration
|
---|---|---|---|---|
2021 |
785
|
13.02 or later
|
Microsoft Windows Server 2022
|
SAN connection, shared disk type, mirror disk type
|
For the hardware and software requirements of SAP NW, see the SAP NW manuals.
2.1.3. Building Procedure¶
The flow of building the SAP NW cluster is shown below.
Installation and basic setup of Windows OS
Setup of disk and network
Installation of EXPRESSCLUSTER
Building of a cluster having CIFS resources and floating IP
Installation of SAP NW
Setup of EXPRESSCLUSTER
2.1.4. HA Database for SAP NW¶
Since SAP NW can run on several database technologies, e.g. SAP HANA, SAP MaxDB, IBM DB2, Oracle, Microsoft SQLSERVER, this guide assumes there is already a high available database setup in place. If you need help how create an HA setup for your database scenario please follow related EXPRESSCLUSTER documents on https://www.nec.com/en/global/prod/expresscluster/.
Throughout this document the HA database setup will be referred to as "database".
2.1.5. New features and improvements¶
The following features and improvements have been released.
EXPRESSCLUSTER Internal Version 11.3x
InternalVersion Contents 11.35 SAP Connector for SAP NetWeaver supports the following OS:- Windows Server 2016
EXPRESSCLUSTER Internal Version 12.0xor later
InternalVersion Contents12.01
The Connector for SAP now supports the following SAP NW:- SAP NetWeaver 7.412.10
The Connector for SAP for SAP NetWeaver supports the following SAP NetWeaver:- SAP NetWeaver Application Server for ABAP 7.5212.10
The Connector for SAP/the bundled scripts for SAP NetWeaver supports the following:- Maintenance mode- Standalone Enqueue Server 212.10
Improved the bundled scripts for SAP NW as follows:- Added parameters to clp_shi_connector.conf and some bundled scripts in order to facilitate the customization- Improved the quality of process alive monitoring of SAP Host Agent12.30
SAP Connector for SAP NetWeaver now supports the following SAP NetWeaver:- ABAP Platform 202013.00
Excluded compatible commands in the bundled scripts for SAP NW.
2.1.6. Corrected information¶
Modification has been performed on the following minor versions.
EXPRESSCLUSTER Internal Version 11.3x
Version in which the problem has been solved/ Version in which the problem occurred Phenomenon Level Occurrence condition/Occurrence frequency 11.35/11.32/11.33/11.34 A failover group for the ERS instance of SAP NetWeaver may not link with a failover group for exclusive control in the same node.* To deal with this problem, it is required to replace the script manually. The sample script can be obtained from the support portal (content ID: 9510100152). M Rarely occurs when starting up a failover group for ASCS instances.
EXPRESSCLUSTER Internal Version 12.0xor later
Version in which the problem has been solved/ Version in which the problem occurred Phenomenon Level Occurrence condition/Occurrence frequency 12.10/11.30, 12.00 When a failure is detected by the Custom monitor resource with the bundled scripts for SAP NetWeaver used, the SAP service is started while it is being stopped. S This problem occurs when stopping the SAP service takes time. 12.10/11.30, 12.00 In the SAP NetWeaver configuration, starting up the ASCS service fails on the failover destination node when the first failover is performed for the ASCS failover group. S This problem occurs when the first failover is performed for the ASCS failover group in the AWS environment. 12.30/12.10 to 12.22If the user starts/stops a SAP instance immediately after enabling the maintenance mode, the following occurs:
The maintenance mode fails to be disabled.
Inconsistency in the startup status occurs between the resource of EXPRESSCLUSTER for the SAP instance and the actual SAP instance.
S This problem occurs when the user starts/stops the SAP instance immediately after enabling the maintenance mode.
2.2. OS Installation and basic settings¶
For the software that needs to be set up and installed before installing SAP NW on Node#1 and Node#2 please refer to the installation guide of SAP NW:
For SAP NOTE about the operating systems, refer to the following:
Microsoft Windows Server 2012
#1730102: Release Restrictions for SAP NetWeaver 7.4
#1732161: SAP Systems on Windows Server 2012 (R2)
#0855498: Installation Prerequisite Checker
#2109662: Windows returns wrong IP address as source IP
Microsoft Windows Server 2016
#2384179: SAP Systems on Windows Server 2016
For examples of settings in this manual refer to "OS Setting Example" in the supplied "SAP NetWeaver Configuration Example ".
3. Installation of EXPRESSCLUSTER and SAP NW¶
3.1. Configuration of EXPRESSCLUSTER¶
Please refer to the "Installation and Configuration Guide" for additional information how to build an EXPRESSCLUSTER environment.
Create a cluster environment with two nodes in the order shown below.
Completely install EXPRESSCLUSTER, build a cluster with a CIFS resource and a floating IP, and start EXPRESSCLUSTER before installing SAP NW.
Preparations before installing SAP NW
Installation of EXPRESSCLUSTER
License Registration
Creating a cluster
Creating a cluster
Creating failover groups
Adding additional group resources
Specifying dependencies between failover groups
For details of the settings used in this manual please refer to "An example of setting EXPRESSCLUSTER" in the supplied "SAP NetWeaver Configuration Example".
After completing the above processes continue with "3.2. Setup of SAP NW Environment", and then "3.3. Setup of EXPRESSCLUSTER".
3.1.1. Installation of EXPRESSCLUSTER¶
Install EXPRESSCLUSTER on Node#1 and Node#2.
For details about the installation of EXPRESSCLUSTER please refer to the following document:
If an older version of this product is already installed, back up the bundled scripts and the configuration file of the Connecter for SAP.
> xcopy C:\Program Files\CLUSTERPRO\etc\clp_shi_connector.conf D:\backup > xcopy C:\Windows\System32\drivers\etc\services D:\backup > xcopy <Folder where sample script is expanded> D:\backup
After installing EXPRESSCLUSTER please install the Connector for SAP. Unzip the Connector for SAP media (clp_shi_connector.zip), and then copy the following files.
<Zip folder>\clp_shi_connector\bin\clp_shi_connector.exe -> C:\Program Files\EXPRESSCLUSTER\bin\clp_shi_connector.exe <Zip folder>\clp_shi_connector\bin\clp_shi_rsc.exe -> C:\Program Files\EXPRESSCLUSTER\bin\clp_shi_rsc.exe <Zip folder>\clp_shi_connector\etc\clp_shi_connector.conf -> C:\Program Files\EXPRESSCLUSTER\etc\clp_shi_connector.conf
3.1.2. License Registration¶
You must register the EXPRESSCLUSTER licenses to use EXPRESSCLUSTER.
For details about how to register the EXPRESSCLUSTER licenses please refer to the following document:
This product consists of the following three licenses.
License Product Name
EXPRESSCLUSTER X for Windows
EXPRESSCLUSTER X Database Agent for Windows
EXPRESSCLUSTER X System Resource Agent for Windows
3.1.3. Creating a cluster¶
Create a cluster from EXPRESSCLUSTER WebManager (hereafter referred to as "WebManager") (for Internal Version 11.3x/12.0x), Cluster WebUI (for Internal Version 12.1xor later).
For details about how to create a cluster please refer to the following document:
Set up the HBA to connect the shared disk by using WebManager or Builder (for Internal Version 11.3x/12.0x), Cluster WebUI (for Internal Version 12.1xor later).
For details, refer to the following document:
- "Server Properties" - "HBA tab"
3.1.4. Creating failover groups¶
Create the failover groups to which each node will belong by using WebManager (for Internal Version 11.3x/12.0x), Cluster WebUI (for Internal Version 12.1xor later).
For details about how to create a failover group please refer to the following document:
In this section, the following failover groups are created:
For the ASCS instance
For the ERS instance (for ENSA2 configuration)
For the ERS1 instance (for ENSA configuration)
For the ERS2 instance (for ENSA configuration)
For the PAS instance
For the AAS instance
For the DA1 instance
For the DA2 instance
For hostexec1
For hostexec2
For Exclusive1 (for ENSA configuration)
For Exclusive2 (for ENSA configuration)
For Node#1, specify " ERS1" for ERS, " DA1" for DA, " hostexec1" for saphostexec, and " Exclusive1" for Exclusive.For Node#2, specify " ERS2" for ERS, " DA2" for DA, " hostexec2" for saphostexec, and " Exclusive2" for Exclusive.
When ENSA2 is used, create only one failover group for the ERS instance.
3.1.4.1. Failover group for exclusively controls¶
When ENSA is used, it is required to create a failover group for exclusive control of ASCS and ERS instances described below.
The name of the failover group that exclusively controls must consist of the failover group name common to all nodes and a number as shown below.
The number following the name must be set in the order of the nodes to which the ERS1 and ERS2 instances have been installed.
<Common failover group name><Number>
Setting examples in this guide:
Exclusive-Group1 (Node#1)
Exclusive-Group2 (Node#2)
Note
The failover group name must not contain any spaces.
Note
If the name the failover group that exclusively controls the ASCS and ERS instances does not conform to the naming conventions, exclusive control cannot be normally performed.
For details about how to control the ASCS and ERS instances exclusively please refer to "2.1.1.5. Illustration of exclusive control of ASCS/ERS instance by EXPRESSCLUSTER".
3.1.4.2. Startup attribute of a failover group¶
Set the startup attribute depending on the configuration as follows:
failover group |
ENSA |
ENSA2 |
---|---|---|
failover group for ERS instance |
Manual Startup |
Auto Startup |
failover group for others |
Auto Startup |
Auto Startup |
3.1.5. Adding additional group resources¶
Add floating IP resources, disk resources and CIFS resources to the failover groups created in the previous section.
For details about how to add group resources please refer to the following document:
In this section, the following group resources are added to each failover group.
ASCS instance group |
- Add floating IP resources, and then assign the IP address that was set in "2.3.2. Network Settings".
- Add disk resources.
- Add CIFS resources.
|
For examples of the settings in this manual please refer to "An example of setting EXPRESSCLUSTER" in the supplied "SAP NetWeaver Configuration Example" document.
3.1.6. Specifying dependencies between failover groups¶
Set up the dependency among the failover groups.
The following shows the dependency (startup order) among SAP NW instances.
Database
->
ASCS
->
ERSPASAAS
Be sure to stop the instances in reverse order.
Note
As outlined in "2.1.4. HA Database for SAP NW" it is assumed there is a database available. This database is a prerequisite for the above dependencies and needs to be available initially. If this is not the case, then you cannot continue from here.
Note
It is not necessary to set up the dependency for DA and hostexec.
For details about how to set up the dependency among failover groups in EXPRESSCLUSTER please refer to the following document:
for Internal Version 11.3x/12.0x
"Reference Guide"- "Starting and stopping a group resource"- "Understanding the settings of dependency among group resources (common to group resources)"
for Internal Version 12.1xor later
3.2. Setup of SAP NW Environment¶
Terminology used in "3.2. Setup of SAP NW Environment", "3.3. Setup of EXPRESSCLUSTER", and "3.4. Connector for SAP".
- SID
SAP System ID
- DASID
Diagnostics Agent System ID
- INO
Instance Number
Master Guide
Installation Guide
The installation guide for each database and OS type supported by SAP NW can be downloaded from the following URL:Please make sure to review the appropriate guide for environment in detail.
Note
Before you start the following procedure please make sure to have the database installed according to SAP's documents and it is available.
Preparing the installation (3.2.1. Preparing the installation)
Creating Operating System Users and Groups (3.2.2. Creating Operating System Users and Groups)
Changing the ASCS instance host name registries (3.2.3. Changing the ASCS Instance Host Name Registries)
Installation of ASCS Instance (Node#1) (3.2.4. Installation of ASCS Instance (Node#1))
Adding sapstartsrv definition (Node#2) (3.2.5. Adding sapstartsrv definition (Node#2))
Installation of ERS Instances (Node#1 and Node#2) (3.2.6. Installation of ERS Instances (Node#1 and Node#2))
Installation of Database Instance (Node#1) (3.2.7. Installation of Database Instance (Node#1))
Installation of PAS Instance (Node#1) (3.2.8. Installation of PAS Instance (Node#1))
Installation of AAS Instance (Node#2) (3.2.9. Installation of AAS Instance (Node#2))
Installation of DA Instances (Node#1 and Node#2) (3.2.10. Installation of DA Instances (Node#1 and Node#2))
Addition of Symbolic Links (Node#1 and Node#2) (3.2.11. Addition of Symbolic Links (Node#1 and Node#2))
Registration of the SAP License registration (3.2.12. Registration of the SAP License)
Changing the SAP Instance Service Settings (3.2.13. Changing the SAP Instance Service Settings)
Disabling Auto Startup for the SAP Instances (3.2.14. Disabling Auto Startup for the SAP Instances)
Enabling Auto Stop for the ERS Instance (3.2.15. Enabling Auto Stop for the ERS Instance)
Changing start parameters for (3.2.16. Changing start parameters for ENSA)
"3.2.4. Installation of ASCS Instance (Node#1)" through "3.2.10. Installation of DA Instances (Node#1 and Node#2)" describe the procedure for installing SAP NW on Node#1 and Node#2.
"3.2.11. Addition of Symbolic Links (Node#1 and Node#2)" describes the procedure for adding symbolic links on Node#1 and Node#2.
"3.2.12. Registration of the SAP License" describes the procedure for registering the SAP license.
"3.2.13. Changing the SAP Instance Service Settings" describes the procedure for changing the SAP service settings.
"3.2.14. Disabling Auto Startup for the SAP Instances" describes the procedure for disabling Auto Startup for the SAP instances.
"3.2.15. Enabling Auto Stop for the ERS Instance" describes the procedure for enabling Auto Stop for the ERS instances.
"3.2.16. Changing start parameters for ENSA" describes the procedure for changing start parameters for SAP Enqueue Server.
For an example of setting up the instance name and instance number in this manual, refer to "An example of setting the SAP NW" in the supplied "SAP NetWeaver Configuration Example" document.
Note
For how to update SAP NW, refer to "4.1. SAP NW Update".
3.2.1. Preparing the installation¶
Before installing SAP NW be sure to start EXPRESSCLUSTER and activate the floating IP resources and CIFS resources on Node#1.
The location to save the SAP software logistics tool including the sapinst command described later depends on your environment and the installation media used (DVD-ROM or downloaded files). The sapinst command is a command used to install SAP NW.
3.2.2. Creating Operating System Users and Groups¶
Perform this work on Node#1 and Node#2.
Run sapinst.
> sapinst
In the Software Provisioning tool select Generic Installation Options > <your database> > Preparations > Operating System Users and Groups to create the OS users and groups.
In this guide NEC is used as SID and DBSID, and only ABAP is selected for Software.
SID |
DBSID |
Based On As |
---|---|---|
NEC |
NEC |
ABAP |
3.2.3. Changing the ASCS Instance Host Name Registries¶
Change the ASCS instance host name registries.
3.2.3.1. Change 1¶
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters DisableStrictNameChecking( <DWORD value> ) Value: 0x1Example:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\LanmanServer\Parameters] "DisableStrictNameChecking"=dword:00000001
3.2.3.2. Change 2¶
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0 BackConnectionHostNames( <multi-line string value> ) Value: <ASCS_Hostname (shortname)> <ASCS_Hostname (FQDN)>
Enter SAP Virtual instance Host associated with the floating IP of ASCS instance in shortname.
Enter SAP Virtual instance Host associated with the floating IP of ASCS instance in FQDN.
Enter the shortname and the FQDN on separate lines.
Example:
3.2.3.3. Change 3¶
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa DisableLoopbackCheck( <DWORD value> ) Value: 0x1Example:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa] "DisableLoopbackCheck "=dword:00000001
3.2.4. Installation of ASCS Instance (Node#1)¶
Perform this work on Node#1.
Run sapinst with specifying a host name associated with the floating IP of ASCS instance for the environment variable SAPINST_USE_HOSTNAME.
> sapinst SAPINST_USE_HOSTNAME=<ASCS_Hostname>
Note
Specify a host name associated with the floating IP of ASCS instance for <ASCS_Hostname>.
In the Software Provisioning tool select <SAP NW to be installed> > <your database> > SAP Systems > Application Server ABAP > Distributed System > ASCS Instance to install the ASCS.
The SID (SAP System ID) and INO (instance number) for the ASCS specified during installation are used in 6.1.1 (ASCS).
In this manual, SID and INO are set as follows:
Instance |
SID |
INO |
Instance name |
---|---|---|---|
ASCS |
NEC |
10 |
ASCS10 |
3.2.5. Adding sapstartsrv definition (Node#2)¶
Perform this work on Node#2.
Copy the
C:\Windows\System32\drivers\etc\services
file of Node#1 to Node#2.This file includes the following port number definitions according to the parameters set at installation.Both nodes of the cluster must share this file.: saphostctrl 1128/tcp # SAPHostControl over SOAP/HTTP saphostctrl 1128/udp # SAPHostControl over SOAP/HTTP saphostctrls 1129/tcp # SAPHostControl over SOAP/HTTPS saphostctrls 1129/udp # SAPHostControl over SOAP/HTTPS sapmsNEC 3610/tcp # SAP System Message Server Port sapdp00 3200/tcp # SAP System Dispatcher Port sapdp01 3201/tcp # SAP System Dispatcher Port sapdp02 3202/tcp # SAP System Dispatcher Port :
Add the ASCS10 sapstartsrv definition to Node#2.
Add the sapstartsrv service to Node#2 by referring to the service definition of Node#1.Use the following command to check the service definition.C:\>sc qc SAP<SID>_<INO> [SC] QueryServiceConfig SUCCESS SERVICE_NAME: SAP<SID>_<INO> TYPE : 10 WIN32_OWN_PROCESS START_TYPE : 3 DEMAND_START ERROR_CONTROL : 1 NORMAL BINARY_PATH_NAME : "S:\usr\sap\<SID>\ASCS<INO>\exe\sapstartsrv.exe" pf="\\<ASCS_Hostname>\sapmnt\<SID>\SYS\profile\<SID>_ASCS<INO>_<ASCS_Hostname>" LOAD_ORDER_GROUP : TAG : 0 DISPLAY_NAME : SAP<SID>_<INO> DEPENDENCIES : RPCSS : LanmanServer SERVICE_START_NAME : AD\SAPService<SID>
Note
By referring to the displayed service definition, run the following command on Node#2 to add the same service definition to Node#2.
C:\>sc create SAP<SID>_<INO> \ binpath= "\"S:\usr\sap\<SID>\ASCS<INO>\exe\sapstartsrv.exe\" \ pf=\"\\<ASCS_Hostname>\sapmnt\<SID>\SYS\profile\<SID>_ASCS<INO>_<ASCS_Hostname>\"" \ type= own start= demand depend= "RPCSS/LanmanServer" \ obj= AD\SAPService<SID> password= "********" [SC] CreateService SUCCESS
Note
3.2.6. Installation of ERS Instances (Node#1 and Node#2)¶
Perform this work on Node#1 and Node#2.
If ENSA is used, execute sapinst as follows:
> sapinst
In the Software Provisioning tool select <SAP NW to be installed> > <your database> > Additional SAP System Instances > Enqueue Replication Server Instance to install the ERS.
If ENSA2 is used, execute sapinst with the specification of the host name associated with the floating IP for the ERS instance to the environment variable SAPINST_USE_HOSTNAME:
> SET SAPINST_USE_HOSTNAME=ERS_Hostname > sapinst
Note
Enter the host name associated with the floating IP of ERS instance for ERS_Hostname.
When ENSA is used, the SID (SAP SID) and instance number for ERS specified along with the installation are used in "3.3.1.2. Setting up the ERS1 (Node#1) resource (for ENSA configuration)" (ERS1) and "3.3.1.3. Setting up the ERS2 (Node#2) resource (for ENSA configuration)" (ERS2). When ENSA2 is used, they are used in "3.3.1.4. Setting up the ERS resource (for ENSA2 configuration)".
In this manual, SID and INO are set as follows:
Instance |
SID |
INO |
Instance name |
---|---|---|---|
ERS |
NEC |
for ENSA configuration (Node#1): 21 |
ERS21 |
ERS |
NEC |
for ENSA configuration (Node#2): 21 |
ERS22 |
ERS |
NEC |
for ENSA2 configuration: 21 |
ERS21 |
3.2.7. Installation of Database Instance (Node#1)¶
Perform this work on Node#1.
> sapinst
In the Software Provisioning tool select <SAP NW to be installed> > <your database> > SAP Systems > Application Server ABAP > Distributed System > Database Instance to install the database instance.
The DBSID (Database ID) specified during installation is used in "Script resources" and "Custom monitor" in the supplied "SAP NetWeaver Configuration Example" document.
In this manual, DBSID is set as follows:
Data Base |
DBSID |
---|---|
SQL Server |
NEC |
3.2.8. Installation of PAS Instance (Node#1)¶
Perform this work on Node#1.
Run sapinst.
> sapinst
In the Software Provisioning tool select <SAP NW to be installed> > <your database> > SAP Systems > Application Server ABAP > Distributed System > Primary Application Server Instance to install the PAS.
The SID (SAP System ID) and INO (instance number) for PAS specified during installation are used in "3.3.1.4. Setting up the ERS resource (for ENSA2 configuration)".
The DASID (Diagnostics Agent System ID) and INO (instance number) for DA specified during installation are used in "3.3.1.7. Setting up the DA1 (Node#1) resource" (DA1).
In this manual, SIDs and INOs are set as follows:
Instance |
SID |
INO |
Instance name |
---|---|---|---|
PAS |
NEC |
31 |
DVEBMGS31 |
DA1 |
DAA |
97 |
SMDA97 |
Note
Depending on the version of SAP Software Provisioning Manager / SWPM the DA instance (Diagnostics Agent) may or may not be installed within the PAS instance installation. If it is not installed within this step, then please manually install as described later, otherwise skip the manual installation.
3.2.9. Installation of AAS Instance (Node#2)¶
Perform this work on Node#2.
Run sapinst.
> sapinst
In the Software Provisioning tool select <SAP NW to be installed> > <your database> > SAP Systems > Application Server ABAP > High-Availability System > Additional Application Server Instance to install the AAS.
The SID (SAP System ID) and INO (instance number) for AAS specified during installation are used in "3.3.1.6. Setting up the AAS resource".
The DASID (Diagnostics Agent System ID) and INO (instance number) for DA specified during installation are used in "3.3.1.8. Setting up the DA2 (Node#2) resource" (DA2).
In this manual, SIDs and INOs are set as follows:
Instance |
SID |
INO |
Instance name |
---|---|---|---|
AAS |
NEC |
32 |
D32 |
DA2 |
DAA |
96 |
SMDA96 |
Note
Depending on the version of SAP Software Provisioning Manager / SWPM the DA instance (Diagnostics Agent) may or may not be installed within the AAS instance installation. If it is not installed within this step, then please manually install as described later, otherwise skip the manual installation.
3.2.10. Installation of DA Instances (Node#1 and Node#2)¶
Perform this work on Node#1 and Node#2.
Note
Perform this step only if sapinst did not offer the installation of DA instances during the PAS / AAS installation as described in the previous two chapters. If the DA instances already got installed, then please skip this chapter and proceed to the next item.
Run sapinst.
> sapinst
In the Software Provisioning tool select <Diagnostics Agent to be installed> > Install Diagnostics Agent with 7.45 Kernel to install the DA.
In this manual, SIDs and INOs are set as follows:
Instance |
SID |
INO |
Instance name |
---|---|---|---|
DA1 |
DAA |
97 |
SMDA97 |
DA2 |
DAA |
96 |
SMDA96 |
3.2.11. Addition of Symbolic Links (Node#1 and Node#2)¶
Perform this work on Node#1 and Node#2.
A setting example in this manual is shown below.
> mklink /d C:usr\sap\NEC\SYS \\<ASCS_Hostname>\sapmnt\NEC\SYS > mklink /d C:usr\sap\trans \\<ASCS_Hostname>\sapmnt\trans
Note
Specify a host name associated with the floating IP of ASCS instance for <ASCS_Hostname>.
3.2.12. Registration of the SAP License¶
For details about how to register the SAP license, refer to the installation guide of SAP NW.
3.2.13. Changing the SAP Instance Service Settings¶
Perform this work on Node#1 and Node#2.
Change the startup type of SAP related processes to manual, because each process of SAP instances should be started by EXPRESSCLUSTER.
Select Start Button > Control Panel > Administrative Tools > Services.
If Startup Type is Automatic for the following instance services, open Properties, and then change Startup Type to Manual.
SAPHostControl
SAPHostExec
SAP<SID>_<INO>
SAP<DASID>_<INO>
3.2.14. Disabling Auto Startup for the SAP Instances¶
Perform this work on Node#1 and Node#2.
To disable auto startup by the SAP interface for the ERS and DA instances, change the relevant configuration in the profile of these instances.
The ERS instance profile is placed in the following location.
<shareddisk>:\usr\sap\<SID>\SYS\profile\<SID>_ERS<INO>_<hostname>
The DA instance profile is placed in the following location.
C:\usr\sap\<DASID>\SYS\profile\<DASID>_SMDA<INO>_<hostname>
Change the Autostart value in each profile to 0.
Autostart=0
3.2.15. Enabling Auto Stop for the ERS Instance¶
Perform this work on Node#1 and Node#2.
To enable auto-stop feature of the ERS instance when ASCS had a failover to the node where the ERS instance is working.
The ERS instance profile is placed in the following location.
<shareddisk>:\usr\sap\<SID>\SYS\profile\<SID>_ERS<INO>_<hostname>
Change the enque/enrep/poll_interval value in each profile to 0.
enque/enrep/poll_interval=0
Add the enque/enrep/hafunc_implementation = script value in each profile.
enque/enrep/hafunc_implementation = script
Comment out the line Restart_Program_03 = local $(_ER) pf=$(_PFL) NR=$(SCSID) in each profile.
Add Start_Program_03 = local $(_ER) pf=$(_PFL) NR=$(SCSID) in each profile.
#Restart_Program_03 = local $(_ER) pf=$(_PFL) NR=$(SCSID)
Start_Program_03 = local $(_ER) pf=$(_PFL) NR=$(SCSID)
3.2.16. Changing start parameters for ENSA¶
Perform this work on Node#1.
<shareddisk>:\usr\sap\<SID>\SYS\profile\<SID>_ASCS<INO>_<hostname>
#Restart_Program_03 = local $(_EN) pf=$(_PFL)
Start_Program_03 = local $(_EN) pf=$(_PFL)
3.3. Setup of EXPRESSCLUSTER¶
3.3.1. Setup of Resources¶
Add the script resource to the failover groups created in "3.1.4. Creating failover groups".
Set up the script resource to control starting and stopping of each instance.
Include the following string in the resource name:
instance_<SID>_<INO>The words in <> indicate the following items:
SID: SAP System ID
INO: Instance number
Note
The resource name must not contain any spaces.
Note
If the resource name does not conform to the naming conventions, starting and stopping of SAP NW instances cannot be normally controlled.
For setting examples in this manual, refer to "An example of setting EXPRESSCLUSTER" and "Script resources" in the supplied "SAP NetWeaver Configuration Example" document.
The sample script specified for the script resource is included in the installation media of this product.
Note
Modify the supplied sample scripts according to your environment.
For how to add the script resource, refer to the following document:
3.3.1.1. Setting up the ASCS resource¶
Add the following two script resources to the group for which the floating IP for ASCS is specified.
Add the script resource for controlling SAP instance services.
Add the script resource for starting SAP instances.
* Include the SID (SAP System ID) and INO (instance number) specified in "3.2.4. Installation of ASCS Instance (Node#1)" in the resource name.Example in this manual
script-ascs-SAP-instance_NEC_10
3.3.1.2. Setting up the ERS1 (Node#1) resource (for ENSA configuration)¶
Add the following two script resources for the group for ERS1.
Add the script resource for controlling SAP instance services.
Add the script resource for starting SAP instances.
* Include the SID (SAP System ID) and INO (instance number) specified in "3.2.6. Installation of ERS Instances (Node#1 and Node#2)" in the resource name.Example in this manual
script-ERS1-SAP-instance_NEC_21
3.3.1.3. Setting up the ERS2 (Node#2) resource (for ENSA configuration)¶
Add the following two script resources for the group for ERS2.
Add the script resource for controlling SAP instance services.
Add the script resource for starting SAP instances.* Include the SID (SAP System ID) and INO (instance number) specified in "3.2.6. Installation of ERS Instances (Node#1 and Node#2)" in the resource name.Example in this manual
script-ERS2-SAP-instance_NEC_22
3.3.1.4. Setting up the ERS resource (for ENSA2 configuration)¶
Add the following three script resources to the group for which the floating IP for ERS is specified.
Add the script resource to check the ASCS startup node
Add the script resource for controlling SAP instance services.
Add the script resource for starting SAP instances.* Include the SID (SAP System ID) and INO (instance number) specified in "3.2.6. Installation of ERS Instances (Node#1 and Node#2)" in the resource name.Example in this manual
script-ERS-SAP-instance_NEC_21
3.3.1.5. Setting up the PAS resource¶
Add the following two script resources for the group for ERS2.
Add the script resource for controlling SAP instance services.
Add the script resource for starting SAP instances.* Include the SID (SAP System ID) and INO (instance number) specified in "3.2.8. Installation of PAS Instance (Node#1)" in the resource name.Example in this manual
script-PAS-SAP-instance_NEC_31
3.3.1.6. Setting up the AAS resource¶
Add the following two script resources for the group for AAS.
Add the script resource for controlling SAP instance services.
Add the script resource for starting SAP instances.* Include the SID (SAP System ID) and INO (instance number) specified in "3.2.9. Installation of AAS Instance (Node#2)" in the resource name.Example in this manual
script-AAS-SAP-instance_NEC_32
3.3.1.7. Setting up the DA1 (Node#1) resource¶
Add the script resource for controlling SAP instance services.
Add the script resource for starting SAP instances.* Include the SID (SAP System ID) and INO (instance number) specified in "3.2.8. Installation of PAS Instance (Node#1)" in the resource name.Example in this manual
script-DA1-instance_DAA_97
3.3.1.8. Setting up the DA2 (Node#2) resource¶
Add the script resource for controlling SAP instance services.
Add the script resource for starting SAP instances.* Include the SID (SAP System ID) and INO (instance number) specified in "3.2.9. Installation of AAS Instance (Node#2)" in the resource name.Example in this manual
script-DA2-instance_DAA_96
3.3.1.9. Setting up the hostexec1 (Node#1) resource¶
Add the following script resource for the group for hostexec1.
Add the script resource for controlling saphostexec.
3.3.1.10. Setting up the hostexec2 (Node#2) resource¶
Add the following script resource for the group for hostexec2.
Add the script resource for controlling saphostexec.
3.3.1.11. Setting up the Exclusive1 (Node#1) resource (for ENSA configuration)¶
Add no group resources.
3.3.1.12. Setting up the Exclusive2 (Node#2) resource (for ENSA configuration)¶
Add no group resources.
For examples of setting up in this manual, refer to "An example of setting EXPRESSCLUSTER" and "Script resources" in the supplied "SAP NetWeaver Configuration Example" document.
Note
Specify a resource name that conforms to the naming conventions for the exec resource that controls starting and stopping of SAP NW instances. If the resource name does not conform to the naming conventions, starting and stopping of SAP NW instances cannot be normally controlled.
3.3.2. Setup of Monitor Resources¶
Add the following monitor resources to the group resources created in the previous section.
Custom monitor resource
Disk TUR monitor resource
CIFS monitor resource
For examples of setting in this guide please refer to "An example of setting EXPRESSCLUSTER" and "Custom monitor" in the supplied "SAP NetWeaver Configuration Example" document.
The sample script specified for the custom monitor resource is included in the installation media of this product.
Note
Modify the supplied scripts according to your environment.
3.3.2.1. Adding custom monitor resources¶
For how to add the custom monitor resources please refer to the following document:
3.3.2.1.1. Setting up the SAP NW instance monitor resources¶
ASCS
ERS1 (for ENSA configuration)
ERS2 (for ENSA configuration)
ERS (for ENSA2 configuration)
PAS
AAS
DA1
DA2
3.3.2.2. Setting up the SAP NW instance service monitor resources¶
ASCS
ERS1 (for ENSA configuration)
ERS2 (for ENSA configuration)
ERS (for ENSA2 configuration)
PAS
AAS
DA1
DA2
hostexec1
hostexec2
3.3.2.3. Adding disk TUR monitor resources¶
The disk TUR monitor resource is used to monitor the shared disk described in this manual.
For how to add the disk TUR monitor resources, refer to the following document:
This resource is automatically registered when a disk resource is added. The disk TUR monitor resource corresponding to each disk resource is automatically registered.
The disk TUR monitor resource has default values. If necessary change them to appropriate values.
Note
This resource cannot be used for a disk or disk interface (HBA) that does not support SCSI Test Unit Ready. Even if your hardware supports it, check the driver specifications because the driver may not support it.
3.3.2.4. Adding a CIFS monitor resource¶
Add a CIFS monitor resource for ASCS.
For how to add the CIFS monitor resources please refer to the following document:
Note
This monitor resource is automatically registered when a CIFS resource is added. The CIFS monitor resource corresponding to each CIFS resource is automatically registered.
Note
When access check is performed, the specified access method must be permitted for the local system account in the CIFS resource to be monitored.
3.4. Connector for SAP¶
3.4.1. Installation of Connector for SAP¶
Perform this work on Node#1 and Node#2.
The Connector for SAP and sample scripts is contained in the installation media.
<media>:\Windows\<Version of EXPRESSCLUSTER>\common\tools\x64\clp_shi_connector.zip
Extract the zip file in any folder. The file constitution of the zip file is shown below.
clp_shi_connector\
clp_shi_connector\
etc\
clp_shi_connector.conf
bin\
clp_shi_connector.exe
clp_shi_rsc.exe
sample\
clpbuilder-w\
scripts\
windows\
SAP-ASCS-instance\
ascs-post-handler.bat
setting.bat
start.bat
stop.bat
SAP-ERS-instance\
exclusive-control.bat
setting.bat
start.bat
stop.bat
SAP-HostExec\
setting.bat
start.bat
stop.bat
SAP-instance\
setting.bat
start.bat
stop.bat
SAP-service\
setting.bat
start.bat
stop.bat
genw\
check-ensa2.bat
ers-mon-preaction.bat
genw-SAP-HostExec.bat
genw-SAP-instance.bat
genw-SAP-service.bat
Copy the extracted Connector for SAP to the following locations in the EXPRESSCLUSTER installation folder.
C:\Program Files\CLUSTERPRO\bin\clp_shi_connector.exe
C:\Program Files\CLUSTERPRO\bin\clp_shi_rsc.exe
C:\Program Files\CLUSTERPRO\etc\clp_shi_connector.conf
3.4.2. Activation of Connector for SAP¶
Perform this work on Node#1 and Node#2.
Perform the following setup to use the Connector for SAP
3.4.2.1. Setting up the SAP profiles¶
Add the following specification to the default profile for SAP instances (DEFAULT.PFL) and the instance profile for each SAP instance to activate the SAP HA Connector and combine it with EXPRESSCLUSTER.
A setting example in this manual is shown below.
The path and a setting example of the default profile
<shareddisk>:\usr\sap\<SID>\SYS\profile\DEFAULT.PFLservice/halib_cluster_connector = C:\Program Files\CLUSTERPRO\bin\clp_shi_connector.exe
The path and a setting example of the instance profile of ASCS instance
<sharedisk>:\usr\sap\<SID>\SYS\profile\<SID>_ASCS<INO>\_\ <ASCS_Hostname>service/halib = <sharedisk>:\usr\sap\<SID>\ASCS<INO>\exe\saphascriptco.dllNote
Specify a host name associated with the floating IP of ASCS instance for <ASCS_Hostname>.
The path and a setting example of the instance profile of ERS instance
<sharedisk>:\usr\sap\<SID>\SYS\profile\<SID>_ERS<INO>_<hostname>
service/halib = C:\usr\sap\<SID>\ERS<INO>\exe\saphascriptco.dll
The path and a setting example of the instance profile of PAS instance
<sharedisk>:\usr\sap\<SID>\SYS\profile\<SID>_DVEBMGS<INO>_<hostname>
service/halib = C:\usr\sap\<SID>\DVEBMGS<INO>\exe\saphascriptco.dll
The path and a setting example of the instance profile of AAS instance
<sharedisk>:\usr\sap\<SID>\SYS\profile\<SID>_D<INO>_<hostname>
service/halib = C:\usr\sap\<SID>\D<INO>\exe\saphascriptco.dll
The path and a setting example of the instance profile of DA instance
c:\usr\sap\<DASID>\SYS\profile\<DASID>_SMDA<INO>_<hostname>
service/halib = c:\usr\sap\<DASID>\SYS\exe\uc\NTAMD64\saphascriptco.dll service/halib_cluster_connector = C:\Program Files\EXPRESSCLUSTER\bin\clp_shi_connector.exe
Note
The SAP instance services need to be restarted after the setting is changed. Restart the cluster for instance.
3.4.2.2. Granting administrator permissions to SAP NW users¶
To make SAP HA Connector executable, give full control permissions to the following registry to SAP NW users (SAP_<SID>_GlobalAdmin, SAP_<DASID>_LocalAdmin).
HKEY_LOCAL_MACHINE\SOFTWARE\NEC\EXPRESSCLUSTER
Note
To combine SAP NW and EXPRESSCLUSTER, give full control permissions to the specified registry to the group that was automatically created when SAP NW was installed. If full control permissions to the registry are not given to SAP NW users, starting and stopping of SAP NW instances cannot be normally controlled.
3.4.3. Configuration File¶
This section describes the configuration file for the Connector for SAP.
The configuration file for the Connector for SAP is placed in the following location. Edit this file using a text editor.
C:\Program Files\EXPRESSCLUSTER\etc\clp_shi_connector.conf
3.4.3.1. Setting items¶
For the Connector for SAP the following parameters in the configuration file can be changed to configure the log level, log size, and group resources for which to refuse the start/stop request from the SAP interface.
Note
Parameter name |
Setting value |
Description |
---|---|---|
LOGLEVEL |
ERROR
WARNING
INFO
TRACE
(default: INFO)
|
Specify the output log level.
ERROR:
Output logs of the error level and information level.
WARNING:
Output logs of the warning level and information level.
INFO:
Output logs of the information level.
TRACE:
Output logs of the internal trace and information level.
|
LOGSIZE |
1 to 2147483647
(default: 1000000)
|
Specify the log size in bytes.
If the size of the current log file exceeds the specified log size, log rotation is performed.
The number of log file generations is two (the current log and the log of the previous generation).
|
REFUSE_START_GROUP_RESOURCE |
Group resource name in EXPRESSCLUSTER
(default: None)
|
As the setting value (group resource name in EXPRESSCLUSTER), specify the name of the script resource that controls a SAP instance for which to refuse the start request from the SAP interface.
To set more than one group resource name, set more than one REFUSE_START_GROUP_RESOURCE parameter.
|
REFUSE_STOP_GROUP_RESOURCE |
Group resource name in EXPRESSCLUSTER
(default: None)
|
As the setting value (group resource name in EXPRESSCLUSTER), specify the name of the script resource that controls a SAP instance for which to refuse the stop request from the SAP interface.
To set more than one group resource name, set more than one REFUSE_STOP_GROUP_RESOURCE parameter.
|
GVI_CHECKCOUNT |
1 - 60
(default: 30)
|
The number of retries EXPRESSCLUSTER will try to obtain product information when the cluster is started. The interval between these attempts is set by GVI_CHECK_INTERVAL as stated below.
Even if the count does not reach to the setting, obtaining product information finishes when one attempt succeeded.
|
GVI_CHECKINTERVAL |
1 - 60
(default: 10)
|
The interval in seconds between EXPRESSCLUSTER attempts to obtain product information. If obtaining product information will be done only once (GVI_CHECKCOUNT=1), then this value will be ignored.
|
FRA_CHECKCOUNT |
1 - 60
(default: 30)
|
The number of retries to check the status of the group resource when the Rolling Kernel Switch is done. The interval between the check is set by FRA_CHECK_INTERVAL as stated below.
Even if the count does not reach to the setting, the status check finishes when one attempt succeeded.
|
FRA_CHECKINTERVAL |
1 - 60
(default: 10)
|
The interval in seconds between checks of the status of the group resource. If the status check will be done only once (FRA_CHECKCOUNT=1), then this value will be ignored.
|
SMM_PATH |
Folder to store the files which the Connector for SAP uses for the maintenance mode (default: none)
|
(Internal Version 12.1x or later can be specified)
Specify this when using maintenance mode.
Set the folder to store the files which the Connector for SAP uses for the maintenance mode. Specify the folder on which each cluster node is allowed to write. Under the specified folder, do not manually create files or folders. Only use up to 240 single-bite characters.
Example:
SMM_PATH=\\<host name for ASCS instance>\sapmnt\sapmm |
ENSA_VERSION |
1, 2
(Default: 1)
|
(Internal Version 12.1x or later can be specified)
Specify the version of ENSA to use.
Make sure that the version of ENSA matches the setting on the SAP NW side.
Set to 1 with ENSA used.
Set to 2 with ENSA2 used.
|
YELLOW_AS_ERROR
|
0, 1
(Default: 0)
|
(Internal Version 12.1x or later can be specified)
The bundled scripts allow you to check the statuses of processes composing each instance with sapcontrol -function GetProcessList. To determine whether the result is abnormal or not, you can choose from the following two patterns:
- Judge the result as abnormal when not all the process statuses are GREEN.
- Judge the result as abnormal when not all the process statuses are GREEN or YELLOW.
Specify either of the patterns with this parameter.
For details, see "Custom monitor" in "SAP NetWeaver Configuration Example".
If a different pattern is to be set for a specific process, customize the bundled script (e.g. for judging the result as normal when Process A is YELLOW, for judging as abnormal when Process B is YELLOW).
Set the value to 1 to judge the result as abnormal when not all the process statuses are GREEN. In this case, YELLOW leads to judging the result as abnormal.
Set the value to 0 to judge the result as normal even if any of the process statuses is YELLOW.
|
Note
Obtaining product information of EXPRESSCLUSTER when the cluster is started may fail due to a timing issue, as well as checking the group resource status when performing a Rolling Kernel Switch. In such a case adjust timing values and repetition count for GVI and FRA parameters.
A configuration example is shown below. This example sets the log level to INFO and the log size to 1000000 bytes.
LOGLEVEL=INFO
LOGSIZE=1000000
3.4.4. Log Files¶
This section describes the log file used for the Connector for SAP.
For information about the logs in EXPRESSCLUSTER, refer to the following document:
3.4.4.1. Log files¶
The Connector for SAP log is output to the following location:
C:\Program Files\EXPRESSCLUSTER\log\clp_shi_connector0.log C:\Program Files\EXPRESSCLUSTER\log\clp_shi_connector1.log
clp_shi_connector0.log is always the current log file; clp_shi_connector1.log is the log file of the previous generation.
3.4.4.2. Log format¶
The format of a log output from the Connector for SAP is shown below.
Log format
yyyy/MM/dd HH:mm:ss.SSS LEVEL[P:PID][T:THREADID] __FILE__:__LINE__ FUNCTION MESSAGE
3.4.4.3. Error messages¶
Error messages that the Connector for SAP outputs to the log file are described below.
Message |
Description |
Action |
---|---|---|
GetModuleFileName() failed. (%1) |
GetModuleFileName() failed. (%1: error code) |
Check whether EXPRESSCLUSTER is correctly installed. |
%1 does not exist. (%2) |
The configuration file is not found. (%1: file path, %2: error code) |
Check whether |
fopen() failed (errno=%1).
|
The file could not be opened. (%1: error code)
|
Check the following possible causes:
- SAPservice<SID> has no access permission for the configuration file.
- The file system is corrupted.
|
fclose() failed (errno=%1). |
The file could not be closed. (%1: error code) |
Check whether the file system is corrupted. |
Length of line exceeds maximum length. see line %1 |
The line is too long. (%1: line number) |
Check the indicated line in the configuration file. |
Value of LOGLEVEL is false. see line %1 |
The value for LOGLEVEL is invalid. (%1: line number) |
Check the indicated line in the configuration file. |
LOGSIZE is not in the setting range. see line %1 |
The value for LOGSIZE is out of the valid range. (%1: line number) |
Check the indicated line in the configuration file. |
Value of LOGSIZE is false. see line %1 |
The value for LOGSIZE is invalid. (%1: line number) |
Check the indicated line in the configuration file. |
The number of REFUSE_START_GROUP_RESOURCE exceeds the maximum number. see line %d |
The number of REFUSE_START_GROUP_RESOURCE parameters is too many. (%1: line number) |
Check the indicated line in the configuration file. |
The number of REFUSE_STOP_GROUP_RESOURCE exceeds the maximum number. see line %d |
The number of REFUSE_STOP_GROUP_RESOURCE parameters is too many. (%1: line number) |
Check the indicated line in the configuration file. |
Hyphen could not be used at beginning and end of group resource name. see line %d |
Hyphen "-" cannot be used at the beginning and end of the group resource name. (%1: line number) |
Check the indicated line in the configuration file. |
Length of group resource name exceeds the maximum length. see line %d |
The group resource name is too long. (%1: line number) |
Check the indicated line in the configuration file. |
Invalid character is used in group resource name. see line %d |
The group resource name contains an invalid character. (%1: line number) |
Check the indicated line in the configuration file. |
Failed to get module file path. |
The module path could not be acquired. |
Check whether EXPRESSCLUSTER is correctly installed. |
Failed to combine path. |
Paths could not be combined. |
Check whether EXPRESSCLUSTER is correctly installed. |
Invalid options.
|
Options for the Connector for SAP are invalid.
|
Correctly specify the option referring to the usage.
|
Module not found. |
The module is not found. |
Check whether EXPRESSCLUSTER is correctly installed. |
Cluster commands are not installed correctly. (%1) |
EXPRESSCLUSTER is not correctly installed. (%1: error code) |
Check whether EXPRESSCLUSTER is correctly installed. |
Error during detection of cluster status. (%1) |
An error has occurred during detection of cluster status. (%1: error code) |
Check whether EXPRESSCLUSTER is correctly installed, as well as whether the cluster configuration information is correct. |
Cluster commands are installed correctly but cluster framework is not running. (%1) |
EXPRESSCLUSTER is correctly installed, but the cluster is not running. (%1: error code) |
Start the cluster. |
Failed to initialize apicl. (%1) |
apicl could not be initialized. (%1: error code) |
Check whether EXPRESSCLUSTER is correctly installed. |
Failed to connect to the server. (%1) |
The server could not be connected. (%1: error code) |
Check whether the cluster configuration information is correct. |
Failed to get group resource status. (%1: %2) |
The group resource status could not be acquired. (%1: error code, %2: group resource name) |
Check whether the cluster configuration information is correct, as well as whether the group resource name conforms to the naming conventions. |
Group resource status is abnormal. (status: %1) |
The group resource status is abnormal. (%1: status) |
Check whether the cluster configuration information is correct. |
Failed to get server names. (%1) |
The server name could not be acquired. (%1: error code) |
Check whether the cluster configuration information is correct. |
Failed to get hostname. (%1) |
The host name could not be acquired. (%1: error code) |
Check whether the cluster configuration information is correct. |
Failed to get cluster group names. (%1) |
The group name could not be acquired. (%1: error code) |
Check whether the cluster configuration information is correct. |
Cluster group does not exist. |
The group is not found. |
Check whether the cluster configuration information is correct. |
Failed to get group resource names. (%1) |
The group resource name could not be acquired. (%1: error code) |
Check whether the cluster configuration information is correct, as well as whether the group resource name conforms to the naming conventions. |
Failed to get cluster group status. (%1: %2) |
The group status could not be acquired. (%1: error code, %2: group name) |
Check whether the cluster configuration information is correct. |
Failed to start group resource %1. (%2) |
The group resource could not be started. (%1: group resource name, %2: error code) |
Check the following possible causes: high system load, memory shortage, and OS resource shortage. |
Failed to start group resource %1 because group resource is not OFFLINE. |
Because the group resource was not offline, the group resource could not be started. (%1: group resource name) |
Check the status of the group resource. |
Failed to stop group resource %1. (%2) |
The group resource could not be stopped. (%1: group resource name, %2: error code) |
Check the following possible causes: high system load, memory shortage, and OS resource shortage. |
Failed to stop group resource %s because group resource is not ONLINE. |
Because the group resource was not online, the group resource could not be stopped. (%1: group resource name) |
Check the status of the group resource. |
Failed to create cluster resource name. (res: %1) |
The group resource name could not be generated. (%1: group resource name) |
Check whether the cluster configuration information is correct, as well as whether the group resource name conforms to the naming conventions. |
Failed to create cluster resource name. (SID: %1, INO: %2) |
The group resource name could not be generated. (%1: SAP System ID, %2: instance number) |
Check whether the cluster configuration information is correct, as well as whether the group resource name conforms to the naming conventions. |
Failed to get resource status. (%1: %2) |
The group resource status could not be acquired. (%1: error code, %2: group resource name) |
Check whether the cluster configuration information is correct, as well as whether the group resource name conforms to the naming conventions. |
Not found resource name. (name: %1) |
The group resource name is not found. (%1: group resource name) |
Check whether the cluster configuration information is correct, as well as whether the group resource name conforms to the naming conventions. |
Not found ONLINE server index. |
The online server index is not found. |
Check whether the cluster configuration information is correct. |
Resource %s is not ONLINE. |
The group resource is not online. (%1: group resource name) |
Check the status of the group resource. |
Failed to get all server status. (%1) |
The status of all servers could not be acquired. (%1: error code) |
Check whether the cluster configuration information is correct. |
Failed to open output file. (filename: %1) |
The output file could not be opened. (%1: output file) |
SAPservice<SID> has no access permission for the output file. |
The file system is corrupted. |
||
Failed to write file. (filename: %1, message; %2:%3:%4:) |
Writing to the file failed. (%1: output file, %2: group resource name, %3: group name, %4: current node) |
Check whether the disk is running out of space. |
Failed to write file. (filename: %1, message; <separate>) |
Writing to the file failed. (%1: output file) |
Check whether the disk is running out of space. |
Failed to write file. (filename: %1, message; %2) |
Writing to the file failed. (%1: output file, %2: active node) |
Check whether the disk is running out of space. |
Failed to write file. (filename: %1, message; %2:%3:%4:%5:) |
Writing to the file failed. (%1: output file, %2: SAP System ID, %3: instance number, %4: group resource name, %5: group name) |
Check whether the disk is running out of space. |
Failed to open log file. (%1, %2)
|
The log file could not be opened. (%1: error code, %2: log file)
|
Check the following possible causes:
- SAPservice<SID> has no access permission for the log file.
- The file system is corrupted.
|
Failed to set the file pointer to the end of log file. (%1) |
The file pointer could not be moved to the end of file. (%1: error code) |
The file system may be corrupted. Check it. |
Failed to rotate logfile.
|
Log rotation could not be performed.
|
Check the following possible causes:
- SAPservice<SID> has no access permission for the log file.
- The file system is corrupted.
|
Failed to get log file size. (%1)
|
The log file size could not be acquired. (%1: error code)
|
Check the following possible causes:
- SAPservice<SID> has no access permission for the log file.
- The file system is corrupted.
|
Failed to delete old log file. (%1)
|
The old log file could not be deleted. (%1: error code)
|
Check the following possible causes:
- SAPservice<SID> has no access permission for the log file.
- The file system is corrupted.
- The old log file is locked.
|
Failed to rename log file. (%1)
|
The log file could not be renamed. (%1: error code)
|
Check the following possible causes:
- SAPservice<SID> has no access permission for the log file.
- The file system is corrupted.
- The old log file exists.
- The current log file is locked.
|
SMM_PATH is not defined.
|
(For Internal Version 12.1x or later)
Parameter SMM_PATH and the value for it have not been defined.
|
Set the value to "SMM_PATH", which is the parameter of clp_shi_connector.conf.
|
Failed to delete lock file. (filename: %1)
|
(For Internal Version 12.1x or later)
The files under the directory specified by the parameter SMM_PATH could not be deleted (%1:file name).
|
Check if the value for the parameter "SMM_PATH" in clp_shi_connector.conf is correct.
|
Failed to resume cluster. (SID: ${sid}, INO: ${ino}).
|
(For Internal Version 12.1x or later)
Cluster could not be resumed.
|
Check the cluster status.
|
Failed to create lock file. (filename: %1)
|
(For Internal Version 12.1x or later)
The files under the directory specified by the parameter SMM_PATH could not be created (%1:file name)
|
Check if the value for the parameter "SMM_PATH" in clp_shi_connector.conf is correct.
|
Failed to suspend cluster. (SID: ${sid}, INO: ${ino})
|
(For Internal Version 12.1x or later)
Cluster could not be suspended.
|
Check the cluster status.
|
4. Miscellaneous¶
4.1. SAP NW Update¶
To update SAP NW, use Software Update Manager (hereafter referred to as SUM). The update procedure with SUM involves restarting SAP instances and therefore it may interfere with EXPRESSCLUSTER which tries to keep the SAP components available. To avoid such interference with EXPRESSCLUSTER, suspend EXPRESSCLUSTER's monitoring for all SAP components that SUM has to restart.
Please select from the following two options to suspend EXPRESSCLUSTER's monitoring.
Suspending the whole cluster
Suspending monitor resources related to SAP instances and instance services
Update SAP NW with SUM while the cluster or the monitor resources are suspended. After the update is completed, resume the suspended cluster or the suspended monitor resources.
For how to suspend and resume a cluster or a monitor resource, please refer to the following document.
for Internal Version 11.3x/12.0x
"Reference Guide"- "Chapter 1 Functions of the WebManager"- "Operations from the WebManager"
for Internal Version 12.1xor later
Cluster WebUI online manual
4.1.1. Maintenance mode (for Internal Version 12.1x or later)¶
The maintenance mode can be switched on/off by the sapcontrol command or from the SAP management console. For details on the maintenance mode and the sapcontrol command, see the SAP documents.
The following is an example to enable the maintenance mode with the sapcontrol command:
sapcontrol -nr <SID> -function HASetMaintenanceMode 1
The following is an example to disable the maintenance mode with the sapcontrol command:
sapcontrol -nr <SID> -function HASetMaintenanceMode 0
In order to use the maintenance mode, add the account SAPService<SID> to the Administrators group of each cluster node.
4.2. Notes and Restrictions¶
- Notes on starting/stopping groups (for Internal Version 11.3x/12.0x)Refer to the following sections in the "Reference Guide":"Group resource details"-> "Group start dependence and group stop dependence"-> "Notes"
Notes on starting/stopping groups (for Internal Version 12.1xor later)
Refer to the following sections in the "Reference Guide":-> "Notes"- Restriction of using spacesA node name , a failover group and a resource name must not contain any spaces. If they contain some spaces, starting and stopping of SAP NW instances cannot be controlled correctly.
- Naming conventions for failover groupsSpecify a failover group name according to the naming conventions for the failover group for exclusive control of ASCS/ERS instance. If the failover group name does not follow the naming conventions, exclusive control of ASCS/ERS instance cannot function correctly.
- Naming conventions for script resourcesSpecify a resource name according to the naming conventions for the exec resource that controls starting and stopping of SAP NW instances. If the resource name does not follow the naming conventions, starting and stopping of SAP NW instances cannot be normally controlled.
- SAP NW instance numberA SAP NW instance number must be unique across the cluster nodes. If some SAP NW instances have duplicate numbers, starting and stopping of the SAP NW instances cannot be controlled correctly.If an instance number is duplicated inside a node or between nodes, reinstall a SAP NW component and reassign an instance number in either of the nodes.
- Note on manual operation of ERS instanceThe ERS instance replicates the lock table from the ASCS instance. The ERS instance must work on the node where the ASCS instance is not performing, to ensure its redundancy.ERS instance should not be launched on the node where ASCS instance is performing even if it is manual operation.Additionally the ERS instance should not be launched on more than two nodes at same time.
- Attention when one node recoversWhen the node where ERS instance was working gets recovered and joins the cluster, then the failover group of the ERS instance is not restarted automatically.You need to validate the node is working healthy and then restart the failover group of ERS instance manually.When the node where the ASCS instance was previously running on crashed and recovers, it may display that it is still sharing the <shareddisk>:\sapmnt folder. However this share is locally inaccessible since it is running on the node, which is currently running the ASCS failover group. In this case it is necessary to manually delete this inaccessible share on the recovered node, otherwise a failback of the ASCS group to this node will fail.
- Restriction of using virtual computer name resourceA virtual computer name resource must not be used with this product. If a virtual computer name resource is used, starting and stopping of SAP NW instances cannot be controlled correctly.
When the disk resource for ASCS is failed over or stopped, the following warning message may be output to WebManager (for Internal Version 11.3x/12.0x), Cluster WebUI (for Internal Version 12.1xor later). Ignore this message.
Type: Warning Module name: sdfunc Event ID: 3201 Message: Disconnection of disk %1 is being retried. The disk may be in use. Check the disk.
The mirror disk or shared disk belonging to the failover group for ASCS also stops, if the failover group for ASCS is stopped while PAS instance or AAS instance is running (e.g. manual failover). As a result, startup of ASCS instance service or ASCS instance on a failover target server may fail.
When creating a cluster on a cloud environment such as AWS and Microsoft Azure, use the AWS virtual ip resources and Azure DNS resources instead of the Floating IP resources. Note that name resolution must be possible for host names associated with virtual IPs for ASCS instances by the AWS virtual ip resource.
- When a shared folder is published by CIFS resource in AWS environment, startup of ASCS instance service on the failover target node may fail.This is due to the fact that a process which accesses <\\Virtual Hostname\Shared Name> during the time period for failover exists and that an access attempt to file share fails when newly connecting to <\\Virtual Hostname\Shared Name> after failover.This phenomenon occurs due to the specification of Windows OS.In this case, it may be avoided by either of the configuration changes in CIFS resources as listed below:
Turn off the "Cache Enable" in CIFS resource
- Change the Cache Setting into "Automatic cache" in CIFS resource
If either of the configuration changes above does not work, customize the bundled script as described below:
- Insert timeout before sapcontrol -function StartService in the bundled script (
SAP-Service\start.bat
) for starting ASCS instance service, so that it waits for the shared folder to be referable (already included in the bundled script for 12.1xor later).
- Maintenance mode (for Internal Version 12.1x or later)When the maintenance mode is used, avoid suspending or resuming the cluster from Cluster WebUI or with the clpcl command in order to avoid conflicts.For the "SMM_PATH" parameter in clp_shi_connector.conf, specify the directory on which each cluster node is allowed to write. Under the specified directory, do not manually create files or directories.
- The minimum SAP kernel patch levelYou must use the following SAP kernel patches level which include fixes for known issues.For details of SAP kernel patches please refer to SAP Note 1693245.
SAP Kernel Version 749: patch level 800 or later
SAP Kernel Version 753: patch level 515 or later
SAP Kernel Version 773: patch level 225 or later
SAP Kernel Version 777: patch level 32 or later
SAP Kernel Version 780: patch level 0 or later
- When a floating IP resource and AWS virtual ip resource adds a floating IP address and a virtual ip address to NIC by using a Windows OS API, the skipassource flag is not set and therefore does not take effect after activating the above resources.Use applications such as PowerShell to set the skipassource flag after activating the above resources.