SAP Host Agent Troubleshooting Guide

Changing the default trace level of SAP Host Agent

SAP Host Agent runs with tracelevel 1 by default. If you need more information to be able to analyze an issue, you can add or change the following profile values:

hostexec/trace
service/trace

Proceed as follows:

1. Open the SAP Host Agent profile ( host_profile ) which is located in the exe directory of the SAP Host Agent (/usr/sap/hostctrl/exe/ or C:\Program Files\SAP\hostctrl\exe\).
2. Add or modify the required values, for example:
   hostexec/trace = 3  
   service/trace = 3
3. Restart SAP Host Agent by executing saphostexec -restart (as a user with root authorization ) or hostexecstart -restart (as user <sapsid>adm)

All SAP Host Agent log files in /usr/sap/hostctrl/work/ (UNIX) or C:\Program Files\SAP\hostctrl\work\ (Windows) now contain much more information.

Start of SAPHostExec Service / saphostexec executable failed. dev_saphostexec log file contains error message "ERROR => Setup of too many communication channels failed"

Reason:

The sapstartsrv process of the Host Agent is not able to start.

Solution(UNIX):

Proceed as follows:

1. Stop the Host Agent with saphostexec -stop .
2. Check that there are no running processes which belong to SAP Host Agent, and kill any remaining Host Agent's sapstartsrv processes.
3. Delete the /tmp/.sapstartsrv99_sapstartsrv.log and content of the Host Agent's work directory.
4. Restart SAP Host Agent by executing saphostexec -restart (as a user with root authorization) or hostexecstart -restart (as as user <sapsid>adm
5. Check the system log for the error messages from the sapstartsrv process of the Host Agent

Note:
Probably, there are multiple sapstartsrv processes running: One from the SAP Host Agent and one from each SAP instance. Just ensure that there is no SAP Host Agent sapstartsrvprocess running. You can identify it by the start parameter, which is pf=/usr/sap/hostctrl/exe/host_profile and by the assigned user, which is sapadm.

Other probable reasons for this error are:

1. /usr/sap/hostctrl/work and/or /usr/sap/hostctrl/work/sapstartsrv.* is not writable for the sapadm user
2. /tmp directory is not writable for the sapadm user or filesystem has no free space left.

The sapstartsrv error message in the system log should give you a hint why it cannot be started.

Solution(Windows):

Check the Application Log in the Windows Event Viewer for the error messages from the SAPHostControl service

Oracle databases cannot be detected on Unix

Symptom:

• Web service method ListDatabases does not return Oracle databases or not all Oracle databases
• Other database Web service methods (for example GetDatabaseStatus) return the error message "Database not found" for Oracle databases.
• With default trace level dev_sapdbctrl contains messages like
  [PID 23258] *** ERROR => Database neither found in '/var/opt/oracle/oratab' nor in inventory and environment of 'ora<sid>' user. Giving up... [dboractrlux.cpp 611]
• With trace level 3 (host_profile: service/trace = 3) the trace file /usr/sap/hostctrl/work/dev_sapdbctrl contains messages like the following:

[PID 10682408] Looking up databases in inventory
[PID 10682408] Retrieving inventory location from '/etc/oraInst.loc'
[PID 10682408] Retrieving inventory data from '/oracle/oraInventory/ContentsXML/inventory.xml'
[PID 10682408] *** ERROR => Failed to open file '/oracle/oraInventory/ContentsXML/inventory.xml': No such file or directory 
[PID 10682408] No installations registered in inventory

or like the following:

[PID 10682408] Looking up databases in inventory
[PID 10682408] Retrieving inventory location from '/etc/oraInst.loc'
[PID 10682408] Retrieving inventory data from '/oracle/oraInventory/ContentsXML/inventory.xml'
[PID 10682408] No installations registered in inventory

Solution:

To correctly detect Non-RAC Oracle databases:

1. The database instance must be maintained in /etc/oratab (Solaris: /var/opt/oracle/oratab)
2. or a central Oracle inventory must exists and the ORACLE_HOMEs of all databases must be registered. Also the central inventory pointer file must exist and contain the correct path to the central inventory.

And for RAC databases:

1. SAP Host Agent 7.21 PL 14 must be installed
2. The GRID/ASM instance must be maintained in /etc/oratab (Solaris: /var/opt/oracle/oratab). E.g.
   +ASM1:/u01/app/12.1.0.2/grid:N
3. or a central Oracle inventory must exists and the ORACLE_HOMEs of at least the GRID/ASM instance must be registered. Also the central inventory pointer file must exist and contain the correct path to the central inventory.

Normally, all this is created by the Oracle/SAP installation tools.

Please note, from Oracle 11.2 onwards the so called runtime home must be maintained in the oratab file. Refer to Wrong Oracle database status on Unix for more details on that.

For details on how to check the central inventory pointer file you can refer to e.g. http://docs.oracle.com/cd/E11882_01/em.112/e12255/oui2_manage_oracle_homes.htm#CJAEHIGJ. For details on how to check whether the central inventory exists and which ORACLE_HOMEs it contains you can refer to e.g. http://docs.oracle.com/cd/E11882_01/em.112/e12255/oui2_manage_oracle_homes.htm#CHDEHFBJ.

If the central inventory does not exist or it doesn't contain all ORACLE_HOMEs, you can create or adapt it with the Oracle runInstaller tool. For example, you can do this by executing the following command:

runInstaller -attachHome ORACLE_HOME="<Oracle_Home_Location>" ORACLE_HOME_NAME="<Oracle_Home_Name>"

For more information, refer to http://docs.oracle.com/cd/E11882_01/em.112/e12255/oui2_manage_oracle_homes.htm.

Wrong Oracle database status on Unix

Although the database is started and running fine it will be reported as stopped. E.g.:

# /usr/sap/hostctrl/exe/saphostctrl -function GetDatabaseStatus -dbname C11 -dbtype ora

Invalid databases status Database Status: Stopped

Component name: Instance (Instance), Status: Stopped (Instance is stopped)

Component name: Database (Database), Status: Stopped (Database is stopped)

Component name: Archiver (Archiver), Status: Stopped (Archiver is stopped)

Component name: Listener (Listener), Status: Running (Listener is started)

Please ensure that the correct Oracle home directory is maintained in /etc/oratab (Solaris: /var/opt/oracle/oratab).

From Oracle 11.2 onwards the so called runtime home must be maintained in the oratab file. E.g.:

C11:/oracle/C11/112_64

You can refer to Note 1524205 for details on the runtime home.

Issues with Oracle 12c databases on Unix

For Oracle 12c the ora<dbsid> user is still needed. Please refer to Note 1915323 for more details on this. In case the user was removed it can be recreated as described in Note 220421.

At least Host Agent 7.21 PL6 is needed for general Oracle 12c support. At least 7.21 PL16 is needed for Oracle 12c Multitenant databases.

SAP Solution Manager Diagnostics (SMD) Agent is not able to connect to SAP Host Agent

Symptom:

The following issues may arise while the SMD agent is connecting to SAP Host Agent for the first time:

Trusted connect is not working

We assume the <smd>adm user is used for the installation of the SMD agent installation, where <smd> is the system ID of the SMD agent ("DAA" by default).

Solution:
Proceed as follows to check if SAP Host Agent is working correctly:

1. Log on on the respective server as <smd>adm.
2. Execute the following command (Note: Instance number is always 99. This is not an example!) :
   on UNIX:
   
   /usr/sap/hostctrl/exe/sapcontrol -nr 99 -user "" "" -function ConfigureLogFileList add /tmp
   
   on MS Windows (replace the <smdadm> and <password for smdadm> with your user and password):
   
   c:\usr\sap\DAA\SMDA97\exe\sapcontrol.exe -user <smdadm> <password for smdadm> -nr 97 -function OSExecute "\"c:\Program Files\SAP\hostctrl\exe\sapcontrol.exe \" -user \"\" \"\" -nr 99 -trace - -function ConfigureLogFileList remove testLogFile.txt" 0 0 c:\temp\tst_out.txt

If the output of the command prompt is as follows, the trusted connect works:

12.12.2012 11:25:27
RequestLogonFile
OK

12.12.2012 11:25:27
ConfigureLogFileList
OK

This generally means that Trusted Connect has been configured correctly on the part of SAP Host Agent.
However, the SMD Agent call might fail. In those cases you should proceed as follows:

Check if ACLs are used. In the host_profile file you can find a configured entry: service/http/acl_file=<filepath>
If yes, check in the ACL files if all the relative IP Addresses of the local machine are configured correctly.

Alternatively you can proceed as follows:

1. In the host_profile, specify service/trace=3
2. Restart Host Agent with the following command: saphostexec -restart
3. Trigger the SMD scenario
4. Check if entries like NiAcl .... denied are contained in the sapstartsrv.log file. If yes, the ACL configuration is wrong and this is the root cause of that issue.

If the sapcontrol call does not return with OK, you have the following options to proceed:

The output is: "FAIL: Permission denied"

The output of the command prompt is as follows:

ConfigureLogFileList
FAIL: Permission denied

Solution:
Proceed as follows:

1. Check if the <smd>adm user is specified in the host_profile in line service/admin_users .
2. If <smd>adm user is not specified there, add it.
3. Restart the SAP Host Agent.

Remark: On Windows (in Unix by default) the service/admin_users in the host_profile are case sensitive and need to be defined exactly as the user running the service. For example service running as <SMD>ADM\SAPServiceDAA cannot be defined as <smd>adm\SAPServiceDAA in the host_profile. The upper and the lower case letters must match.

The output is: "HTTP/1.1 401 Unauthorized"

The output of the command prompt is as follows:

ConfigureLogFileList
FAIL: HTTP error, HTTP/1.1 401 Unauthorized

Analysis:

User <smd>adm seems not to be able to read the logon file. The cause might be that file system permissions or ownership are wrong.

Solution:

Ensure that the file permission and ownership are as follows:

# ls -lda /usr/sap/hostctrl/work/sapcontrol_logon
drwx--x--x 2 sapadm sapsys 4096 28. Apr 06:33 /usr/sap/hostctrl/work/sapcontrol_logon

Ensure that the <smd>adm user is member of the sapsys group and the user is able to access files in /usr/sap/hostctrl/work/sapcontrol_logon.

Error message "Remote access not permitted" in SMD agent logs

Ensure that you have at least Host Agent 7.20 patch level 179 installed.

Autoupgrade from shared directory not working on Windows

For windows, the Autoupgrade has certain prerequisites:

1. The Host Agent system must have access to the share. The SAPHostExec service is doing the autoupgrade and this service is running as Local System Account. ( One can read more about the Local System Account on http://msdn.microsoft.com/en-us/library/windows/desktop/ms684190 ) Hence, the <System>$ account must have access to the shared directory.
2. Autoupgrade is working with UNC paths only. "Mounted" directories (e.g. as Drive X:) do not work.

If the Host Agent System and the System with the shared directory are located in different windows domains, there must be a one way trust to access the share.That means, the domain of system A (the system with the shared directory) has to trust the domain of system B (the system with the Host Agent).

See http://msdn.microsoft.com/en-us/library/cc237016%28v=prot.10%29.aspx for more information on how to setup a domain trust.

Unable to detect Microsoft IIS applications

Symptom:

The command to retrieve list of IIS web applications fails with error "Invalid namespace":

"c:\Program Files\SAP\hostctrl\exe\saphostctrl.exe" -function GetCIMObject -namespace WebAdministration -enuminstances Application -user sapadm <password>

Error: CIM Request EnumerateInstances for classes Application failed: EnumerateInstances for Application failed: Invalid namespace

Solution:

Please ensure that the "IIS Management Scripts and Tools" are installed. For example, with Windows 2008 this can be checked via Server Manager -> Roles -> Web Server -> Add Role Services.

Username/Password-based authentication fails on Unix

Symptom:

When a Web service method of the Host Agent is called or a Web-UI is accessed via the Host Agent (e.g. SUM) Username/Password-based authentication fails although the correct credentials were supplied.

Solution:

Ensure that the Host Agent is correctly installed/updated as described in Note 1031096. A common error is that the host agent files are manually copied into /usr/sap/hostctrl/exe or the host agent archive is just directly extracted into /usr/sap/hostctrl/exe. This results in invalid file ownership and permission, which in turn breaks various functionality like e.g. the Username/Password-based authentication.

To fix this install the Host Agent as described in Note 1031096.

Host Agent fails to load clean root user environment on startup or installation/upgrade

When starting or upgrading the Host Agent you see the following messages:

Executing "'/bin/su' '-' '-c' 'echo SAPHOSTAGENT-START;env;echo SAPHOSTAGENT-END' " *** WARNING => Timeout, helper didn't answered within 30 sec... *** ERROR => Unable to load clean root environement.

Starting with 7.21 PL8 the Host Agent is trying to load a clean root user environment during startup. This is done by running a (non-interactive) root user login shell. This step hangs and is therefore interrupted after some timeout.

Typical reasons for that are some prompts issued by shell login scripts requesting manual user input. You can check this by e.g. running the following command:

/bin/su - root -c env </dev/null

Please ensure that these non-interactive root user login shells don't hang/require any user input.

E.g. in the shell login scripts you can use the 'tty -s' command to check whether the shell is interactive and only in this case execute commands that require user input.

SAP Diagnostics Agent Maintenance Procedures

Diagnostics Agent Installations

Finding Previously Installed Diagnostics Agents

To double check, if a Diagnostics Agent is installed you should have a look at all existing / installed SAP systems that are operational at (logical) host level.

The Diagnostics Agents are in general identifiable by a System ID, as follows:

• For agents shipped with / for SAP NetWeaver 7.1 EhP1 and higher, the Diagnostics Agent System ID is usually: DA? (DAA, DAB, etc.)
• For agents based on older versions, the Diagnostics Agent System ID is usually: SMD

Diagnostics Agent Installer

Use the latest SWPM 1.0 Support Package (and associated Patch Level), when installing a new or an additional Diagnostics Agent. See SAP note 1833501 for the latest Diagnostics Agent Installation & Setup Guide.

SAP NetWeaver System Rename

SWPM based System Rename is not supported for the Diagnostics Agent. Instead, the below described Diagnostics Agent Replacement procedure must be used.

SAP NetWeaver System Copy

SWPM based System Copy is not supported and not relevant for the Diagnostics Agent. Instead, perform a fresh Diagnostics Agent installation on the target system.

Have Up-To-Date SAP Host Agents

Checking the SAP Host Agent Version

• Solution Manager 7.1 currently requires having at least SAP Host Agents 7.20 SP147 in the Managed system landscape.
• However, when using Solution Manager 7.1 SP10 or higher and Agents On-the-fly, deploy SAP Host Agents 7.20 SP163, at least on High Available/Fail-Over hosts. With this version, the logical hostname switches are detected faster.
• Consider using the SAP Host Agents 7.20 SP179, in case you encounter errors related to the connectivity with the SAP Host Agent. See also: SAP Host Agent troubleshooting.
• Further below you find more details, including hints to find out the version of existing SAP Host Agents or how to upgrade a SAP Host Agent.

Installing SAP Host Agent

• Install one SAP Host Agent, per physical host, if not yet installed. See SAP note 1031096.

Upgrading SAP Host Agent

• Upgrading the SAP Host Agent according to SAP note 1031096, if the version is not sufficient
  • Check the version at OS level (search in the output for "SAPHOSTAGENT information"):
    • For Windows: "%ProgramFiles%"\SAP\hostctrl\exe\saphostctrl -function ExecuteOperation -name versioninfo
    • For Unix: /usr/sap/hostctrl/exe/saphostctrl -function ExecuteOperation -name versioninfo
  • The currently installed SAP Host Agent version is also visible under the host details, if “Managed System Configuration” has been performed with solman_setup
    • Navigate to Solution Manager Administration work center → Landscape → Hosts → <search for your host> → Host Operations → <expand "More Details" section> → Status
• Implement automatic upgrade process (especially if you plan SAP Host Agent related changes)
  • SAP note 1473974 describes ways to centrally and automatically upgrade running SAP Host Agents

Diagnostics Agent Upgrade

• There is no need to upgrade Diagnostics Agents (business logic) explicitly.
• Diagnostics Agents update their business logic (coding) from the SAP Solution Manager system to which they are connected. All connected Diagnostics Agents that are running in the landscape, are centrally patched, when deploying an LM-SERVICE Java Software Component Archive (SCA) on the SAP Solution Manager system.
  • Diagnostics Agents installed with Software Provisioning Manager 1.0 SP3, SP4, SP5 and SP6, initially have a version like 7.10.8.xxxxxx
  • When connecting to a SAP Solution Manager system 7.10 SP5, the freshly installed agent will downgrade to a version like 7.10.5.xxxxxx
  • When connecting to a SAP Solution Manager system 7.10 SP10, the freshly installed agent will upgrade to a version like 7.10.10.xxxxxx
• Diagnostics Agent installation SAP notes document procedures to resolve known issues. Alternatively, you can use the below Diagnostics Agent Replacement procedure.

SAP Solution Manager 7.2:

• Diagnostics Agents which were previously connected against a SAP Solution Manager 7.1 system, and which have been migrated/reconnected to a SAP Solution Manager 7.2 system, will also automatically update their business logic (coding) from the SAP Solution Manager system to which they are connected.

 Diagnostics Agent Kernel Patch

If you have problems with the SAP Kernel which is used by the Diagostics Agent, you may need to apply a SAP Kernel patch for the Diagnostics Agent. The procedure for Linux/Unix and for Windows is described in SAP note 2147092.

Diagnostics Agent Kernel Upgrade

In rare cases, you may be asked to upgrade the SAP Kernel used by the Diagnostics Agent (see SAP note 1707141).

The same can be achieved, by applying the below Diagnostics Agent Replacement procedure.

Diagnostics Agent JVM Upgrade

In rare cases, you may be asked to upgrade the SAP JVM used by the Diagnostics Agent (see SAP note 1774669).

The same can be achieved, by applying the below Diagnostics Agent Replacement procedure.

Diagnostics Agent Replacement

To replace a Diagnostics Agent, install a new Diagnostics Agent, and uninstall the previous Diagnostics Agent. This procedure can be relevant in the following scenarios:

• Replace a Diagnostics Agent, which has problems that cannot be easily resolved.
• “Upgrade” a Diagnostics Agent system, e.g.
• install a new Diagnostics Agent using a newer installer and a newer kernel, and get a Diagnostics Agent that has a new SAP JVM and a new SAP Kernel.
• uninstall the previous Diagnostics Agent

Follow the following procedure:

Reset server name Login to the SAP Solution Manager system to which the agent to be replaced is connected Navigate to the Agent Administration and note the server name of the previous Diagnostics Agent Reset the server name using the context menu of the Diagnostics Agent Stop the Diagnostics Agent Install new Diagnostics Agent using a different SID Assign the same server name to the new Diagnostics Agent, using the smdsetup script. For smdsetup script usage, see the latest Diagnostics Agent Installation & Setup Guide, accessible via SAP note 1833501. Java based Managed Systems require important additional actions(see next section) Uninstall the previous Diagnostics Agent using SWPM.

This procedure will:

• Provide you with a newly installed Diagnostics Agent
• The Diagnostics Agent SAP Kernel will have been upgraded to the version that you have provided
• The Diagnostics Agent SAP JVM will have been upgraded to the version that you have provided via the SAP Kernel media
• The SAP Host Agent on your system will have been upgraded to a more recent version if a more recent version was provided via the SAP Kernel media

Diagnostics Agent Replacement in the context of a Java Managed System

This procedure updates the Managed System JVM parameters so that the Introscope Byte Code Adapter binaries are no longer taken from the previous Diagnostics Agent directory structure.

• In case the “Deploy Byte Code Adapter under Managed System Instance(s)” option was not used in the past (before 7.1 SP12 this option was called "Ensure HA virtualization compatibility"):
  • • Again configure the Wily Introscope Agent (Byte Code Adapter Installation) via transaction solman_setup → Managed System Configuration → Configure Automatically
      • • Pre-requisite since Solution Manager 7.1 SP05:
          • • Additionally, mark the option "Deploy Byte Code Adapter under Managed System Instance(s)" within step "Enter System Parameters". SAP recommends providing a relative path name like ByteCodeAdapter for the "Introscope BC Adapter deployment path" field, in order to locate the Byte Code Adapter binaries and configuration files inside the Managed system directory structure (/usr/sap/<ManagedSID>/<InstanceID>).
            • The Diagnostics Agent OS user must have write permissions for this Managed System path location. For details see SAP note 1163751.
    • Restart the Java Managed System

If you experience problems with the Byte Code Adapter (Managed System does not restart), please check Introscope Troubleshooting (search for "uninstall").

Diagnostics Agent Removal

To remove a Diagnostics Agent instance installed for a specific host name:

• In case the solman_setup “Deploy Byte Code Adapter under Managed System Instance(s)” option was used in the past to set up the Byte Code Adapter of the Java Managed System(s) running on the host (before 7.1 SP12 this option was called "Ensure HA virtualization compatibility"):
• • Optionally: Also uninstall the Byte Code Adapter by removing the JVM parameters listed in the relevant Instroscope Setup Guide from Introscope Home → Documentation.
• Else, take the following actions (in case the solman_setup “Deploy Byte Code Adapter under Managed System Instance(s)” option was not used in the past during the Managed System Configuration):
• • IMPORTANT: Remove the JVM parameters used to start/run the Introscope Byte Code Adapter from the Java Managed System. These parameters listed in the relevant Instroscope Setup Guide from Introscope Home → Documentation.
  • NOTE: The Managed System JVM parameters (to be removed) show that the Introscope Byte Code Adapter executed by the Java Managed System uses files from the Diagnostics Agent directory structure, e.g.:
    • • Xbootclasspath/p:D:/usr/sap/DAA/SMDA98/SMDAgent/applications.config/com.sap.smd.agent.application.wily/BytecodeAgent/ISAGENT.xxxxxxxxxx/wily/connectors/AutoProbeConnector.jar; D:/usr/sap/DAA/SMDA98/SMDAgent/applications.config/com.sap.smd.agent.application.wily/BytecodeAgent/ISAGENT.xxxxxxxxxx/wily/Agent.jar 
  • Restart the Java Managed system to no longer run the Introscope Byte Code Agent.
• Run the Diagnostics Agent installer SWPM, to uninstall the Diagnostics Agent

If you experience problems with the Byte Code Adapter (Managed System does not restart), please check Introscope Troubleshooting (search for "uninstall").

After the Diagnostics Agent has been uninstalled, apply the option "Delete Offline Agent Entry" in the Agent Administration.