SAP Host Agent Troubleshooting Guide

Changing the default trace level of SAP Host Agent

SAP Host Agent runs with tracelevel 1 by default. If you need more information to be able to analyze an issue, you can add or change the following profile values:

hostexec/trace
service/trace

Proceed as follows:

1. Open the SAP Host Agent profile ( host_profile ) which is located in the exe directory of the SAP Host Agent (/usr/sap/hostctrl/exe/ or C:\Program Files\SAP\hostctrl\exe\).
2. Add or modify the required values, for example:
   hostexec/trace = 3  
   service/trace = 3
3. Restart SAP Host Agent by executing saphostexec -restart (as a user with root authorization ) or hostexecstart -restart (as user <sapsid>adm)

All SAP Host Agent log files in /usr/sap/hostctrl/work/ (UNIX) or C:\Program Files\SAP\hostctrl\work\ (Windows) now contain much more information.

Start of SAPHostExec Service / saphostexec executable failed. dev_saphostexec log file contains error message "ERROR => Setup of too many communication channels failed"

Reason:

The sapstartsrv process of the Host Agent is not able to start.

Solution(UNIX):

Proceed as follows:

1. Stop the Host Agent with saphostexec -stop .
2. Check that there are no running processes which belong to SAP Host Agent, and kill any remaining Host Agent's sapstartsrv processes.
3. Delete the /tmp/.sapstartsrv99_sapstartsrv.log and content of the Host Agent's work directory.
4. Restart SAP Host Agent by executing saphostexec -restart (as a user with root authorization) or hostexecstart -restart (as as user <sapsid>adm
5. Check the system log for the error messages from the sapstartsrv process of the Host Agent

Note:
Probably, there are multiple sapstartsrv processes running: One from the SAP Host Agent and one from each SAP instance. Just ensure that there is no SAP Host Agent sapstartsrvprocess running. You can identify it by the start parameter, which is pf=/usr/sap/hostctrl/exe/host_profile and by the assigned user, which is sapadm.

Other probable reasons for this error are:

1. /usr/sap/hostctrl/work and/or /usr/sap/hostctrl/work/sapstartsrv.* is not writable for the sapadm user
2. /tmp directory is not writable for the sapadm user or filesystem has no free space left.

The sapstartsrv error message in the system log should give you a hint why it cannot be started.

Solution(Windows):

Check the Application Log in the Windows Event Viewer for the error messages from the SAPHostControl service

Oracle databases cannot be detected on Unix

Symptom:

• Web service method ListDatabases does not return Oracle databases or not all Oracle databases
• Other database Web service methods (for example GetDatabaseStatus) return the error message "Database not found" for Oracle databases.
• With default trace level dev_sapdbctrl contains messages like
  [PID 23258] *** ERROR => Database neither found in '/var/opt/oracle/oratab' nor in inventory and environment of 'ora<sid>' user. Giving up... [dboractrlux.cpp 611]
• With trace level 3 (host_profile: service/trace = 3) the trace file /usr/sap/hostctrl/work/dev_sapdbctrl contains messages like the following:

[PID 10682408] Looking up databases in inventory
[PID 10682408] Retrieving inventory location from '/etc/oraInst.loc'
[PID 10682408] Retrieving inventory data from '/oracle/oraInventory/ContentsXML/inventory.xml'
[PID 10682408] *** ERROR => Failed to open file '/oracle/oraInventory/ContentsXML/inventory.xml': No such file or directory 
[PID 10682408] No installations registered in inventory

or like the following:

[PID 10682408] Looking up databases in inventory
[PID 10682408] Retrieving inventory location from '/etc/oraInst.loc'
[PID 10682408] Retrieving inventory data from '/oracle/oraInventory/ContentsXML/inventory.xml'
[PID 10682408] No installations registered in inventory

Solution:

To correctly detect Non-RAC Oracle databases:

1. The database instance must be maintained in /etc/oratab (Solaris: /var/opt/oracle/oratab)
2. or a central Oracle inventory must exists and the ORACLE_HOMEs of all databases must be registered. Also the central inventory pointer file must exist and contain the correct path to the central inventory.

And for RAC databases:

1. SAP Host Agent 7.21 PL 14 must be installed
2. The GRID/ASM instance must be maintained in /etc/oratab (Solaris: /var/opt/oracle/oratab). E.g.
   +ASM1:/u01/app/12.1.0.2/grid:N
3. or a central Oracle inventory must exists and the ORACLE_HOMEs of at least the GRID/ASM instance must be registered. Also the central inventory pointer file must exist and contain the correct path to the central inventory.

Normally, all this is created by the Oracle/SAP installation tools.

Please note, from Oracle 11.2 onwards the so called runtime home must be maintained in the oratab file. Refer to Wrong Oracle database status on Unix for more details on that.

For details on how to check the central inventory pointer file you can refer to e.g. http://docs.oracle.com/cd/E11882_01/em.112/e12255/oui2_manage_oracle_homes.htm#CJAEHIGJ. For details on how to check whether the central inventory exists and which ORACLE_HOMEs it contains you can refer to e.g. http://docs.oracle.com/cd/E11882_01/em.112/e12255/oui2_manage_oracle_homes.htm#CHDEHFBJ.

If the central inventory does not exist or it doesn't contain all ORACLE_HOMEs, you can create or adapt it with the Oracle runInstaller tool. For example, you can do this by executing the following command:

runInstaller -attachHome ORACLE_HOME="<Oracle_Home_Location>" ORACLE_HOME_NAME="<Oracle_Home_Name>"

For more information, refer to http://docs.oracle.com/cd/E11882_01/em.112/e12255/oui2_manage_oracle_homes.htm.

Wrong Oracle database status on Unix

Although the database is started and running fine it will be reported as stopped. E.g.:

# /usr/sap/hostctrl/exe/saphostctrl -function GetDatabaseStatus -dbname C11 -dbtype ora

Invalid databases status Database Status: Stopped

Component name: Instance (Instance), Status: Stopped (Instance is stopped)

Component name: Database (Database), Status: Stopped (Database is stopped)

Component name: Archiver (Archiver), Status: Stopped (Archiver is stopped)

Component name: Listener (Listener), Status: Running (Listener is started)

Please ensure that the correct Oracle home directory is maintained in /etc/oratab (Solaris: /var/opt/oracle/oratab).

From Oracle 11.2 onwards the so called runtime home must be maintained in the oratab file. E.g.:

C11:/oracle/C11/112_64

You can refer to Note 1524205 for details on the runtime home.

Issues with Oracle 12c databases on Unix

For Oracle 12c the ora<dbsid> user is still needed. Please refer to Note 1915323 for more details on this. In case the user was removed it can be recreated as described in Note 220421.

At least Host Agent 7.21 PL6 is needed for general Oracle 12c support. At least 7.21 PL16 is needed for Oracle 12c Multitenant databases.

SAP Solution Manager Diagnostics (SMD) Agent is not able to connect to SAP Host Agent

Symptom:

The following issues may arise while the SMD agent is connecting to SAP Host Agent for the first time:

Trusted connect is not working

We assume the <smd>adm user is used for the installation of the SMD agent installation, where <smd> is the system ID of the SMD agent ("DAA" by default).

Solution:
Proceed as follows to check if SAP Host Agent is working correctly:

1. Log on on the respective server as <smd>adm.
2. Execute the following command (Note: Instance number is always 99. This is not an example!) :
   on UNIX:
   
   /usr/sap/hostctrl/exe/sapcontrol -nr 99 -user "" "" -function ConfigureLogFileList add /tmp
   
   on MS Windows (replace the <smdadm> and <password for smdadm> with your user and password):
   
   c:\usr\sap\DAA\SMDA97\exe\sapcontrol.exe -user <smdadm> <password for smdadm> -nr 97 -function OSExecute "\"c:\Program Files\SAP\hostctrl\exe\sapcontrol.exe \" -user \"\" \"\" -nr 99 -trace - -function ConfigureLogFileList remove testLogFile.txt" 0 0 c:\temp\tst_out.txt

If the output of the command prompt is as follows, the trusted connect works:

12.12.2012 11:25:27
RequestLogonFile
OK

12.12.2012 11:25:27
ConfigureLogFileList
OK

This generally means that Trusted Connect has been configured correctly on the part of SAP Host Agent.
However, the SMD Agent call might fail. In those cases you should proceed as follows:

Check if ACLs are used. In the host_profile file you can find a configured entry: service/http/acl_file=<filepath>
If yes, check in the ACL files if all the relative IP Addresses of the local machine are configured correctly.

Alternatively you can proceed as follows:

1. In the host_profile, specify service/trace=3
2. Restart Host Agent with the following command: saphostexec -restart
3. Trigger the SMD scenario
4. Check if entries like NiAcl .... denied are contained in the sapstartsrv.log file. If yes, the ACL configuration is wrong and this is the root cause of that issue.

If the sapcontrol call does not return with OK, you have the following options to proceed:

The output is: "FAIL: Permission denied"

The output of the command prompt is as follows:

ConfigureLogFileList
FAIL: Permission denied

Solution:
Proceed as follows:

1. Check if the <smd>adm user is specified in the host_profile in line service/admin_users .
2. If <smd>adm user is not specified there, add it.
3. Restart the SAP Host Agent.

Remark: On Windows (in Unix by default) the service/admin_users in the host_profile are case sensitive and need to be defined exactly as the user running the service. For example service running as <SMD>ADM\SAPServiceDAA cannot be defined as <smd>adm\SAPServiceDAA in the host_profile. The upper and the lower case letters must match.

The output is: "HTTP/1.1 401 Unauthorized"

The output of the command prompt is as follows:

ConfigureLogFileList
FAIL: HTTP error, HTTP/1.1 401 Unauthorized

Analysis:

User <smd>adm seems not to be able to read the logon file. The cause might be that file system permissions or ownership are wrong.

Solution:

Ensure that the file permission and ownership are as follows:

# ls -lda /usr/sap/hostctrl/work/sapcontrol_logon
drwx--x--x 2 sapadm sapsys 4096 28. Apr 06:33 /usr/sap/hostctrl/work/sapcontrol_logon

Ensure that the <smd>adm user is member of the sapsys group and the user is able to access files in /usr/sap/hostctrl/work/sapcontrol_logon.

Error message "Remote access not permitted" in SMD agent logs

Ensure that you have at least Host Agent 7.20 patch level 179 installed.

Autoupgrade from shared directory not working on Windows

For windows, the Autoupgrade has certain prerequisites:

1. The Host Agent system must have access to the share. The SAPHostExec service is doing the autoupgrade and this service is running as Local System Account. ( One can read more about the Local System Account on http://msdn.microsoft.com/en-us/library/windows/desktop/ms684190 ) Hence, the <System>$ account must have access to the shared directory.
2. Autoupgrade is working with UNC paths only. "Mounted" directories (e.g. as Drive X:) do not work.

If the Host Agent System and the System with the shared directory are located in different windows domains, there must be a one way trust to access the share.That means, the domain of system A (the system with the shared directory) has to trust the domain of system B (the system with the Host Agent).

See http://msdn.microsoft.com/en-us/library/cc237016%28v=prot.10%29.aspx for more information on how to setup a domain trust.

Unable to detect Microsoft IIS applications

Symptom:

The command to retrieve list of IIS web applications fails with error "Invalid namespace":

"c:\Program Files\SAP\hostctrl\exe\saphostctrl.exe" -function GetCIMObject -namespace WebAdministration -enuminstances Application -user sapadm <password>

Error: CIM Request EnumerateInstances for classes Application failed: EnumerateInstances for Application failed: Invalid namespace

Solution:

Please ensure that the "IIS Management Scripts and Tools" are installed. For example, with Windows 2008 this can be checked via Server Manager -> Roles -> Web Server -> Add Role Services.

Username/Password-based authentication fails on Unix

Symptom:

When a Web service method of the Host Agent is called or a Web-UI is accessed via the Host Agent (e.g. SUM) Username/Password-based authentication fails although the correct credentials were supplied.

Solution:

Ensure that the Host Agent is correctly installed/updated as described in Note 1031096. A common error is that the host agent files are manually copied into /usr/sap/hostctrl/exe or the host agent archive is just directly extracted into /usr/sap/hostctrl/exe. This results in invalid file ownership and permission, which in turn breaks various functionality like e.g. the Username/Password-based authentication.

To fix this install the Host Agent as described in Note 1031096.

Host Agent fails to load clean root user environment on startup or installation/upgrade

When starting or upgrading the Host Agent you see the following messages:

Executing "'/bin/su' '-' '-c' 'echo SAPHOSTAGENT-START;env;echo SAPHOSTAGENT-END' " *** WARNING => Timeout, helper didn't answered within 30 sec... *** ERROR => Unable to load clean root environement.

Starting with 7.21 PL8 the Host Agent is trying to load a clean root user environment during startup. This is done by running a (non-interactive) root user login shell. This step hangs and is therefore interrupted after some timeout.

Typical reasons for that are some prompts issued by shell login scripts requesting manual user input. You can check this by e.g. running the following command:

/bin/su - root -c env </dev/null

Please ensure that these non-interactive root user login shells don't hang/require any user input.

E.g. in the shell login scripts you can use the 'tty -s' command to check whether the shell is interactive and only in this case execute commands that require user input.