Symptoms

Sometimes default log level does not allow to find the exact reason of the error. It is necessary to increase logs verbosity in order to receive more information about exact operation.

Resolution.

PPAB/BA logs can be found in the directory /var/log/pa on Billing Service Node/BA Aplication server.

The list of containers for BA 6.x: WWW, XMLRPCD, _amt_service_, csmapi, generic_worker, logsrv_ctl, pcpapi, scheduler, upsellapi.

for BA 7.x: billing, billing._amt_, billing.scheduler, billing.www, billing.xmlrpcd, logsrv_ctl

With the following command you can retrieve current log level of the PPAB/BA processes:

# /usr/local/stellart/bin/logsrv_ctl --command list
Current log levels (* - default): :4 billing:4 billing._amt_:3 billing.scheduler:4 billing.www:3 billing.xmlrpcd:3 logsrv_ctl:3

PPAB/BA log level is an integer value from 0 to 7:

  • 0 - FATAL, only fatal error that cannot be, even theoretically, processed by the application, for example, a system error like 'unable to allocate memory' are written to the log.
  • 1 - ERR, any error that can be theoretically processed by the application, for example, by the caller, but the module which caught it cannot process the error, is written to log.
  • 2 - WARN, any error that was caught and processed by the module.
  • 3 - NOTE, is used for logging key application execution points, like incoming RPC calls.
  • 4 - TRACE, is the DEFAULT logging level, when LogLevels is set to "*:4". Diagnostic messages are displayed by calling 'tracing arguments'. WARNING: If you remove LogLevels variable from the .global.conf and restart the service 'pba', then new default level is set as '3' ('NOTE')
  • 5 - INFO, records the information about key code branching points, e.g. calls to external systems with their addresses.
  • 6 - DEBUG, records all key code points with the data, for example, 'entering critical section' with the value of mutex used. At this level, database statements are recorded. The level is often used for troubleshooting.
  • 7 - ALL, is the last chance before starting a debugger data values, for example, it requires full response text from external systems or data fetched from the database. Rarely used for troubleshooting as a lot of excessive data might appear in the log.

logsrv_ctl utility allows to change log level without application restart. For example, to set log level of the billing container to 7:

# /usr/local/stellart/bin/logsrv_ctl --command set --level 7 --container billing
Log level 7 for container billing has been changed
Current log levels (* - default): :4 billing:7 billing._amt_:3 billing.scheduler:4 billing.www:3 billing.xmlrpcd:3 logsrv_ctl:3

Upon completion of work with logs change the log level to the default level:

# /usr/local/stellart/bin/logsrv_ctl --command set-default
Default log levels has been restored
Current log levels (* - default): :4 billing:4 billing._amt_:3 billing.scheduler:4 billing.www:3 billing.xmlrpcd:3 logsrv_ctl:3

General command description:

# logsrv_ctl --help
logsrv_ctl options:
--help produce help message
--command arg set - sets new log level for specific container.
list - shows current log level for all containers.
set-default - returns log levels of all containers to their default values.
--container arg Container name to apply log level.
--level arg Log Level to be applied: 0 - 7 (0 - FATAL, 1 - ERR, 2 -
WARN, 3 - NOTE, 4 - TRACE, 5 - INFO, 6 - DEBUG, 7 - ALL)

IMPORTANT! There is a bug #PBA-63879 ("Database queries are not always logged") which prevents the database queries being logged when the log level changed from default to debug with the logsrv_ctl command. The bug is fixed in OSA 7.3 (#PBA-83185). If you need to see the queries in lower product versions you have 2 options:

  1. Change the log level with restart.
  2. Remove LogLevels parameter from global.conf and .global.conf files, restart pba service (WARNING: new default level will be '3' then, so you might want then to add in the /etc/init.d/pba the command logsrv_ctl with the required log level different from '3'). Then you may use logsrv_ctl to change log level "on-the-fly".

Internal content