Symptoms

  • Domain registration/transfer orders are not processed, are staying in the statuses "Provisioning", "Long Operation Running" statuses.

  • The corresponding subscriptions service status history shows last messages "Submitting for registration", "Checking registartion status", "Checking transfer status", "Submitting for ns creation" etc, but the date of the last message is more than an hour ago.

Cause

It is possible that the regular event DomainProcessing is stuck. You may check if there is an active task of Event Type DomainProcessing, method ProcessDomains:

BA 5.4: Home > Configuration Director > Event Manager > Active Tasks

BA 5.5 and later: Home > Operations > Tasks > Active Tasks

If there is such event with big "Wait Time", it may mean it hang, as usually it takes several seconds and is scheduled for execution each minute.

Resolution

Delete the hanged task from Active Tasks list, it will be restarted automatically. Check that the orders started processing - the new requests to the registrar are sending.

Additional information

The reason of the issue is usually some custom plugin. You may check it from the logs:

  1. less /usr/local/bm/log/DOMAINGATE.log (for BA >6.0), /usr/local/bm/log/scheduler.log (for BA 6.0.x), /var/log/pa/billing.scheduler.log (for BA 7.0.x)
  2. find the last record of DOMAINGATE entering the method ProcessDomains (before the task deleting), the record shall look like:

    [14-02-12 10:36:14.969 Worker1.3   RQ3090642 NTE] Entering method DOMAINGATE_Container::Worker1.3.ProcessDomains(user = -1, transaction = 2307662, SID = 0, lang = en, LP)
    [14-02-12 10:36:14.969 Worker1.3   RQ3090642 NTE]       1 input arguments :
            string: NULL
    

    then, at some of the checks within a method there shall be a call to the external plugin, e.g:

    [14-02-12 10:36:15.020 Worker1.3   RQ3090642 TRC]  +++[1] void _process_domains(DOMAINGATE*, Int, Int, Int, void (bDomainProcessing::*)(Int), const char*)(Find domains to be checked for registration status)
    <... - here it finds such domain and sends a request to the custom plugin MYPLUGIN>
    [14-02-12 10:36:15.053 RO_3:21739  RQ3090642 TRC]      +++[5] DataPacket* RemoteObject::Call(const std::string&, DataPacket&, int)((RO_3:21739) method = GetDomainStatus MYPLUGIN_Container::MYPLUGIN_Object, tmout = 86400, user = -1, transaction = 2307664, SID = 0, lang = en, LP)
    

    but due to whatever reason the request GetDomainStatus was not completed on the third-party plugin side (can be seen in the plugin's log), did not return neither success, no error message. As a result ProcessDomains hangs, as it waits for an answer from the custom plugin which is never received.

    If the described situation appears to be the reason of the issue, you may implement some kind of timeout in the custom plugin, if the method does not reply within some period of time, it returns an error. There is also a feature request #PBA-47672 (ProcessDomains improvement) to implement some reasonable timeout in the ProcessDomains event on calls to GATEs.

Internal content