Symptoms
During mailboxes import, the error is thrown:
Database deadlock detected while executing
From poa.log:
Sep 10 10:10:10 hostname : DBG [UI:347cb53e:844125:1476240117052 1:19744:ed8d1b90 Kernel]: [txn:7418506 DBS::checkSQLError] Diagnostic info about SQL error: sqlreturn=0 state=HY000, nativeCode=7, description=ERROR: deadlock detected; Error while executing the query
Sep 10 10:10:10 hostname : DBG [UI:347cb53e:844125:1476240117052 1:19744:ed8d1b90 lib]: [txn:7418506 execQuery] {module_id="db_service"; code="100"} Database deadlock detected while executing ''.
And the following information on appeared deadlock can be found in the Postgres SQL logs:
ERROR: deadlock detected
DETAIL: Process 29408 waits for ShareLock on transaction 3523533654; blocked by process 8194.
Process 8194 waits for ShareLock on tuple (5,6) of relation 130563444 of database 16406; blocked by process 29401.
Process 29401 waits for ShareLock on transaction 3523525476; blocked by process 29408.
Process 29408: INSERT INTO actdir_group_membership (group_id, entity_id, sync_status) SELECT 31403 , entity_id, 1 FROM actdir_entities WHERE entity_id = 69777 AND NOT EXISTS (SELECT 1 FROM actdir_group_membership WHERE entity_id = 69777 AND group_id = 31403 )
Process 8194: INSERT INTO exch_mailboxes ( mailbox_id, mailbox_type, is_hidden, sync_status, ops_scheduled, ds_id, name, warning_limit, p_send_limit, p_send_rec_limit, i_item_max_size, o_item_max_size, rcpts_max_num, msg_retention, imap4_status, pop3_status, http_status, mapi_status, unified_messaging_status, active_sync_status, store_id, hsu_id, alias, migrated ) VALUES ( 69799, 0, 0, 0, 0, 706, E'John.Doe', 92160, 102400, 102400, 25600, 25600, 100, 365, 1, 1, 1, 1, 0, 1, 173, 226887, E'John.Doe', E'n' )
Process 29401: UPDATE exch_domain_services SET oab_next_rebuild_time = '2016-10-10 04:00:00'::timestamp , oab_retries_left = 2 WHERE enabled='y' AND oab_enabled='y' AND ds_id = 706
Cause
The root cause of such behaviour is software related issue #POA-64469 which leads to simultaneously running 'Create mailbox' tasks and import operation. Both actions require exclusive lock on the same table and cause the deadlock.
Resolution
#POA-64469 has been fixed in Odin Operations Automation version 6.0. Consider to upgrade the platform to at least 6.0 version.
As a workaround, import mailboxes in bunches during the hours of low load on the platform. The right way to do it is to import a bunch of mailboxes then wait until all related tasks are processed, and only then import a new bunch of mailboxes.