Symptoms

APS 2.0 tasks are failing:

Network error: SSL connection unexpectedly closed. Application endpoint: 192.0.2.2. Request: DELETE ...

where 192.0.2.2 is the application instance endpoint node.

The /var/log/httpd/error_log file contains Segmentation fault error messages corresponding to task failures by time.

Cause

httpd service is unstable and faces segmentation faults during processing APS 2.0 REST requests.

Resolution

As a workaround, restart Apache service to stop the errors:

# systemctl restart httpd

To allow future root cause diagnostics:

  • change core pattern:

    # mkdir /tmp/coredumps
# sysctl -w kernel.core_pattern="/tmp/coredumps/core-%e-sig%s-user%u-group%g-pid%p-time%t"
# echo 'kernel.core_pattern="/tmp/coredumps/core-%e-sig%s-user%u-group%g-pid%p-time%t"' >> /etc/sysctl.conf

  • create /etc/security/limits.d/core.conf file with some reasonable limit, e.g. 10 core dumps:

    *       hard        core        10
*       soft        core        10

  • modify /etc/systemd/system.conf and add:

    DefaultLimitCORE=infinity

  • modify /etc/systemd/system/multi-user.target.wants/httpd.service and add:

    LimitCORE=infinity

  • commit the changes:

    # systemctl daemon-reexec
# systemctl stop httpd
# systemctl start httpd

Internal content