Question

How to properly restart Apache on NG cluster and check whether the restart completed successfully?

Answer

Instead of service pemhttpd restart, like on Legacy shared hosting, service httpd restart should be used as on usual Apache server.

Use it the following way:

~# service httpd stop
~# service httpd start

Message like one below should appear:

Starting httpd: [Wed Sep 11 08:45:57 2013] [notice] POA VHosts: module has registered its hooks.
                                                           [  OK  ]

After that check Apache error log file for details:

[Wed Sep 11 08:44:14 2013] [notice] caught SIGTERM, shutting down
[Wed Sep 11 08:45:58 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Sep 11 08:45:58 2013] [notice] SSL FIPS mode disabled
[Wed Sep 11 08:45:58 2013] [notice] POA VHosts: module has registered its hooks.
[Wed Sep 11 08:45:58 2013] [notice] Digest: generating secret for digest authentication ...
[Wed Sep 11 08:45:58 2013] [notice] Digest: done
[Wed Sep 11 08:45:59 2013] [notice] SSL FIPS mode disabled
[Wed Sep 11 08:45:59 2013] [notice] Apache configured -- resuming normal operations

No errors in log means that Apache was started successfully.

Last thing - check if httpd processes are running:

~# ps ax | grep httpd
 106906 ?        Ss     0:00 /usr/sbin/httpd.worker
 106909 ?        S      0:00 /usr/sbin/daemon_sucgid 106906 /etc/httpd/logs.106906.0 1 0
 106910 ?        S      0:00 /usr/sbin/daemon_sucgid 106906 /etc/httpd/logs.106906.1 1 0
 106911 ?        S      0:00 /usr/sbin/daemon_sucgid 106906 /etc/httpd/logs.106906.2 1 0
 106912 ?        S      0:00 /usr/sbin/daemon_sucgid 106906 /etc/httpd/logs.106906.3 1 0
...

There should be many of them. If apache is stopped or crashed, there are none:

~# ps ax | grep httpd | grep -v grep
~#

Internal content