If VDS was scaled (from Name Based to IP based) while "Set site's mail server as outgoing mail server." was unchecked (disabled), then it's impossible to enable "POP before SMTP" for this VDS.
The problem occurs since that the out_mail="false" value in hostdir.xml of the scaled VDS was kept after the scaling and the "POP before SMTP" related code doesn't override it's value while upgradingto IP based VDS.
In order to fix this issue it's necessary to set "out_mail" value in hostdir.xml to "true".
The problem found in SD v4.1 but it could be the same in SD v3.7.