To replace the App Volumes Manager SSL certificates, you should follow VMware KB article 2095969.
However, there is one important step missing in the KB article: After having replaced the original svserver.crt and svserver.key files with your own certificate files you should also edit the nginx.conf file to use svserver.crt and svserver.key instead of appvol_self_vmware.com.crt and appvol_self_vmware.com.key
I have asked VMware to adapt the KB article to add this important step.