Issue
I am unable to call my HTTPS endpoint using a self-signed certificate in Appsmith. Even though I followed the steps mentioned in the documentation, the server is unable to find a valid certification path to the requested target. I shared the server logs with Appsmith but they have identified it as a bug and are working on resolving it.
Resolution
The first question was whether PKCS12 self-signed certificates can be used instead of PEM format. The answer is no, as Appsmith only supports PEM format for SSL certificates.
The second question was related to calling a HTTPS REST endpoint and adding a self-signed certificate in the Authenticated Datasource option. The answer is yes, this is the correct way to do it in Appsmith.
However, the user encountered an error while trying to call the endpoint, with an SSLHandshakeException due to Appsmith not being able to find a valid certification path. This was identified as a bug and a GitHub issue was opened to fix it.
Overall, the solution to calling a self-signed HTTPS endpoint in Appsmith is to attach the PEM certificate in the Authenticated Datasource option.