After many hours of trial&error I found the solution.
Created a Shared folder with the name ResilioSync.
Created a sub-folder with the name cert.
Copied cert.pem and privkey.pem (own trusted certificate) into sub-folder cert.
Used SSH to get into the Synology.
Went to /volume1/@appstore/resiliosync/var
In this folder you will find the configuration file sync.conf
sudo vi sync.conf
Enter password
Add the 2 ssl lines as below:
"force_https": true,
"ssl_certificate" : "/volume1/ResilioSync/cert/cert.pem",
"ssl_private_key" : "/volume1/ResilioSync/cert/privkey.pem"
},
Save the file and restart (Stop/Start) Sync in the Package Center.
Now refresh the Web UI page and the certificate issue should be solved.