SSL Warnings

You will have to install a publicly recognizable certificate which can cost money.

There are however, free variants:

1. Letsencrypt

2. Startssl (free but you will have to renew every year)

I don't think that will fix the problem. The name of the UTM still won't match the name of the SFTP server.

I'm wondering what policy will force the tester to look the server.

On which open port is it detecting the certificate?  UTM is known to open ports that are not wanted, notably SMTP proxy opens port 25 on all confogured addresses.  

The name of the UTM doesn't have to match the server. Think of webservers. They can have multiple certificates but many can reside on one host.

If you have a host called utm.mydomain.com at 1.1.1.1 and it has a certificate loaded into it called ftp.mydomain.com, as long as ftp.mydomain.com is mapped to 1.1.1.1, the browser will not complain. It will complain however if you type in utm.mydomain.com because it doesn't have the matching certificate.
If you need multiple certificates eg for sftp, you will need additional public ip addresses rather than using the utm interface certificate

Would I be right in assuming you are only using one public ip address?

No, we have a number of public IPs. The server is using an IP that's used for nothing else and the only port allowed through is 22.

I would think the pentest simply uses the external IP of the UTM for it's warnings, not port 22.

Would be interesting on which ports/protocols it complains about using a self-signed certificate? Is eventually user portal active for Any IPv4? Or WebAdmin allowed from anywhere outside (what would be a definately a "DON'T DO THAT")?

Those services use a self-signed certificate if you didn't upload a signed one.

Normally, SSH/SFTP doesn't use any x509 certificates but rsa keys that must be known to the SSH server to allow connection or not.

SSH uses encryption for authentication/login, SSL for transportation to be trusted.

As Kevin has mentioned, is there anything running on that public ip?

The pen test is obviously getting a reply from something. Are there anymore details ie what port eg 443, 4444

I would suggest re-running your tests to be certain that an outside entity really receives the UTM certificate when it connects on that IP address and port 22.   If so, then one of the UTM proxy modules has intercepted the traffic, but I don't see that UTM has a proxy module for SSH/SFTP, so I don't see how this is possible.

Your PCI-compliance testing service is not likely to be satisfied with any of the following:   (a) unencrypted connections, (b) encrypted connections that do not use a commercial CA certificate or DANE to prove identity, because encryption does not matter until you know that you are talking to the right endpoint, or (c) encrypted connections that support weak ciphers or MACs.  

Since you said that your SFTP server does not have a CA certificate, nor does your UTM, you have a problem because of (b).   I think your pentest vendor will object, with or without UTM in the network path.

I've been able to find a little more information from the vendor. It's finding the cert on Port 3400, which is the port REDs use. The cert on the UTM is there for that exact purpose. What I don't know is why the UTM is grabbing traffic sent to the server's IP address or how to fix it.

The RED port should only be open if RED management is active (and I guess an active RED-configuration is detectet as the port is not automatically in "listen" state if you simply enable "RED management").

If you have "RED management" enabled it automatically creates a new Signing CA called "Remote Ethernet Device CA" and that one is oviously found in your pentest. If you have REDs connected to your UTM you won't be able to fix that, I'm afraid since there is no way so select a publically signed certificate within RED management.