RDP Packet loss via SSL-VPN in UTM9 / SG230

Hello everybody,

we are using an application based on the Microsoft RDP protocol, namely the IRDPSRAPI. The "sharer"-part of the application initiates a connection via IRDPSRAPISharingSession which is answered by the "viewer"-part of the application via IRDPSRAPIViewer. The catch: This works in the following cases:
- Both sharer and viewer are directly connected to the network (non-VPN)
- The sharer is directly connected to the network (non-VPN) and the viewer is using Sophos SSL-VPN of the UTM

but *not* if the sharer is the one using SSL-VPN, regardless of the viewer being directly connected or also using SSL-VPN. While connecting, the sharer sends several packets, including a "PUSH" packet, which is then lost in the UTM and the connection can't be established.

We have tried several versions of the UTM 9-Software (9-5xx to 9-6xx) and SSL clients, both Windows and MAC based as well as Sophos REDs.

The software vendor has analyzed the problem and concluded that the packet gets lost in the SSL stack of the UTM. Does this make sense? (How) Can this be verified? Is this a misconfiguration on our end? Regular Remote Desktop sessions work flawlessly.

Thank you in advance and regards,
Ken

Edit: Formatting fixed
  • Hallo Ken,

    I suspect that you need a Static Route.  I'm having trouble visualizing what works and what doesn't work with SSL VPN Remote Access.  Perhaps you could show us a diagram including subnets of what doesn't work as well as a picture of the Edit of the SSL VPN Profile.

    Cheers - Bob

  • Is your SSL VPN configured to use UDP?   If not, change to UDP and re-test.

    There are some perverse things that can happen if you run one TCP session inside another TCP session.  Basically, retransmits in one layer can trigger unwanted retransmits in the other layer.   Do a web search for"TCP Meltdown" if you want more details.

  • In reply to BAlfson:

    Hi Bob,

    I made three diagrams showing what works and what doesn't. I hope this works within the forum (no Image upload…)

    This works: https://ibb.co/MZdxZVF

    This also works: https://ibb.co/NC4H4hg

    This doesn't work: https://ibb.co/KqP4JCz

    We set up wireshark and it looks like something happens to the packets within the UTM. This is the wireshark trace on the viewer's side: https://ibb.co/PYMKn1G

    As I said, "regular" RDP works in every case.

    Regards, Ken
  • In reply to chronowerx:

    Getting closer, Ken.  What are 172.16.2.22 and .56?  What subnet does "VPN Pool (SSL)" contain?  What IP are the SSL VPN clients connecting to?  What port(s) does the sharer use?

    Please show the Edit of the SSL VPN Profile.  You can copy and paste an image directly here and we'll know that it can't be modified to contain malware by some hacker.  I opened your links in a sandbox just in case.  We can't know if an external site is properly protected. The only malware I've gotten in over 10 years was from an external link to a picture in this forum about six years ago.

    Let me say something in my own words to see if I understand what's not working.  When client 1, connected via VPN, shares a session to directly-connected client 2, Client 2 receives the sharing offer and initiates a viewing session of that offer.  Wireshark shows the viewing session packet arriving on the UTM but not leaving the UTM.  There is nothing in the Firewall or Intrusion Protection log indicating that the UTM blocked the packet.

    Cheers - Bob

     

  • In reply to BAlfson:

    Hi Bob, first of all thank you for your continuous help. The good news is - I figured it out. The bad news: I could have figured it out a lot sooner by following Rule #1 in Rulz more precisely ;-) I only now checked the IPS logs, since I didn't expect it to react within an internal-defacto-internal (VPN) connection.

    Bottom line: "OS-WINDOWS Microsoft Windows Terminal server RDP over non-standard port attempt". After I disabled the IPS for two test clients, the software works flawlessly.

    Sidenote: Contrary to the images posted, the connection VPN->Internal worked but not the connection Internal->VPN. My mistake.

    The question remains: Why does regular RDP work in both directions? As I understand, RDP is using its standard port 3389 to establish a connection and to negotiate which ports are being used e. g. for image transportation, which is fine with the IDS. So if "our" application uses non-standard ports for the connection part already - which I assume happens - it is being flagged by the IDS. A design flaw? An "overreaction" of the IDS?

    Regards, Ken
  • In reply to chronowerx:

    "Working as intended," let's say - glad you found the problem!  Even when I "know" I won't find anything, I always do #1 in Rulz to leave no doubt.

    Cheers - Bob