IPsec VPN: best policy for encryption?

"Best encryption" usually means "hardest to intercept", not "best performance".   So you need to decide what you want.  Better security requires more CPU, so you need to determine if your perceived performance problem is because of a CPU limitation.

For security:

• 3DES is considered barely adequate these days.   AES-128 is a minimum and AES-256 is preferred.
• MD5 and SHA1 are considered broken, and you should use one of the SHA2 variants.

But you also need to address MTU.   When a packet enters a VPN tunnel, it receives an envelope that makes the packet bigger.   If the new packet is too big for the outbound interface, it has to be fragmented.   Fragmentation costs you CPU overhead at both ends, consumes bandwidth because of extra packet overhead, and introduces latency,   To prevent this problem, you need to change the MTU on the INSIDE interfaces, so that the maximum packet size plus the VPN overhead still fits in one outbound packet.

 To test this, try ping.   In Windows syntax the option -l nnnn says to use packets of size nnnn.   -f says to prevent packet fragmentation -- if the packet is too big, the ping will fail.   Find the largest value for -l that does not fail, and set your inside MTU to not more than that value.   I think some of the overhead is variable, so a value somewhat less than the maximum is probably wise.

Another option is to switch from TCP to UPD.   TCP guarantees error-free lossless communication, but adds bandwidth and latency to do so.   UTM doesn't care if an occasional packet gets lost, so it is faster because of less overhead and less latency.   Streaming services work best on UDP.   I use TCP, although for an RDP session, an occasional dropped packet might be as inconsequential as a screen blip or jumpy mouse movements.

"Best encryption" usually means "hardest to intercept", not "best performance".   So you need to decide what you want.  Better security requires more CPU, so you need to determine if your perceived performance problem is because of a CPU limitation.

For security:

• 3DES is considered barely adequate these days.   AES-128 is a minimum and AES-256 is preferred.
• MD5 and SHA1 are considered broken, and you should use one of the SHA2 variants.

But you also need to address MTU.   When a packet enters a VPN tunnel, it receives an envelope that makes the packet bigger.   If the new packet is too big for the outbound interface, it has to be fragmented.   Fragmentation costs you CPU overhead at both ends, consumes bandwidth because of extra packet overhead, and introduces latency,   To prevent this problem, you need to change the MTU on the INSIDE interfaces, so that the maximum packet size plus the VPN overhead still fits in one outbound packet.

 To test this, try ping.   In Windows syntax the option -l nnnn says to use packets of size nnnn.   -f says to prevent packet fragmentation -- if the packet is too big, the ping will fail.   Find the largest value for -l that does not fail, and set your inside MTU to not more than that value.   I think some of the overhead is variable, so a value somewhat less than the maximum is probably wise.

Another option is to switch from TCP to UPD.   TCP guarantees error-free lossless communication, but adds bandwidth and latency to do so.   UTM doesn't care if an occasional packet gets lost, so it is faster because of less overhead and less latency.   Streaming services work best on UDP.   I use TCP, although for an RDP session, an occasional dropped packet might be as inconsequential as a screen blip or jumpy mouse movements.