S/MIME and OpenPGP - risks and questions

Just ran into a complication.   I started to deploy encryption between two divisions, one that is behind UTM and one that is external to UTM.   The planned UTM implementation is for outbound encryption to occur at the client, and inbound decryption to occur at UTM.   

I was hoping to use UTM to generate keys for both user groups.   Happily, it will generate user keys for a user unrelated to any incoming mail domain.   Unhappily, it does not allow export of private keys, for security reasons.   I cannot argue with the security concern, but it means that I have to buy commercial certificates or build an in-house key or certificate generation process.

I have constructed a grid of encryption configuration options and the related configuration tasks.  It helped me to organize my thoughts and plan my strategy.  I include Password Protected Files and

My related assumptions and conclusions

• I assume that signing and encryption occur together. Similarly, signature verification and decryption occur together.
• Encrypting every outbound message to a user seems to be too much of a burden on the recipient, given an assumption that many messages will not require encryption, and that every decryption operation requires inconvenience to the recipient because they must activate their encryption key, with a password, a special program, or both. This makes me reluctant to use encryption at UTM, except for outbound SPX.  This decision requires me to bypass Antispam checking on outbound mail for authorized users, but I assume that my network is generally clean and is therefore risk I can accept.
• Allowing inbound decryption requires a significant weakening of defenses. I must either disable the global setting to “Quarantine encrypted and unscannable messages” or disable “Antispam Checking” for either the allowed users or the allowed senders.  Granting any exception to a sender, even a trusted one, seems unacceptable because sender identity is so difficult to prove decisively, and I do not want to risk granting an exception to a fraudulent user.   Disabling “Antispam Checking” for all message coming to a user seems equally unacceptable.   So disabling Quarantine of encrypted messages for all users seems the least objectionable.  However, this option also means that encrypted messages bypass all spam checking and that gives me heartburn.
• Given the above, my most appealing scenario is to encrypt at the client and decrypt at the UTM. This allows me to quarantine encrypted material that cannot be decrypted at UTM, and allows me to keep antispam defenses in place for all users.
• If I must accept password-protected files for any users, the Quarantine Encrypted/Unscannable option must be off. This includes SPX files from other Sophos clients, since SPX generates on a password-protected PDF.   One can hope that if an encrypted file is hostile, the user will not know how to decrypt it, or will choose not to do so. 
• PGP seems difficult to deploy and administer because of the difficulty of collecting and distributing keys. Implementing PGP in my suggested configuration is even more difficult, because it requires collecting the necessary keys at both the client and all of the user devices.   Keeping user devices and UTM will be challenging, especially if keyservers are not used because of the concerns in my previous post.
• Ultimately, the decision between S/MIME and PGP will be driven by the encryption method that communication partners are willing to deploy. If given the choice, S/MIME seems preferable to PGP.

I hope somebody finds this useful.   It took awhile to get my head around the complexity of these options.   I would love to hear from someone who already has this working.

Table 1:   Outbound Encryption, Options at Client device

Table 2:   Outbound Encryption, Options at UTM

Table 3:  Inbound Decryption, Options at Client

Table 4:  Inbound Decryption, Options at UTM

Discussed some of these issues at my ISC2 chapter meeting this week.   Since it was not a group of UTM users, the conversation focused on the risks of allowing encrypted messages to the end user.   We concluded that incoming encrypted messages really are a problem.   We also concluded that for key-based technologies like S/MIME and PGP, you need to attempt to control who acquires the public key:

• Because the encrypted payload has been allowed to bypass the perimeter defenses, you need really good endpoint protection.   Unfortunately, really good endpoint protection is really hard to construct.
  
  
• Once the public key is released to anyone for any purpose, the key owner cannot prevent its distribution outside of his control, although he could retire and replace it.   Therefore, we need to worry about unwanted senders of encrypted messages, just as we worry about unwanted senders of clear-text messages.   To my thinking, this means a whitelist of allowed sender-recipient pairs, a feature which UTM does not offer.   But authorized senders are difficult to define because of the problems with misrepresentation in email.
  
  
• Signing every outbound message with S/MIME or PGP seems like a terrible idea, because this releases your public key to people who do not need it.   Wikipedia says that really smart people use one keypair for signing, and a different keypair for encryption.   This would allow you to sign all outgoing emails (to prove authenticity) without worrying about the signature public key ever being used for unwanted incoming encryption. 
  
  
• If a PGP or S/MIME message is "triple-wrapped", the receiving spam filter could potentially use the outer signature to test for allowed-sender status.

Overall, this conversation ratified my perceptions that:

• Quarantining of encrypted/unscannable content is critically important for network security.
  
  
• PGP and S/MIME traffic must be decrypted at the UTM, not at the client device, if it is used at all.