Different tunnels for different users

I have AWS VPC tunnels set up to four different AWS accounts (test, staging, production, build tools). These are working great, but I would now like to restrict access to these networks based on group or user. Is that possible? Can I create different tunnel configurations on a per group basis, or different firewall rules based on the group? The goal is to create a groups as follows:

Devs - Can access test and build-tools only

QA - Can access test and staging only

Admin - Can access all networks

Is that even possible?

Thanks,

Martin