Unable to ssh to slave node

Further information:  this evening, I ssh'd to the UTM and connected to node 2 which was the active node at the time.  As root, I issued a passwd loginuser command to reset the password once more on that node.

I then triggered a takeover via the web UI, and to my surprise, found that my SSH session remained active and connected to node 2, which was now the slave.  I fired up another ssh session to the UTM, which connected to node 1 which was now master, and tried hautils ssh once more, specifying the password I had just set on node 2.

It still failed.

So I reset the password on node2 again, in case it had somehow gotten overwritten when the roles switched, but no, I still get Permission denied when attempting to   ha_utils ssh from the active master node.

It sure seems to me like something ain't working as designed!

<M> astaro:/root # ha_utils status
- Status -----------------------------------------------------------------------
Current mode: HA MASTER with id 2 in state ACTIVE
-- Nodes -----------------------------------------------------------------------
MASTER: 2 esxi55 198.19.250.2 9.705003 ACTIVE since Tue May 11 00:51:50 2021
SLAVE: 1 esxi3 198.19.250.1 9.705003 ACTIVE since Tue May 11 00:56:50 2021
-- Load ------------------------------------------------------------------------
Node  2: [1m] 0.15  [5m] 0.07  [15m] 0.06
Node  1: [1m] 0.00  [5m] 0.02  [15m] 0.05

- Kernel -----------------------------------------------------------------------
Current mode: enabled master
interface: eth3
Local ID: 198.19.250.2
debug: off
verbose: off
ppp sync: off
port smtp: 25
port pop3: 8110
port ftp: 2121

- IPSec ------------------------------------------------------------------------
000 HA System active on eth3/224.0.0.82. Current mode is Master. Seqdiff in: 256 Seqdiff out: 4096

- PostgreSQL ------------------------------------------------------------------------
 primary | standby |     lag      |  bytelag
---------+---------+--------------+----------
       2 |       1 | 00:00:00.993 |        0

<M> astaro:/root # passwd loginuser
Changing password for loginuser.
New Password:
Reenter New Password:
Password changed.

<M> astaro:/root #
<S> astaro:/root # ha_utils status
- Status -----------------------------------------------------------------------
Current mode: HA SLAVE with id 2 in state ACTIVE
-- Nodes --------------------------------------------------- eth0 alive --------
SLAVE: 2 esxi55 198.19.250.2 9.705003 ACTIVE since Wed May 12 02:04:36 2021
MASTER: 1 esxi3 198.19.250.1 9.705003 ACTIVE since Wed May 12 01:59:36 2021
-- Load ------------------------------------------------------------------------
Node  2: [1m] 0.00  [5m] 0.01  [15m] 0.08
Node  1: [1m] 0.01  [5m] 0.04  [15m] 0.09

- Kernel -----------------------------------------------------------------------
Current mode: enabled slave
interface: eth3
Local ID: 198.19.250.2
Master ID: 1
debug: off
verbose: off
ppp sync: off
port smtp: 25
port pop3: 8110
port ftp: 2121

- IPSec ------------------------------------------------------------------------
000 HA System active on eth3/224.0.0.82. Current mode is Slave. Seqdiff in: 256 Seqdiff out: 4096

- PostgreSQL ------------------------------------------------------------------------
 primary | standby |     lag      |  bytelag
---------+---------+--------------+----------
       1 |       2 | 00:00:00.587 |        0
<S> astaro:/root # passwd loginuser
Changing password for loginuser.
New Password:
Reenter New Password:
Password changed.
<S> astaro:/root #

<M> astaro:/root # ha_utils status
- Status -----------------------------------------------------------------------
Current mode: HA MASTER with id 1 in state ACTIVE
-- Nodes -----------------------------------------------------------------------
MASTER: 1 esxi3 198.19.250.1 9.705003 ACTIVE since Wed May 12 01:59:36 2021
SLAVE: 2 esxi55 198.19.250.2 9.705003 ACTIVE since Wed May 12 02:04:37 2021
-- Load ------------------------------------------------------------------------
Node  1: [1m] 0.05  [5m] 0.04  [15m] 0.05
Node  2: [1m] 0.06  [5m] 0.03  [15m] 0.05

- Kernel -----------------------------------------------------------------------
Current mode: enabled master
interface: eth3
Local ID: 198.19.250.1
debug: off
verbose: off
ppp sync: off
port smtp: 25
port pop3: 8110
port ftp: 2121

- IPSec ------------------------------------------------------------------------
000 HA System active on eth3/224.0.0.82. Current mode is Master. Seqdiff in: 256 Seqdiff out: 4096

- PostgreSQL ------------------------------------------------------------------------
 primary | standby |     lag      |  bytelag
---------+---------+--------------+----------
       1 |       2 | 00:00:00.299 |        0
<M> astaro:/root # ha_utils ssh

Connecting to slave 198.19.250.2
loginuser@198.19.250.2's password:
Permission denied, please try again.
loginuser@198.19.250.2's password:

<M> astaro:/root #

Hi JonEtkins,

Did you configure public key authentication under Management > System settings > Shell Access > Authentication? 

To change it to root user, are you using "su -"? 

What is the current system status under Management > High Availability > System Status? 

Thanks,

I have both password authentication and public key authentication enabled.  I have "Allow root login" set to "Root access but only with SSH key."  I have an authorized key specified for root, but none for loginuser.  When I connect to the active node, I log in directly as root, using that key, so there is no need to su.  HA status is as follows:

Oh, and in case it's relevant, I have the ssh port set to 2222.

Thanks,
  Jon

Hi JonEtkins,

Would it be possible for you to turn off the public key authentication for testing and try to ssh into the secondary unit and let us know if that works. 

Thanks,

Well we're getting somewhere, but I'm not sure where. :)  After disabling key authentication, I am of course unable to log in to the master as root, so I tried logging in as loginuser, and...Access denied.

So it would appear that the problem is not tied to the secondary node ssh feature, but is instead some sort of problem authenticating or authorizing the loginuser account for any ssh access.  I have double-checked that the password I have set is correct, and that it meets all the criteria I have set under Authentication Services - Advanced (length is 8 chars and it contains at least one character from each of the four groups).  The account is not locked, and is not denied in /etc/ssh/sshd_config.

Not sure where to go next.  I don't see any errors in /var/log/login.log or system.log, but I do get email notification of the failed attempt.  I could change the debug level in /etc/ssh/sshd_config, but I don't know how to restart the sshd daemon since I don't see an entry for it in /etc/init.d.

Hi, Harsh.  Not sure if you've seen my update below; is there some further tracing and/or logging I can enable to try to see what's blocking this login?

Hi JonEtkins,

Apologies for the delayed response. 

After removing the public key authentication, you should be able to ssh into the secondary unit, at least I was able to ssh to the secondary unit. 

If you want to restart the sshd service use the following command: 

<M> h_patel:/root # /var/mdw/scripts/sshd restart
:: Stopping SSH done
:: Starting SSH starting SSH daemon done
:: Restarting SSH
<M> h_patel:/root #

Authentication logs are stored in aua.log file. 

Thanks,

Hi JonEtkins,

Apologies for the delayed response. 

After removing the public key authentication, you should be able to ssh into the secondary unit, at least I was able to ssh to the secondary unit. 

If you want to restart the sshd service use the following command: 

<M> h_patel:/root # /var/mdw/scripts/sshd restart
:: Stopping SSH done
:: Starting SSH starting SSH daemon done
:: Restarting SSH
<M> h_patel:/root #

Authentication logs are stored in aua.log file. 

Thanks,

Well I found the cause of the problem, but not sure how best to resolve it.  Apparently it's because I have OTP authentication enabled for myself and the SuperAdmins group, and I guess Loginuser must be included in that group even though it's not visible in the UI. 

2021:05:21-11:45:18 astaro-1 aua[3871]: id="3006" severity="info" sys="System" sub="auth" name="Running _cleanup_up_children with max_run_time: 20"
2021:05:21-11:45:18 astaro-1 aua[2530]: id="3006" severity="info" sys="System" sub="auth" name="OTP verification did not succeed, failing authentication."
2021:05:21-11:45:18 astaro-1 aua[2530]: id="3005" severity="warn" sys="System" sub="auth" name="Authentication failed" srcip="0.0.0.0" host="xxxxxxxx" user="loginuser" caller="sshd" reason="DENIED"

I tried to log in to the user portal as loginuser, but was unable, so I'm not sure how I can obtain an OTP for that account.  I can work around it by disabling OTP for shell access, but I would rather not, so I'm open to other suggestions.  

Circling back to this after a couple of weeks.  Does anyone have any suggestions for a way around the the inability to set an OTP for loginuser?