SG 230 im HA 9.7.19-3 Slave bleibt im Syncing-Mode

Hallo Community,

ich habe vor 2 Monaten ein Problem mit dem Netzteil meines Slaves gehabt, wodurch der Slave sich nicht mehr starten lies (Display aus und kein anlaufen der Lüfter), habe dann ein anderes Netzteil verbaut und dann lief der Slave wieder. Allerdings hatte sich das fast 2 Wochen gezogen und der Slave war in der Zeit offline.

Nach dem Einbau des Netzteils, Slave hochgefahren und er wurde auch erkannt und hat mit dem Master gesynct, erstmal alles ok. Dann hab ich ein paar Tage später immer wieder Benachrichtigungen bekommen das sich die Root-Partition füllt (teilweise 94%). Dr. Google hat mir dann einige Ansätze geliefert (Log-Files durchschauen, PostGreSQL Datenbank wiederherstellen, Neustart) Ich habe dann die Datenbank wiederhergestellt und dem Slave einen Neustart gegeben. Danach hat erstmal alles auch gut ausgesehen. Master Active und Slave Ready, alles gut.

Jetzt hat mir die UTM Anfang der Woche das Update 9.720-5 angezeigt, was ich dann gestern angestoßen hatte. Er wollte als erstes auf dem Slave installieren und normalerweise öffnet sich dann ja das Infofenster mit dem Updateverlauf. Dieses war dann nur weiß ohne jegliche Info. Ich dachte es sei ein Darstellungsfehler und hab mich dann erstmal mit anderen Dingen weiter beschäftigt. Als ich aber nach 3 Stunden nachgesehen habe, war das Fenster immer noch weiß und es ging nichts voran.

Ich habe den Slave dann Neu gestartet und seither bleibt er beim Syncing hängen. Habe ich mir nun das Dateisystem zerschossen? Ich hoffe nicht.

Ich danke euch schon mal für eure Unterstützung :)

Hier das Log:

2024:10:22-10:18:54 mail-2 repctl[9805]: [i] setup_replication(278): checkinterval 300
2024:10:22-10:20:34 mail-2 repctl[9805]: [i] recheck(1057): got ALRM: replication recheck triggered Setup_replication_done = 0
2024:10:22-10:20:34 mail-2 repctl[9805]: [i] execute(1768): pg_ctl: no server running
2024:10:22-10:20:36 mail-2 ha_daemon[4277]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 680 36.289" name="HA control: cmd = 'sync start 1 database'"
2024:10:22-10:20:36 mail-2 ha_daemon[4277]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 681 36.289" name="Activating sync process for database on node 1"
2024:10:22-10:20:36 mail-2 repctl[9805]: [i] execute(1768): pg_ctl: PID file "/var/storage/pgsql92/data/postmaster.pid" does not exist
2024:10:22-10:20:36 mail-2 repctl[9805]: [i] execute(1768): Is server running?
2024:10:22-10:20:42 mail-2 repctl[9805]: [i] start_backup_mode(744): starting backup mode at 0000000100000AA600000052
2024:10:22-10:20:42 mail-2 ha_daemon[4277]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 682 42.897" name="HA control: cmd = 'sync start 1 database'"
2024:10:22-10:20:42 mail-2 ha_daemon[4277]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 683 42.897" name="Activating sync process for database on node 1"
2024:10:22-10:21:57 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 53 57.341" name="HA control: cmd = 'reboot 2'"
2024:10:22-10:21:57 mail-2 ha_daemon[4277]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 684 57.342" name="Received command to reboot!"
2024:10:22-10:22:01 mail-2 ha_daemon[4277]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 685 01.076" name="HA daemon shutting down (SIGTERM)"
2024:10:22-10:22:01 mail-2 ha_daemon[4277]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 686 01.076" name="Executing (nowait) /etc/init.d/ha_mode disable"
2024:10:22-10:22:01 mail-2 ha_daemon[4277]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 687 01.076" name="--- Node is disabled ---"
2024:10:22-10:22:01 mail-2 ha_daemon[4277]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 688 01.076" name="Executing (nowait) /etc/init.d/ha_mode shutdown"
2024:10:22-10:22:01 mail-2 ha_mode[11470]: calling shutdown
2024:10:22-10:22:01 mail-2 ha_mode[11469]: calling disable
2024:10:22-10:22:01 mail-2 ha_mode[11469]: disable: waiting for last ha_mode done
2024:10:22-10:22:01 mail-2 ha_mode[11469]: Switching disable mode
2024:10:22-10:22:01 mail-2 ha_mode[11469]: disable done (started at 10:22:01)
2024:10:22-10:22:01 mail-2 repctl[9805]: [i] execute(1768): pg_ctl: PID file "/var/storage/pgsql92/data/postmaster.pid" does not exist
2024:10:22-10:22:01 mail-2 repctl[9805]: [i] execute(1768): Is server running?
2024:10:22-10:22:01 mail-2 repctl[9805]: [i] execute(1768): starting server anyway
2024:10:22-10:22:01 mail-2 repctl[9805]: [i] execute(1768): waiting for server to start...
2024:10:22-10:22:01 mail-2 repctl[9805]: [i] execute(1768): .
2024:10:22-10:22:01 mail-2 ha_mode[11470]: shutdown: waiting for last ha_mode done
2024:10:22-10:22:01 mail-2 ha_mode[11470]: /var/mdw/scripts/confd-sync: /usr/local/bin/confd-sync stopped
2024:10:22-10:22:01 mail-2 ha_mode[11470]: shutdown done (started at 10:22:01)
2024:10:22-10:22:01 mail-2 ha_daemon[4277]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 689 01.188" name="HA daemon exits (SIGTERM)"
2024:10:22-10:22:02 mail-2 repctl[9805]: [i] execute(1768): stopped waiting
2024:10:22-10:22:02 mail-2 repctl[9805]: [i] execute(1768): pg_ctl: could not start server
2024:10:22-10:22:02 mail-2 repctl[9805]: [i] execute(1768): Examine the log output.
2024:10:22-10:22:02 mail-2 repctl[9805]: [i] terminate(2321): exit due to signal TERM
2024:10:22-10:22:04 mail-1 ha_daemon[4258]: id="38C1" severity="error" sys="System" sub="ha" seq="M: 54 04.746" name="Node 2 is dead, received no heart beats"
2024:10:22-10:22:04 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 55 04.747" name="Executing (wait) /usr/local/bin/confd-setha mode master master_ip 198.19.250.1 slave_ip ''"
2024:10:22-10:22:04 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 56 04.861" name="Executing (nowait) /etc/init.d/ha_mode topology_changed"
2024:10:22-10:22:04 mail-1 ha_mode[18461]: calling topology_changed
2024:10:22-10:22:04 mail-1 ha_mode[18461]: topology_changed: waiting for last ha_mode done
2024:10:22-10:22:05 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 57 05.258" name="Reading cluster configuration"
2024:10:22-10:22:05 mail-1 ha_mode[18461]: repctl[18489]: [i] daemonize_check(1480): daemonized, see syslog for further messages
2024:10:22-10:22:05 mail-1 repctl[18489]: [i] daemonize_check(1480): daemonized, see syslog for further messages
2024:10:22-10:22:05 mail-1 repctl[18489]: [i] daemonize_check(1497): trying to signal daemon and exit
2024:10:22-10:22:05 mail-1 repctl[28781]: [i] recheck(1057): got HUP: replication recheck triggered Setup_replication_done = 1
2024:10:22-10:22:05 mail-1 ha_mode[18461]: topology_changed done (started at 10:22:04)
2024:10:22-10:22:20 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 58 20.576" name="Monitoring interfaces for link beat: eth5 eth4 eth0"
2024:10:22-10:22:26 mail-1 ha_daemon[4258]: id="38A3" severity="debug" sys="System" sub="ha" seq="M: 59 26.738" name="Netlink: Lost link beat on eth3!"
2024:10:22-10:22:26 mail-1 conntrack-tools[4655]: no dedicated links available!
2024:10:22-10:22:35 mail-1 ha_daemon[4258]: id="38A3" severity="debug" sys="System" sub="ha" seq="M: 60 35.051" name="Netlink: Found link beat on eth3 again!"
2024:10:22-10:22:43 mail-1 conntrack-tools[4655]: no dedicated links available!
2024:10:22-10:22:43 mail-1 ha_daemon[4258]: id="38A3" severity="debug" sys="System" sub="ha" seq="M: 61 43.054" name="Netlink: Lost link beat on eth3!"
2024:10:22-10:22:53 mail-1 ha_daemon[4258]: id="38A3" severity="debug" sys="System" sub="ha" seq="M: 62 53.511" name="Netlink: Found link beat on eth3 again!"
2024:10:22-10:22:56 mail-1 conntrack-tools[4655]: no dedicated links available!
2024:10:22-10:22:56 mail-1 ha_daemon[4258]: id="38A3" severity="debug" sys="System" sub="ha" seq="M: 63 56.078" name="Netlink: Lost link beat on eth3!"
2024:10:22-10:22:58 mail-1 ha_daemon[4258]: id="38A3" severity="debug" sys="System" sub="ha" seq="M: 64 58.947" name="Netlink: Found link beat on eth3 again!"
2024:10:22-10:23:03 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 65 03.547" name="Access granted to remote node 2!"
2024:10:22-10:23:03 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 66 03.636" name="Node 2 changed version! 0.000000 -> 9.719003"
2024:10:22-10:23:03 mail-1 ha_daemon[4258]: id="38C0" severity="info" sys="System" sub="ha" seq="M: 67 03.636" name="Node 2 is alive"
2024:10:22-10:23:03 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 68 03.636" name="Node 2 changed state: DEAD(2048) -> SYNCING(2)"
2024:10:22-10:23:03 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 69 03.636" name="Node 2 changed role: DEAD -> SLAVE"
2024:10:22-10:23:03 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 70 03.636" name="Executing (wait) /usr/local/bin/confd-setha mode master master_ip 198.19.250.1 slave_ip 198.19.250.2"
2024:10:22-10:23:03 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 71 03.740" name="Executing (nowait) /etc/init.d/ha_mode topology_changed"
2024:10:22-10:23:03 mail-1 ha_mode[18809]: calling topology_changed
2024:10:22-10:23:03 mail-1 ha_mode[18809]: topology_changed: waiting for last ha_mode done
2024:10:22-10:23:04 mail-1 repctl[28781]: [i] recheck(1057): got HUP: replication recheck triggered Setup_replication_done = 1
2024:10:22-10:23:04 mail-1 ha_mode[18809]: repctl[18826]: [i] daemonize_check(1480): daemonized, see syslog for further messages
2024:10:22-10:23:04 mail-1 repctl[18826]: [i] daemonize_check(1480): daemonized, see syslog for further messages
2024:10:22-10:23:04 mail-1 repctl[18826]: [i] daemonize_check(1497): trying to signal daemon and exit
2024:10:22-10:23:04 mail-1 ha_mode[18809]: topology_changed done (started at 10:23:03)
2024:10:22-10:23:04 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 72 04.318" name="Reading cluster configuration"
2024:10:22-10:23:08 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 73 08.746" name="Set syncing.files for node 2"
2024:10:22-10:23:11 mail-1 ha_daemon[4258]: id="38A1" severity="warn" sys="System" sub="ha" seq="M: 74 11.899" name="filesync_end(): initial sync failed, status = 0x200"
2024:10:22-10:23:11 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 75 11.899" name="Clear syncing.files for node 2"
2024:10:22-10:23:19 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 76 19.757" name="Monitoring interfaces for link beat: eth5 eth4 eth0"
2024:10:22-10:25:37 mail-2 repctl[4285]: [i] execute(1768): pg_ctl: PID file "/var/storage/pgsql92/data/postmaster.pid" does not exist<30>Oct 22 10:25:37 repctl[4285]: [i] execute(1768): Is server running?
2024:10:22-10:25:37 mail-2 repctl[4285]: [e] clear_backup(983): found leftover backup_label
2024:10:22-10:25:38 mail-2 repctl[4285]: [i] stop_backup_mode(765): stopped backup mode at 0000000100000AA600000053
2024:10:22-10:25:39 mail-2 repctl[4285]: [i] start_backup_mode(744): starting backup mode at 0000000100000AA600000055
2024:10:22-10:25:39 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 37 39.551" name="HA control: cmd = 'sync start 1 database'"
2024:10:22-10:25:39 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 38 39.551" name="Activating sync process for database on node 1"
2024:10:22-10:26:54 mail-2 repctl[4285]: [i] stop_backup_mode(765): stopped backup mode at 0000000100000AA600000055
2024:10:22-10:26:54 mail-2 repctl[4285]: [i] execute(1768): waiting for server to start...
2024:10:22-10:26:54 mail-2 repctl[4285]: [i] execute(1768): .
2024:10:22-10:26:55 mail-2 repctl[4285]: [i] execute(1768): stopped waiting
2024:10:22-10:26:55 mail-2 repctl[4285]: [i] execute(1768): pg_ctl: could not start server
2024:10:22-10:26:55 mail-2 repctl[4285]: [i] execute(1768): Examine the log output.
2024:10:22-10:26:55 mail-2 repctl[4285]: [e] prepare_secondary(352): prepare_secondary: Local db(pg_ctl) start failed
2024:10:22-10:26:56 mail-2 repctl[4285]: [i] start_backup_mode(744): starting backup mode at 0000000100000AA600000057
2024:10:22-10:26:56 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 39 56.838" name="HA control: cmd = 'sync start 1 database'"
2024:10:22-10:26:56 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 40 56.838" name="Activating sync process for database on node 1"
2024:10:22-10:27:35 mail-2 repctl[4285]: [i] stop_backup_mode(765): stopped backup mode at 0000000100000AA600000057
2024:10:22-10:27:35 mail-2 repctl[4285]: [i] execute(1768): waiting for server to start...
2024:10:22-10:27:35 mail-2 repctl[4285]: [i] execute(1768): .
2024:10:22-10:27:36 mail-2 repctl[4285]: [i] execute(1768): stopped waiting
2024:10:22-10:27:36 mail-2 repctl[4285]: [i] execute(1768): pg_ctl: could not start server
2024:10:22-10:27:36 mail-2 repctl[4285]: [i] execute(1768): Examine the log output.
2024:10:22-10:27:36 mail-2 repctl[4285]: [e] prepare_secondary(352): prepare_secondary: Local db(pg_ctl) start failed
2024:10:22-10:27:36 mail-2 repctl[4285]: [i] start_backup_mode(744): starting backup mode at 0000000100000AA600000059
2024:10:22-10:27:36 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 41 36.980" name="HA control: cmd = 'sync start 1 database'"
2024:10:22-10:27:36 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 42 36.980" name="Activating sync process for database on node 1"
2024:10:22-10:28:04 mail-1 repctl[28781]: [i] recheck(1057): got ALRM: replication recheck triggered Setup_replication_done = 1
2024:10:22-10:28:08 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 77 08.746" name="Set syncing.files for node 2"
2024:10:22-10:28:10 mail-1 ha_daemon[4258]: id="38A1" severity="warn" sys="System" sub="ha" seq="M: 78 10.813" name="filesync_end(): initial sync failed, status = 0x200"
2024:10:22-10:28:10 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 79 10.813" name="Clear syncing.files for node 2"
2024:10:22-10:28:14 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 43 14.296" name="Monitoring interfaces for link beat: eth5 eth4 eth0"
2024:10:22-10:28:15 mail-2 repctl[4285]: [i] stop_backup_mode(765): stopped backup mode at 0000000100000AA600000059
2024:10:22-10:28:15 mail-2 repctl[4285]: [i] execute(1768): waiting for server to start....
2024:10:22-10:28:16 mail-2 repctl[4285]: [i] execute(1768): stopped waiting
2024:10:22-10:28:16 mail-2 repctl[4285]: [i] execute(1768): pg_ctl: could not start server
2024:10:22-10:28:16 mail-2 repctl[4285]: [i] execute(1768): Examine the log output.
2024:10:22-10:28:16 mail-2 repctl[4285]: [e] prepare_secondary(352): prepare_secondary: Local db(pg_ctl) start failed
2024:10:22-10:28:16 mail-2 repctl[4285]: [c] prepare_secondary(360): failed to get database up, waiting for retry
2024:10:22-10:28:16 mail-2 repctl[4285]: [c] setup_replication(274): setup_replication was not properly executed
2024:10:22-10:28:16 mail-2 repctl[4285]: [i] setup_replication(278): checkinterval 300
2024:10:22-10:33:04 mail-1 repctl[28781]: [i] recheck(1057): got ALRM: replication recheck triggered Setup_replication_done = 1
2024:10:22-10:33:08 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 80 08.746" name="Set syncing.files for node 2"
2024:10:22-10:33:10 mail-1 ha_daemon[4258]: id="38A1" severity="warn" sys="System" sub="ha" seq="M: 81 10.772" name="filesync_end(): initial sync failed, status = 0x300"
2024:10:22-10:33:10 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 82 10.772" name="Clear syncing.files for node 2"
2024:10:22-10:33:16 mail-2 repctl[4285]: [i] recheck(1057): got ALRM: replication recheck triggered Setup_replication_done = 0
2024:10:22-10:33:16 mail-2 repctl[4285]: [i] execute(1768): pg_ctl: no server running
2024:10:22-10:33:18 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 44 18.278" name="HA control: cmd = 'sync start 1 database'"
2024:10:22-10:33:18 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 45 18.278" name="Activating sync process for database on node 1"
2024:10:22-10:33:18 mail-2 repctl[4285]: [i] execute(1768): pg_ctl: PID file "/var/storage/pgsql92/data/postmaster.pid" does not exist
2024:10:22-10:33:18 mail-2 repctl[4285]: [i] execute(1768): Is server running?
2024:10:22-10:33:25 mail-2 repctl[4285]: [i] start_backup_mode(744): starting backup mode at 0000000100000AA60000005C
2024:10:22-10:33:25 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 46 25.307" name="HA control: cmd = 'sync start 1 database'"
2024:10:22-10:33:25 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 47 25.307" name="Activating sync process for database on node 1"
2024:10:22-10:35:01 mail-2 repctl[4285]: [i] stop_backup_mode(765): stopped backup mode at 0000000100000AA60000005C
2024:10:22-10:35:01 mail-2 repctl[4285]: [i] execute(1768): waiting for server to start....
2024:10:22-10:35:02 mail-2 repctl[4285]: [i] execute(1768): stopped waiting
2024:10:22-10:35:02 mail-2 repctl[4285]: [i] execute(1768): pg_ctl: could not start server
2024:10:22-10:35:02 mail-2 repctl[4285]: [i] execute(1768): Examine the log output.
2024:10:22-10:35:02 mail-2 repctl[4285]: [e] prepare_secondary(352): prepare_secondary: Local db(pg_ctl) start failed
2024:10:22-10:35:10 mail-2 repctl[4285]: [i] start_backup_mode(744): starting backup mode at 0000000100000AA60000005E
2024:10:22-10:35:10 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 48 10.359" name="HA control: cmd = 'sync start 1 database'"
2024:10:22-10:35:10 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 49 10.359" name="Activating sync process for database on node 1"
2024:10:22-10:35:50 mail-2 repctl[4285]: [i] stop_backup_mode(765): stopped backup mode at 0000000100000AA60000005E
2024:10:22-10:35:50 mail-2 repctl[4285]: [i] execute(1768): waiting for server to start...
2024:10:22-10:35:50 mail-2 repctl[4285]: [i] execute(1768): .
2024:10:22-10:35:51 mail-2 repctl[4285]: [i] execute(1768): stopped waiting
2024:10:22-10:35:51 mail-2 repctl[4285]: [i] execute(1768): pg_ctl: could not start server
2024:10:22-10:35:51 mail-2 repctl[4285]: [i] execute(1768): Examine the log output.
2024:10:22-10:35:51 mail-2 repctl[4285]: [e] prepare_secondary(352): prepare_secondary: Local db(pg_ctl) start failed
2024:10:22-10:35:59 mail-2 repctl[4285]: [i] start_backup_mode(744): starting backup mode at 0000000100000AA600000060
2024:10:22-10:35:59 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 50 59.463" name="HA control: cmd = 'sync start 1 database'"
2024:10:22-10:35:59 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 51 59.463" name="Activating sync process for database on node 1"
2024:10:22-10:36:39 mail-2 repctl[4285]: [i] stop_backup_mode(765): stopped backup mode at 0000000100000AA600000060
2024:10:22-10:36:39 mail-2 repctl[4285]: [i] execute(1768): waiting for server to start...
2024:10:22-10:36:39 mail-2 repctl[4285]: [i] execute(1768): .
2024:10:22-10:36:40 mail-2 repctl[4285]: [i] execute(1768): stopped waiting
2024:10:22-10:36:40 mail-2 repctl[4285]: [i] execute(1768): pg_ctl: could not start server
2024:10:22-10:36:40 mail-2 repctl[4285]: [i] execute(1768): Examine the log output.
2024:10:22-10:36:40 mail-2 repctl[4285]: [e] prepare_secondary(352): prepare_secondary: Local db(pg_ctl) start failed
2024:10:22-10:36:40 mail-2 repctl[4285]: [c] prepare_secondary(360): failed to get database up, waiting for retry
2024:10:22-10:36:40 mail-2 repctl[4285]: [c] setup_replication(274): setup_replication was not properly executed
2024:10:22-10:36:40 mail-2 repctl[4285]: [i] setup_replication(278): checkinterval 300
2024:10:22-10:38:04 mail-1 repctl[28781]: [i] recheck(1057): got ALRM: replication recheck triggered Setup_replication_done = 1
2024:10:22-10:38:08 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 83 08.746" name="Set syncing.files for node 2"
2024:10:22-10:38:11 mail-1 ha_daemon[4258]: id="38A1" severity="warn" sys="System" sub="ha" seq="M: 84 11.100" name="filesync_end(): initial sync failed, status = 0x300"
2024:10:22-10:38:11 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 85 11.100" name="Clear syncing.files for node 2"
2024:10:22-10:38:16 mail-2 repctl[4285]: [i] recheck(1057): got ALRM: replication recheck triggered Setup_replication_done = 0
2024:10:22-10:38:16 mail-2 repctl[4285]: [i] execute(1768): pg_ctl: no server running
2024:10:22-10:38:20 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 52 20.234" name="HA control: cmd = 'sync start 1 database'"
2024:10:22-10:38:20 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 53 20.234" name="Activating sync process for database on node 1"
2024:10:22-10:38:20 mail-2 repctl[4285]: [i] execute(1768): pg_ctl: PID file "/var/storage/pgsql92/data/postmaster.pid" does not exist
2024:10:22-10:38:20 mail-2 repctl[4285]: [i] execute(1768): Is server running?
2024:10:22-10:38:26 mail-2 repctl[4285]: [i] start_backup_mode(744): starting backup mode at 0000000100000AA600000062
2024:10:22-10:38:26 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 54 26.883" name="HA control: cmd = 'sync start 1 database'"
2024:10:22-10:38:26 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 55 26.883" name="Activating sync process for database on node 1"
2024:10:22-10:39:21 mail-2 repctl[4285]: [i] stop_backup_mode(765): stopped backup mode at 0000000100000AA600000062
2024:10:22-10:39:21 mail-2 repctl[4285]: [i] execute(1768): waiting for server to start....
2024:10:22-10:39:22 mail-2 repctl[4285]: [i] execute(1768): stopped waiting
2024:10:22-10:39:22 mail-2 repctl[4285]: [i] execute(1768): pg_ctl: could not start server
2024:10:22-10:39:22 mail-2 repctl[4285]: [i] execute(1768): Examine the log output.
2024:10:22-10:39:22 mail-2 repctl[4285]: [e] prepare_secondary(352): prepare_secondary: Local db(pg_ctl) start failed
2024:10:22-10:39:29 mail-2 repctl[4285]: [i] start_backup_mode(744): starting backup mode at 0000000100000AA600000064
2024:10:22-10:39:29 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 56 29.736" name="HA control: cmd = 'sync start 1 database'"
2024:10:22-10:39:29 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 57 29.736" name="Activating sync process for database on node 1"
2024:10:22-10:40:09 mail-2 repctl[4285]: [i] stop_backup_mode(765): stopped backup mode at 0000000100000AA600000064
2024:10:22-10:40:09 mail-2 repctl[4285]: [i] execute(1768): waiting for server to start....
2024:10:22-10:40:10 mail-2 repctl[4285]: [i] execute(1768): stopped waiting
2024:10:22-10:40:10 mail-2 repctl[4285]: [i] execute(1768): pg_ctl: could not start server
2024:10:22-10:40:10 mail-2 repctl[4285]: [i] execute(1768): Examine the log output.
2024:10:22-10:40:10 mail-2 repctl[4285]: [e] prepare_secondary(352): prepare_secondary: Local db(pg_ctl) start failed
2024:10:22-10:40:17 mail-2 repctl[4285]: [i] start_backup_mode(744): starting backup mode at 0000000100000AA600000066
2024:10:22-10:40:17 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 58 17.481" name="HA control: cmd = 'sync start 1 database'"
2024:10:22-10:40:17 mail-2 ha_daemon[4238]: id="38A0" severity="info" sys="System" sub="ha" seq="S: 59 17.481" name="Activating sync process for database on node 1"
2024:10:22-10:40:57 mail-2 repctl[4285]: [i] stop_backup_mode(765): stopped backup mode at 0000000100000AA600000066
2024:10:22-10:40:57 mail-2 repctl[4285]: [i] execute(1768): waiting for server to start...
2024:10:22-10:40:57 mail-2 repctl[4285]: [i] execute(1768): .
2024:10:22-10:40:58 mail-2 repctl[4285]: [i] execute(1768): stopped waiting
2024:10:22-10:40:58 mail-2 repctl[4285]: [i] execute(1768): pg_ctl: could not start server
2024:10:22-10:40:58 mail-2 repctl[4285]: [i] execute(1768): Examine the log output.
2024:10:22-10:40:58 mail-2 repctl[4285]: [e] prepare_secondary(352): prepare_secondary: Local db(pg_ctl) start failed
2024:10:22-10:40:58 mail-2 repctl[4285]: [c] prepare_secondary(360): failed to get database up, waiting for retry
2024:10:22-10:40:58 mail-2 repctl[4285]: [c] setup_replication(274): setup_replication was not properly executed
2024:10:22-10:40:58 mail-2 repctl[4285]: [i] setup_replication(278): checkinterval 300
2024:10:22-10:43:04 mail-1 repctl[28781]: [i] recheck(1057): got ALRM: replication recheck triggered Setup_replication_done = 1
2024:10:22-10:43:08 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 86 08.746" name="Set syncing.files for node 2"
2024:10:22-10:43:10 mail-1 ha_daemon[4258]: id="38A1" severity="warn" sys="System" sub="ha" seq="M: 87 10.884" name="filesync_end(): initial sync failed, status = 0x200"
2024:10:22-10:43:10 mail-1 ha_daemon[4258]: id="38A0" severity="info" sys="System" sub="ha" seq="M: 88 10.884" name="Clear syncing.files for node 2"