Essential task grandpa-voter failed. Shutting down service. #6656

Posting this log as an FYI...

We had a node crash with the following error. The server came back up via systemd, and there have been no issues since.

Jan 31 13:27:42 GS-Prod-VA2-VM19 polkadot[646170]: 2023-01-31 13:27:42 ✨ Imported #16434322 (0x38c9…8e04)
Jan 31 13:27:44 GS-Prod-VA2-VM19 polkadot[646170]: 2023-01-31 13:27:44 � Idle (40 peers), best: #16434322 (0x38c9…8e04), finalized #16434319 (0xef3e…f2b0),>
Jan 31 13:27:48 GS-Prod-VA2-VM19 polkadot[646170]: 2023-01-31 13:27:48 ✨ Imported #16434323 (0x128b…93d7)
Jan 31 13:27:49 GS-Prod-VA2-VM19 polkadot[646170]: 2023-01-31 13:27:49 � Idle (40 peers), best: #16434323 (0x128b…93d7), finalized #16434320 (0x3cc2…dbb9),>
Jan 31 13:27:54 GS-Prod-VA2-VM19 polkadot[646170]: 2023-01-31 13:27:54 � Idle (40 peers), best: #16434323 (0x128b…93d7), finalized #16434321 (0x5e4b…323d),>
Jan 31 13:27:55 GS-Prod-VA2-VM19 polkadot[646170]: 2023-01-31 13:27:55 ✨ Imported #16434324 (0xba48…bbca)
Jan 31 13:27:55 GS-Prod-VA2-VM19 polkadot[646170]: 2023-01-31 13:27:55 Advanced session window for approvals update=Advanced { prev_window_start: 27835, pre>
Jan 31 13:27:59 GS-Prod-VA2-VM19 polkadot[646170]: 2023-01-31 13:27:59 � Idle (40 peers), best: #16434324 (0xba48…bbca), finalized #16434322 (0x38c9…8e04),>
Jan 31 13:28:00 GS-Prod-VA2-VM19 polkadot[646170]: 2023-01-31 13:28:00 ✨ Imported #16434325 (0xe07b…b367)
Jan 31 13:28:04 GS-Prod-VA2-VM19 polkadot[646170]: 2023-01-31 13:28:04 � Idle (40 peers), best: #16434325 (0xe07b…b367), finalized #16434322 (0x38c9…8e04),>
Jan 31 13:28:04 GS-Prod-VA2-VM19 polkadot[646170]: 2023-01-31 13:28:04 ♻️  Reorg on #16434325,0xe07b…b367 to #16434324,0x6e4d…69ec, common ancestor #16434323>
Jan 31 13:28:04 GS-Prod-VA2-VM19 polkadot[646170]: 2023-01-31 13:28:04 ✨ Imported #16434324 (0x6e4d…69ec)
Jan 31 13:28:04 GS-Prod-VA2-VM19 polkadot[646170]: 2023-01-31 13:28:04 GRANDPA voter error: safety invariant has been violated: SelectChain returned a final>
Jan 31 13:28:04 GS-Prod-VA2-VM19 polkadot[646170]: 2023-01-31 13:28:04 Essential task `grandpa-voter` failed. Shutting down service.
Jan 31 13:28:05 GS-Prod-VA2-VM19 polkadot[646170]: Error:
Jan 31 13:28:05 GS-Prod-VA2-VM19 polkadot[646170]:    0: Other: Essential task failed.
Jan 31 13:28:05 GS-Prod-VA2-VM19 polkadot[646170]: Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Jan 31 13:28:05 GS-Prod-VA2-VM19 polkadot[646170]: Run with RUST_BACKTRACE=full to include source snippets.
Jan 31 13:28:05 GS-Prod-VA2-VM19 systemd[1]: polkadot.service: Main process exited, code=exited, status=1/FAILURE
Jan 31 13:28:05 GS-Prod-VA2-VM19 systemd[1]: polkadot.service: Failed with result 'exit-code'.
Jan 31 13:28:05 GS-Prod-VA2-VM19 systemd[1]: polkadot.service: Consumed 2w 2d 22h 47min 18.325s CPU time.
Jan 31 13:29:05 GS-Prod-VA2-VM19 systemd[1]: polkadot.service: Scheduled restart job, restart counter is at 1.
Jan 31 13:29:05 GS-Prod-VA2-VM19 systemd[1]: Stopped Kusama Node.
Jan 31 13:29:05 GS-Prod-VA2-VM19 systemd[1]: polkadot.service: Consumed 2w 2d 22h 47min 18.325s CPU time.
Jan 31 13:29:05 GS-Prod-VA2-VM19 systemd[1]: Started Kusama Node.
Jan 31 13:29:05 GS-Prod-VA2-VM19 polkadot[706190]: 2023-01-31 13:29:05 ----------------------------
Jan 31 13:29:05 GS-Prod-VA2-VM19 polkadot[706190]: 2023-01-31 13:29:05 This chain is not in any way
Jan 31 13:29:05 GS-Prod-VA2-VM19 polkadot[706190]: 2023-01-31 13:29:05       endorsed by the
Jan 31 13:29:05 GS-Prod-VA2-VM19 polkadot[706190]: 2023-01-31 13:29:05      KUSAMA FOUNDATION
Jan 31 13:29:05 GS-Prod-VA2-VM19 polkadot[706190]: 2023-01-31 13:29:05 ----------------------------
Jan 31 13:29:05 GS-Prod-VA2-VM19 polkadot[706190]: 2023-01-31 13:29:05 Parity Polkadot
Jan 31 13:29:05 GS-Prod-VA2-VM19 polkadot[706190]: 2023-01-31 13:29:05 ✌️  version 0.9.37-645723987cf
Jan 31 13:29:05 GS-Prod-VA2-VM19 polkadot[706190]: 2023-01-31 13:29:05 ❤️  by Parity Technologies <admin@parity.io>, 2017-2023
Jan 31 13:29:05 GS-Prod-VA2-VM19 polkadot[706190]: 2023-01-31 13:29:05 � Chain specification: Kusama
Jan 31 13:29:05 GS-Prod-VA2-VM19 polkadot[706190]: 2023-01-31 13:29:05 �  Node name: �LuckyFriday-KSM-09�
Jan 31 13:29:05 GS-Prod-VA2-VM19 polkadot[706190]: 2023-01-31 13:29:05 � Role: AUTHORITY