OT: Linux guru's

[the_msp]

I have a linux server (on prosumer hardware), debian stretch 10.

There was some utility works in my area and they hit the power line. The server is on a UPS but it drained as I was away.

Ever since, the server only boots for a while then shuts itself down. About 17 minutes after booting.

I pulled the log from dmesg, attached, can anyone spot anything that points to a specific piece of hardware before I start randomly replacing components?

I've tried it on mains only, the UPS isn't the issue. All the other hardware in the same server rack is fine.

 

[drbitboy]

it may not be shutting down, it's looks like it's going into suspend mode. See the entry at around ~1244s, 994s (there's the 17 minutes) after the last boot-ish entry at ~250s.

It does seem to wake again though. It sounds like it is not normally connected (comms, not power) to the UPS, so it can do a clean shutdown when UPS loses the mains.

What are the power settings? One thing that would trigger this would be the lid of a laptop closing, but this does not sound like a laptop.


This entry

[ 2464.311619] Suspending console(s) (use no_console_suspend to debug)

​

suggests there might be a way to get more diagnostics.

 

[drforsythe]

I am not a Linux guru. It seems odd that it stays running for a time period then shuts down. Is that time repeatable? If it is repeatable, is it possible it is looking for a good signal from the UPS? Are there any tasks designed to run on time if a certain input is seen (or not seen)? I am obviously just guessing. The fact that it runs for a while then shuts down after that much time is intriguing.

 

[drbitboy]

This link is a google search that might allow you to rule in or rule out whether suspend is an issue.

 

[the_msp]

Hi doctors Brian and Forsythe,

yes Brian you are correct. I checked the syslog and saw this:
Feb 11 16:49:46 octo gnome-shell[1035]: Screen lock is locked down, not locking
Feb 11 16:49:46 octo systemd[1]: Reached target Sleep.
Feb 11 16:49:46 octo systemd[1]: Starting Suspend...
Feb 11 16:49:46 octo systemd-sleep[3551]: Suspending system...
Feb 11 16:49:46 octo kernel: [ 3670.491866] PM: suspend entry (deep)
Feb 11 16:49:46 octo kernel: [ 3670.491868] PM: Syncing filesystems ... done.


I google-foo'd and ran this:

matrix@octo:~$ sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target
Created symlink /etc/systemd/system/sleep.target → /dev/null.
Created symlink /etc/systemd/system/suspend.target → /dev/null.
Created symlink /etc/systemd/system/hibernate.target → /dev/null.
Created symlink /etc/systemd/system/hybrid-sleep.target → /dev/null.


Then went to the server (it has Gnome too) and checked the GUI, the only thing in Power settings is display off time which is set to never.

Odd that it put in a shutdown timer after a power cut by itself...

 

[the_msp]

I am not a Linux guru. It seems odd that it stays running for a time period then shuts down. Is that time repeatable? If it is repeatable, is it possible it is looking for a good signal from the UPS? Are there any tasks designed to run on time if a certain input is seen (or not seen)? I am obviously just guessing. The fact that it runs for a while then shuts down after that much time is intriguing.

There is no link from the UPS to initiate a shutdown, just the C13/C14 cable.

 

[drbitboy]

Maybe a power supply issue? how hard would it be to replace the PS?

 

[the_msp]

Not too hard, have a spare here, just need to rack it out (4U chassis). Trying software checks before I do that

 

[drbitboy]

P.S. I am only guessing; the suspend-related text may be a canard.

See this also: https://01.org/blogs/rzhang/2015/best-practice-debug-linux-suspend/hibernate-issues

 

[Peter Nachtwey]

Use SSH or similar to see if the server is still running. SSH is used by IT people to access remote servers or network attached storage because NAS normally don't have a user interface.

 

[dmroeder]

Maybe a power supply issue? how hard would it be to replace the PS?

I have always built computers myself, for probably 20 years. Not just for me, but for friends and family. Computers shutting down without being told has almost always been, if not always, the power supply.

Could be that after about 17 minutes, it is sufficiently warm and bites the dust.

 

[Peter Nachtwey]

I have always built computers myself, for probably 20 years. Not just for me, but for friends and family. Computers shutting down without being told has almost always been, if not always, the power supply.

Could be that after about 17 minutes, it is sufficiently warm and bites the dust.

In that case SSH won't work either.

 

[dmroeder]

In that case SSH won't work either.

Oh I agree. I was assuming that it was actually shutting down. Maybe I didn't fully comprehend what is going on.

EDIT: So it is not shutting down and the server can still be accessed via SSH?

 

[the_msp]

I am using PuTTy to SSH in, after it boots. I ran a ping with timestamp in PowerShell on my Win 10 desktop to determine the ~17 minutes.

So, it is shutting down, SSH tunnel disconnects.

I have also been building computers since 2003, just not as au fait with Linux as I am with Windoze.

I'll do the PSU soon.

 

[chelton]

Have you checked the syslog? /var/log/syslog it might give you some insight into whats calling shutdown