Postgresql kept shutting down at odd times. Then it became predictable. I googled, went through mailing lists, archives, old threads, Stack overflow, friends, family, my bosses, my old colleagues, forums, chat groups and paid redhat support.

Apparently, I'm the only one and the first on this planet to have this problem. And it's scary.

Did you ulimit the postgres user? Like cpu time?

@nbamaral I haven't tried that. What I have tried:

- checked syslogs, pgsql logs, kern logs
- tried dmesg but it looks cryptic
- checked the CRM resource and logs
- checked pacemakerd git clusterlabs for solution
- checked space, free, top, ps -ef
- tried to get dtrace utility installed but that didn't work.
- tried to tcpdump during the predictable db shutdown but yielded nothing

The only clue I got is the child_timeout_callback where a lrmd PID timed out and won't die.

So I guess I should try ulimit tomorrow

@agentQ
Noo, don't try that!
I was asking if you by any chance did that, it has the effect of killing the processes owned by the user once the allotted cpu time is reached.
It's an often overlooked aspect of Unix, I'd call it legacy nowadays, but since you were looking for weird causes, I thought you could check limits.conf.

The funny thing is this rant is one of the few results 😄

@nbamaral lol well before I tried ulimit, I looked at what it can do. Basically it did help me look up if there are any restrictions. Sadly no. Most of it is unlimited.

Stuck on this bug ticket since June 2017. Sigh.

@agentQ
Good luck man,
Around that time, I tripped over a bug in glibc too, that basically let the system without be able to euid a process, no non root services would run.
Still haven't completely solved that mystery, and it bited me twice on the same server. 😊