Let's see if this kitty survives the code review and makes it to production.

Waaaay too many but let's go with this one for now.

At my previous job there was a web application which was generating about 1gb of log data a second. Server was full and the 'fullstack engineers' we called had zero clue about backend stuff and couldn't fix it.

Me and another engineer worked our asses off to figure this out but eventually the logging stopped and it went back to normal.

Great, right?

For that moment. I was the on-call server engineer and at like 3am I got called awake because this shit was happening again.
Sleep drunk with my phone I ssh'd into the server, not sure about what to do at first but then suddenly: let's chattr the goddamn log file...

$ chattr +i /var/log/logfile

Bam, worked, done, back to sleep.

(this comment + param marks the file in a way that it can only be read until the mark is removed, so you can't write to it or move it or remove it or whatever)

TL;DR my first vps got hacked, the attacker flooded my server log when I successfully discovered and removed him so I couldn't use my server anymore because the log was taking up all the space on the server.

The first Linux VPN I ever had (when I was a noob and had just started with vServers and Linux in general, obviously) got hacked within 2 moths since I got it.
As I didn't knew much about securing a Linux server, I made all these "rookie" mistakes: having ssh on port 22, allowing root access via ssh, no key auth...

So, the server got hacked without me even noticing. Some time later, I received a mail from my hoster who said "hello, someone (probably you) is running portscans from your server" of which I had no idea... So I looked in the logs, and BAM, "successful root login" from an IP address which wasn't me.

After I found out the server got hacked, I reinstalled the whole server, changed the port and activated key auth and installed fail2ban.
Some days later, when I finally configured everything the way I wanted, I observed I couldn't do anything with that server anymore. Found out there was absolutely no space on the server. Made a scan to find files to delete and found a logfile. The ssh logfile. I took up a freaking 95 GB of space (of a total of 100gb on the server). Turned out the guy who broke into my server got upset I discovered him and bruteforced the shit out of my server flooding the logs with failed login attempts...

I guess I learnt how to properly secure a server from this attack 💪

I made a bash script for my website that anonymises the visitor IPs in the Awstats logs by replacing the last octet with 0. It can either process all logfiles except the one of the current month, or only the one of the previous month. The latter mode is how I put it in a cron job to be called on the first day of each month.

Everything worked flawlessly with test data, but on the server, some visitor IPs were not anonymised. I noticed that all of them were from the last day of the previous month. Looking at the time stamp of the logfile, it was indeed from the first of the current month, but not from 00:21 where my cron job runs - instead, it was modified around 14:30.

Then I realised that the Awstats engine seems to be configured to batch add the log entries once per day at 14:30 so that when my cron job ran, the visitor data from between 14:30 and 00:00 were not yet in the file!

Solution: batch process all previous logfiles once to clean them up, and schedule the cron job on the 2nd of each month at 00:21.

How do you define a seniority in a corporate is beyond me.
This guy is supposed to be Tier3, literally "advanced technical support". Taking care of network boxes, which are more or less linux servers. The most knowledgable person on the topic, when Tier1 screws something and it's not BAU/Tier2 can't fix it.
In the past hour he:
- attempted to 'cd' to a file and wondered why he got an error
- has no idea how to spell 'md5sum'
- syntax for 'cp' command had to be spelled out to him letter by letter
- has only vague idea how SSH key setup works (can do it only if sombody prepares him the commands)
- was confused how to 'grep' a string from a logfile

This is not something new and fancy he had no time to learn yet. These things are the same past 20-30 years. I used to feel sorry for US guys getting fired due to their work being outsourced to us but that is no longer the case. Our average IT college drop-out could handle maintenance better than some of these people.

Currently I have to devop a Verilog Module. Somehow there seems to be a bug in my environment and once I start the compile process this fucking compiler hangs for a solid 30 minutes or longer and I don't even get an output. It just stops with an Error that says I shall check the logfile, but it is empty.

I spent half a day trying to figure out why the app on the staging server does not log in the app log file while it does on the dev server.
Server log said log config file found but could not find the root logger.

Problem was that the directory was readable for the app, but not the logfile configuration file.

Dear devs, when a file is not readable that might be some interesting information one could write into a log. AT LEAST MORE INTERESTING THAN "APPLICATION STARTING..."

How does one convert an .img system image for Android to the format readable by Heimdall on Linux... Only thing I found so far requires Windows... Why make Heimdall if you need Windows anyway to make the image... Just use Odin on Windows then... Any ideas? Don't have Windows... And all I need is to read a bloody logfile (btsnoop_hci.log). Thanks, "secure" Android Oreo!!!