Good Morning!, its time for practiseSafeHex's most incompetent co-worker!

Todays contestant is a very special one.

*sitcom audience: WHY?*

Glad you asked, you see if you were to look at his linkedin profile, you would see a job title unlike any you've seen before.

*sitcom audience oooooooohhhhhh*

were not talking software developer, engineer, tech lead, designer, CTO, CEO or anything like that, No No our new entrant "G" surpasses all of those with the title ..... "Software extraordinaire".

*sitcom audience laughs hysterically*

I KNOW!, wtf does that even mean! as a previous dev-ranter pointed out does this mean he IS quality code? I'd say he's more like a trash can ... where his code belongs

*ba dum tsssss*

Ok ok, lets get on with the show, heres some reasons why "G" is on the show:

One of G's tasks was to build an analytics gathering library for iOS, similar to google analytics where you track pages and events (we couldn't use google's). G was SO good at this job he implemented 2 features we didn't even ask for:

- If the library was unable to load its config file (for any reason) it would throw an uncatchable system integrity error, crashing the app.
- If anything was passed into any of the functions that wasn't expected (null, empty array etc.) it would crash the app as it was "more efficient" to not do any sanity checks inside the library.

This caused a lot of issues as some of the data needed to come from the clients server. The day we launched the app, within the first 3 hours we had over 40k crash logs and a VERY angry client.

Now, what makes this story important is not the bugs themselves, come on how many times have we all done something stupid? No the issue here was G defended all of this as the right thing to do!

.. and no he wasn't stoned or drunk!

G claimed if he couldn't get the right settings / params he wouldn't be able to track the event and then our CEO wouldn't have our usage data. To which I replied:

"So your solution was to not give the client an app instead? ... which also doesn't give the CEO his data".

He got very angry and asked me "what would you do then?". I offered a solution something like why not have a default tag for "error" or "unknown" where if theres an issue, we send up whatever we have, plus the file name and store it somewhere else. I was told I was being ridiculous as it wasn't built to track anything like that and that would never work ... his solution? ... pull the library out of the app and forget it.

... once again giving everyone no data.

G later moved onto another cross-platform style project. Backend team were particularly unhappy as they got no spec of what needed to be done. All they knew was it was a single endpoint dealing with very complex model. There was no Java classes, super classes, abstract classes or even interfaces, just this huge chunk of mocked data. So myself and the lead sat down with him, and asked where the interfaces for the backend where, or designs / architecture for them etc.

His response, to this day frightens me ... not makes me angry, not bewilders me ... scares the living shit out of me that people like this exist in the world and have successful careers.

G: "hhhmmm, I know how to build an interface, but i've never understood them ... Like lets say I have an interface, what now? how does that help me in any way? I can't physically use it, does it not just use up time building it for no reason?"

us: "... ... how are the backend team suppose to understand the model, its types, integrate it into the other systems?"

G: "Can I not just tell them and they can write it down?"

**
I'll just pause here for a moment, as you'll likely need to read that again out of sheer disbelief
**

I've never seen someone die inside the way the lead did. He started a syllable and his face just dropped, eyes glazed over and he instantly lost all the will to live. He replied:

" wel ............... it doesn't matter ... its not important ... I have to go, good luck with the project"

*killed the screen share and left the room*

now I know you are all dying in suspense to know what happened to that project, I can drop the shocking bombshell that it was in fact cancelled. Thankfully only ~350 man hours were spent on it

... yep, not a typo.

G's crowning achievement however will go down in history. VERY long story short, backend got deployed to the server and EVERYTHING broke. Lead investigated, found mistakes and config issues on every second line, load balancer wasn't even starting up. When asked had this been tested before it was deployed:

G: "Yeah I tested it on my machine, it worked fine"

lead: "... and on the server?"

G: "no, my machine will do the same thing"

lead: "do you have a load balancer and multiple VM's?"

G: "no, but Java is Java"

... and with that its time to end todays episode. Will G be our most incompetent? ... maybe.

Tune in later for more practiceSafeHex's most incompetent co-worker!!!

D: “Did the attackers exfiltrate any data?”

M: “I can’t say for sure, but most likely based on—”

D: “—but did you find any undeniable evidence of it?”

M: “Keep in mind that the absence of evidence isn’t necessarily evidence of absence. There was very limited logging to begin with and the attacker erased artifacts and logs.”

D: “If there’s no evidence, then there was no exfiltration.”

M: “If a business doesn’t have cameras on its front door and then gets robbed, it can’t claim there was no robbery just because they didn’t video-record it.”

D: “That’s a poor analogy. Nothing’s missing here. I couldn’t care less if a robber made a *copy* of my money. That isn’t robbery.”

M: “... If the Titanic really hit an iceberg, then how come no pieces of an iceberg were ever found in the wreckage?”

Dear self proclaimed wordpress 'developers/programmers', kindly go fuck yourself.

I'm not talking about wordpress devs/designers who don't claim to have a better skillset than they have and are actually willing to learn, those are very much fine.

I'm talking about those wordpress people who claim that they're developers, programmers or whatever kind of bullshit which they're obviously not.

"A client's site crashed, you have to fix it!!!!!" sorry, come again? It's YOUR client's site. It's hosted on our hosting platform meaning that WE are responsible for KEEPING THE SERVERS UP AND FUNCTIONING.
You call yourself a wordpress 'developer' with 'programming experience' for 10 years but the second one of your shitty sites crashes, you come to us because 'it's your responsibility!!!'.
No, it's not. Next to that fact, the fact that you have to ask US why the site is crashing while you could easily login to your control panel, go to the fucking error logs and see that one of your facebook plugins crashes with a quite English error message, shows me that you definitely don't have 10 years of programming experience. And if you can't find that fucking article which tells you exactly where the motherfucking error logs are, don't come crying to us asking to fix your own fucking bullshit.

"My clients site got hacked, you have to clean it up and get it online again ASAP!!!!" - Nah, sorry, not my responsibility. The fact that you explicitly put your wordpress installation on 'no automatic updates' also doesn't help with my urge to fucking end you right now.
Add to that that we have some quite clear articles on wordpress security which you appearantly found too difficult (really? basic shit like 'set a strong fucking password' is too difficult for you?), you're on your own.

"I'm getting an error, please explain what's going wrong as soon as you can! this is a prio 1!!!!" - Nope. You were a wordpress dev/programmer right? Please act like one.

I'm not your personal wordpress agent.

I'm not your personal hacked wordpress site cleanup guy.

I'm not even a fucking wordpress professional. No, I'd rather jump off a bridge than develop wordpress bullshit for a living.

That you chose to do this, not a problem. Just don't rely on me for fixing your shit.

I'm sick of cleaning up your bullshit.

I'm done with answering your high prio tickets about bullshit which any dev could find out with just a few minutes of searching.

Oh your wordpress site isn't showing up so high in google? Yeah sure, shoot a ticket at us blaming us for your own SEO mess. I'm a fucking sysadmin, not a SEO expert.

I'm fucking done with you.

Go die in a fucking corner.

C: application not working
Me: k. What changed?
C: we didn't make changes
Me: k... *gets a tech team (W) on the phone*
W: Hey, what's broken?
Me: C's application. How do things look?
W: running healthy. I'll check logs.
Me: thanks. *gets tech team (S) on the line*
S: hey, everything clear on our end, will check logs.
Me: thanks *gets tech team (U)*
U: hey! They asked us to deploy their new version today during normal deployment time. Is it acting up?
Me: C, what did you change?
C: nothing major, just how we connect to W and S...
W&S: are you shitting me???
Me: U, will you please roll it back?
C: no! Must stay on this version, you need to fix your side!!
Me: nope. *calls U boss (UG)*
UG: U, you have my permission to roll back, they need to fix. C, if your boss doesn't like it, have them call me.
*rollback fixes problem*

IF I FUCKING ASK YOU WHAT THE FUCK YOU CHANGED, YOU BETTER TELL ME THE TRUTH, OR I WILL STRIP YOUR CODE OFF OUR FUCKING SYSTEMS AND SHOVE IT DOWN YOUR THROAT. MY JOB IS TO HELP YOU AND YOU NEED TO BACK TO FUCK UP AND NOT GET IN THE WAY OF MY JOB OR YOU WON'T HAVE ONE ANYMORE.

Hacking/attack experiences...
I'm, for obvious reasons, only going to talk about the attacks I went through and the *legal* ones I did 😅 😜

Let's first get some things clear/funny facts:
I've been doing offensive security since I was 14-15. Defensive since the age of 16-17. I'm getting close to 23 now, for the record.

First system ever hacked (metasploit exploit): Windows XP.
(To be clear, at home through a pentesting environment, all legal)
Easiest system ever hacked: Windows XP yet again.

Time it took me to crack/hack into today's OS's (remote + local exploits, don't remember which ones I used by the way):
Windows: XP - five seconds (damn, those metasploit exploits are powerful)
Windows Vista: Few minutes.
Windows 7: Few minutes.
Windows 10: Few minutes.
OSX (in general): 1 Hour (finding a good exploit took some time, got to root level easily aftewards. No, I do not remember how/what exactly, it's years and years ago)
Linux (Ubuntu): A month approx. Ended up using a Java applet through Firefox when that was still a thing. Literally had to click it manually xD
Linux: (RHEL based systems): Still not exploited, SELinux is powerful, motherfucker.

Keep in mind that I had a great pentesting setup back then 😊. I don't have nor do that anymore since I love defensive security more nowadays and simply don't have the time anymore.

Dealing with attacks and getting hacked.

Keep in mind that I manage around 20 servers (including vps's and dedi's) so I get the usual amount of ssh brute force attacks (thanks for keeping me safe, CSF!) which is about 40-50K every hour. Those ip's automatically get blocked after three failed attempts within 5 minutes. No root login allowed + rsa key login with freaking strong passwords/passphrases.

linu.xxx/much-security.nl - All kinds of attacks, application attacks, brute force, DDoS sometimes but that is also mostly mitigated at provider level, to name a few. So, except for my own tests and a few ddos's on both those domains, nothing really threatening. (as in, nothing seems to have fucked anything up yet)

How did I discover that two of my servers were hacked through brute forcers while no brute force protection was in place yet? installed a barebones ubuntu server onto both. They only come with system-default applications. Tried installing Nginx next day, port 80 was already in use. I always run 'pidof apache2' to make sure it isn't running and thought I'd run that for fun while I knew I didn't install it and it didn't come with the distro. It was actually running. Checked the auth logs and saw succesful root logins - fuck me - reinstalled the servers and installed Fail2Ban. It bans any ip address which had three failed ssh logins within 5 minutes:
Enabled Fail2Ban -> checked iptables (iptables -L) literally two seconds later: 100+ banned ip addresses - holy fuck, no wonder I got hacked!

One other kind/type of attack I get regularly but if it doesn't get much worse, I'll deal with that :)

Dealing with different kinds of attacks:

Web app attacks: extensively testing everything for security vulns before releasing it into the open.
Network attacks: Nginx rate limiting/CSF rate limiting against SYN DDoS attacks for example.
System attacks: Anti brute force software (Fail2Ban or CSF), anti rootkit software, AppArmor or (which I prefer) SELinux which actually catches quite some web app attacks as well and REGULARLY UPDATING THE SERVERS/SOFTWARE.

So yah, hereby :P

Funny story about the first time two of my servers got hacked. The fun part is how I noticed it.
So I purchased two new vps's for proxy server goals and thought like 'I can setup fail2ban tomorrow, I'll be fine.'

Next day I wanted to install NginX so I ran the command and it said that port 80 was already in use!

I was sitting there like no that's not possible I didn't install any server software yet. So I thought 'this can't be possible' but I ran 'pidof apache2' just to confirm. It actually returned a PID! It was a barebones Debian install so I was sure it was not installed yet by ME. Checked the auth logs and noticed that an IP address had done a huge brute force attack and managed to gain root access. Simply reinstalled debian and I put fail2ban on it RIGHT AWAY.

Checked about two seconds later if anyone tried to login again (iptables -L and keep in mind that fail2ban's default config needs six failed attempts within I think five minutes to ban an ip) and I already saw that around 8-10 addresses were banned.

Was pretty shaken up but damn I learned my lesson!

Manager: THE SERVER IS DOWN THE SERVER IS DOWN!!!!

Dev: Ok I’ll look into it

*5 mins later

Dev: Wow these are really strange logs, it’s like config values are being changed all over the place while I’m looking at it

Manager: Well I figured while you were looking into it I’d go i to the server settings and change everything I could find in order to try and get the server back up again. Two sets of hands are better than one, Is it up yet???

Dev: …No.

Manager: I THOUGT YOU SAID YOU’D LOOK INTO THIS. I NEED ANSWERS NOW. WHAT IS TAKING SO LONG?!?!?

Dev: …

My first job: The Mystery of The Powered-Down Server

I paid my way through college by working every-other-semester in the Cooperative-Education Program my school provided. My first job was with a small company (now defunct) which made some of the very first optical-storage robotic storage systems. I honestly forgot what I was "officially" hired for at first, but I quickly moved up into the kernel device-driver team and was quite happy there.

It was primarily a Solaris shop, with a smattering of IBM AIX RS/6000. It was one of these ill-fated RS/6000 machines which (by no fault of its own) plays a major role in this story.

One day, I came to work to find my team-leader in quite a tizzy -- cursing and ranting about our VAR selling us bad equipment; about how IBM just doesn't make good hardware like they did in the good old days; about how back when _he_ was in charge of buying equipment this wouldn't happen, and on and on and on.

Our primary AIX dev server was powered off when he arrived. He booted it up, checked logs and was running self-diagnostics, but absolutely nothing so far indicated why the machine had shut down. We blew a couple of hours trying to figure out what happened, to no avail. Eventually, with other deadlines looming, we just chalked it up be something we'll look into more later.

Several days went by, with the usual day-to-day comings and goings; no surprises.

Then, next week, it happened again.

My team-leader was LIVID. The same server was hard-down again when he came in; no explanation. He opened a ticket with IBM and put in a call to our VAR rep, demanding answers -- how could they sell us bad equipment -- why isn't there any indication of what's failing -- someone must come out here and fix this NOW, and on and on and on.

(As a quick aside, in case it's not clearly coming through between-the-lines, our team leader was always a little bit "over to top" for me. He was the kind of person who "got things done," and as long as you stayed on his good side, you could just watch the fireworks most days - but it became pretty exhausting sometimes).

Back our story -
An IBM CE comes out and does a full on-site hardware diagnostic -- tears the whole server down, runs through everything one part a time. Absolutely. Nothing. Wrong.

I recall, at some point of all this, making the comment "It's almost like someone just pulls the plug on it -- like the power just, poof, goes away."

My team-leader demands the CE replace the power supply, even though it appeared to be operating normally. He does, at our cost, of course.

Another weeks goes by and all is forgotten in the swamp of work we have to do.

Until one day, the next week... Yes, you guessed it... It happens again. The server is down. Heads are exploding (will at least one head we all know by now). With all the screaming going on, the entire office staff should have comped some Advil.

My team-leader demands the facilities team do a full diagnostic on the UPS system and assure we aren't getting drop-outs on the power system. They do the diagnostic. They also review the logs for the power/load distribution to the entire lab and office spaces. Nothing is amiss.

This would also be a good time draw the picture of where this server is -- this particular server is not in the actual server room, it's out in the office area. That's on purpose, since it is connected to a demo robotics cabinet we use for testing and POC work. And customer demos. This will date me, but these were the days when robotic storage was new and VERY exciting to watch...

So, this is basically a couple of big boxes out on the office floor, with power cables running into a special power-drop near the middle of the room. That information might seem superfluous now, but will come into play shortly in our story.

So, we still have no answer to what's causing the server problems, but we all have work to do, so we keep plugging away, hoping for the best.

The team leader is insisting the VAR swap in a new server.

One night, we (the device-driver team) are working late, burning the midnight oil, right there in the office, and we bear witness to something I will never forget.

The cleaning staff came in.

Anxious for a brief distraction from our marathon of debugging, we stopped to watch them set up and start cleaning the office for a bit.

Then, friends, I Am Not Making This Up(tm)... I watched one of the cleaning staff walk right over to that beautiful RS/6000 dev server, dwarfed in shadow beside that huge robotic disc enclosure... and yank the server power cable right out of the dedicated power drop. And plug in their vacuum cleaner. And vacuum the floor.

We each looked at one-another, slowly, in bewilderment... and then went home, after a brief discussion on the way out the door.

You see, our team-leader wasn't with us that night; so before we left, we all agreed to come in late the next day. Very late indeed.

Had this a few weeks ago. A customer send an email telling that her emails didn't load anymore in the web interface.

Fair enough, tested it and indeed, no emails while there were definitely emails on the server and nothing in the logs or anything.

Replied saying that I'd look into it.

Few hours later she messaged back all happy about it working again and thanking me so much for all the effort I put into getting it to work again!

My thought at that moment: "oh shit completely forgot about that ticket... Wait it's working again? Oo I didn't do anything... 😅"

Love it how some things just solve themselves haha

pm: our client wants a proprietary pdf compression app.
me: Okay gimme 3 days and some sample PDFs.
pm: they won't supply any sample PDFs because they contain confidential information.
me: okay fine, I'll download some from the interwebs.

** 3 days later **

me: here is the pdf compression app. all done and works with all of about 100 PDFs we tested with.
pm: okay great I'll have the client take a look.

** half and hour later **

pm: the client said that the compression app errors out.
me: okay I'll go look at the server logs to see what's up.

** 10 seconds later **

me: what the shit is a "foxit phantompdf" file.
pm: it's the proprietary pdf format that they are using.
me: oh joy. I'll go try to find some sample files and see if I can fix it.

** 1 hour later, no sample files found **

pm: got anything?
me: *sobs obnoxiously*

So I own a webshop together with a guy I met at one of my previous contract jobs. He said he had a great idea to sell product X because he can get them very cheap from another European country. Actually it is a great idea so we decided to work together on this: I do everything tech related, he does the non tech stuff.

Now we are more than 1 year in business. I setup a VPS, completely configured it, installed and setup the complete webshop, built 2 custom PrestaShop modules, built many customizations, built a completely new order proces (both front and back end), advertised quite some products, did some link building, ensured everything is in place to do proper SEO, wrote some content pages, did administration and tax declarations, rewrote a part of a PrestaShop component because it was so damn inefficient and horribly slow, and then some more. Much more.

He did customer relation management, supplier management and some ad words campaigns. Promised me many times to write the content for our product pages. This guy has an education in marketing but literally said: I'm not gonna invest in creating some marketing plan. I have no ambition in online marketing.

What?! You have the marketing knowledge and skills but refuse to use it to market our webshop and business? What the fuck is wrong with you?!

Today he says to me: 'Hey man, this is becoming an expensive hobby as we don't sell much and have lots of costs. I don't understand why I should be the one to write these content pages. Everything you did in the past 8 months can be done in less than 20 hours! You are a joke and just made it a big deal by spreading your work over so many months. I know for sure because I currently work at a company where I'm surrounded by front end devs! Are you fucking crazy?! You're a liar.'

He talks like this to me every 2 months or so while he can't even deliver the content for 1 single product in 6 fuckin' months! We even had to refund a few of our customers because Mr. client relations manager didn't respond to their e-mails within 1 fucking week!! So I asked him how could that have happened as you do the client relations and support. Well, he replied to me: 'Why didn't YOU respond to our clients? You don't log on in our back office at least once a day?!'.

Of course I do asshole. But YOU don't. He replied that I was lying just like I was lying about what I did for our business.

So, asshole, let's have a look at PrestaShops logs to see who's logging in daily. Well, you can probably guess who's IP was there in most of the entries. It wasn't his.

So, what the fuck have you been doing then?! You can't even manage to respond quickly to a client?!! We have maybe 50 clients and if we get 1 question a month by email it is already a lot. But you keep bitching, complaining and insulting me instead?!!!

Last time he literally admitted on a WhatsApp conversation that he had and still has the hope that he could just sit back and relax and watch me do ALL the work.

Well, guess what you fucking moron. That's not what we agreed upon. You fuckin' retard think you're so smart but you say EVERYTHING on WhatsApp! Including your promises to me. Thank you you fuckin' piece of dog shit because now I have hard evidence and will hand it over to my lawyer to make you pay every god damn cent for all the hours I've spent working on our business. Oh, and I'll take over the webshop and make it a success on my own because I know damn well how to get relevant traffic and thus customers.

You just go get yourself fucked in the ass without lubricant you fuckin' asshole. I have told you you shouldn't fuck with me because I take business very seriously. I even warned you when you were crossing a line again. Well, if you don't listen... You will pay for the consequences. I will be so damn happy to tell you 'I told you so' with a very very big smile on my face. That momemt WILL come, 'partner'.

Fuck you. You will be fucked. Count on that. Fucking asshole.

Hey, Root? How do you test your slow query ticket, again? I didn't bother reading the giant green "Testing notes:" box on the ticket. Yeah, could you explain it while I don't bother to listen and talk over you? Thanks.

And later:
Hey Root. I'm the DBA. Could you explain exactly what you're doing in this ticket, because i can't understand it. What are these new columns? Where is the new query? What are you doing? And why? Oh, the ticket? Yeah, I didn't bother to read it. There was too much text filled with things like implementation details, query optimization findings, overall benchmarking results, the purpose of the new columns, and i just couldn't care enough to read any of that. Yeah, I also don't know how to find the query it's running now. Yep, have complete access to the console and DB and query log. Still can't figure it out.

And later:

Hey Root. We pulled your urgent fix ticket from the release. You know, the one that SysOps and Data and even execs have been demanding? The one you finished three months ago? Yep, the problem is still taking down production every week or so, but we just can't verify that your fix is good enough. Even though the changes are pretty minimal, you've said it's 8x faster, and provided benchmark findings, we just ... don't know how to get the query it's running out of the code. or how check the query logs to find it. So. we just don't know if it's good enough.

Also, we goofed up when deploying and the testing database is gone, so now we can't test it since there are no records. Nevermind that you provided snippets to remedy exactly scenario in the ticket description you wrote three months ago.

And later:

Hey Root: Why did you take so long on this ticket? It has sat for so long now that someone else filed a ticket for it, with investigation findings. You know it's bringing down production, and it's kind of urgent. Maybe you should have prioritized it more, or written up better notes. You really need to communicate better. This is why we can't trust you to get things out.

*twitchy smile*

Sister = bee ( who isn't a stranger to Ubuntu)
Me = Cee

Bee: can I use your laptop?
Cee : why ? Use yours ,it's works fine.
Bee : no I want to use yours and I need to work with windows.
Cee: 🤯
Bee : my work can only be done using windows.
Cee : fine do whatever ( doesn't want to argue )
* Le bee opens MS word, and starts her work *
Cee : 😤😤Seriously?
Bee : I don't like libre
Cee : 😑😑😑^∞
* Few moments later *
Bee : my work is done ,you can have your laptop,btw it's updating.
Cee : 😑😑😑😑😑
* 2000 years later *
*Opens Ubuntu *
*Getting a weird bug*
*Tried to fix *
*Can't open OS files * 👏👏👏🎆
* Windows not shutdown properly *
* Opens windows *
* Not able to login via pin *
* Password ? not accepted *
* Changes outlook password *
* Please chose a password you haven't chosen before *
* Logs in *
* types old pin to change pin *
*You've entered wrong pin too many times *
*System hanging a lot *
* Removes pin *
* Gets huge mcAfee restart system popups , every 10 sec *
* Just shutdown , feels irritated for the rest of the day*

* Regrets dual booting, shd have wiped the windows partition 😫😫*

*Wonders,what the hell did my sister even do to my laptop ?*

It's maddening how few people working with the internet don't know anything about the protocols that make it work. Web work, especially, I spend far too much time explaining how status codes, methods, content-types etc work, how they're used and basic fundamental shit about how to do the job of someone building internet applications and consumable services.

The following has played out at more than one company:

App: "Hey api, I need some data"
API: "200 (plain text response message, content-type application/json, 'internal server error')"
App: *blows the fuck up

*msg service team*

Me: "Getting a 200 with a plaintext response containing an internal server exception"
Team: "Yeah, what's the problem?"
Me: "...200 means success, the message suggests 500. Either way, it should be one of the error codes. We use the status code to determine how the application processes the request. What do the logs say?"
Team: "Log says that the user wasn't signed in. Can you not read the response message and make a decision?"
Me: "That status for that is 401. And no, that would require us to know every message you have verbatim, in this case, it doesn't even deserialize and causes an exception because it's not actually json."
Team: "Why 401?"
Me: "It's the code for unauthorized. It tells us to redirect the user to the sign in experience"
Team: "We can't authorize until the user signs in"
Me: *angermatopoeia* "Just, trust me. If a user isn't logged in, return 401, if they don't have permissions you send 403"
Team: *googles SO* "Internet says we can use 500"
Me: "That's server error, it says something blew up with an unhandled exception on your end. You've already established it was an auth issue in the logs."
Team: "But there's an error, why doesn't that work?"
Me: "It's generic. It's like me messaging you and saying, "your service is broken". It doesn't give us any insight into what went wrong or *how* we should attempt to troubleshoot the error or where it occurred. You already know what's wrong, so just tell me with the status code."
Team: "But it's ok, right, 500? It's an error?"
Me: "It puts all the troubleshooting responsibility on your consumer to investigate the error at every level. A precise error code could potentially prevent us from bothering you at all."
Team: "How so?"
Me: "Send 401, we know that it's a login issue, 403, something is wrong with the request, 404 we're hitting an endpoint that doesn't exist, 503 we know that the service can't be reached for some reason, 504 means the service exists, but timed out at the gateway or service. In the worst case we're able to triage who needs to be involved to solve the issue, make sense?"
Team: "Oh, sounds cool, so how do we do that?"
Me: "That's down to your technology, your team will need to implement it. Most frameworks handle it out of the box for many cases."
Team: "Ah, ok. We'll send a 500, that sound easiest"
Me: *..l.. -__- ..l..* "Ok, let's get into the other 5 problems with this situation..."

Moral of the story: If this is you: learn the protocol you're utilizing, provide metadata, and stop treating your customers like shit.

"Hey, Root, someone screwed up and now all of our prod servers are running this useless query constantly. I know I already changed your priorities six times in the past three weeks, but: Go fix it! This is higher priority! We already took some guesses at how and supplied the necessary code changes in the ticket, so this shouldn't take you long. Remember, HIGH PRIORITY!"

1. I have no idea how to reproduce it.
2. They have no idea how to reproduce it.
3. The server log doesn't include queries.
4. The application log doesn't include queries.
5. The tooling intercepts and strips out some log entries the legendary devs considered useless. (Tangent: It also now requires a tool to read the logs because log entries are now long json blobs instead of plain text.)
6. The codebase uses different loggers like everywhere, uses a custom logger by default, and often overwrites that custom logger with the default logger some levels in. gg
7. The fixes shown in the ticket are pretty lame. (I've fixed these already, and added one they missed.)
8. I'm sick and tired and burned out and just can't bring myself to care. I'm only doing this so i don't get fired.
9. Why not have the person who screwed this up fix it? Did they quit? I mean, I wouldn't blame them.

Why must everything this company does be so infuriatingly complicated?

My team handles infrastructure deployment and automation in the cloud for our company, so we don't exactly develop applications ourselves, but we're responsible for building deployment pipelines, provisioning cloud resources, automating their deployments, etc.

I've ranted about this before, but it fits the weekly rant so I'll do it again.

Someone deployed an autoscaling application into our production AWS account, but they set the maximum instance count to 300. The account limit was less than that. So, of course, their application gets stuck and starts scaling out infinitely. Two hundred new servers spun up in an hour before hitting the limit and then throwing errors all over the place. They send me a ticket and I login to AWS to investigate. Not only have they broken their own application, but they've also made it impossible to deploy anything else into prod. Every other autoscaling group is now unable to scale out at all. We had to submit an emergency limit increase request to AWS, spent thousands of dollars on those stupidly-large instances, and yelled at the dev team responsible. Two weeks later, THEY INCREASED THE MAX COUNT TO 500 AND IT HAPPENED AGAIN!

And the whole thing happened because a database filled up the hard drive, so it would spin up a new server, whose hard drive would be full already and thus spin up a new server, and so on into infinity.

Thats probably the only WTF moment that resulted in me actually saying "WTF?!" out loud to the person responsible, but I've had others. One dev team had their code logging to a location they couldn't access, so we got daily requests for two weeks to download and email log files to them. Another dev team refused to believe their server was crashing due to their bad code even after we showed them the logs that demonstrated their application had a massive memory leak. Another team arbitrarily decided that they were going to deploy their code at 4 AM on a Saturday and they wanted a member of my team to be available in case something went wrong. We aren't 24/7 support. We aren't even weekend support. Or any support, technically. Another team told us we had one day to do three weeks' worth of work to deploy their application because they had set a hard deadline and then didn't tell us about it until the day before. We gave them a flat "No" for that request.

I could probably keep going, but you get the gist of it.

We had the most fucking retarded client today. No, seriously, if you ever beat their level you have a serious mental issue.

They had a mail problem for which they'd need to check at the side of another company since we don't have those fucking logs.

Their statements:
- they entered an email address In the text field of mail-tester.com and were furious that they didn't get the results sent.
Note: it says right on that page that YOU JUST NEED TO SEND THE EMAIL ADDRESS WHICH IS PRE-ENTRRED IN THAT TEXT FIELD AN EMAIL.
- their company has been a reputable 'conservative' company which hasn't done anything wrong since 19xx so the fact that they'd end up on a blacklist was FUCKING OUTRAGEOUS and bullshit.
- our support wasn't willing to help and only willing to tell them outrageous lies.
- the other it company was only reachable at a premium number and thus expensive to call.

Emails back and forth and finally they CC'd the other company. They're reply was fucking priceless:

"we never had a premium number. Feel free to call us on *number* any time during the week between *time* and *time*.

Then he told us that we should just go back to sleep.

It was way worse than that but due to privacy and my own memory this is all I can tell.

Just wow.

Victory!

Today I finally closed a 'Nessy' bug (A scary bug you can't reproduce typically only sighted by one person). Below is my response...

"There were no errors in the error logs because writing to the error log was causing the error."

Last half hour of last Friday my Cinnamon desktop started fucking up every time I booted after like 20 seconds of uptime.

Logs said something about xorg errors but after like 15 reboots I said fuck it.

Was quite nervous this morning because of that issue and possibly not being able to open a terminal for installing a different DE before everything started to get screwed again.

Booted up. No freezes. No errors. It just works.

I guess my computer needed weekend too 😋

My last wk93 story, the time we discovered school faculty was spying on students and we uncovered student's deepest secrets.

I call it, kiddiegate.

So if you've read my past rants you've noticed I did some pretty childish and reckless stuff with my highschool's systems when I was younger, but nothing compares to this thing.

After resetting the sysadmin account pwd on some machines it occurred to me I could write a keylogger to capture teachers Moodle accounts and so on, I decided to try it out on a regular lab computer first.

Imagine my surprise when I found a hidden keylogger already installed! I couldn't believe it but then I thought, what if other PC's have it? So I recruited my mates and teached them the process to check if a PC had been infected...ALL PCs were, over 30 computers we checked had been logging for over 3 months! That damn sysadmin! >:[

We were shocked and angry, but then I thought "hey. . . My work has been done for me, better take advantage"

So we did, we extracted each log and then removed it from the PCs along with the keyloggers. There were hundreds of records and then one day we started snooping into the fb accounts of some students (we shouldn't have) we uncovered so many nasty, shocking secrets...

One of the school's lady's man had a drunk one nighter with one of our gay friends, the most secluded and shy guy was sexting like crazy with 15 chicks at the same time, things like that...we promised to never say a word and deleted the logs.

After that we didn't do much and continued highschool as every teenage minor should, getting drunk and avoiding responsibilities, though we could never see many of our classmates the same way. The sysadmin was fired shortly after I graduated, no reason was stablished.

I want to clear out we were minors and laws in my country weren't clearly stablished at the time plus no harm was ever done. I don't condone hacking or any kind of illegal activity, just thought I'd share.

Root: Fleshes out missing data in some factories. Tests affected code and finds the change breaks some specs (but shouldn’t).

Root: Reaches out to spec author.

Root: Messages thundercunt (the ticket’s code reviewer) on slack about the specs and the reaching out. No response.

Root: Works on another ticket while blocked.

Root: Logs off.

Root: Talks with spec author chick in the morning. Decide to pair on specs later.

TC: Still no slack response.

Root: Gives update in standup. Mentions factories and broken specs. Mentions pairing with spec chick.

TC: Still no slack response.

Root: Pulled off tickets in favor of prod issue. Gets ignored by everyone else diagnosing prod issue. Investigates prod issue by herself. Discovers prod issue isn’t from bad code, but bad requirements — code works as requested. Communicates this with details. Gets ignored by people still diagnosing prod issue. Tries again. Gets ignored. Gives up. Works on non-blocked tickets instead.

TC: Still no slack response.

Hours later:

TC: Comments on PR telling me I broke specs (how did I not notice?), that I need to reach out to spec chick and work with her, and that I can’t resolve the ticket until it’s fixed and passes code review.

TC: Still no slack response. (21 hours later at this point)

TC: Logs off. Still no response (25 hours at this point)

———

Ignoring the prod issue for the moment…

I broke specs. No shit.
I need to talk with spec chick. No shit.
I can’t resolve the ticket. No shit!

Bitch, I told you all of this 21 fucking hours prior, and again 3 hours prior during standup. But no, I clearly “don’t communicate” and obviously have no bloody clue what I’m doing, either, so I need everything spelled out for me.

And no, I didn’t resolve the fucking ticket. Why the fuck would I if it still has pending changes? Do you even check? Ugh!

And what the fuck with that prod issue? I’m literally giving you the answer. fucking listen! Stupid cunts.

Why is it all of the women I work with are useless or freaking awful people? Don’t get me wrong, many of the men are, too, but I swear it’s every single one of the women. (Am I awful, too?)

Just. Ugh.
I can’t wait to leave this sewer of a company.

Oddly still a good day, though. Probably because I talked to recruiters and sent out my resume again.

* I send an APK so that my client can test*

My client: I can't install the application

Me: Did you download and install it from the email I sent you?

client: No I downloaded from the play store.

(In my mind: Why the fuck do you Download it from the play store? I fucking sent you an email to 3 of your fucking email addresses so that you can fucking test the fucking APK that I fucking fixed it for you! You fucking worthless peice of shit!)

But I reacted as: No no, you should download the apk from the email I sent. I've sent it to all your email addresses.

client: I can't find the APK In your email.

(In my mind: Wow! I just don't get it! How can you be so stupid? I'm just wondering how your company hired you as a 'director for X')

*I send him a Screenshot proving that the apk exists*

Turns out that this idiot doesn't know to use outlook for Android! He then logs in to his gmail and finds the apk.(Coz I had emailed it to his outlook and gmail accounts)

M just wondering, should I drop this guy? Or charge him 2x for this shit?

Dev: “Ughh..look at this –bleep- code! When I execute the service call, it returns null, but the service received a database error.”
Me: “Yea, that service was written during a time when the mentality was ‘Why return a service error if the client can’t do anything about it?’”
Dev: “I would say that’s a misunderstanding of that philosophy.”
Me: “I would say it’s a perfectly executed example of a deeply flawed philosophy.”
Dev: “No, the service should just return something that tells the client the operation failed.”
Me: “They did. It was supposed to return a valid result, and the developer indicated a null response means the operation failed. How you deal with the null response is up to you.”
Dev: “That is stupid. How am I supposed to know a null response means the operation failed?”
Me: “OK, how did you know the operation failed?”
Dev: “I had to look at the service error logs.”
Me: “Bingo.”
Dev: “This whole service is just a –bleep-ing mess. There are so many things that can go wrong and the only thing the service returns is null when the service raises an exception.”
Me: “OK, what should the service return?”
Dev: ”I don’t know. Error 500 would be nice.”
Me: “Would you know what to do with error 500?”
Dev: ”Yea, I would look at the error log”
Me: “Just like you did when the service returned null?”
<couple of seconds of silence>
Dev: “I don’t know, it’s a –bleep-ing mess.”
Me: “You’re in the code, change it.”
Dev: “Ooohhh no, not me. The whole thing will have to be re-written. It should have been done correctly the first time. If we had time to do code reviews, I would have caught this –bleep- before the service was deployed.”
Me: “Um, you did.”
<a shocked look from Dev>
Dev: “What…no, I’ve never seen this code.”
Me: “I sat next to Chuck when you were telling him he needed to change the service to return null if an exception was raised. I remember you telling him specifically to pop-up an error dialog ‘Service request failed’ to the user when the service returned null.”
Dev: “I don’t remember any of that.”
Me: “Well, Chuck did. He even put it in the check-in comments. See…”
<check in comments stated Dev’s code review and dictated the service return null on exceptions>
Dev: “Hmm…I guess I did. –bleep- are you a –bleep-ing elephant? You –bleep-ing remember everything.”

<what I wanted to say>
No, I don’t remember everything, but I remember all the drive-by <bleep>-ed up coding philosophies you tried to push to the interns and we’re now having all kinds of problems I spend waaaaay too much time fixing.
<what I said, and lied a little bit>
Me: “No, I was helping Nancy last week troubleshoot the client application last week with the pop-up error. Since the service returned a null, she didn’t know where to begin to look for the actual error.”
Dev: “Oh.”

So, in the printing industry, FTP has a long and storied history as the standard method of sending art assets. But as time has gone on, more and more people are utterly incapable of handling FTP.

Customer: "I sent you the file. It's called xyz.zip"

PM: "I don't see the file."

Customer: "I know I sent it."

PM: "Let me check with IT."

I check the logs. No such file was uploaded.

PM: "What program did you use to send the file?"

Customer: "Firefox"

Every. Fucking. Time.

It turns out the Germans actually have a word for this:

!!pointless story

Bug report comes in from a coworker. "Cloudinary uploads aren't working. I can't sign up new customers."

"I'll look into it" I say.

I go to one of our sites, and lo! No Cloudinary image loads. Well that can't be good.

I check out mobile app -- our only customer-facing platform. None of the images load! Multiple "Oops!" snackbars from 500 errors on every screen / after every action.

"None of our Cloudinary images load, even in the mobile app," I report.

Nobody seems to notice, but they're probably busy.

I go to log into the Cloudinary site, and realize I don't have the credentials.

"What are the Cloudinary credentials, @ceo?" I ask.

I'm met with more silence. I use this opportunity to look through the logs, try different URLs/transforms directly. Oddly, everything seems fine except on our site.

I check Slack again, and see nothing's changed, so I set about trying to guess the credentials.

Let's see... the ceo is basically illiterate when it come to tech, so it's probably not his email. It's a startup, and custom emails for things cost money, and haven't been a thing here forever, so it's probably oen of the CTO's email aliases. he likes dots and full names so that narrows it down. Now for the password.... his are always crappy (so they're "easy to remember") and usually have the abbreviated company name in them. He also likes adding numbers, generally two-digit numbers, and has a thing for 7s and 9s. Mix in some caps, spaces, order...

Took me a few minutes, but I managed to figured it out.

"Nevermind, I guessed them." I reported.

After getting into Cloudinary, I couldn't find anything amiss. Everything looked great. No outage warnings, metrics looked fine, images all loaded. Ex-cto didn't revoke payment or cancel the account.

I checked our app; everything started loading -- albeit slowly.

I checked the aforementioned site; after a few minutes, everything loaded there, too.

Not sure what else to do, and with everything appearing to work, I said "Fixed!" and closed the issue.

About 20 minutes later, the original person said "thanks" -- never did hear anything from the ceo. I've heard him chatting away in the other room the entire time.

Regardless, good thing for crappy passwords, eh?

micromanager: "Quick and easy win! Please have this done in 2-3 days to start repairing your reputation"

ticket: "Scrap this gem, and implement your own external service wrapper using the new and vastly different Slack API!"

slack: "New API? Give me bearer tokens! Don't use that legacy url crap, wth"

prev dev: "Yeah idk what a bearer token is. Have the same url instead, and try writing it down so you don't forget it?"

Slack admin: "I can't give you access to the slack integration test app, even though it's for exactly this and three others have access already, including your (micro)manager."

Slack: "You can also <a>create a new slack app</a>!" -- link logs me into slack chat instead. After searching and finding a link elsewhere: doesn't let me.

Slack admin: "You want a new test slack app instead? Sure, build it the same as before so it isn't abuseable. No? Okay, plan a presentation for it and bring security along for a meeting on Friday and I'll think about it. I'm in some planning meetings until then."

asdfjkagel.

This job is endless delays, plus getting yelled at over the endless delays.

At least I can start on the code while I wait. Can't test anything for at least a week, though. =/

So, some time ago, I was working for a complete puckered anus of a cosmetics company on their ecommerce product. Won't name names, but they're shitty and known for MLM. If you're clever, go you ;)

Anyways, over the course of years they brought in a competent firm to implement their service layer. I'd even worked with them in the past and it was designed to handle a frankly ridiculous-scale load. After they got the 1.0 released, the manager was replaced with some absolutely talentless, chauvinist cuntrag from a phone company that is well known for having 99% indian devs and not being able to heard now. He of course brought in his number two, worked on making life miserable and running everyone on the team off; inside of a year the entire team was ex-said-phone-company.

Watching the decay of this product was a sheer joy. They cratered the database numerous times during peak-load periods, caused $20M in redis-cluster cost overrun, ended up submitting hundreds of erroneous and duplicate orders, and mailed almost $40K worth of product to a random guy in outer mongolia who is , we can only hope, now enjoying his new life as an instagram influencer. They even terminally broke the automatic metadata, and hired THIRTY PEOPLE to sit there and do nothing but edit swagger. And it was still both wrong and unusable.

Over the course of two years, I ended up rewriting large portions of their infra surrounding the centralized service cancer to do things like, "implement security," as well as cut memory usage and runtimes down by quite literally 100x in the worst cases.

It was during this time I discovered a rather critical flaw. This is the story of what, how and how can you fucking even be that stupid. The issue relates to users and their reports and their ability to order.

I first found this issue looking at some erroneous data for a low value order and went, "There's no fucking way, they're fucking stupid, but this is borderline criminal." It was easy to miss, but someone in a top down reporting chain had submitted an order for someone else in a different org. Shouldn't be possible, but here was that order staring me in the face.

So I set to work seeing if we'd pwned ourselves as an org. I spend a few hours poring over logs from the log service and dynatrace trying to recreate what happened. I first tested to see if I could get a user, not something that was usually done because auth identity was pervasive. I discover the users are INCREMENTAL int values they used for ids in the database when requesting from the API, so naturally I have a full list of users and their title and relative position, as well as reports and descendants in about 10 minutes.

I try the happy path of setting values for random, known payment methods and org structures similar to the impossible order, and submitting as a normal user, no dice. Several more tries and I'm confident this isn't the vector.

Exhausting that option, I look at the protocol for a type of order in the system that allowed higher level people to impersonate people below them and use their own payment info for descendant report orders. I see that all of the data for this transaction is stored in a cookie. Few tests later, I discover the UI has no forgery checks, hashing, etc, and just fucking trusts whatever is present in that cookie.

An hour of tweaking later, I'm impersonating a director as a bottom rung employee. Score. So I fill a cart with a bunch of test items and proceed to checkout. There, in all its glory are the director's payment options. I select one and am presented with:

"please reenter card number to validate."

Bupkiss. Dead end.

OR SO YOU WOULD THINK.

One unimportant detail I noticed during my log investigations that the shit slinging GUI monkeys who butchered the system didn't was, on a failed attempt to submit payment in the DB, the logs were filled with messages like:

"Failed to submit order for [userid] with credit card id [id], number [FULL CREDIT CARD NUMBER]"

One submit click later and the user's credit card number drops into lnav like a gatcha prize. I dutifully rerun the checkout and got an email send notification in the logs for successful transfer to fulfillment. Order placed. Some continued experimentation later and the truth is evident:

With an authenticated user or any privilege, you could place any order, as anyone, using anyon's payment methods and have it sent anywhere.

So naturally, I pack the crucifixion-worthy body of evidence up and walk it into the IT director's office. I show him the defect, and he turns sheet fucking white. He knows there's no recovering from it, and there's no way his shitstick service team can handle fixing it. Somewhere in his tiny little grinchly manager's heart he knew they'd caused it, and he was to blame for being a shit captain to the SS Failboat. He replies quietly, "You will never speak of this to anyone, fix this discretely." Straight up hitler's bunker meme rage.

I wrote a database migration to add a column to a table and populated that column upon record creation.
But the code is so freaking convoluted that it took me four days of clawing my eyes out to manage this.
BUT IT'S FINALLY DONE.
FREAKING YAY.

Why so long, you ask? Just how convoluted could this possibly be? Follow my lead ~

There's an API to create a gift. (Possibly more; I have no bloody clue.)
I needed the mobile dev contractor to tell me which APIs he uses because there are lots of unused ones, and no reasoning to their naming, nor comments telling me what they do.

This API takes the supplied gift params, cherry-picks a few bits of useful data out (by passing both hashes by reference to several methods), replaces a couple of them with lookups / class instances (more pass-by-reference nonsense). After all of this, it logs the resulting (and very different) mess, and happily declares it the original supplied params. Utterly useless for basically everything, and so very wrong.

It then uses this data to call GiftSale#create, which returns an instance of GiftSale (that's actually a Gift; more on that soon).

GiftSale inherits from Gift, and redefines three of its methods.

GiftSale#create performs a lot of validations / data massaging, some by reference, some not. It uses `super` to call Gift#create which actually maps to the constructor Gift#initialize.

Gift#initialize calls Gift#pre_init (passing the data by reference again), which does nothing and returns null. But remember: GiftSale inherits from Gift, meaning GiftSale#pre_init supersedes Gift#pre_init, so that one is called instead. GiftSale#pre_init returns a Stripe charge object upon success, or a Gift (and a log entry containing '500 Internal') upon failure. But this is irrelevant because the return value is never actually used. Pass by reference, remember? I didn't.

We're now back at Gift#initialize, Rails finally creates a Gift object using the args modified [mostly] in-place by all of the above.

Another step back and we're at GiftSale#create again. This method returns either the shiny new Gift object or an error string (???), and the API logic branches on its type. For further confusion: not all of the method's returns are explicit, and those implicit return values are nested three levels deep. (In Ruby, a method will return the last executed line's return value automatically, allowing e.g. `def add(a,b); a+b; end`)

So, to summarize: GiftSale#create jumps back and forth between Gift five times before finally creating a Gift instance, and each jump further modifies the supplied params in-place.

Also. There are no rescue/catch blocks, meaning any issue with any of the above results in a 500. (A real 500, not a fake 500 like last time. A real 500, with tragic consequences.)

If you're having trouble following the above... yep! That's why it took FOUR FREAKING DAYS! I had no tests, no documentation, no already-built way of testing the API, and no idea what data to send it. especially considering it requires data from Stripe. It also requires an active session token + user data, and I likewise had no login API tests, documentation, logging, no idea how to create a user ... fucking hell, it's a mess.)

Also, and quite confusingly:
There's a class for GiftSale, but there's no table for it.
Gift and GiftSale are completely interchangeable except for their #create methods.

So, why does GiftSale exist?
I have no bloody idea.
All it seems to do is make everything far more complicated than it needs to be.

Anyway. My total commit?
Six lines.
IN FOUR FUCKING DAYS!

AHSKJGHALSKHGLKAHDSGJKASGH.

My brother made a PC game for his game dev class.

You are a hamster with a hammer
- acorns drop and u crack them for points
- you have to avoid rolling logs and eagles that swoop down
- the game gets faster

Dad: I just played the game, I think it has potential like Angry Birds, can you convert it to Android?

Me: no and no
Dad: why and why?
Me: **facepalm and takes a deep breath**

Explains why very simply and that I'm not a game dev... And if he thinks it's that good, go pay for a freelancer...

3 rants for the price of 1, isn't that a great deal!

1. HP, you braindead fucking morons!!!

So recently I disassembled this HP laptop of mine to unfuck it at the hardware level. Some issues with the hinge that I had to solve. So I had to disassemble not only the bottom of the laptop but also the display panel itself. Turns out that HP - being the certified enganeers they are - made the following fuckups, with probably many more that I didn't even notice yet.

- They used fucking glue to ensure that the bottom of the display frame stays connected to the panel. Cheap solution to what should've been "MAKE A FUCKING DECENT FRAME?!" but a royal pain in the ass to disassemble. Luckily I was careful and didn't damage the panel, but the chance of that happening was most certainly nonzero.
- They connected the ribbon cables for the keyboard in such a way that you have to reach all the way into the spacing between the keyboard and the motherboard to connect the bloody things. And some extra spacing on the ribbon cables to enable servicing with some room for actually connecting the bloody things easily.. as Carlos Mantos would say it - M-m-M, nonoNO!!!
- Oh and let's not forget an old flaw that I noticed ages ago in this turd. The CPU goes straight to 70°C during boot-up but turning on the fan.. again, M-m-M, nonoNO!!! Let's just get the bloody thing to overheat, freeze completely and force the user to power cycle the machine, right? That's gonna be a great way to make them satisfied, RIGHT?! NO MOTHERFUCKERS, AND I WILL DISCONNECT THE DATA LINES OF THIS FUCKING THING TO MAKE IT SPIN ALL THE TIME, AS IT SHOULD!!! Certified fucking braindead abominations of engineers!!!

Oh and not only that, this laptop is outperformed by a Raspberry Pi 3B in performance, thermals, price and product quality.. A FUCKING SINGLE BOARD COMPUTER!!! Isn't that a great joke. Someone here mentioned earlier that HP and Acer seem to have been competing for a long time to make the shittiest products possible, and boy they fucking do. If there's anything that makes both of those shitcompanies remarkable, that'd be it.

2. If I want to conduct a pentest, I don't want to have to relearn the bloody tool!

Recently I did a Burp Suite test to see how the devRant web app logs in, but due to my Burp Suite being the community edition, I couldn't save it. Fucking amazing, thanks PortSwigger! And I couldn't recreate the results anymore due to what I think is a change in the web app. But I'll get back to that later.

So I fired up bettercap (which works at lower network layers and can conduct ARP poisoning and DNS cache poisoning) with the intent to ARP poison my phone and get the results straight from the devRant Android app. I haven't used this tool since around 2017 due to the fact that I kinda lost interest in offensive security. When I fired it up again a few days ago in my PTbox (which is a VM somewhere else on the network) and today again in my newly recovered HP laptop, I noticed that both hosts now have an updated version of bettercap, in which the options completely changed. It's now got different command-line switches and some interactive mode. Needless to say, I have no idea how to use this bloody thing anymore and don't feel like learning it all over again for a single test. Maybe this is why users often dislike changes to the UI, and why some sysadmins refrain from updating their servers? When you have users of any kind, you should at all times honor their installations, give them time to change their individual configurations - tell them that they should! - in other words give them a grace time, and allow for backwards compatibility for as long as feasible.

3. devRant web app!!

As mentioned earlier I tried to scrape the web app's login flow with Burp Suite but every time that I try to log in with its proxy enabled, it doesn't open the login form but instead just makes a GET request to /feed/top/month?login=1 without ever allowing me to actually log in. This happens in both Chromium and Firefox, in Windows and Arch Linux. Clearly this is a change to the web app, and a very undesirable one. Especially considering that the login flow for the API isn't documented anywhere as far as I know.

So, can this update to the web app be rolled back, merged back to an older version of that login flow or can I at least know how I'm supposed to log in to this API in order to be able to start developing my own client?

Story time!

A little over a year ago I was in the hiring process with a new company and countered their initial offer. I was told by the CTO that it was no problem and they would get back to me soon.

A couple days go by and I'm then informed that they're hiring a new IT director and would like me to interview with him as well. It felt kinda lame since I'd already been offered the job but I rolled with it.

When I showed up to the office for an interview I tried to call and let them know I was there and couldn't get a hold of anyone. 30 minutes later I get a call from the CTO saying they couldn't find the new IT director and when they got him to answer the phone he said he had left early and would call me to do a phone interview.

Obviously the whole experience so far has been pretty lame but I stuck with it because I knew the CTO personally. I did the phone interview and quickly realized this dude was a prick, and would be a terrible boss, but I spoke with the CTO again who told me to stick with it and eventually I did get the job.

Fast forward about a month and it's clear the new director is trash. He literally bragged about firing a dude over an accidental outage (wtf!?).

He had the technical experience you'd expect of a junior help desk and his management skills were pretty clearly sub-par.

He was also, for whatever reason, completely unable to communicate with the only woman on our team. When assigning work he would always feel the need to ask if she could 'handle it' rather than just assigning it to her like it's done for everyone else. He was pretty clearly sexist.

The whole team hates this dude by this point but he's somehow managed to woo the executives into thinking he shits gold.

I was helping him set up a Python venv on his machine when I noticed another VPN client installed which certainly piqued my interest. After a bit of digging it was clear he was using company time and company equipment to continue working for his previous employer.

We turned over logs and he was fired the next day. He tried to add me on LinkedIn afterwards and I have never declined something quicker.

Moral of the story is don't be a dickhead.

@JoshBent suggested that I'd make a blog about security.

Nice idea, fair enough!

*registers domain at provider with discounts at the moment*
*tries to find whois protection option*

"You can add WHOIS protection to your account as an upgrade"

*requests authorization token*

*logs into usual domain name provider account*

*transfers domain name*

*anonymizes WHOIS details within two seconds*

I could've stayed and ask them about the cost etc but the fact that they even HAVE a price for protecting WHOIS data is a no-go for me.

Fuck domain name resellers which ask money for protecting ones WHOIS information (where possible).

One of my older servers just went down. It's been hacked. How do I know this you ask? Is it mining bitcoin?

No - windows event viewer has no errors in the logs for the last 48 hours.

> Customer logs Jira ticket claiming app is not working
< I restart the app, investigate and explain tht their server has issues
ø Client closes the ticket as Resolved
-- a couple of days pass by ---

> Customer logs Jira ticket claiming app is not working
< I restart the app, investigate and explain tht their server has issues
ø Client closes the ticket as Resolved
-- a couple of days pass by ---

> Customer logs Jira ticket claiming app is not working
< I restart the app, investigate and explain tht their server has issues
ø Client closes the ticket as Resolved
-- a couple of days pass by ---
<...>

< I log a JIRA ticket explaining what and how is wrong with the server with suggestions how to fix the problem so the app will not crash any longer (client own the server, has his own sysadmins -- I don't even had permissions to open syslog.. had to hack dmesg on their PROD server to pin-point the issue)
> no reaction from customer for weeks. I ping the ticket

× app crashes again

> no reaction from customer for weeks. I ping the ticket
> customer leaves a comment that their sysadmins are looking at it trying to figure out what might be wrong (ignoring what I wrote in ticket's description??? srsly?)

× app crashes again
< I post detail investigation details: snips from logs, screenshots, everything with crystal clear explanations.

> no reaction for weeks
......

well that's fun..

I swear to god, I'm going to track down the dipshit who just made my day hilariously painful.

So here I am, finishing up this project that's been going on for what feels like an eternity, when I get an email "why doesn't order X show up in this other system?".

I mean, it's a common thing they can take 15 minutes to push across, so the usual quick glance and what do you know, it's just sitting there as if it's waiting to be pushed through, than an hour later... it's still there, so I start digging, maybe a data issue, nope looks all good, customer details, payment details, products...

just another order, jump on the logs and all looks fi......... wait.... why does this postcode have 3 digits and not 4 , Australia has 4 digit postal codes fyi, looks at order again, 3 digits, look at log, 3....hold on why's it only 3 digits, checks code, handled as string... ok..... where the fuck would it drop a digit.... frontend requires 4 digits, validation requires 4 digits... how the fuck did you get 3 digits in... I can't see anything anywhere that logically makes sense for this🤔

Drops address into google and it's a postcode starting with 0.

Jumps on DB and the fucker is an int in the postcode table. For all you playing at home 0123 <> 123

I don't know if I should feel bad, or impressed, it's been 7 years since this table was created, and 7 years before someone managed to live in one of these parts of the country with a leading 0.

QA didn't spot this years ago,
No one tested this exact scenario,
The damn thing isn't even documented as a required delivery area, but here we are!

Kudos good sir, you broke it! 🤜 🤛
You sir may get your order now!

"IT BROKEN!", the QA tester spoke in unintelligible broken English.

The developer asked for more details.

Then the QA person attempted to explain the problem in a surge of verbal diarrhea and horrible English.

Why do we hire people who don't speak the language of the development team as our QA people? I have nothing against devs and qa guys from India...but it makes my job really difficult when I can't even begin to understand what you are telling me, or even worse...you just tell me "IT BROKEN!" and don't give me a single bit of useful information on how to reproduce the error.

There was this wonderful QA person I used to work with. Her name was Ranjana. She was a beautiful Indian girl with two children, and the best QA person I ever worked with. She took screenshots, grabbed logs, and gave steps to reproduce everything she found. And then one day at stand up we were told she had died. And since then...there has been no one who has ever come close to her level of excellence.

Our university has a rather small gym, and it tends get pretty crowded. They have an online counter, so I wrote a Python script that queries the current number of people every minute, and logs it in a CSV (no need to get fancy). Hopefully in a week I’ll have enough data to spot the quiet times 😎

5 hours on the call. No end in sight. Nothing in logs, errors, or captures. Want to cry.

I think this is so far one of the most priceless WTF moments I encountered at my current work:

A coworker of mine came up to me explaining the problem he had with russian characters in the filename. He explained in detail that everything works ok (the other part of the code he was fixing) if he changes the name of the file to test1.xlsx for example which doesn't use russian characters. OK great.

Then he goes on to show me how he fixed the other stuff and of course everything blows up. The file he used for demonstration was of course the original file our cusotomer provided, he just deleted the obvious russian chars and left the rest.

МТС != MTC

I cracked up: but you still have russian chars in the name.
The guy: no way, I deleted them all.
Me: but what about that МТС in the name?! Guy: what about it?
Me: did you actually typed that in or you left it there?! Those are russian chars that are fucking things up for you.
Guy: no way, it's MTC.
Me: checked the logs, you have ??? In the filename instead of МТС..don't you find that at least a little bit suspicious?!
Guy: but it looks the same. How does it (the computer) know it is in russian?!? //Why doesn't it understand?!

O.o I still can't believe it.. Is it just me & my high standards, or should it be normal for coders to know things such as character encoding & stuff?!?

I almost died of laughter, he and some other guy had problems finding customers in the software due to not being able to type the russian chars << happened more then once before, even after I told them about a quick hack on how to use google translate onboard keyboard & other stuff to make proper chars so they can get a match..

I think when they bury me, I'll still be facepalming and laughing over this incident. 🤣🤣🤣🤣🤣🤣🤣

From my work -as an IT consultant in one of the big 4- I can now show you my masterpiece

INSIGHTS FROM THE DAILY LIFE OF A FUNCTIONAL ANALIST IN A BIG 4 -I'M NOT A FUNCTIONAL ANALYST BUT THAT'S WHAT THEY DO-

- 10:30, enter the office. By contract you should be there at 9:00 but nobody gives a shit
- First task of the day: prepare the power point for the client. DURATION: 15 minutes to actually make the powerpoint, 45 minutes to search all the possible synonyms of RESILIENCE BIG DATA AGILE INTELLIGENT AUTOMATION MACHINE LEARNING SHIT PISS CUM, 1 hour to actually present the document.
- 12:30: Sniff the powder left by the chalks on the blackboards. Duration: 30 minutes, that's a lot of chalk you need to snort.

13:00, LUNCH TIME. You get back to work not one minute sooner than 15.00

- 15:00, conference with the HR. You need to carefully analyze the quantity and quality of the farts emitted in the office for 2 hours at least
- 17:00 conference call, a project you were assigned to half a day ago has a server down.
The client sent two managers, three senior Java developers, the CEO, 5 employees -they know logs and mails from the last 5 months line by line-, 4 lawyers and a beheading teacher from ISIS.
On your side there are 3 external ucraininans for the maintenance, successors of the 3 (already dead) developers who put the process in place 4 years ago according to God knows which specifications. They don't understand a word of what is being said.
Then there's the assistant of the assistant of a manager from another project that has nothing to do with this one, a feces officer, a sys admin who is going to watch porn for the whole conference call and won't listen a word, two interns to make up a number and look like you're prepared. Current objective: survive. Duration: 2 hours and a half.
- 19:30, snort some more chalk for half an hour, preparing for the mail in which you explain the associate partner how because of the aforementioned conference call we're going to lose a maintenance contract worth 20 grands per month (and a law proceeding worth a number of dollars you can't even read) and you have no idea how could this happen
- 20:00, timesheet! Compile the weekly report, write what you did and how long did it take for each task. You are allowed to compile 8 hours per day, you worked at least 11 but nobody gives a shit. Duration: 30 minutes
- 20:30, update your consultant! Training course, "tasting cum and presenting its organoleptic properties to a client". Bearing with your job: none at all. Duration: 90 minutes, then there's half an hour of evaluating test where you'll copy the answers from a sheet given to you by a colleague who left 6 months ago.
- 22:30, CHANCE CARD! You have a new mail from the HR: you asked for a refund for a 3$ sandwich, but the receipt isn't there and they realized it with a 9 months delay. You need to find that wicked piece of paper. DURATION: 30 minutes. The receipt most likely doesn't even exist anymore and will be taken directly from your next salary.
- 23:00 you receive a message on Teams. It's the intern. It's very late but you're online and have to answer. There's an exception on a process which have been running for 6 years with no problems and nobody ever touches. The intern doesn't know what to do, but you wrote the specifications for the thing, 6 years ago, and everything MUST run tonight. You are not a technician and have no fucking clue about anyhing at all. 30 minutes to make sure it's something on our side and not on the client side, and in all that the intern is as useful as a confetto to wipe your ass. Once you're sure it's something on our side you need to search for the senior dev who received the maintenance of the project, call him and solve the problem.
It turns out a file in a shared folder nobody ever touches was unreachable 'cause one of your libraries left it open during the last run and Excel shown a warning modal while opening it; your project didn't like this last thing one bit. It takes 90 minutes to find the root of the problem, you solve it by rebooting one of your machines. It's 01:00.

You shower, watch yourself on the mirror and search for the line where your forehead ends and your hair starts. It got a little bit back from yesterday; the change can't be seen with the naked eye but you know it's there.

You cry yourself to sleep. Tomorrow is another day, but it's going to be exactly like today.

Girl I work with says she's going to make a change on the "back-end of the website"... logs into Wordpress... no comment, just keep swimming

Rather than singling out one person, I wanna present what I see as incompetent/stupid/ignorant:
- no will to learn
- failure to follow the very specific instructions & later asking for help when they FUBR sth & not even knowing what they did to fuck up in the first place
- asking how to solve stuff, then ignoring the suggestions & doing sth totally against recommendations
- failure to remember most basic stuff, especially if not writing it down to look at later when needed
- failure to check logs & 'google' stuff before asking why something isn't working the way they want it
- after two weeks, asking me how feature xy works, mind you they coded it, not me
- asking me why they did something in a specific way - WTF, am I a mind reader?! Who designed that crap?! Me or you?!!
- being passive/aggressive & snarky when told to do something or being asked why isn't it done already
- not testing their shit properly
- not making backups when upgrading (production) servers
- not checking the input value, no validation.. even after many many debacles on production with null ref exceptions
- failure to admit they fucked up
- not learning from (their) mistakes

Received a urgent email from a business client saying that the application we support is completely broken. Their staff said they used the app to send several submissions that day but they did not come through. This is a major issue as these submissions need to occur daily.

I understand that this is a priority so I immediately check everything. I test the app, the server, check the database. Everything seems fine, but there's no record of these submissions. Maybe it's the specific device that was used. I reply saying that everything seems to be in order. Can I please be provided with more information about what occurred? What time were the submissions sent?

Client replies saying that the submissions were definitely sent and that the staff swear by it.

I now know something is up, so I remote into the the devices in question and check the logs. The app was not even used that day! I've got them! Those liars!

I am now quite pissed off, but remain professional and reply saying that we log all app events and that the logs show that the app had not been used at all that day. Now they have to own up to their lie. Right?

Wrong. Client replies with: The issue has been fixed. Thanks.

Can you believe the bloody nerve? The client doesn't even have the decency to apologise but rather insinuates that it was all our fault.

Well I'm not having that. I reply: It is great that the app is functioning correctly. However, I believe it is important to understand the cause of the issue as to prevent it from occuring again.

Client: No reply.

Well, if you want to waste other people's time, here's the fat bill.

Moral of the story. Don't trust anything that the client says and for any issue, debug the user before doing anything else.

So I looked at our dashboard and noticed a banner mentioning scheduled maintenance set for 7:00 AM. And I thought to myself, "I never released an update, and even if I had, the maintenance would be performed 15 minutes after the build finished, not at 7:00 AM." So I emailed my coworkers, asking if they had put up the banner, no, no. I started pulling my hair out trying to figure out what caused this banner to be created. Was there some old job that was just now running? I combed through the server logs, thousands of entries later, and I found the banner was installed by some user with the IP 172.18.0.1...which was the local machine. I went through all the users on the system, running atq to see if anyone had jobs scheduled. And there was one job scheduled, under the root user. At that moment, I legit thought to myself, "have we been hacked? How is that possible?" It's wasn't! Then I looked under /var/spool/atjobs to see what the job actually was. And then I saw it. My weekly updater cron job had installed updates and had scheduled a maintenance window to reboot the system. And I smiled, realizing that my code was now sentient.

The website for our biggest client went down and the server went haywire. Though for this client we don’t provide any infrastructure, so we called their it partner to start figuring this out.
They started blaming us, asking is if we had upgraded the website or changed any PHP settings, which all were a firm no from us. So they told us they had competent people working on the matter.

TL;DR their people isn’t competent and I ended up fixing the issue.

Hours go by, nothing happens, client calls us and we call the it partner, nothing, they don’t understand anything. Told us they can’t find any logs etc.

So we setup a conference call with our CXO, me, another dev and a few people from the it partner.
At this point I’m just asking them if they’ve looked at this and this, no good answer, I fetch a long ethernet cable from my desk, pull it to the CXO’s office and hook up my laptop to start looking into things myself.

IT partner still can’t find anything wrong. I tail the httpd error log and see thousands upon thousands of warning messages about mysql being loaded twice, but that’s not the issue here.
Check top and see there’s 257 instances of httpd, whereas 256 is spawned by httpd, mysql is using 600% cpu and whenever I try to connect to mysql through cli it throws me a too many connections error.

I heard the IT partner talking about a ddos attack, so I asked them to pull it off the public network and only give us access through our vpn. They do that, reboot server, same problems.

Finally we get the it partner to rollback the vm to earlier last night. Everything works great, 30 min later, it crashes again. At this point I’m getting tired and frustrated, this isn’t my job, I thought they had competent people working on this.

I noticed that the db had a few corrupted tables, and ask the it partner to get a dba to look at it. No prevail.

5’o’clock is here, we decide to give the vm rollback another try, but first we go home, get some dinner and resume at 6pm. I had told them I wanted to be in on this call, and said let me try this time.

They spend ages doing the rollback, and then for some reason they have to reconfigure the network and shit. Once it booted, I told their tech to stop mysqld and httpd immediately and prevent it from start at boot.

I can now look at the logs that is leading to this issue. I noticed our debug flag was on and had generated a 30gb log file. Tail it and see it’s what I’d expect, warmings and warnings, And all other logs for mysql and apache is huge, so the drive is full. Just gotta delete it.

I quietly start apache and mysql, see the website is working fine, shut it down and just take a copy of the var/lib/mysql directory and etc directory just go have backups.

Starting to connect a few dots, but I wasn’t exactly sure if it was right. Had the full drive caused mysql to corrupt itself? Only one way to find out. Start apache and mysql back up, and just wait and see. Meanwhile I fixed that mysql being loaded twice. Some genius had put load mysql.so at the top and bottom of php ini.

While waiting on the server to crash again, I’m talking to the it support guy, who told me they haven’t updated anything on the server except security patches now and then, and they didn’t have anyone familiar with this setup. No shit, it’s running php 5.3 -.-

Website up and running 1.5 later, mission accomplished.

I had a huge epiphany on Friday... not all developers enjoy coding.

Discovered when they brought down 2 of our environments, well told them what was wrong with the changes in their code that caused the environments to break, gave them links directly to the file in the gitlab repo that needed to be updated, and...

They fucking went home. The change would’ve taken all of about 30-45 seconds to update and they fucking left.

This person’s team lead come storming in pissed off because her manager is furious about 2 environments going down and preventing everyone else from being able to deploy their changes.

We provide the exact same details to the team lead about what needs to be changed, and advise that her team member took off....

30 mins later, her manager is storming up to us (devops/sre) livid as hell.

Explain the situation for a third time... manager is like, why can’t you guys fix it?

Look here you dense motherfuckers, we can fix the code. We can be the plumbers that clean up your shit. But what value do you gain as a developer if you don’t understand how the systems work and you keep pushing shit in?

Made the changes, fixed the environments, done right? Wrong.

The original developer made more changes not knowing what would happen and thoroughly fucked the environments again.

This dumb-fucking dumpster fire of a dude then sends us a slack message. “It’s down again, can you fix it?”

Our manager steps in and tells us to send him a link to the logs and have him fix it himself!

Thank goodness we have a badass manager.

Send logs, send repo file links (again), and send line numbers in the logs to try and help just a bit more. Dude goes almost the whole day without fixing it, environments are down, other devs are pissed, we throw this dude to the wolves. His manager starts to head over and was about to talk with my team lead when our manager steps out of his office and tells him the in’s and out’s of the situation and that our job isn’t to play log parser/error fixer for the developers. This dude that’s breaking the environments needs to be the one to fix the issue and his team lead should be aware of the problems and should have been able to correct his errors before it ever came to us.

The amount of hand-holding we do is ridiculous.

(Disclaimer, this one guy making some mistakes doesn’t sound too bad, but this is actually a common occurrence for like 40% of all of our developers)

We literally have interns still in college running circles around some of our full time devs. I know I’m not a developer, but for anyone that’s new-ish to developing, when you see shit like that please don’t lose hope. Those ass-hats got into programming purely for a paycheck, not because of passion.

Stick with it and your greatness will know no bounds 👍

As for you craptastic dipstick lickers, FUCK YOU!!! Go back to school and learn how to give a damn.

mysql server crashes every 18 days, no oom, no crash logs, no sigkill being sent (used auditd). so I figure it's a unknown corner case bug in mysql. now I use a cron job to restart the damn thing every week at 3am, not a problem anymore

*launch software*
> goes tits up with no info
*restart machine, launch software*
> still goes tits up
*su to the user it runs as and run it manually because fuck you shitstaind*
> still goes tits up
*launch with debug logs enabled*
> suddenly works

What is this black magic?!

*plays game for 10h consecutive*
.. yeah yeah I know what you're thinking. This guy doesn't have a life. And you're probably right.

*gets hungry*
… I could really use a hamburger right now 🤔
… But the fast food tent is ~30 minutes walking distance away, and this game automatically logs you out after 30 mins inactivity...

What if I could program in some delayed input?

*jazz hand routine engages*
Hmm.. so if I do something like:
PS C:\Users\Condor> $wshell.AppActivate('BlueStacks'); Sleep 1; $wshell.SendKeys('abusing this chat~'); Sleep 1; $wshell.SendKeys('for upkeep of 10h play~'); Sleep 1; $wshell.SendKeys('while I get myself a hamburger~'); Sleep 1; $wshell.SendKeys('sorry~');
that should work, no?

Le output:
abusing this chat
sorry

Well, even for PowerShit.. good enough, right? It gets the message across 🙃

Hmm.. let's just put an afk message instead, as I'm using the guild chat and don't want any of the members to think that I'm a freak

PS C:\Users\Condor> Sleep 1; $wshell.AppActivate('BlueStacks'); Sleep 1; $wshell.SendKeys('afk~');

.. which seems to work like a charm.. alright, perhaps I can entrust PowerShell to do that again after a 900 second delay, which should give me enough time to get that hamburger.

*comes back home*

"Logged out due to 30 minutes of inactivity."

MICROSOFT POWERSHIT, YOU'VE HAD BUT ONE FUCKING JOB!!!!

Well, guess I'll do that no-life 10+h gaming session somewhere next year again then. Thanks Powercunt!

> dockerized gitea stops working 502,
> other gitea with same config works just fine
> is the same config the issue? maybe the network names can't be the same?
> no
> any logs from the reverse proxy?
> no
> does it return anything at all on that port?
> no
> any logs inside the container?
> no
> maybe it logs to the wrong file?
> no others exist
> try to force custom log levels
> ignored
> try to kill the running pid
> it instantly restarts
> try to run a new instance with specifying the new config
> ignores config
> check if theres anything even listening
> nothing is listening on that port, but is listening in the other working gitea container
> try to destroy the container and force a fresh container
> still the same issue
> maybe the recent docker update broke it? try to make a new one and move only necessary
> mkdir gitea2
> all files seem necessary
> guess I'll try to move the same folder here
> it works
> it is exactly the same files as in gitea1, just that the folder name is different
>

I was on vacation when my employer’s new fiscal year started. My manager let me take vacation because it’s not like anything critical was going to happen. Well, joke was on us because we didn’t foresee the stupidity of others…

I had to update a few product codes in the website’s web config and deploy those changes. I was only going to be logged in for 30 minutes to complete that.

I get messaged by one of our database admins. He was doing testing and was unable to complete a payment on the website. That was strange. There was a change pushed by our offsite dev agency, but that was all frontend changes (just updating text) and wouldn’t affect payments.

We don’t want to enlist the dev agency for debugging work, especially when it’s not likely that it’s a code issue. But I was on vacation and I couldn’t stay online past the time I had budgeted for. So my employer enlists the dev agency for help. It’s going to be costly because the agency is in Lithuania, it was past their business hours, and it was emergency support.

Dev agency looks at error logs. There are Apple Pay errors, but that doesn’t explain why non Apple Pay transactions aren’t going through. They roll back my deployment and theirs, but no change. They tell my employer to contact our payment processor.

My manager and the Product Manager contact Payroll, who is the stakeholder for our payment gateways. Payroll contacts our payment gateway and finds out a service called Decision Manager was recently configured for our account. Decision Manager was declining all payments. Payroll was not the person who had Decision Manager installed and our account using this service was news to her.

Payroll works with our payment processor to get payments working again. The damage is pretty severe. Online payments were down for at least 12 hours. Our call center had logged reports from customers the night before.

At our post mortem, we had to find out who ok’d Decision Manager without telling anyone. Luckily, it was quick work. The first stakeholder up was for the Fundraising Dept. She said it wasn’t her or anyone on her team. Our VP of Analytics broke it to her that our payment processor gave us the name of the person who ok’d Decision Manager and it was someone on the Fundraising team. Fundraising then starts backtracking and says that oh yes she knew about it but transactions were still working after the Decision Manager had been configured. WTAF.

Everyone is dumbfounded by this. How could you make a big change to our payment processor and not tell anyone? How did our payment processor allow you to make this change when you’re not the account admin (you’re just a user)?

Our company head had to give an awkward speech about communication and how it’s important. The web team can’t figure out issues if you don’t tell us what you did. The company head was pissed because it was a shitty way to start off the new fiscal year. Our bill for the dev agency must have been over $1000 for debugging work that wasn’t helpful.

Amazingly, no one was fired.

So my previous alma mater's IT servers are really hacked easily. They run mostly in Microsoft Windows Server and Active Directory and only the gateway runs in Linux. When I checked the stationed IT's computer he was having problems which I think was another intrusion.

I asked the guy if I can get root access on the Gateway server. He was hesitant at first but I told him I worked with a local Linux server before. He jested, sent me to the server room with his supervision. He gave me the credentials and told me "10 minutes".

What I did?

I just installed fail2ban, iptables, and basically blocked those IP ranges used by the attacker. The attack quickly subsided.

Later we found out it was a local attack and the attacker was brute forcing the SSH port. We triaged it to one kid in the lobby who was doing the brute forcing connected in the lobby WiFi. Turns out he was a script kiddie and has no knowledge I was tracking his attacks via fail2ban logs.

Moral of lesson: make sure your IT secures everything in place.

Earlier I signed up on this forum called NulledBB. Basically some hacker skiddie forum that had a dump of an archive I wanted, unfortunately behind a paywall which I didn't want to bother with.

On signup I noticed that I couldn't use my domain as an email address, as I usually do (the domain is a catch-all which means that mail addresses can be made up for each service I sign up to on the fly, super useful). They did expose the regex that they accepted email as however, which included something along the lines of "@live.*".

So I figured, why not register a subdomain live.nixmagic.com real quick and put that into the mail servers? Didn't take too long and that's what I eventually went with, and registered as somepissedoffsysop@live.nixmagic.com (which I have no trouble putting on a public forum as you'll see in a minute).

Still didn't manage to get that archive I wanted but I figured, fuck it. It's a throwaway account anyway. But eventually that email address started to receive spam. Stupid motherfucker of a forum operator with his Kali skidmachine probably leaked it.

Usually I just blacklist the email address in SpamAssassin by adding an additional spam score of 100 to email sent to such addresses. But in that case it didn't even sit on the main domain, thanks to that stupid regex block from earlier... 😏

*Logs into my domain admin panel*
*Le rm on the live.nixmagic.com record*

Null routed entirely.. nulled, if you will! 🙃

A few years ago I was browsing Bash.org, and a user posted that he'd physically lost a machine.

A few weeks ago, I'd switched my router out for OPNSense. I figured it was time to start cleaning up my network.

Over the course of tracking down IP addresses and assigning statics to mac addresses, I spotted an IP I didn't recognize.

Being a home network, I'm pretty familiar with everything on the network by IP, so was a little taken aback.

I did some testing, found out that it was a Linux box. Cool.

I can SSH into it. Ok.

Logs show that it's running fine, no CPU/Memory/Harddrive issues. Nice.

So where is it?

Traceroute shows its connected directly to the router... Maybe over an unmanaged switch...

Hostname is "localhost"... That's no help.

I've walked the network 4 times now, and God knows where it is.

I think maybe I'll just leave it alone. If it ain't broke...

TLDR - you shouldn't expect common sense from idiots who have access to databases.

I joined a startup recently. I know startups are not known for their stable architecture, but this was next level stuff.
There is one prod mongodb server.
The db has 300 collections.
200 of those 300 collections are backups/test collections.
25 collections are used to store LOGS!! They decided to store millions of logs in a nosql db because setting up a mysql server requires effort, why do that when you've already set up mongodb. Lol 😂
Each field is indexed separately in the log.
1 collection is of 2 tb and has more than 1 billion records.
Out of the 1 billion records, 1 million records are required, the rest are obsolete. Each field has an index. Apparently the asshole DBA never knew there's something called capped collection or partial indexes.

Trying to get approval to clean up the db since 3 months, but fucking bureaucracy. Extremely high server costs plus every week the db goes down since some idiot runs a query on this mammoth collection. There's one single set of credentials for everything. Everyone from applications to interns use the same creds.
And the asshole DBA left, making me in charge of handling this shit now. I am trying to fix this but am stuck to get approval from business management. Devs like these make me feel sad that they have zero respect for their work and inability to listen to people trying to improve the system.

Going to leave this place really soon. No point in working somewhere where you are expected to show up for 8 hours, irrespective of whether you even switch on your laptop.
Wish me luck folks.

Me:
Totally riffing to my new playlist....
the ideas are just flowing.....
Code flying...
changing in my brain....
I think I've got I might have it.....
...... RING RING ITS THE MOTHERFUCKING BOSS,
Boss:
Why is the whole website down?

Me: WTF, looks fine here, all logs are clear.

Boss: I just got an email saying the whole thing is fucked. Stop everything and fix it now.

Me: but we just agreed dev is taking priority over any support issues within sla and I've checked from everywhere there are no issues, just data issues probably from user error.

Boss: Just get it back and figure it out!!!!! Why are you being difficult?

Me: okay whatever, let's patch each of these shits.

COULDVE SENT THIS ANYWHERE BUT NOW MY IDEA IS GOOOONEEE!!!!!! NULL FUCKING DATA FIELD ON A SINGLE FUCKING EMAIL....FRAAAAACKKK THIS

So I have that custom-made wifi router I've built. And it uses a USB wifi adapter with AC (wifi5) capability - the fastest one I could find in AliExpress.

I set it up a while ago - the internet access works fine, although speeds are somewhat sluggish. But hey, what to expect from a cheapo on Ali! Not to mention it's USB, not a PCIe...

A few days ago I ran a few speedtest.net tests with my actual AC router and the one I've built. Results were so different I wanted to cry :( some pathetic 23Mbps with my custom router :(

This evening I had some time on my hands and finally decided to have an umpteenth look.

nmcli d wifi
this is what caught my eye first. The RATE column listed my custom router as 54Mbps, whereas the actual router had 195Mbps.

I have reviewed the hostapd configuration sooo many times - this time nothing caught my eye as well.

Googling did not give anything obvious as well.

What do we do next? Yes, that's right - enable debug and read the logs.

> VHT (IEEE 802.11ac) with WPA/WPA2 requires CCMP/GCMP to be enabled, disabling VHT capabilities

This is one of the lines at the top of the log. Waaaaiiitttt.. VHT is something I definitely want with ac -- why does it disable that??? Sounds like a configuration fuckup rather than the HW limitation! And config fuckups CAN be fixed!

Turns out, an innocently looking
`wpa_pairwise=TKIP`
change into
`wpa_pairwise=TKIP CCMP`

made a world of a difference!

:wq
!hostapd
connect to the hostapd hotspot and run that iperf3 test again, and... Oh my. Oh boi! My pants fell off -- the speed increased >3x times!

A quick speedtest.net test deems my custom router's download speeds hardly any worse than the speeds obtained using my LInksys!!

The moral of the story: no matter how innocent some configurations look, they might make a huge difference. And RTFL [read the fucking logs]

In the pic -- left - my actual router, right - my custom-built router with a USB wifi adapter. Not too shabby!

Okay so this happened ages ago (nearly five years) but this suddenly came to my mind again.

It was in the first year of my study (currently in my 5th and last year).

I was experimenting around with php and mysql during some free hours. All the insert,delete and so on statements worked perfectly find except for one update statement. Started to debug of course and after a little while of no results I was like "oh yeah right, something like logs exists of course". Looked in the logs but nothing. No matter how I altered my code (rewrote it numerous times for some 'clean starts') it just would not run the update statement.

Alright, time for some class mate help. After multiple hours of debugging with a few classmates, there was still no result at all.

Time to bring in one or more teachers. After hours of debugging, still no result even with the help of a few good teachers.

Decided to give it a rest for that day.

Two weeks later it still was not updating anything/working and I finally gave up.

Till today, I still have no clue what went wrong and it still bugs me from time to time :/

We've had a bunch of flaky tests in our repo for a while now that no one could be bothered to fix; we'd just re-run ci until it's green. Today I looked into it and I was inspired to make this meme, because I lived through it.

Adding logs to investigate just lowered the fail rate making investigation more difficult. I do have an idea of what it might be though so, we'll see tomorrow.

So, a few years ago I was working at a small state government department. After we has suffered a major development infrastructure outage (another story), I was so outspoken about what a shitty job the infrastructure vendor was doing, the IT Director put me in charge of managing the environment and the vendor, even though I was actually a software architect.
Anyway, a year later, we get a new project manager, and she decides that she needs to bring in a new team of contract developers because she doesn't trust us incumbents.
They develop a new application, but won't use our test team, insisting that their "BA" can do the testing themselves.
Finally it goes into production.
And crashes on Day 1. And keeps crashing.
Its the infrastructure goes out the cry from her office, do something about it!
I check the logs, can find nothing wrong, just this application keeps crashing.
I and another dev ask for the source code so that we can see if we can help find their bug, but we are told in no uncertain terms that there is no bug, they don't need any help, and we must focus on fixing the hardware issue.
After a couple of days of this, she called a meeting, all the PMs, the whole of the other project team, and me and my mate. And she starts laying into us about how we are letting them all down.
We insist that they have a bug, they insist that they can't have a bug because "it's been tested".
This ends up in a shouting match when my mate lost his cool with her.
So, we went back to our desks, got the exe and the pdb files (yes, they had published debug info to production), and reverse engineered it back to C# source, and then started looking through it.
Around midnight, we spotted the bug.
We took it to them the next morning, and it was like "Oh". When we asked how they could have tested it, they said, ah, well, we didn't actually test that function as we didn't think it would be used much....
What happened after that?
Not a happy ending. Six months later the IT Director retires and she gets shoed in as the new IT Director and then starts a bullying campaign against the two of us until we quit.

Watch out for these fucking bug bounty idiots.

Some time back I got an email from one shortly after making a website live. Didn't find anything major and just ran a simple tool that can suggest security improvements simply loading the landing page for the site.

Might be useful for some people but not so much for me.

It's the same kind of security tool you can search for, run it and it mostly just checks things like HTTP headers. A harmless surface test. Was nice, polite and didn't demand anything but linked to their profile where you can give them some rep on a system that gamifies security bug hunting.

It's rendering services without being asked like when someone washes your windscreen while stopped at traffic but no demands and no real harm done. Spammed.

I had another one recently though that was a total disgrace.

"I'm a web security Analyst. My Job is to do penetration testing in websites to make them secure."
"While testing your site I found some critical vulnerabilities (bugs) in your site which need to be mitigated."
"If you have a bug bounty program, kindly let me know where I should report those issues."
"Waiting for response."

It immediately stands out that this person is asking for pay before disclosing vulnerabilities but this ends up being stupid on so many other levels.

The second thing that stands out is that he says he's doing a penetration test. This is illegal in most major countries. Even attempting to penetrate a system without consent is illegal.

In many cases if it's trivial or safe no harm no foul but in this case I take a look at what he's sending and he's really trying to hack the site. Sending all kinds of junk data and sending things to try to inject that if they did get through could cause damage or provide sensitive data such as trying SQL injects to get user data.

It doesn't matter the intent it's breaking criminal law and when there's the potential for damages that's serious.

It cannot be understated how unprofessional this is. Irrespective of intent, being a self proclaimed "whitehat" or "ethical hacker" if they test this on a site and some of the commands they sent my way had worked then that would have been a data breach.

These weren't commands to see if something was possible, they were commands to extract data. If some random person from Pakistan extracts sensitive data then that's a breach that has to be reported and disclosed to users with the potential for fines and other consequences.

The sad thing is looking at the logs he's doing it all manually. Copying and pasting extremely specific snippets into all the input boxes of hacked with nothing to do with the stack in use. He can't get that many hits that way.

After 'Dev' deployed a service using Azure ServiceBus, a particular queue/client was receiving errors.

Dev: "Looking at the logs, client is getting faulted."
Me: 'What is the error being logged?'
Dev: 'Client is faulted'
Me: 'No, that is our error when the client is either unable to connect or there is an exception in the middle of sending a message. What is the exception from Azure?'
Dev: 'Client is faulted. That's it. I'm going to have to re-engineer the code to implement a retry policy.'
<OK, I smell someone cooking up some solution finding, so I dig into the logs a little further>
Me: "Looks like an invalid connection string. The actual exception being thrown and logged is from the Azure client connection string builder. The value cannot be null."
Dev: "No, I'm looking right at the connection string in the config. Looks fine."
Me: "Looks correct on your machine, but what is actually being deployed to the server?"
<I could tell he was getting agitated>
<Dev clicks around, about 10 min. later>
Dev: "Aha!..I found it. The connection string in the config on the main branch is wrong, in fact, the entry is missing."
<dev fixes, re-deploys, life is good, I document the error and the root cause>
Boss: "Great job Dev."

*sigh* ..go teamwork?

And this, ladies and gentlemen, is why you need properly tested backups!

TL;DR: user blocked on old gitlab instance cascade deleted all projects the user was set as owner.

So, at my customer, collegue "j" reviews gitlab users and groups, notices an user who left the organisation
"j" : ill block this user
> "j" blocks user
> minutes pass away, working, minding our own business
> a wild team devops leader "k" appears
k: where are all the git projects?
> waitwut?.jpg
> k: yeah all git projects where user was owner of, are deleted
> j.feeling.despair() ; me.feeling.despair();
> checks logs on server, notices it cascade deletes all projects to that user
> lmgt log line
> is a bugreport reported 3(!) years ago
> gitlab hasnt been updated since 3 years
> gitlab system owner is not present, backup contact doesnt know shit about it
> i investigate further, no daily backup cron tasks, no backup has been made whatsoever.
> only 'backups' are on file system level, trying to restore those
> gitlab requires restore of postgres db
> backup does not contain postgres since the backup product does not support that (wtf???)
> fubar.scene
> filesystem restore finished...
> backup product did not back up all files from git tree, like none of refs were stored since the product cannot handle such filenames .. Git repo's completely broken

Fuck my life

We made a simple SMS application - when there is alert in a building, SMS is sent to specific range of numbers, based on the alert type. After a month we received invoice of approx. 10K € from the client, my colleague was supposed to pay
.
.
.
.
.
.
.
.
.
.
.
After checking the logs and running tests on module, we found out it was not our fault, client then told us he played with SIM-card while module was sending SMS and somehow he managed to fu*k up. I still don't get it, partially.. Mobile service provider doesn't give a sh!t about sending tons of SMS/hour? No warning, nothing..

Ahh.. Clients, right?

For the love of god, I spent 2,5 hours debugging why Minecraft from the windows store doesn't work...

The game just shows a red message telling you it didn't work.

I checked the logs, nothing just warnings

I re-installed the game, nothing, same error

Updated java and all parts of the store, nothing....

Obviously I had to install Something called the "xbox identity Provider"... You know... On a PC... For a distinctly PC game to work... Installed by the store... And the provider is also on the store... But it doesn't auto-install with the game

Ever since you migrated to the Microsoft Auth the login experience is awful (I ranted about that already)

How about you do the bare fucking Minimum of an User experience and Install the fucking dependencies when I re-installed something your fucking store??!!!

The fucking bare minimum that every package manager ever created fucking has as a basic requirement?! Are you kidding me?

Rename your fucking services so they make sense and please don't waste everyone's time by having both shitty logs and no dep management for your own apps... Fucks sake

OMFG I don't even know where to start..
Probably should start with last week (as this is the first time I had to deal with this problem directly)..

Also please note that all packages, procedure/function names, tables etc have fictional names, so every similarity between this story and reality is just a coincidence!!

Here it goes..

Lat week we implemented a new feature for the customer on production, everything was working fine.. After a day or two, the customer notices the audit logs are not complete aka missing user_id or have the wrong user_id inserted.
Hm.. ok.. I check logs (disk + database).. WTF, parameters are being sent in as they should, meaning they are there, so no idea what is with the missing ids.

OK, logs look fine, but I notice user_id have some weird values (I already memorized most frequent users and their ids). So I go check what is happening in the code, as the procedures/functions are called ok.

Wow, boy was I surprised.. many many times..
In the code, we actually check for user in this apps db or in case of using SSO (which we were) in the main db schema..
The user gets returned & logged ok, but that is it. Used only for authentication. When sending stuff to the db to log, old user Id is used, meaning that ofc userid was missing or wrong.

Anyhow, I fix that crap, take care of some other audit logs, so that proper user id was sent in. Test locally, cool. Works. Update customer's test servers. Works. Cool..

I still notice something off.. even though I fixed the audit_dbtable_2, audit_dbtable_1 still doesn't show proper user ids.. This was last week. I left it as is, as I had more urgent tasks waiting for me..

Anyhow, now it came the time for this fuckup to be fixed. Ok, I think to myself I can do this with a bit more hacking, but it leaves the original database and all other apps as is, so they won't break.
I crate another pck for api alone copy the calls, add user_id as param and from that on, I call other standard functions like usual, just leave out the user_id I am now explicitly sending with every call.
Ok this might work.

I prepare package, add user_id param to the calls.. great, time to test this code and my knowledge..

I made changes for api to incude the current user id (+ log it in the disk logs + audit_dbtable_1), test it, and check db..
Disk logs fine, debugging fine (user_id has proper value) but audit_dbtable_1 still userid = 0.

WTF?! I go check the code, where I forgot to include user id.. noup, it's all there. OK, I go check the logging, maybe I fucked up some parameters on db level. Nope, user is there in the friggin description ON THE SAME FUCKING TABLE!!
Just not in the column user_id...

WTF..Ok, cig break to let me think..

I come back and check the original auditing procedure on the db.. It is usually used/called with null as the user id. OK, I have replaced those with actual user ids I sent in the procedures/functions. Recheck every call!! TWICE!! Great.. no fuckups. Let's test it again!
OFC nothing changes, value in the db is still 0. WTF?! HOW!?

So I open the auditing pck, to look the insides of that bloody procedure.. WHAT THE ACTUAL FUCK?!
Instead of logging the p_user_sth_sth that is sent to that procedure, it just inserts the variable declared in the main package..
WHAT THE ACTUAL FUCK?! Did the 'new guy' made changes to this because he couldn't figure out what is wrong?! Nope, not him. I asked the CEO if he knows anything.. Noup.. I checked all customers dbs (different customers).. ALL HAD THIS HARDOCED IN!!! FORM THE FREAKING YEAR 2016!!! O.o

Unfuckin believable.. How did this ever work?!

Looks like at the begining, someone tried to implement this, but gave up mid implementation.. Decided it is enough to log current user id into BLABLA variable on some pck..
Which might have been ok 10+ years ago, but not today, not when you use connection pooling.. FFS!!

So yeah, I found easter eggs from years ago.. Almost went crazy when trying to figure out where I fucked this up. It was such a plan, simple, straight-forward solution to auditing..

If only the original procedure was working as it should.. bloddy hell!!

Did a bunch more cowboy coding today as I call it (coding in vi on production). Gather 'round kiddies, uncle Logan's got a story fer ya…

First things first, disclaimer: I'm no sysadmin. I respect sysadmins and the work they do, but I'm the first to admit my strengths definitely lie more in writing programs rather than running servers.

Anyhow, I recently inherited someone else's codebase (the story of my profession career, but I digress) and let me tell you this thing has amateur hour written all over it. It's written in PHP and JavaScript by a self-taught programmer who apparently discovered procedural programming and decided there was nothing left to learn and stopped there (no disrespect to self-taught programmers).

I could rant for days about the various problems this codebase has, but today I have a very specific story to tell. A story about errors and logs.

And it all started when I noticed the disk space on our server was gradually decreasing.

So today I logged onto our API server (Ubuntu running Apache/PHP) and did a df -h to check the disk space, and was surprised to see that it had noticeably decreased since the last time I'd checked when everything was running smoothly. But seeing as this server does not store any persistent customer data (we have a separate db server) and purely hosts the stateless API, it should NOT be consuming disk space over time at all.

The only thing I could think of was the logs, but the logs were very quiet, just the odd benign message that was fully expected. Just to be sure I did an ls -Sh to check the size of the logs, and while some of them were a little big, nothing over a few megs. Nothing to account for gigabytes of disk space gradually disappearing.

What could it be? I wondered.

cd ../..
du . | sort --sort=numeric

What's this? 2671132 K in some log folder buried in the api source code? I cd into it and it turns out there are separate PHP log files in there, split up by customer, so that each customer of ours (we have 120) has their own respective error log! (Why??)

Armed with this newfound piece of (still rather unbelievable) evidence I perform a mad scramble to search the codebase for where this extra logging is happening and sure enough I find a custom PHP error handler that is capturing (most) errors and redirecting them to these individualized log files.

Conveniently enough, not ALL errors were being absorbed though, so I still knew the main error_log was working (and any time I explicitly error_logged it would go there, so I was none the wiser that this other error-catching was even happening).

Needless to say I removed the code as quickly as I found it, tail -f'd the error_log and to my dismay it was being absolutely flooded with syntax errors, runtime PHP exceptions, warnings galore, and all sorts of other things.

My jaw almost hit the floor. I've been with this company for 6 months and had no idea these errors were even happening!

The sad thing was how easy to fix all the errors ended up being. Most of them were "undefined index" errors that could have been completely avoided with a simple isset() check, but instead ended up throwing an exception, nullifying any code that came after it.

Anyway kids, the moral of the story is don't split up your log files. It makes absolutely no sense and can end up obscuring easily fixable bugs for half a year or more!

Happy coding.

Why does FireFox has the shittiest dev tools?
Working on my website and it kept throwing "TypeError: Failed to fetch"
with no other info

Opened Chrome and that thing gave me the entire error without even modifying my logs code, and now I can peacefully solve the problem -.-

Might be a loose interpretation of 'vacation', but I was running a marathon using my phone for tunes, when suddenly I got a call from my boss; our application server had died and he had no idea how to restart it. So while running the race I was timing my exhales to give him the step-by-step instructions for reset-to-restart. The good news is that the miles just flew by as he read the logs, and I responded with commands. Suddenly I was at mile 22 and was actually feeling pretty good; didn't finish the race with a PR but was happy with the result and did get the server back up.

In the before time (late 90s) I worked for a company that worked for a company that worked for a company that provided software engineering services for NRC regulatory compliance. Fallout radius simulation, security access and checks, operational reporting, that sort of thing. Given that, I spent a lot of time around/at/in nuclear reactors.

One day, we're working on this system that uses RFID (before it was cool) and various physical sensors to do a few things, one of which is to determine if people exist at the intersection of hazardous particles, gasses, etc.

This also happens to be a system which, at that moment, is reporting hazardous conditions and people at the top of the outer containment shell. We know this is probably a red herring or faulty sensor because no one is present in the system vs the access logs and cameras, but we have to check anyways. A few building engineers climb the ladders up there and find that nothing is really visibly wrong and we have an all clear. They did not however know how to check the sensor.

Enter me, the only person from our firm on site that day. So in the next few minutes I am also in a monkey suit (bc protocol), climbing a 150 foot ladder that leads to another 150 foot ladder, all 110lbs of me + a 30lb diag "laptop" slung over my shoulder by a strap. At the top, I walk about a quarter of the way out, open the casing on the sensor module and find that someone had hooked up the line feed, but not the activity connection wire so it was sending a false signal. I open the diag laptop, plug it into the unit, write a simple firmware extension to intermediate the condition, flash, reload. I verify the error has cleared and an appropriate message was sent to the diagnostic system over the radio, run through an error test cycle, radio again, close it up. Once I returned to the ground, sweating my ass off, I also send a not at all passive aggressive email letting the boss know that the next shift will need to push the update to the other 600 air-gapped, unidirectional sensors around the facility.

When I was at university in my last semester of my bachelor's, I was doing a game programming paper and our last assignment was to group up and make a game. So I go with one of the guys I know and this other dude since his previous game was really neat. Then two randoms joined that from my first impressions of their games wasn't much at all (one guy made four buttons click and called it a game in Java when we had to make games in c++ and the other guy used an example game and semi modded it.

Anyways we get to brain storming, totally waste too much time getting organised because the guy that volunteered (4 buttons guy) was slow to getting things sorted. Eventually we get to making the game and 4 buttons guy hasn't learnt how to use git, I then end up spending 3 hours over Skype explaining to him how to do this. He eventually learns how to do things and then volunteers to do the AI for the game, after about a week (this assignment is only 5 weeks long) he hasn't shown any progress, we eventually get to our 3rd week milestone no progress from him and the modder, with only three classes left we ask them both to get stuff done before a set deadline (modder wanted to do monsters and help 4 buttons with AI) both agreed and deadline rolls up and no work is shown at all, modest shows up extremely late and shows little work.

4 buttons guy leaves us a Skype message the day of our 2nd to last class,, saying he dropped the paper...

Modder did do some work but he failed to read all the documentation I left him (the game was a 2d multiplayer crafting game, I worked so hard to make a 2d map system with a world camera) he failed to read everything and his monsters used local coordinates and were stuck on screen!

With about a week left and not too many group meetings left we meet up to try and get stuff done, modder does nothing to help, the multiplayer is working my friend has done the crafting and weapon system and the map stuff is working out well. We're missing AI and combat, with our last few hours left we push to get as much stuff done, I somehow get stuck doing monster art, AI is done by the other two and I try to getting some of the combat and building done.

In the end we completely commented all of modders work because well it made us look bad lol. He later went to complain to my free claiming I did it and was a douchebag for doing so. We had to submit our developer logs and the three of us wrote about how shitty it was to deal with these two.

We tried out best not to isolate ourselves from them and definitely tried to help but we were swamped with our other assignments and what we had to work on.

In the end leaving and not helping right when the deadline is close was what I call the most shittiest thing team mates can do, I think sticking together even if we were to fail was at least a lot better.

So we have an API that my team is supposed send messages to in a fire and forget kind of style.

We are dependent on it. If it fails there is some annoying manual labor involved to clean that mess up. (If it even can be cleaned up, as sometimes it is also time-sensitive.)

Yet once in a while, that endpoint just crashes by letting the request vanish. No response, no error, nothing, it is just gone.

Digging through the log files of that API nothing pops up. Yet then I realize the size of the log files. About ~30GB on good old plain text log files.

It turns out that that API has taken the LOG EVERYTHING approach so much too heart that it logs to the point of its own death.

Is circular logging such a bleeding edge technology? It's not like there are external solutions for it like loggly or kibana. But oh, one might have to pay for them. Just dump it to the disk :/

This is again a combination of developers thinking "I don't need to care about space! It's cheap!" and managers thinking "100 GB should be enough for that server cluster. Let's restrict its HDD to 100GB, save some money!"

And then, here I stand trying to keep my sanity :/

//little Story of a sys admin

Wondered why a Server on my Linux Root couldn't build a network connection, even when it was running.

Checked iptables and saw, that the port of the Server was redirected to a different port.

I never added that rule to the firewall. Checked and a little script I used from someone else generated traffic for a mobile game.

OK beginn the DDoS Penetration. Over 10 Gbit/s on some small servers.

Checked Facebook and some idiot posted on my site:
Stop you little shithead or I will report you to the police!!!

Checked his profile page and he had a small shitty android game with a botnet.

Choose one:
1. let him be
2. Fuck him up for good

Lets Sudo with 2.

I scaled up my bandwith to 25 Gbit/s and found out that guys phone number.

Slowly started to eat away his bandwith for days. 3 days later his server was unreachable.

Then I masked my VoIP adress and called him:

Me: Hi, you know me?
He: No WTF! Why are you calling me.
Me: I love your're game a lot, I really love it.
He: What's wrong with you? Who are you?
Me: I'm teach
He: teach?
Me: Teach me lesson
He: Are you crazy I'm hanging up!
Me: I really love you're game. I even took away all your bandwith. Now you're servers are blocked, you're game banned on the store.
He: WHAT, WHAT? (hearing typing)

Me: Don't fuck with the wrong guys. I teached you a lesson, call me EL PENETRATO

He: FUCK Fuck Fuck you! Who are you???!!! I'm going to report you!
Me: How?
He: I got you're logs!
Me: Check it at Utrace...
He: Holy shit all around the world

Me: Lemme Smash Bitch
*hung up*

Never mess with a motivated developer. I will make your life difficult in return.

Me: we need server logs and stats daily for analysis
DBA: to get those, you need to open a ticket
Me: can't you just give me SFTP access and permissions to query the stats from the DB?
DBA: No.

*OK.... 🤔🤔🤔*

*Writes an Excel Template file that I basically just need to copy and paste from to create a ticket*

This process should not take me more than 2mins 👍😁😋🙂😙😙😙😙😙😙😙😙

For them.... 😈😈😈😈😈😈😈😈😈😈😈

Partner of ours claimed they are going to update their api. No breakage. My hopes were low and they did not disappoint.

Soon after the new version of their api went live, of course, loads of breakage. And the email contact with them is really fun.

Me: "Hello, since your update we get the issue A. Here's the complete communication."

Them: "We did not change the existing behavior. You are doing X wrong. Repeat that one call during the step and you should be fine."

Me: "Thank you, if I repeat the call, it does indeed work, albeit slower, since we are now repeating calls. Furthermore, our application was consuming your api for years and we did not change anything. So why is that step necessary now? Only after your update do our logs show errors from your API. And by the way, we now also have a issue with B. Why is that?"

Them: "Oh that's because your query the endpoint with "Fnord", try "Baz".

Me: "Yes, I do know that we query it with "Fnord" as that is what a previous endpoint of yours is responding to us. Why are we getting "Fnord"? What request do I have to make to get a "Baz" back?"

It feels like a game of wackamole. Squash one issue, ten more will pop up. I am one step away from becoming active-aggressive.

Ok so I've been working on this bug for the past four days, fucking non-stop. I wanted to fucking die, was wishing I could just "pkill -f mylife". I tried fucking everything, did what the documentation told me to, stack overflow, tried different versions of the API, read through more lines of documentation than lines in the bible, to no avail. Start comparing screenshots of error logs from the past four days, notice that I started getting a line saying that it's connecting to the config file in a different location from default. I realize that the config file does not match the config file provided by the package installed, so I switch it to the default location. IT FUCKING WORKED, I've tested it nearly 10 times now and I am still in disbelief. It was a rollercoaster of emotions fixing the bug but now I'm just smiling like a fool in my chair at work now.

LPT: NEVER accept a freelance job without looking at the project's source first

Client: I have a project made by a company that is now abandoning it, I want you to fix some bugs

Me: Okay, can you:
1) Give me a build to test the current state of the game
2) Tell me what the bugs are
3) Show me the source
4) Tell me your budget

Client: *sends a list of 10 bugs* Here's the APK and to give you the project I'll need you to sign an NDA

Me: Sure...

*tests build*
*sees at least 20 bugs*
*still downloading source*
*bugs look quite easy to fix should be done under an hour*

Me: Okay, so, I can fix each bug for $10 and I can do 2 today

Client: Okay can you fix 8 bugs today for $40??

*sigh*

Me: No I cannot.

Client: okay then 2 today for $20 is fine, I want a refund if you can't fix them today

*sigh*

Me: Look dude, this isn't the first time I am doing this, aight? I'll fix the bugs today you can pay me after check they are done, savvy?

Client: okay

*source is downloaded*
*literal apes wrote the scripts, commented out code EVERYWHERE
Debug logs after every line printing every frame causing FPS drops, empty objects in the scene
multiple unused UI objects
everything is spaghetti*

*give up, after 2 hours of hell*
*tfw averted an order cancellation by not taking the order and telling client that they can pay me after I am done*

Attached is an image of a level object pool
It's an array with each element representing a level.
The numbers and "Final" are ids for objects in an object pool
The whole string is .Split(',') into an array (RIP MEMORY BTW) and then a loop goes through each element in the split array and instantiates the object from an object pool

The application has a system for sending reports and errors.
Client: "I have a problem with the app, I can not log in (android phone)"
I check the logs: "sent from iPhone"

(the person has no other account and registered telephone)

Finally finished the screwdriver followup ticket. I think.

I spent almost two full days (14 hours) on a seemingly simple bug on Friday, and then another four hours yesterday. Worse yet: I can’t test this locally due to how Apple notifications work, so I can only debug this on one particular server that lives outside of our VPN — which is ofc in high demand. And the servers are unreliable, often have incorrect configuration, missing data, random 504s, and ssh likes to disconnect. Especially while running setup scripts, hence the above. So it’s difficult to know if things are failing because there’s a bug or the server is just a piece of shit, or just doesn’t like you that day.

But the worst fucking part of all? The bug appeared different on Monday than it did on Friday. Like, significantly different.

On Friday, a particular event killed all notifications for all subsequent events thereafter, even unrelated ones, and nothing would cause them to work again. This had me diving through the bowels of several systems, scouring the application logs, replicating the issue across multiple devices, etc. I verified the exact same behavior several times over, and it made absolutely no sense. I wrote specs to verify the screwdriver code worked as expected, and it always did. But an integration test that used consumer-facing controller actions exhibited the behavior, so it wasn’t in my code.

On Monday while someone else was watching: That particular event killed all notifications but ONLY FOR RELATED EVENTS, AND THEY RESUMED AFTER ANOTHER EVENT. All other events and their notifications worked perfectly.

AKL;SJF;LSF

I think I fixed it — waiting on verification — and if it is indeed fixed, it was because two fucking push event records were treated as unique and silently failing to save, run callbacks, etc.

BUT THIS DOESN’T MATCH WHAT I VERIFIED MULTIPLE TIMES! ASDFJ;AKLSDF

I’m so fucking done with this bs.

That strange moment an entire PHP project no longer throws any errors, warnings or notices and you don’t know if you should believe it or not.

It’s been a long day trawling the logs file by file, action by action, surely I’ve missed something or it’s just waiting to break like hell when I commit the changes.

Took me 6+ full days.

The feature does not work. Repro is unknown, so only prod is experiencing the issue.. Which rules out the debugger option. Sometimes there's an entry seen in logs: "java.lang.StringIndexOutOfBoundsException". Nothing more - just that. No stack, no class, no nothing. Is it my code that's buggy? Is it some config? Integration? Unexpected response...? A bug in a lib? Is dimm faulty ir maybe server's shared libs are off?

Turns out I used a closing parentheses instead a closing curly bracket in an error message that's supposed to be interpolated...

String message = "{some-business-rule-related-error-message-key)";

took me 6+ full days... But I found it. Took the rest of that Friday off to walk in a park and enjoy my life :)

Another incident which made a Security Researcher cry
[ NOTE : Check profile to read older incidents ]
-----------------------------------------------------------
So this all started when I was at my home (bunked the office that day xD) and I got a call from a..... Let's call him Fella as I always do . So here we go . And yeah , our Fella is a SysAdmin .
-----------------------------------------------------------
Fella - Hey man sup!
Me - Good going mate , bunked the office , weather's nice , gonna spend time with my girl today . So what's goinon?
Fella - Bruh my network sharing folders ain't working no more .
Me - Did you changed or modified anything?
Fella - Nope
Me - Okay , gimme your login creds lemme check .
Fella - Check your inbox *texts me the credentials*

*I logged in and what I'm seeing is that server runs on Windows2008R2 , checked the event logs , everything's fine and all of a sudden what I found is fucking embarrassing , this wise man closed SMB service*
Me - Did you closed SMB service?
Fella - Yeah
Me - You know what it does?
Fella - Yeah it's a protocol , I turned it off to protect the server from Wannacry .
Me - Fuckerrrr!!!!! Asshole dumbass you fuckin piece of Dodo's shit!! SMB is the service responsible for files and network sharing!!!
Fella - But....I just wanted protection
Me - 😭😭😭

*A long conversation continues with a lot of specially made words to decrease the rate of frustration which I used already*

Fella - Okay I'm turning it on .
Me - Go on....... Asshole
Fella - It worked! Thanks a lot bro
Me - Just leave me and my soul away from evil and hang up .

*Now the question is , who the hell gives them the post of SysAdmin? While thinking this question , I almost thought of committing suicide​ but then my girl came with coffee and my rubber duck*

A project I'm working on uses Elastic for internal monitoring and logs. The customer asked to access those logs - not something we'd normally do, but it's isolated from other things we use and there's no critical data there, so what the heck, let them have it.

Ever since, we're getting tons of questions like "There are tons of [insert random info message] all the time, do you have any plans to resolve them?" and it gets to the point where I'm just about ready to scream back "NO, SUZAN, BOOKING NOT COMPLETED MANS THE USER F###ING CANCELLED IT, IT'S NOT SOMETHING I CAN FIX IN THE CODE"

Edit: the customer's name isn't actually Suzan

Java dev here. I rewrote an app and replaced a system call to ssh with a modern jaxrs post for uploading a file and (new) some additional data.
I even used a stream.

1 hour in production, first client doesn't get his file. Log says OutOfMemoryError: heap.

Me: wtf? I already use streams.
Looking at the Jersey library. Docs say nothing. An issue from 2013 says: oh if you silly don't use the Apache httpclient addon, we disable chunking and buffer the whole body, because our tests fail with the jdk included http client otherwise.

Me: meh.

No warning in the logs. Thank you soooooo much! Who could have known?

This begs for a rant... [too bad I can't post actual screenshots :/ ]

Me: He k8s team! We're having trouble with our k8s cluster. After scaling up and running h/c and Sanity tests environment was confirmed as Healthy and Stable. But once we'd started our load tests k8s cluster went out for a walk: most of the replicas got stoped and restarted and I cannot find in events' log WHY that happened. Could you please have a look?

k8s team [india]: Hello, thank you for reaching out to k8s support. We will check and let you know.

Me: Oh, you're welcome! I'll be just sitting here quietly and eagerly waiting for your reply. TIA! :slightly_smiling_face:

<5 minutes later>

k8s team India: Hi. Could you give me a list of replicas that were failing?

Me: I gave you a Grafana link with a timeframe filter. Look there -- almost all apps show instability at k8s layer. For instance APP_1 and APP_2 were OK. But APP_3, APP_4 and APP_5 were crashing all over the place

k8s team India: ok I will check.

<My shift has ended. k8s team works in different timezone. I've opened up Slack this morning>

k8s team India: HI. APP_1 and APP_2 are fine. I don't even see any errors from logs, no restarts. All response codes are 200.

Me: 🤦‍♂️ .... Man, isn't that what I've said? ... 🤦‍♂️

This was some time ago. A Legendary bug appeared. It worked in the dev environment, but not in the test and production environment.

It had been a week since I was working on the issue. I couldn't pinpoint the problem. We CANNOT change the code that was already there, so we needed to override the code that was written. As I was going at it, something happened.

---
Manager: "Hey, it's working now. What did you do?"
Me: *Very confused because I know I was nowhere close to finding the real source of the problem* Oh, it is? Let me check.
Also me: *Goes and check on the test and prod environment and indeed, it's already working*
Also me to the power of three: *Contemplates on life, the meaning of it, of why I am here, who's going to throw out the trash later, asking myself whether my buddies and I will be drinking tonight, only to realize that I am still on the phone with my manager*
Me again: "Oh wow, it's working."
Manager: "Great job. What were the changes in the code?"
Me: "All I did was put console logs and pushed the changes to test and prod if they were producing the same log results."
Manager: "So there were no changes whatsoever, is that what you mean?"
Me: "Yep. I've no idea why it just suddenly worked."
Manager: "Well, as long as it's working! Just remove those logs and deploy them again to the test and prod environment and add 'Test and prod fix' to the commit comment."
Me: "But what if the problem comes up again? I mean technically we haven't resolved the issue. The only change I made were like 20 lines of console logs! "
Manager: "It's working, isn't it? If it becomes a problem, we'll work it out later."
---

I did as I was told, and Lo and Behold, the problem never occurred again.

Was the system playing a joke on me? The system probably felt sorry for me and thought, "Look at this poor fucker, having such a hard time on a problem he can't even comprehend. That idiotic programmer had so many sleepless nights and yet still couldn't find the solution. Guess I gotta do my job and fix it for him. I'm the only one doing the work around here. Pathetic Homo sapiens!"

Don't get me wrong, I'm glad that it's over but..

What the fuck happened?

Rant!!!!!!!
When you work hard on building frontend and suddenly, you realise whenever you restart your localhost, some URLs don't work. And it's random. Error logs also seem meaningless as the latest error report keeps changing the error location from file to file. Wasted hours to identify the abnormal behaviour.
I always had the mentality to keep its programmers fault in order to always consider all possible flaws.
But realised later that it was the OS setting issue. Did a stacktrace about 300 lines and found out the root cause(hopefully as no issues till now). The bug was related to total allowed open files at a time.

ALWAYS read warnings guys.
Story time !
A client of ours has a synchronization app (we wrote it) between his inhouse DB and our app. (No, no APIs on their end. It’s a schelduled task).
Because we didn’t want to ask them for logs every single time, the app writes logs to disk (normal) and in Applications Insights in Azure.
When needed, I can go in portal, get all logs for last execution in a nice CSV file.
Well, recently we added more logs (Some problems were impossible to track).
So client calls us : “problem with XXX”
Me : Goes to Azure, does the same manipulation as always. Dismiss a smaaaaaalish warning without reading. Study logs. Conclusion: “The XXX is not even in the logs, check your DB”.
Little I knew, the warning was telling me “Results are truncated at 10.000 lines”.
So client was right, I was wrong and I needed to develop a small app to get logs with more than 10.000 lines. (It’s per execution. Every 3 hours)

Still dealing with the web department and their finger pointing after several thousand errors logged.

SeniorWebDev: “Looks like there were 250 database timeout errors at 11:02AM. DBAs might want to take a look.”

I look at the actual exceptions being logged (bulk of the over 1,600 logged errors)..

“Object reference not set to an instance of an object.”

Then I looked the email timestamp…11:00AM. We received the email notification *before* the database timeout errors occurred.

I gather some facts…when the exceptions started, when they ended, and used the stack trace to find the code not checking for null (maybe 10 minutes of junior dev detective work). Send the data to the ‘powers that be’ and carried on with my daily tasks.
I attached what I found (not the actual code, it was changed to protect the innocent)
Couple of hours later another WebDev replied…

WebDev: “These errors look like a database connectivity issue between the web site and the saleitem data service. Appears the logging framework doesn’t allow us to log any information about the database connection.”

FRACK!!...that Fracking lying piece of frack! Our team is responsible for the logging framework. I was typing up my response (having to calm down) then about a minute later the head DBA replies …

DBA: “Do you have any evidence of this? Our logs show no connectivity issues. The logging framework does have the ability to log an extensive amount of data regarding the database transaction. Database name, server, login, command text, and parameter values. Everything we need to troubleshoot. This is the link to the documentation …. If you implement the one line of code to gather the data, it will go a long way in helping us debug performance and connectivity issue. Thank you.”

DBA sends me a skype message “You’re welcome :)”

Ahh..nice to see someone else fed up with their lying bull...stuff.

Okay, so today I've taught a colleague how to use a simple office ruler to measure AWS server's CPU usage :) We needed to figure out whether CPU% spikes correlate with error message in logs an d latency spikes. Once again a ruler was the perfect tool for the job.

P.S. no, CPU% spikes did not correlate to errors in logs

Want to make someone's life a misery? Here's how.

Don't base your tech stack on any prior knowledge or what's relevant to the problem.

Instead design it around all the latest trends and badges you want to put on your resume because they're frequent key words on job postings.

Once your data goes in, you'll never get it out again. At best you'll be teased with little crumbs of data but never the whole.

I know, here's a genius idea, instead of putting data into a normal data base then using a cache, lets put it all into the cache and by the way it's a volatile cache.

Here's an idea. For something as simple as a single log lets make it use a queue that goes into a queue that goes into another queue that goes into another queue all of which are black boxes. No rhyme of reason, queues are all the rage.

Have you tried: Lets use a new fangled tangle, trust me it's safe, INSERT BIG NAME HERE uses it.

Finally it all gets flushed down into this subterranean cunt of a sewerage system and good luck getting it all out again. It's like hell except it's all shitty instead of all fiery.

All I want is to export one table, a simple log table with a few GB to CSV or heck whatever generic format it supports, that's it.

So I run the export table to file command and off it goes only less than a minute later for timeout commands to start piling up until it aborts. WTF. So then I set the most obvious timeout setting in the client, no change, then another timeout setting on the client, no change, then i try to put it in the client configuration file, no change, then I set the timeout on the export query, no change, then finally I bump the timeouts in the server config, no change, then I find someone has downloaded it from both tucows and apt, but they're using the tucows version so its real config is in /dev/database.xml (don't even ask). I increase that from seconds to a minute, it's still timing out after a minute.

In the end I have to make my own and this involves working out how to parse non-standard binary formatted data structures. It's the umpteenth time I have had to do this.

These aren't some no name solutions and it really terrifies me. All this is doing is taking some access logs, store them in one place then index by timestamp. These things are all meant to be blazing fast but grep is often faster. How the hell is such a trivial thing turned into a series of one nightmare after another? Things that should take a few minutes take days of screwing around. I don't have access logs any more because I can't access them anymore.

The terror of this isn't that it's so awful, it's that all the little kiddies doing all this jazz for the first time and using all these shit wipe buzzword driven approaches have no fucking clue it's not meant to be this difficult. I'm replacing entire tens of thousands to million line enterprise systems with a few hundred lines of code that's faster, more reliable and better in virtually every measurable way time and time again.

This is constant. It's not one offender, it's not one project, it's not one company, it's not one developer, it's the industry standard. It's all over open source software and all over dev shops. Everything is exponentially becoming more bloated and difficult than it needs to be. I'm seeing people pull up a hundred cloud instances for things that'll be happy at home with a few minutes to a week's optimisation efforts. Queries that are N*N and only take a few minutes to turn to LOG(N) but instead people renting out a fucking off huge ass SQL cluster instead that not only costs gobs of money but takes a ton of time maintaining and configuring which isn't going to be done right either.

I think most people are bullshitting when they say they have impostor syndrome but when the trend in technology is to make every fucking little trivial thing a thousand times more complex than it has to be I can see how they'd feel that way. There's so bloody much you need to do that you don't need to do these days that you either can't get anything done right or the smallest thing takes an age.

I have no idea why some people put up with some of these appliances. If you bought a dish washer that made washing dishes even harder than it was before you'd return it to the store.

Every time I see the terms enterprise, fast, big data, scalable, cloud or anything of the like I bang my head on the table. One of these days I'm going to lose my fucking tits.

Client: I need the push notification to have purple edges with blue background.

Me: I'm sorry. Apple does not have a support for such customization. What we can do is modify title, description and also add an overview image.

Client: No, you can make it purple edges with blue background. I saw it on dribble!

Me: *Logs out*.

La me working on a new chrome extension:

- ok, this page has some hidden divs, I need to tell the extension to make windows scroll to the bottom while there are still elements with a hidden class
- creates a while(1) loop with a condition inside it to break if no elements with hidden class are longer there.
- happy with the code
- uploads the extension
- goes to page
- brings out developer tools
- goes to console
- clicks on extension on chrome
- right clicks the extension and then inspect
- ok here we go: la me click on button inside extension popup
- console shows some logs
- nice it's still looking.
.
.
.
- wait! Why is the page not scrolling ???
- looks at logs, WTF nothing changes in logs .....
- OMMMMG a infinite loop .... infinite loop inside chrome ....
- OMMMMMG my pc's gonna crash .
-stop please stop stop.
- wait! how do I stop this?
- tries CTRL+C ... nothing
- tries CTRL+Z ... nothing ...
.
.
.
.
Abort abort Aboooooort.
.
.
.
- Deletes extension from chrome.
-..... loop still running
- clicks on X to close Chrome.....
- not closing O_o
- Oh God, i need to do something before Chrome sucks all the RAM left.
- remembers the savior...
.
.
.
- Task Manager heeeelp me.
- opens Task manager
- chrome is consuming ~ 2 GB of RAM.
- WTF! Kills chrome.

Thanks for reading my lil adventure 😅

why people around me act like dump. i have recently worked with this site, which is written in php.

customer: (yelling) my website is hacked, fix it immediately
me : ok sir, we will restore your site immediately

after finishing talk with customer. i have checked website, there is no sign of website being hacked. i have checked server logs and website for security breach, there is no sign.

me: your website is not hacked, sir. can you please tell me where you have seen hacked page.
customer: look at those pages

after seen that page i facepalmed myself. it's a bug, person who created that page just splitted string without using any multibyte function, so page is showing with corrupted characters. i fixed it and problem solved. i have told about that bug, to the person who created that page.

me: hey you have used this function which is not able to handle multibyte characters, you should use multibyte character functions for that one.
person: every characters are the same. we shouldn't need to handle that way.

he is actually a senior developer. who don't even know the difference between unicode and ascii characters.

setting up email2print kind of setup for office, Its an internal hackathon happening.. gave up last night when no jobs were being printed.. tried all debugging, logs, whatever came to the sleepy mind. Just woke and started again. found this gold on stackkverflow "check the printer cable, it might be faulty".. oh man!! finally some progress

App fails, Check logs...No error logged. Check source code and debug....
And then you see following piece of code....

try{
//Code to hit an API
}catch(Exception ex){
/*DO NOTHING. Not even log stack trace*/
}

Goodbye, a night of work!

I just typed "rm * .sh.*" instead of "rm *.sh.*" for deleting the logs from a bunch of qsubs. Yes, I removed the logs... as well as the rest of the files in the folder.

Now, probably because of the lack of sleep, I'm laughing to keep myself from crying.

No more code for today!

So we have a portal for all our University courses, where the professors also upload the presentations, etc.
Professor sends Email: Please participate in this questionnaire.
*clicks link*
"Please log in to continue"
*logs in*
Error (while I am already logged in): "Please log in to continue"
????

No wonder some people nick name the thing stupid. <.<

This just happened....

Tester: My cluster is not working properly!!!
Me: What's wrong?
Tester: I don't know. I've checked all the logs available on the entire cluster. All i know is that node 1 and 7 is broken.
*ssh into the cluster*
node1
# less /var/log/<affected application log>.log
*no errors here everything is working properly*
node7
# less /var/log/<affected application log>.log
*goes down to the bottom and scrolls up a few lines*
<insert massive error here>

Checked all the logs eh?

tl : "hey dotenv, we have a presentation with VP tomorrow, do you want to present any of your achievements in product?"

me: "umm, what achievements ?"

tl : "you know, something that you added in app which made a good impact to various metrics like DAU, MAU, less bad reviews etc"

me: "umm... i coded the tasks and features created by you folks. they got shipped at some point of your liking, and are now being tracked by you for its success failure. So i am not sure what to take credit for"

TL: "no, no.. i mean like any bugs or issues that you fixed outside of your daily jira tasks which you tracked to be a sucess"

me: "well as far as tracking is concerned, then neither i know how to track them nor i did. but yea, i identified a bug where an outdated payload was generating bad request and giving a silent failure instead of success which recently got shipped. maybe its helping users get actual response instead of "we will get back to you in some time" , so this might get considered?

TL : "oh that? that we have already added as one of the team's achievements (=PM+TL's achievement) and have tracked it to be a succes"

me : "what th- okay. then how about that api failure which was identified by AVP as "something is not right" in which the api was intermittently taking a long time to respond. he tagged me and i set up logs to identify which type of users got that issue and the actual cause of that api failure. that was definitely a good fox for app as we ended up with good reviews on playstore for our new release?"

TL : "oh that? how can you take credit for that fix? it was identified by AVP, you just added similar logs that we were using for tracking errors and implemented a fix when it came to you as a sprint task? its a team achievement"

me : "but you guys didn't identified the cause through your logs!? my log was more granular. and even if that's the case, we aren't allowed to pick any task just as is, without getting it added to sprint , right?"

TL : "nah, that was a team win"

*6 months later, during appraisal time"
TL : "Hey dotenv, you haven't displayed any leadership skills and haven't gone put of the box to improve the product. Here's your peanut appraisal 🗑️"

me : 🥲🔫🤯🪦

------------

fuck this stupid neaurocrst structure. i hate being a selfish prick than a team player, but either give credits as well as punishment to the team or gove credits as well as punishment to the single person. but wtf is thos culture of giving reward to team and punishment to individual? fckin communists

------

The company considers the project manager I work with to be the best. After working with him, I consider him to be everything that is wrong with project management.

This PM injects himself into everything and has a way of completely over-complicating the smallest of things. I will give an example:

We needed to receive around 1000 rows of data from our vendor, process each row, and host an endpoint with the data in json. This was a pretty simple task until the PM got involved and over complicated the shit out of it. He asks me what file format I need to receive the data. I say it doesnt really matter, if the vendor has the data in Excel, I can use that. After an hour long conversation about his concerns using Excel he decides CSV is better. I tell him not a problem for me, CSV works just as good. The PM then has multiple conversations with the Vendor about the specific format he wants it in. Everything seems good. The he calls me and asks how am I going to host the JSON endpoints. I tell him because its static data, I was probably going to simply convert each record into its own file and use `nginx`. He is concerned about how I would process each record into its own file. I then suggest I could use a database that stores the data and have an API endpoint that will retrieve and convert into JSON. He is concerned about the complexities of adding a database and unnecessary overhead of re-processing records every time someone hits the endpoint. No decision is made and two hours are wasted. Next day he tells me he figured out a solution, we should process each record into its own JSON file and host with `nginx`. Literally the first thing I said. I tell him great, I will do that.

Fast forward a few days and its time to receive the payload of 1000 records from the Vendor. I receive the file open it up. While they sent it in CSV format the headers and column order are different. I quietly without telling the PM, adjust my code to fit what I received, ran my unit test to make sure it processed correctly, and outputted each record into its own json file. Job is now done and the project manager gets credit for getting everything to work on the first try.

This is absolutely ridiculous, the PM has an absurd 120 hours to this task! Because of all the meetings, constant interruptions, and changing of his mind, I have 35 hours to this task. In reality the actual time I spent writing code was probably 2-3 hours and all the rest was dealing with this PM's meetings and questions and indecisiveness. From a higher level, he appears to be a great PM because of all the hours he logs but in reality he takes the easiest of tasks and turns them into a nightmare. This project could have easily been worked out between me and vendor in a 30 min conversation but this PM makes it his business to insert himself into everything. And then he has the nerve to complain that he is so overwhelmed with all the stuff going on. It drives me crazy because this inefficacy and unwanted help makes everything he touches turn into a logistical nightmare but yet he is viewed as one of the companies top Project Managers.

LEGIT FUCK ME!

So I use KDE Neon RN. One day, it randomly doesn't boot. Checked logs, sddm was fucked. Reinstalled. Everything worked fine for a day, next day: fucked. Reinstalled, disabled sddm (auto login) and then a day later: basically no services start / are missing and random parts of the os randomly crash.

Wtf do I do 😢

I work on a warehouse dev team. One day this past year, I was trying to deploy a new build to a QA server. Earlier that day I had been looking at the logs on the production server and had left the ssh session open. I had been working for less than a year out of college at this point and shouldn't have had access to deploy to the production server.

Long story short I deployed my QA build to the production server and saw there were problems connection to our production database. Then my heart dropped in my chest as I realized I had just brought down our production server.

I managed to get the server back up by rolling back in about 5 minutes and no one ever knew except some people on my team.

I felt horrible for the longest time. Later in the year another guy that joined my team that has about 20 years of experience under his belt did the exact same thing, but needed help rolling it back. Needless to say, that made me feel a lot better. 😂

Definitely the worst moment of my year.

As usual a rather clickbait title, because only the chrome extensions (as always) seem to be vulnerable:

"Warning – 3 Popular VPN Services Are Leaking Your IP Address"

"Researchers found critical vulnerabilities in three popular VPN services that could leak users' real IP addresses and other sensitive data."

"VPN Mentor revealed that three popular VPN service providers—HotSpot Shield, PureVPN, and Zenmate"

"PureVPN is the same company who lied to have a 'no log' policy, but a few months ago helped the FBI with logs that lead to the arrest of a Massachusetts man in a cyberstalking case."

"Hijack all traffic (CVE-2018-7879) "
"DNS leak (CVE-2018-7878)"
"Real IP Address leak (CVE-2018-7880)"

Parents had asked me to assemble some furniture, fix the pc and so some other "brain" work

Furniture
Me: *Stops to check something in the manual*
Parents: Are you stuck? Maybe try assembling the door upside down
Me: No, it is the right way
Parents: *Ramble, ramble*
Me: Just let me work, shut the hell up

PC
Me: *Checks cables, checks logs, ect*
Parents: Maybe it is the problem with (insert random tech word there)?
Me: No.
Parents: W h y?
Me: Let me finish, it isnt that, (tech word) isnt even correlated to that
Parents: But, but, you never talk to me... (ramble ramble)
Me: Get the hell outta the room
*Argumemt breaks out*
30min later

Me: *Finnaly manages to fix it after the heated argument*
Me: I finished, everything works
Parents: Great, but you are mean
Me: I managed to finish the work in 15 min, you dont even have enough strengh to call for a specialist, (but knowing me i wouldnt allow it anyway because a lot of them make a poor job), so in order to make it properly and to relieve you from learning how to fucking google i want you to stay out of this so i can just finish my job. Your interruptions waste time and i dont need your help at all. Everytime you tried to fix stuff you always managed to fuck shit up when you tried to do anything.
Parents: (ramble)
Me: SHUT THE FUCK UP, LEAVE MY ROOM
Parents:
Me:
Parents: *Leave the room*

1h later i get 25$ for the job perfectly done

Sometime i wish they were tech independent, so i can save my sanity and time but money is nice.

If anybody tries to argue that i should respect them:
I tried talking to them nicely countless times through years, but they always force themself to a project and they always fuck shit up because of it. I tried telling them about my problems and they tried helping me but after it didnt work they retured to the old: "it is the pc fault" and similar. Even if they couldnt help me i juat wanted them to understand my situation, but no that didnt happened.

First i fix my life then i will fix the relationship

But but greeeeeg, relationships should be cared for always!
Eat shit. There is time for family and there is time for me especialy when my life can suddenly colapse due to my problems.

There's nothing like the fresh smell of emails on Monday morning, soooooooo fucking many random emails, actually, 2,708 fucking emails, 99.9999999% are stupid useless logs or alerts that have no meaning to me, and yet, I have to setup outlook rules to filter out this shit.

Ah, another glorious Monday 😤

I think I made someone angry, then sad, then depressed.

I usually shrink a VM before archiving them, to have a backup snapshot as a template. So Workflow: prepare, test, shrink, backup -> template, document.

Shrinking means... Resetting root user to /etc/skel, deleting history, deleting caches, deleting logs, zeroing out free HD space, shutdown.

Coworker wanted to do prep a VM for docker (stuff he's experienced with, not me) so we can mass rollout the template for migration after I converted his steps into ansible or the template.

I gave him SSH access, explained the usual stuff and explained in detail the shrinking part (which is a script that must be explicitly called and has a confirmation dialog).

Weeeeellll. Then I had a lil meeting, then the postman came, then someone called.

I had... Around 30 private messages afterwards...

- it took him ~ 15 minutes to figure out that the APT cache was removed, so searching won't work
- setting up APT lists by copy pasta is hard as root when sudo is missing....
- seems like he only uses aliases, as root is a default skel, there were no aliases he has in his "private home"
- Well... VIM was missing, as I hate VIM (personal preferences xD)... Which made him cry.
- He somehow achieved to get docker working as "it should" (read: working like he expects it, but that's not my beer).

While reading all this -sometimes very whiney- crap, I went to the fridge and got a beer.

The last part was golden.

He explicitly called the shrink script.
And guess what, after a reboot... History was gone.

And the last message said:

Why did the script delete the history? How should I write the documentation? I dunno what I did!

*sigh* I expected the worse, got the worse and a good laugh in the end.

Guess I'll be babysitting tomorrow someone who's clearly unable to think for himself and / or listen....

Yay... 4h plus phone calls. *cries internally*

When there is a fire alarm where I work, someone has to go print out the door access logs so we have a list of who is (theoretically) on site and make sure no one is still inside. The printer is half way across the building. And breaks down at least once a day.

We are all going to die.

Set up a 2GB upload to run and a 6GB folder to compress while I went to do an errand. Came back to find computer had rebooted itself while I was out. No reason for it in event logs. Just a random reboot for giggles, I guess. File upload aborted with no resume and I’m unsure if the full folder compressed. Have to start over.

Saturday 9.00 AM. I was sleeping, my colleague (on holiday) sent me a text: "We got a problem on our system, probably we ran out of space". I checked the log and found out that several cron jobs failed due to not enough space on the disk. I started deleting some unnecessary logs (we're paranoid) and ended up to squeeze the vm like a lemon to save some space. Sent an email to the sysadmin, "We got to add more space ASAP, users are getting 500 errror for almost everything". Silence. I thought to myself: "Until monday we're safe..". I did a df (96%) and sent a screen to the sysadmin, just to be sure that we understood each other. Finally monday comes, nobody worries about the issue. At noon I literally takled the guy of IT dept. "Yeah, we read your email. I think the sysadmin didn't take you seriously". "Why? Which part of 'we're running out of space' isn't serious?!!!". "He just told me that we have unlimited space on that vm". Unlimited space...sure.... "Right.....the disk is at 96%, buuuuut if he said so No news to worry. Don't call me if everything burns. Have a good day!!!"

TL;DR my first vps got hacked, the attacker flooded my server log when I successfully discovered and removed him so I couldn't use my server anymore because the log was taking up all the space on the server.

The first Linux VPN I ever had (when I was a noob and had just started with vServers and Linux in general, obviously) got hacked within 2 moths since I got it.
As I didn't knew much about securing a Linux server, I made all these "rookie" mistakes: having ssh on port 22, allowing root access via ssh, no key auth...

So, the server got hacked without me even noticing. Some time later, I received a mail from my hoster who said "hello, someone (probably you) is running portscans from your server" of which I had no idea... So I looked in the logs, and BAM, "successful root login" from an IP address which wasn't me.

After I found out the server got hacked, I reinstalled the whole server, changed the port and activated key auth and installed fail2ban.
Some days later, when I finally configured everything the way I wanted, I observed I couldn't do anything with that server anymore. Found out there was absolutely no space on the server. Made a scan to find files to delete and found a logfile. The ssh logfile. I took up a freaking 95 GB of space (of a total of 100gb on the server). Turned out the guy who broke into my server got upset I discovered him and bruteforced the shit out of my server flooding the logs with failed login attempts...

I guess I learnt how to properly secure a server from this attack 💪

So, I work in a game development studio, right?
We're trying to launch the title on as many platforms as reasonable, because as a social VR app we're kinda rowing upstream.
So far, Steam and Oculus have been fairly reasonable, if oddly broken and inconsistent.
Enter store 3.
Basically no in-game transaction support (our asking prompted them to *start* developing it. No, it's not very complete). No patch-update system (You want an update? Gotta download the whole fsckin' thing!). No beta-testing functionality for most of their stuff ("Just write the code like the example, it will work, trust us!"). No tools besides the buggy SDK (Wanna upload that new build? Say hello to this page in your web browser!).
So, in other words: Fun.

We've been trying to get actively launched for two months now. Keep in mind that the build has been up on Steam and Oculus for over a year and half a year (respectively), so the actual binary functionality is, presumably fine.

The best feedback we get back tends to be "Well, when we click the Launch button it crashes, so fail."
Meanwhile we're going back and forth, dealing with other-side-of-the-world timezone lag, trying to figure out what is so different from their machines as ours. Eventually we get them to start sending logs (and no, Windows Event logs are not sufficient for GAMES, where did you even get that idea????) except the logs indicate that the program is getting killed so terribly that the engine's built-in crash handler can't even kick in to generate memory dumps or even know it died.

All this boils down to today, where I get a screenshot of their latest attempt.

I just can't even right now.

I am currently looking for a DAW (Digital Audio Workstation), because my music projects are starting to get a little too complex for Audacity.

So I started looking for a good, easy-to-learn, ideally free program, and quickly learned that Avid now has a free version of Pro Tools called First.

So I go to their site and fill out the registration form to get the download. In addition to creating an account with Avid, you also need to create one with iLok, which apparently has something to do with how they manage their licenses. Kinda overkill for a free program, but okay...

I download the program (about 3gigs...), install it and try to start it. It gives me an error message about missing some service. Okay? I'm confused because I notice that an 'Application Manager' service has appeared in my tray, and when I open that I can log into my new account just fine. But it still doesn't work.

There's a link in the error message to the iLok website, and it looks like ai need to dowload and install another component. Why didn't that get installed with the program if it's required?

Hmm...

So I go to the iLok site, download it and install it. Pro Tools First still won't start. I realize that the PTF installer asked me to reboot, which I didn't do because: a) I always have a lot of windows open, and b) How often is a reboot ACTUALLY required? Why would you need to reboot?

So I (begrudgingly) reboot, and now the program seems to start initializing... but then it throws an error message about some plugin that it can't load because it doesn't work for the 64 bit version. Then... why are you even looking for it?
And then it says something like: 'I can't handle that, I'm just gonna shut down'.

What?

I try starting it again. Same error appears, but then it gets past it this time... Only to throw another error message about something else it can't load, and therefore it must shut down.

Deep breath.

Third time is the charm, the program actually made it to the project create/load screen! Huzzah!
So I look around a bit, but don't do much. It doesn't seem too intuitive to me, so I start watching some tutorials on YouTube from Avid themselves. It's a little late by now, so I don't get my hands dirty that day.

Next time I want to try out the program I start it up, still get error messages, but it does seem to initialize okay. But then the 'Create project' button doesn't react when I press it.

It turns out that the program takes a looong time to log in to the avid account, even though the manager service is running and logged in...

When it finally logs on I create a new blank project, but it doesn't ask me where to save it to. I see there is a counter saying 1/3 and looking around I find some info about 'cloud based projects'.

It would seem that this program only supports saving projects to the cloud, and you get only 3 projects total. Three. THREE?

Ahem...

I add an instrument track to my new project and select the one and only plugin, which is a synth. I don't see the plugin window, like in the tutorials I watched. I fiddle around with the windows, but I only manage to get the layout fucked up. There's a handy 'Window' menu, but none of the options resets the view. The main window is now sporting a WINDOWS FUCKING 7 BORDER! And partially blocking the view of the top menu.

Blaaargh!

Frustrated, I shut the program down and restart it. I now select one of the project templates (after waiting for it to LOG IN AGAIN!) in the hope that I might have a bit more luck with that starting point.

But when the template has loaded, out of nowhere, the program goes from maximized to windowed mode! And the fucking Win7 border is back again, still messing with the main menu!

FFS!

I get the sucker maximized again and select one of the synth tracks, and Lo and Behold! The synth plugin window actually shows up! But of course there is no sound produced when I play, neither with the keyboard or my midi keyboard.

Oh no, that would have been too easy.

I see some the meters moving when I play, but no sound is produced. I check the options menu, but find out nothing useful except for the fact that the program only support 48kHz sample rate. That's pretty disappointing when you have a 192kHz/24bit soundcard.

I'm done. This piece of shit software is NOT for me. It's bloated, complicated to sign up for and install, extremely limited and buggy as hell!

The final insult is that it takes 5 minutes to uninstall because there is no uninstall option in the so-called 'Application Manager' (of course fucking not!), and doing it through Programs & Features there are 5 (FIVE!!) different apps and services to uninstall, one by one.

0/10, would not recommend.

TLDR; I was editing the wrong file, let's go to bed.

We have this huge system that receives data from an API endpoint, does a whole bunch of stuff, going through three other servers, and then via some calculation based on the data received from the UI, and data received from the endpoint, it finally sends the calculated fields to the UI via websocket.

Poor me sitting for over 4 hours debugging and changing values in the logic file trying to understand why one of the fields ends up being null.

Of course every change needs a reboot to all the 4 servers involved, and a hard refresh of the UI.

I even tried to search for the word null in that file, but to no avail.

After scattering hundreds of console logs, and pulling my hair out, I found out that I am editing the wrong file.

I guess it's time for some sleep.

What is it with devs who try to bloody "cost optimise" everything to within an inch of its life when there's no reason to do so?! This ain't your personal pocket money project here. This is a real commercial app with real consequences.

Seriously, saving £100 a month might seem like a lot to you, but this is a multi million pound project we're talking about. That's bloody nothing, and no-one will care. If a Fargate spot instance restarts at the wrong time and causes downtime though, or if we need logs going back a week, and don't have them because the log retention period is a few days, then everyone will be royally pissed. All because you thought "it should be ok", or it "seemed like the right thing to do". Sod off.

apparently my bank's password length limit of 10,000 is too much for paypal's app to handle and it fucking imploded. sadly, no screenshots were allowed by the app, but it fucking broke so hard it spat logs and shit at me with no formatting or anything. it was NUTS dude

Today's GDPR-Bullshittery.

So we are using an open source remote update system for updating our embedded devices.
And today we learned that, that system logs ip-adress'. And low and behold mr.GDPR says that is a no no.

So either we completely drops it, finds a new update system and implements it..
Sift through all the source code of the update system "fix" it and recompile it.
Or we setup a Man in the middle attack on ourselves. To mask the ip-adress'.

GDPR encouraging hacking ourselves I fucking love it!

Sometimes I don't know if my co-worker is that stupid or...
Well, he came to me with an strange problem with mongoose.
I looked at the error message. And guess what the database was not reachable. Asked him, did you check the mongo db service. No. Of course the service was not running. Told him to restart it. Then he restarted robo t not the service itself. Major face palm. He then asked me if I knew why his service was not running. Do I look like some kind of wizard? Told him to check the logs. Long story short, his drive ran out of space....

At the institute I did my PhD everyone had to take some role apart from research to keep the infrastructure running. My part was admin for the Linux workstations and supporting the admin of the calculation cluster we had (about 11 machines with 8 cores each... hot shit at the time).

At some point the university had some euros of budget left that had to be spent so the institute decided to buy a shiny new NAS system for the cluster.

I wasn't really involved with the stuff, I was just the replacement admin so everything was handled by the main admin.

A few months on and the cluster starts behaving ... weird. Huge CPU loads, lots of network traffic. No one really knows what's going on. At some point I discover a process on one of the compute nodes that apparently receives commands from an IRC server in the UK... OK code red, we've been hacked.

First thing we needed to find out was how they had broken in, so we looked at the logs of the compute nodes. There was nothing obvious, but the fact that each compute node had its own public IP address and was reachable from all over the world certainly didn't help.
A few hours of poking around not really knowing what I'm looking for, I resort to a TCPDUMP to find whether there is any actor on the network that I might have overlooked. And indeed I found an IP adress that I couldn't match with any of the machines.

Long story short: It was the new NAS box. Our main admin didn't care about the new box, because it was set up by an external company. The guy from the external company didn't care, because he thought he was working on a compute cluster that is sealed off behind some uber-restrictive firewall.

So our shiny new NAS system, filled to the brink with confidential research data, (and also as it turns out a lot of login credentials) was sitting there with its quaint little default config and a DHCP-assigned public IP adress, waiting for the next best rookie hacker to try U:admin/P:admin to take it over.

Looking back this could have gotten a lot worse and we were extremely lucky that these guys either didn't know what they had there or didn't care.

I run update without where on mysql console on production database Today.

CLASSIC

Just because I needed to fix database after bug fix on the backend of the application.
I thought I wrote good sql statement after executing it on my local machine and then everything got bad.

Luckily it was only one column with some cached statistics data and I checked that it was not important data before I actually started fixing stuff but still ...
Almost got hard attack afterwards.
Made a script to fix this column and it took me only 15 minutes but still...

Bug was caused in part I got no unit tests and application grow after 3 years of development from simple one for one customer and volumes of documents around 50k to over 40 customers and volumes over 2mil per month, don’t know how many pages each, just in one year after we completed all needed features.

I have daily backups and logs of every api operation but still.
I think this got to far for one backend developer.

I got scared that I will loose money cause I am contractor and the only backend developer working on it.
I am so tired of this right now I think I need a break from work.

Responsibility is killing me so hard right now.
It will take a week to get back to normal.

> Am writing code
> Life is good
> Add debugger keyword
> Script pauses
> Type in var name... Undefined.
> ...What?
> Check out local scope. It's there. What the fuck?
> Add console.log(myVariable)
> Refresh
> Logs variable no problem. Cool.
> Type in my var name
> Undefined

FFFUUUUUUU-

1 Week ago I took down the development version of an internal demo, leaving only the prod one (temporary space issue). Colleagues were told n+15 times via email, face to face, comments on tickets etc. The prod one has also been live for weeks, and again, they were told when it went up.

This just happened:

Colleague 1: practiseSafeHex can you help me, the demo doesn't seem to be working.

Me: *logs in*, *click around* ... seems fine dude. Are you using the correct URL: <prod-url>.

Colleague 1: let me try again and see.
Colleague 2: practiseSafeHex i'm with Colleague 1, were trying to use the demo, and its not working. Can you have a look, we need it.

Me: I just told him, its fine, I think you have the wrong URL <prod-url>. Can you try again.

Colleague 2: No I have the right one, can you check it.

Me: *does nothing*, yep i've had a look at it, can you try again: <prod-url>.

Colleague 2: Ah its back now, cheers.

They are with a customer now, so I won't say anything, when they get back, one of them is being castrated.

Welp, this made my night and sorta ruined my night at the same time.

He decided to work on a new gaming community but has limited programming knowledge, but has enough to patch and repair minor issues. He's waiting for an old friend of his to come back to start helping him again, so this leads to me. He needed a custom backend made for his server, which required pulling data from an SQL/API and syncing with the server, and he was falling behind pace and asked for my help. He's a good friend that I've known for a while, and I knew it wouldn't take to long to create this, so I decided to help him. Which lead to an interesting find, and sorta made my night.

It wasn't really difficult, got it done within an hour, took some time to test and fix any bugs with his SQL database. But this is where it get's interesting, at least for me. He had roughly a few hundred people that did beta testing of the server, anyways, once the new backend was hooked in and working, I realized that the other developer he works with had created a 'custom' script to make sure there are no leaks of the database. Well, that 'custom' script actually begins wiping rows/tables (Depends on the sub-table, some get wiped row by row, some just get completely dropped), I just couldn't comprehend what had happened, as rows/tables just slowly started disappearing. It took me a while of checking, before checking his SQL query logs (At least the custom script did that properly and logged every query), to realize it just basically wiped the database.

Welp, after that, it began to restrict the API I was using, and due to this it identified the server as foreign access (Since it wasn't using the same key as his plugin, even though I had an API key created just so it could only access ranks and such, to prevent abuse) and begin responding not with denied, but with a lovely "Fuck you hacker!" This really made my night, I don't know why, but I was genuinely laughing pretty hard at this response.

God, I love his developer. Luckily, I had created a backup earlier, so I patched it and just worked around the plugin/API to get it working. (Hopefully, it's not a clusterfuck to read, writing this at 2 am with less than an hour of sleep, bedtime! Goodnight everyone.)

Considering mullvad is based in sweden (xkeyscore, 14 eyes, ..) and has never seemingly had any public court record proving its claims, is it really to be trusted, instead of e.g. BlackVPN/NordVPN? does the server location matter if they are based in sweden?

Is it just again an excuse like "even if that happens, I am only hiding from X, I have nothing to hide from Y"? so e.g. your neighbour can't snoop, but a court if they decide everybody that visited devrant is a criminal - is alright?

PIA is based in the US (no discussion level of bad), but atleast got tested twice already and each time could not return any logs, even though I like mullvads model and it is clearly better than being US based, it still makes one question if mullvad is not yet another PureVPN in fancier clothes?

I often read articles describing developer epiphanies, where they realized, that it was not Eclipse at fault for a bad coding experience, but rather their lack of knowledge and lack of IDE optimization.

No. Just NO.

Eclipse is just horrendous garbage, nothing else. Here are some examples, where you can optimize Eclipse and your workflow all you like and still Eclipse demonstrates how bad of an IDE it is:

- There is a compilation error in the codebase. Eclipse knows this, as it marks the error. Yet in the Problems tab there is absolutely nothing. Not even after clean. Sometimes it logs errors in the problems tab, sometimes t doesn't. Why? Only the lord knows.

- Apart from the fact that navigating multiple Eclipse windows is plain laughable - why is it that to this day eclipse cannot properly manage windows on multi-desktop setups, e.g. via workspace settings? Example: Use 3 monitors, maximize Eclipse windows of one Eclipse instance on all three. Minimize. Then maximize. The windows are no longer maximized, but spread somehow over the monitors. After reboot it is even more laughable. Windows will be just randomly scrabled and stacked on top of each other. But the fact alone that you cannot navigate individual windows of one instance.. is this 2003?

- When you use a window with e.g. class code on a second monitor and your primary Eclipse window is on the first monitor, then some shortcuts won't trigger. E.g. attempting to select, then run a specific configuration via ALT+R, N, select via arrows, ALT+R won't work. Eclipse cannot deal with ALT+R, as it won't be able to focus the window, where the context menus are. One may think, this has to do with Eclipse requiring specific perspectives for specific shortcuts, as shortcuts are associated with perspectives - but no. Because the perspective for both windows is the same, namely Java. It is just that even though Shortcuts in Eclipse are perspective-bound, but they are also context-sensitive, meaning they require specific IDE inputs to work, regarldless of their perspective settings. Is that not provided, then the shortcut will do absolutely nothing and Eclipse won't tell you why.

- The fact alone that shortcut-workarounds are required to terminate launches, even though there is a button mapping this very functionality. Yes this is the only aspect in this list, where optimizing and adjusting the IDE solves the problem, because I can bind a shortcut for launch selection and then can reliably select ant trigger CTRL+F2. Despite that, how I need to first customize shortcuts and bind one that was not specified prior, just to achieve this most basic functionality - teminating a launch - is beyond me.

Eclipse is just overengineered and horrendous garbage. One could think it is being developed by people using Windows XP and a single 1024x768 desktop, as there is NO WAY these issues don't become apparent when regularily working with the IDE.

Come on guys, use those JSON schemas properly. The number of times I see people going "err, few strings here, any other properties ok, no properties required, job done." Dahhh, that's pointless. Lock that bloody thing down as much as you possibly can.

I mean, the damn things can be used to fail fast whenever you misspell properties, miss required properties, format dates wrong - heck, even when you want to validate the set format of an array - and then libraries will throw back an error to your client (or logs if you're just on backend) and tell you *exactly what's wrong.* It's immensely powerful, and all you have to do is craft a decent schema to get it for free.

If I see one more person trying to validate their JSON manually in 500 lines of buggy code and throwing ambiguous error messages when it could have been trivially handled by a schema, I'm going to scream.

I was working on a new feature for this legacy project.. Just minding my own buisness then all out of the blue I got an email from our client (before the email I exported the live database because I needed to reproduce a weird bug) saying the sync stopped working
<backstory>
The database needs to sync every 15 minutes because it has a master-master relationship with a 3rd party database..
</backstory>
So I was like shiiit! Did I do that!? So I checked the logs, nothing... I called the 3rd part to check if they have problems, nothing.. Then I checked the network logs... Again all fine... 30 nervous minutes later, I got a new mail... Saying it finally synced... Still have no clue what happened or if it ever was a problem... God damn clients man...

So today I found a file share containing some super super sensitive information accessible to what I think was our entire user base (6,500 users) if you knew the server name and had an interest in nosing around.

I reported it to our head of IT and heard nothing after, although 5 mins after reporting I could no longer access...

I suspect the infrastructure lead is going to be a dick (because his one of them awkward non team player kind of guys) and not thank me for preventing our company from being in national news papers... but try to spin it on why am I nosing around his servers in the first place..

I actually feel 50/50 about if I should of told or not.. but on flip side, I guess the access logs of me listing the files as I flick through to confirm my suspicions would of caused s bigger headache.

Fucking useless infrastructure engineers!

Spend literally two days trying to figure out why I have a 2 hour offset in my timezones for a lamp web app. This isn't even close to my first timezone rodeo.

Check logs, reset Apache/MySQL/PHP timezones in like 100 places. Use 3rd party server side and client side timezone libraries. Moment.js you say? Shit works like a charm... but is, of course, still two hours off.

MySQL is right. PHP is right. Apache is right. PHP libs are in place. Finally convert the entire damn project to use epoch time because I have a deadline, I have no more time to read backwater AWS docs and try to figure out why the hell this Ubuntu EC2 is fucked up, and I literally cannot figure out why in the hell the damn clock is off.

Several days later notice a variable in the main .config file... right in root... 2 hour timezone offset.

Fuuuuuuuuuuuuuuuuuuuuck.

No really. Logs have the answer, or at least point you to one.

In a conference video call of our tech team, someone remarked how he used to believe his laptop is so powerful until the day he installed Android Studio. To which I remarked, it takes somewhere around 10seconds on my machine to build this up. No wonder it became a butt of jokes, someone would say my laptop has a nitro-booster, or I build Hello World Programs. I literally had to show them screenshots and gradle build logs.

PouchDB.

It promised full-blown CRDT functionality. So I decided to adopt it.

Disappointment number one: you have to use CouchDB, so your data model is under strict regulations now. Okay.

Disappointment number two: absolutely messed up hack required to restrict users from accessing other users’ data, otherwise you have to store all the user data in single collection. Not the most performant solution.

Disappointment number three: pagination is utter mess. Server-side timestamps are utter mess. ANY server-side logic is utter mess.

Just to set it to work, you need PouchDB itself, websocket adapter (otherwise only three simultaneous syncs), auth adapter (doesn’t work via sockets), which came out fucking large pile of bullshit at the frontend.

Disappointment number four, the final one: auth somehow works but it doesn’t set cookie. I don’t know how to get access.

GitHub user named Wohali, number one CouchDB specialist over there, doesn’t know that either.

It also doesn’t work at Incognito mode, doesn’t work at Firefox at all.

So, if you want to use PouchDB, bear that in mind:
1. CouchDB only
2. No server-side logic
3. Authorization is a mess
4. Error logs are mess too: “ERROR 83929629 broken pipe” means “out of disk space” in Erlang, the CouchDB language.
5. No hosting solutions. No backup solutions, no infrastructure around that at all. You are tied to bare metal VPS and Ansible.
6. Huge pile of bullshit at frontend. Doesn’t work at Incognito mode, doesn’t work at Firefox.

>Discovers a new low level profiling tool that could help us at work with stuck process debugging and gets all hyped
>Installs on test machine, tool doesn't work
>Wonders why. Oh. Needs a kernel module to work, compiled and loaded
>"Well, its my test machine... Guess that's no problem..." but... my hype died down a bit. Kernel module installation just for a new tool that aggregates all other commonly used tools? eh... Maybe it will blow me out of my shoes still

>Installs and loads the module
>Tool works. Turns out its just a htop-like tool, with shortcuts to launch specific other profiling tools like strace/ltrace/lsof/netstat/ss etc...
"Oh... That's boring. Maybe it has all those tools built in at least?"
>Tries to run ltrace - tool exits as ltrace is not installed

Lol
>Installs ltrace and launches tool again. Tries to ltrace a process and
>Nothing. Nothing happens. For seconds... Then kicks me off of SSH
WTF?
>Tries to ping machine... silence
Did... our net go down again? (Having issues due to a storm going over our area these few days)
>Pings google and... gets instant reply
More wtf
>Pings the hypervisor the machine was running on
Works like normal

Oh... Oh no. Please tell me it didn't!
>Logs into the hypervisor UI, checks machine state
Running OK
>Opens machine console aaaaand... Yep. Stacktrace as well as a lot of kernel mumbo-jumbo... It took the machine down to kernel panic.

I never went so quick from "We need this tool deployed everywhere" to "Omg I need to get rid of this crap as soon as possible" lol.

And just for those wondering, it was sysdig.

>Be a customer
>Ask to reduce number of servers you rent at our company
>Agree to a date when the one server you no longer needs will be due to be disconnected and taken apart
>Date comes
>I have the honors of sending the final /sbin/poweroff
>All goes neatly... until...

The web that used to be there, now moved to another one of their machines, goes down, wtf???

Oh. a 500. What?
Checks logs...
Cannot connect to Database.
Wtf? Local database works... Oh. OH. OH MY GOD.

>Turn the server back on and tell the customer to fix the app to no longer connect to that machine

Sometimes, being a sysadmin can be a real fun!

So I did a code review for a colleagues pull request and I've noticed that he hasn't written the PHPDocs for a lot of the classes and functions. One minor thing I wrote is to add the author for the class.

About 2 mins after writing that comment he came over to tell me why should he write the author in the comments when people can just go look at the git commit logs. I was like WTF? I asked why would he do that, his answer was that if there's an issue, we can just use git blame to identify the author. To me that makes no sense as git blame isn't supposed to be used like that.

It's guys like these are the ones who don't document anything whether in an online document or even in code. And they just make work harder for the rest of us.

Just got off the phone with a csr about a bug they found. No biggie, I said I'd fix it. Basically until it gets fixed I told them that when they do their process to make sure to do "foo" first, then "bar" second. As soon as I got off the phone, had to poop so I went to the bathroom, and as soon as I sat down I get a message from the same CSR, "Hey I did bar first, can I type foo then bar again?"

WTF DID I JUST SAY LITERALLY 2 MINUTES AGO ON THE PHONE. TBH IT WOULD BE BETTER IF YOU JUST DIDNT DO ANYTHING FOR 15 MINUTES BUT NOW I HAVE TO COMB CLOUD FUNCTION LOGS, FIND THE DOC UID YOU CREATED, FIND THE DOC YOU MADE, DELETE IT, DELETE THE ASSIGNMENT IN YOUR TRASH ASS WORKPLACE PORTAL, AND STILL FIX THE SAID BUG