Guy called in because he wanted to get an IP white listed on a server. He wasn't authorized so i told him to send an email from an authorized email address.

He didn't like that very much and asked if another engineer was available (he talked to him more often so he thought that engineer would just do it. We need those kind of requests by email.)

Walked over to my colleague and explained what that client asked for.
'let him send an email!'

Told him i ready told the client that but that the client wanted to talk to him instead.

'sure, connect him through and then come back so you can hear him after i ask him to mail us!'

Connected him through. Client explained the situation.

Then he says with the sweetest voice and a 'get rekt' face: 'could you send me an email about that? 😊'

Let's just say that the client sounded everything but happy xD

So, I needed a package installed on one of our Unix servers. The package manager--which is obsolete garbage--was failing with a message which can only be described as a variant of "Go fuck yourself". A quick Google search didn't help.

3 espressos and an eternity later, I have descended into a manic state. My hair has turned grey and I have started lactating. As a last-ditch effort, I try a new search query on Google, and the first link takes me to a forum with a thread discussing a similar issue. The last post in the thread has a solution which works for me. After fixing the issue, everything in the world feels right and I decide to thank the generous poster, who is like an angel to me at this point.

Guess what? The poster is none other than me. 8 months back, I had created a user account on the forum just to post the solution to a similar issue I had on another server.

Was doing some work on a server today and removing loads of stuff.

rm -rf file1
Etc
Etc
Etc

Went into another directory with very important data. Wanted to do ls -la but my fingers went:
rm -rf ./
.
.
*1 milisecond later*
😶
FUCK FUCK FUCK FUCK FUCK FUCK FUCK FUCK FUCK FUCK FUCK FUCK
CTRL+C
CTRL+C
CTRL+C
CTRL+C
CTRL+C
CTRL+C
CTRL+C
CTRL+C
CTRL+C
CTRL+C
CTRL+C
CTRL+C
CTRL+C
CTRL+C
CTRL+C
*VIGOROUSLY CHECKS FILES*

Everything still there 😅

So I got the job. Here's a story, never let anyone stop you from accomplishing your dreams!

It all started in 2010. Windows just crashed unrecoverably for the 3rd time in two years. Back then I wasn't good with computers yet so we got our tech guy to look at it and he said: "either pay for a windows license again (we nearly spend 1K on licenses already) or try another operating system which is free: Ubuntu. If you don't like it anyways, we can always switch back to Windows!"

Oh well, fair enough, not much to lose, right! So we went with Ubuntu. Within about 2 hours I could find everything. From the software installer to OpenOffice, browsers, email things and so on. Also I already got the basics of the Linux terminal (bash in this case) like ls, cd, mkdir and a few more.

My parents found it very easy to work with as well so we decided to stick with it.

I already started to experiment with some html/css code because the thought of being able to write my own websites was awesome! Within about a week or so I figured out a simple html site.

Then I started to experiment more and more.

After about a year of trial and error (repeat about 1000+ times) I finally got my first Apache server setup on a VirtualBox running Ubuntu server. Damn, it felt awesome to see my own shit working!

From that moment on I continued to try everything I could with Linux because I found the principle that I basically could do everything I wanted (possible with software solutions) without any limitations (like with Windows/Mac) very fucking awesome. I owned the fucking system.

Then, after some years, I got my first shared hosting plan! It was awesome to see my own (with subdomain) website online, functioning very well!

I started to learn stuff like FTP, SSH and so on.

Went on with trial and error for a while and then the thought occured to me: what if I'd have a little server ONLINE which I could use myself to experiment around?

First rented VPS was there! Couldn't get enough of it and kept experimenting with server thingies, linux in general aaand so on.

Started learning about rsa key based login, firewalls (iptables), brute force prevention (fail2ban), vhosts (apache2 still), SSL (damn this was an interesting one, how the fuck do you do this yourself?!), PHP and many other things.

Then, after a while, the thought came to mind: what if I'd have a dedicated server!?!?!?!

I ordered my first fucking dedicated server. Damn, this was awesome! Already knew some stuff about defending myself from brute force bots and so on so it went pretty well.

Finally made the jump to NginX and CentOS!

Made multiple VPS's for shitloads of purposes and just to learn. Started working with reverse proxies (nginx), proxy servers, SSL for everything (because fuck basic http WITHOUT SSL), vhosts and so on.

Started with simple, one screen linux setup with ubuntu 10.04.

Running a five monitor setup now with many distro's, running about 20 servers with proxies/nginx/apache2/multiple db engines, as much security as I can integrate and this fucking passion just got me my first Linux job!

It's not just an operating system for me, it's a way of life. And with that I don't just mean the operating system, but also the idea behind it :).

Hello everyone, found this place recently, decided to bore you with one (or many) Navy story... tech Navy story. I'll start from the end.
Little backstory: I've deployed a simple domain setup on the ship I served, nothing fancy, a server, a switch, 10 computers, all Windows (details on that at another rant). I enter the ship Monday morning, and the XO tells me that he can't access his online folders.
OK, I say, I'll get to it. I fire up my laptop, try to RDP to the server (I know, I know, burn me at the stake later) no connection. WTF? Is the service down? I try pinging. No luck. I tried pinging the switch. OK. Looking at the switch admin panel, I see the server's port is dead. "OK, probably the cable." (we have old ethernet cables)
So, I drag my ass over to the server (same room with ship comms) with the cable tester to confirm that. What do I see?
The IMBECILES had pulled the plug from the server so that they could charge their mobile phones. I literally slammed my head against the door (calming exercise in case of spontaneous murder impulses - the things you learn at the Academy). My CO was nearby, and lucky for the guys, he heard me yell at them, while throwing mobiles and chargers around.
"But we thought it was OK, we just wanted to charge our-"
I kid you not, I reached for the firefighter's axe.
My CO grabbed me by the collar and dragged me to his room. I explained to him (between two cigarettes) that we MUST get a UPS and a server cabinet (budget constraints in the military are something that will give you people nightmares, trust me). I carefully explained to him that unless we got those, nothing would prevent the next moron from destroying confidential data and me from murdering him.
I plugged in and booted the server, after installing a multi socket extension. Two days after, surprise surprise, the server was off again. That was the first time I opened the door to the CO's room with a low kick. I must have looked like a psycho on drugs, he gave approval for the purchase in twenty seconds flat.
After that, I installed the UPS and the cabinet. Everything went inside, from the UPS to the very plugs. Just a locked box with cables coming out.
One of the guys came to my room, and asked if I could unlock the cabinet so that they could plug a "device" they needed.
I actually reached for my folding knife.
Disclaimer: The story above is TRUE. Even the almost violent parts.

Production is down, a coworker got himself locked in his own apartment so he can’t leave and another is late, the phone won’t stop ringing

And I don’t have the credentials to access the production server

Just a monday morning, everything’s fine 🔥😊🔥

So a friend of Mine asked me to check their Mail server because some emails got lost. Or had a funny signature.

Mails were sent from outlook so ok let's do this.

I go create a dummy account, and send/receive a few emails. All were coming in except one and some had a link appended. The link was randomly generated and was always some kind of referral.

Ok this this let's check the Mail Server.

Nothing.

Let's check the mail header. Nothing.

Face -> wall

Fml I want to cry.

Now I want to search for a pattern and write a script which sends a bunch of mails on my laptop.
Fuck this : no WLAN and no LAN Ports available. Fine let's hotspot the phone and send a few fucking mails.

Guess what? Fucking cockmagic, no funny mails appear!

At that moment I went out and was like chainsmoking 5 cigarettes.

BAM!

It hit me! A feeling like a unicorn vomiting rainbows all over my face.

I go check their firewall. Shit redirected all email ports from within the network to another server.

Yay nobody got credentials because nobody new it existed. Damn boy.

Hook on to the hostmachine power down the vm, start and hack yourself a root account before shit boots. Luckily I just forgot the credentials to a testvm some time ago so I know that shit. Lesson learned: fucking learn from your mistakes, might be useful sometimes!

Ok fucker what in the world are you doing.

Do some terminal magic and see that it listens on the email ports.

Holy cockriders of the galaxy.

Turns out their former it guy made a script which caught all mails from the server and injected all kind of bullshit and then sent them to real Webserver. And the reason why some mails weren't received was said guy was too dumb to implement Unicode and some mails just broke his script.

That fucker even implented an API to pull all those bullshit refs.

I know your name "Matthias" and I know where you live and what you've done... And to fuck you back for that misery I took your accounts and since you used the same fucking password for everything I took your mail, Facebook and steam account too.

Git gut shithead! You better get a lawyer

The DNS server I'm writing in PHP (largely taken from another project) is starting to work!

Next to just blocking queries it logs every blocked query so I can have stats :3

A little terminal output:

I had just started my new job and deleted 3 years of data that the client had spent over £450,000 collecting 😱

another developer used my PC to quickly access the clients database while I was out the room as I had sql management studio open. I went back to my PC thinking I was connected to my local database, did a few truncate tables to test my software and :0 minutes later I get a call asking why there was no data on the server!

Thank god for backups 😓

A typical demo...
Me: We added validation, server communication, caching....
Customer: Meh...
Me: We fixed bugs, sped up queries, implemented X features.
Customer: Meh...
Me: We surpassed the speed of light, transcended to another plane of reality, cured cancer, brought peace to galaxy.
Customer: Meh...

UI Designer: I prepared these sketches for the UI
Customer: Wow, so innovative, look at that beautiful transitions, even mobile design, just wow

Me: * dies *

My first job: The Mystery of The Powered-Down Server

I paid my way through college by working every-other-semester in the Cooperative-Education Program my school provided. My first job was with a small company (now defunct) which made some of the very first optical-storage robotic storage systems. I honestly forgot what I was "officially" hired for at first, but I quickly moved up into the kernel device-driver team and was quite happy there.

It was primarily a Solaris shop, with a smattering of IBM AIX RS/6000. It was one of these ill-fated RS/6000 machines which (by no fault of its own) plays a major role in this story.

One day, I came to work to find my team-leader in quite a tizzy -- cursing and ranting about our VAR selling us bad equipment; about how IBM just doesn't make good hardware like they did in the good old days; about how back when _he_ was in charge of buying equipment this wouldn't happen, and on and on and on.

Our primary AIX dev server was powered off when he arrived. He booted it up, checked logs and was running self-diagnostics, but absolutely nothing so far indicated why the machine had shut down. We blew a couple of hours trying to figure out what happened, to no avail. Eventually, with other deadlines looming, we just chalked it up be something we'll look into more later.

Several days went by, with the usual day-to-day comings and goings; no surprises.

Then, next week, it happened again.

My team-leader was LIVID. The same server was hard-down again when he came in; no explanation. He opened a ticket with IBM and put in a call to our VAR rep, demanding answers -- how could they sell us bad equipment -- why isn't there any indication of what's failing -- someone must come out here and fix this NOW, and on and on and on.

(As a quick aside, in case it's not clearly coming through between-the-lines, our team leader was always a little bit "over to top" for me. He was the kind of person who "got things done," and as long as you stayed on his good side, you could just watch the fireworks most days - but it became pretty exhausting sometimes).

Back our story -
An IBM CE comes out and does a full on-site hardware diagnostic -- tears the whole server down, runs through everything one part a time. Absolutely. Nothing. Wrong.

I recall, at some point of all this, making the comment "It's almost like someone just pulls the plug on it -- like the power just, poof, goes away."

My team-leader demands the CE replace the power supply, even though it appeared to be operating normally. He does, at our cost, of course.

Another weeks goes by and all is forgotten in the swamp of work we have to do.

Until one day, the next week... Yes, you guessed it... It happens again. The server is down. Heads are exploding (will at least one head we all know by now). With all the screaming going on, the entire office staff should have comped some Advil.

My team-leader demands the facilities team do a full diagnostic on the UPS system and assure we aren't getting drop-outs on the power system. They do the diagnostic. They also review the logs for the power/load distribution to the entire lab and office spaces. Nothing is amiss.

This would also be a good time draw the picture of where this server is -- this particular server is not in the actual server room, it's out in the office area. That's on purpose, since it is connected to a demo robotics cabinet we use for testing and POC work. And customer demos. This will date me, but these were the days when robotic storage was new and VERY exciting to watch...

So, this is basically a couple of big boxes out on the office floor, with power cables running into a special power-drop near the middle of the room. That information might seem superfluous now, but will come into play shortly in our story.

So, we still have no answer to what's causing the server problems, but we all have work to do, so we keep plugging away, hoping for the best.

The team leader is insisting the VAR swap in a new server.

One night, we (the device-driver team) are working late, burning the midnight oil, right there in the office, and we bear witness to something I will never forget.

The cleaning staff came in.

Anxious for a brief distraction from our marathon of debugging, we stopped to watch them set up and start cleaning the office for a bit.

Then, friends, I Am Not Making This Up(tm)... I watched one of the cleaning staff walk right over to that beautiful RS/6000 dev server, dwarfed in shadow beside that huge robotic disc enclosure... and yank the server power cable right out of the dedicated power drop. And plug in their vacuum cleaner. And vacuum the floor.

We each looked at one-another, slowly, in bewilderment... and then went home, after a brief discussion on the way out the door.

You see, our team-leader wasn't with us that night; so before we left, we all agreed to come in late the next day. Very late indeed.

A: "Hey let's move our server rack to this room tomorrow so we can set it up."
Me: "sure, 11:00?"
A: "11:00 it is"
Me: arrives 11:00, waits till 12:00.
A: arrives at 12:00 "sup man, sorry I'm late, let's do this"
Me: annoyance-level 6/10 "sure... let's go"
A: "hold on, this guy wants something"
Me: waits another 45 minutes.
annoyance-level 9/10

I just went to cafeteria.
Fuck you, wasting almost 2h.
I stood up early for this bs.

My worst fear while being the on-call/standby server engineer is that I have to call another engineer awake at night because I can't figure something out 😅

Had a former customer call a few (10) years ago, furious that we shut down their website.

Me: but you moved your website to another provider 2 years ago? We dont even provide webhosting any more?

Turns out, when they moved we put a redirect to their new site on our server.

During the summer this server was decommisoned due to a failing harddrive but since we no longer had any active customers on it we just pulled the plug.

The customer had never actually redirected their domain name. :)

When we sent them a copy of their own cancellation letter we got an "oh hell, sorry".

:D

One of the linux engineers gets up with: 'I'm sorry, I just pushed a windows server sale to the sales guy :/'

Linux engineer 1: WHAT HAVE YOU DONE D:
Linux engineer 2: WHY WOULD YOU DO THAT @_@
Linux engineer 3: Why the FUCK would we want to manage yet another piece of problems?!

Engineer who said it: 'But it's still a sale guys....'

All engineers: THAT'S NOT A VALID EXCUSE O_o

😆

I know that my coworker can't write a single fucking operable line of code. So I wrote a script that is called everytime someone pushes new commits. If the commits contain the username of my coworker, create a ticket in YouTrack with the Label "Rewrite", and assign it to the files changed.
So I had that running for a longer time, and my dumbfuck of coworker hardcoded the credentials of the server in a networking library. One of the credentials was his username. He then updated the copyright on the whole project(which adds a copyright in the top of every file), also in the included librarys(!). The script had a check if the files are related to the project or just librarys. In the end, he pushed all of that with another account(in fact, a reporter account), which had another name(and didn't even belong him). So the files didn't belong to the project, the script sees his username anyways, the script assigns a rewrite, and in the end, everyone in the team thinks I'm mad because I(the script with my account) assigned a rewrite to a HUGE library.

PS: It was great fun to remove these copyright notices.

So, here's how online payment works in my country:

1. The selling website creates a hidden <form> and populates it with product ID, price and etc.

2. Some Javascript posts the mentioned form to the bank from the browser.

3. User enters credit info, and submits. If all is right, the bank creates another hidden <form>, and populates is with status code and an invoice ID.

4. Said form is then posted back to the selling website.

I don't know how the programmers behind this scenario call themselves programmers if they don't know basic things about server-side only verifications, but thanks to them I've been buying a lot of products for free these past years. 😂😂

How? 1. Just install Requestly, Tampermonkey and enable Chrome's dev tools.
2. Change price to zero, and the bank's response code to success.
3. Profit!

P. S. I have notified the people behind this, but they don't listen and go fix their codes. Oh well, serves them right.

Biggest scaling challenge I've faced?

Around 2006~2007 the business was in double-digit growth thanks to the eCommerce boom and we were struggling to keep up with the demand.

Upper IT management being more hardware focused and always threw more hardware at the problem. At its worst, we had over 25 web servers (back then, those physical tall-rectangle boxes..no rack system yet) and corresponding SQL server for each (replicated from our main sql server)

Then business boomed again and projected the need for 40 servers (20 web servers, 20 sql servers) over the next 5 years. Hardware+software costs (they were going to have to tear down a wall in order to expand the server room) were going to be in the $$ millions.

Even though we were making money, the folks spending it didn't seem to care, but I knew this trajectory was not sustainable, so I started utilizing (this was 2007) WCF services and Microsoft's caching framework Velocity. Started out small, product lookup data (description, price, the simple stuff) and within a month, I was able to demonstrate the web site could scale with less than half of our current hardware infrastructure.

After many political battles (I've ranted about a few of those), the $$ won and even with the current load, we were able to scale back to 5 web servers and 2 sql servers. When the business increased in the double-digits again, and again...we were still the same hardware for almost 5 years. We only had to add another service server when the international side of the business started taking off.

Challenge wasn't the scaling issue, the challenge was dealing with individuals who resisted change.

I worked in the same building as another division in my organization, and they found out I had created a website for my group. They said, “We have this database that was never finished. Do you think you could fix it?”

I asked, “What was it developed in?”
He replied, “Well what do you know?”
I said, “LAMP stack: PHP, MySQL, etc.” [this was over a decade ago]

He excitedly exclaimed, “Yeah, that’s it! It’s that S-Q-L stuff.”

I’m a little nervous at this point but I was younger than 20 with no degree, entirely self-taught from a book, and figured I’d check it out - no actual job offer here yet or anything.

They logged me on to a Windows 2000 Server and I become aware it’s a web application written in VB / ASP.NET 2.0 with a SQL Server backend. But most of the fixes they wanted were aesthetic (spelling errors in aspx pages, etc.) so I proceeded to fix those. They hired me on the spot and asked when I could start. I was a wizard to them and most of what they needed was quite simple (at first). I kept my mouth shut and immediately went to a bookstore after work that day and bought an ASP.NET book.

I worked there several years and ended up rewriting that app in C# and upgrading the server and ASP.NET framework, etc. It stored passwords in plaintext when I started and much more horrific stuff. It was in much better shape when I left.

That job was pivotal in my career and set the stage for me to be where I am today. I got the job because I used the word “SQL” in a sentence.

Hello Monday:
0.Arrive late due to traffic.(Apparently a car hit a cow crossing the road)
1. Try upgrading php5 to php7 and break stuff in the process and waste 2 hours fixing things.(Poor connection so ssh sessions hung occasionally)
2.PHP fixed,open Gmail and get over 100 emails from clients about the server being down(because of (0)).Ignore all.Find a snaglist of over 20 TODOs.
3.Open Android Studio, update to 2.3 and everything becomes broken.Each time i open it ,it crashes and i have to "Report to Google"
4.Spend the next 1 hour reinstalling AS.It finally works.
5.Open Project and the libraries are broken.Spend another hour upgrading build tools.
6.Leave SDK to update and decide to check my Google Cloud console.$50 bill pending.Shit.
7.Try XCode. Remember the project is still in Swift 2 and I have to upgrade it(Would take eternity).Immediately closes xcode.
8.Gives up on life and decides to log into Devrant.

What an awful day :(

The server where I host my 4 clients websites crashed.

Unable to reboot from the console.

I contact the support. 15 minutes later: "we'll look at this"

No news for 1 week despite my messages.

Then... 1st ticket escalation... 2nd ticket escalation... 3rd ticket escalation...

Answer: "Sorry, your server is down and cannot be repaired."

Fuck.

I ask "is there any way to get my data back?". Answer: "No, because we would shutdown the whole bay and all our clients would be impacted".

Fuck.

I subscribe to another server, at another provider.

I look at my backups... shit, the last one is 4 month ago!!

I restore the first website: OK

I restore the second website: OK

I restore the third website: My new server is "too recent" and not compatible. with this old Wordpress. Fuck! I'll look at this later...

I restore the fourth website: database is empty!! What??? I look at the SQL backup for this site... it failed...

I lost ALL my 4th client data!!!

I'm sooooo piece of crap!

Waaaay too many but let's go with this one for now.

At my previous job there was a web application which was generating about 1gb of log data a second. Server was full and the 'fullstack engineers' we called had zero clue about backend stuff and couldn't fix it.

Me and another engineer worked our asses off to figure this out but eventually the logging stopped and it went back to normal.

Great, right?

For that moment. I was the on-call server engineer and at like 3am I got called awake because this shit was happening again.
Sleep drunk with my phone I ssh'd into the server, not sure about what to do at first but then suddenly: let's chattr the goddamn log file...

$ chattr +i /var/log/logfile

Bam, worked, done, back to sleep.

(this comment + param marks the file in a way that it can only be read until the mark is removed, so you can't write to it or move it or remove it or whatever)

Soms week ago a client came to me with the request to restructure the nameservers for his hosting company. Due to the requirements, I soon realised none of the existing DNS servers would be a perfect fit. Me, being a PHP programmer with some decent general linux/server skills decided to do what I do best: write a small nameservers which could execute the zone transfers... in PHP. I proposed the plan to the client and explained to him how this was going to solve all of his problems. He agreed and started worked.

After a few week of reading a dozen RFC documents on the DNS protocol I wrote a DNS library capable of reading/writing the master file format and reading/writing the binary wire format (we needed this anyway, we had some more projects where PHP did not provide is with enough control over the DNS queries). In short, I wrote a decent DNS resolver.

Another two weeks I was working on the actual DNS server which would handle the NOTIFY queries and execute the zone transfers (AXFR queries). I used the pthreads extension to make the server behave like an actual server which can handle multiple request at once. It took some time (in my opinion the pthreads extension is not extremely well documented and a lot of its behavior has to be detected through trail and error, or, reading the C source code. However, it still is a pretty decent extension.)

Yesterday, while debugging some last issues, the DNS server written in PHP received its first NOTIFY about a changed DNS zone. It executed the zone transfer and updated the real database of the actual primary DNS server. I was extremely euphoric and I began to realise what I wrote in the weeks before. I shared the good news the client and with some other people (a network engineer, a server administrator, a junior programmer, etc.). None of which really seemed to understand what I did. The most positive response was: "So, you can execute a zone transfer?", in a kind of condescending way.

This was one of those moments I realised again, most of the people, even those who are fairly technical, will never understand what we programmers do. My euphoric moment soon became a moment of loneliness...

More sysadmin focused but y’all get this stuff and I need a rant.

TLDR: Got the wrong internship.

Start working as a sysadmin/dev intern/man-of-many-hats at a small finance company (I’m still in school). Day 1: “Oh new IT guy? Just grab a PC from an empty cubicle and here’s a flash drive with Fedora, go ahead and manually install your operating system. Oh shit also your desktop has 2g of ram, a core2 duo, and we scavenged your hard drive for another dev so just go find one in the server room. And also your monitor is broken so just take one from another cubicle.”

Am shown our server room and see that someone is storing random personal shit in there (golf clubs propped against the server racks with heads mixed into the cabling, etc.). Ask why the golf clubs etc. are mixed in with the cabling and server racks and am given the silent treatment. Learn later that my boss is the owners son, and he is storing his personal stuff in our server room.

Do desktop support for end users. Another manager asks for her employees to receive copies of office 2010 (they’re running 2003 an 2007). Ask boss about licensing plans in place and upgrade schedules, he says he’ll get back to me. I explain to other manager we are working on a licensing scheme and I will keep her informed.

Next day other manager tells me (*the intern*) that she spoke with a rich business friend whose company uses fake/cracked license keys and we should do the same to keep costs down. I nod and smile. IT manager tells me we have no upgrade schedule or licensing agreement. I suggest purchasing an Office 365 subscription. Boss says $150 a year per employee is too expensive (Company pulls good money, has ~25 employees, owner is just cheap) I suggest freeware alternatives. Other manager refuses to use anything other than office 2010 as that is what she is familiar with. Boss refuses to spend any money on license keys. Learn other manager is owners wife and mother of my boss. Stalemate. No upgrades happen.

Company is running an active directory Windows Server 2003 instance that needs upgrading. I suggest 2012R2. Boss says “sure”. I ask how he will purchase the license key and he tells me he won’t.

I suggest running an Ubuntu server with LDAP functionality instead with the understanding that this will add IT employee hours for maintenance. Bosses eyes glaze over at the mention of Linux. The upgrade is put off.

Start cleaning out server room of the personal junk, labeling server racks and cables, and creating a network map. Boss asks what I’m doing. I show him the organized side of the server room and he says “okay but don’t do any more”.

... *sigh* ...

My team handles infrastructure deployment and automation in the cloud for our company, so we don't exactly develop applications ourselves, but we're responsible for building deployment pipelines, provisioning cloud resources, automating their deployments, etc.

I've ranted about this before, but it fits the weekly rant so I'll do it again.

Someone deployed an autoscaling application into our production AWS account, but they set the maximum instance count to 300. The account limit was less than that. So, of course, their application gets stuck and starts scaling out infinitely. Two hundred new servers spun up in an hour before hitting the limit and then throwing errors all over the place. They send me a ticket and I login to AWS to investigate. Not only have they broken their own application, but they've also made it impossible to deploy anything else into prod. Every other autoscaling group is now unable to scale out at all. We had to submit an emergency limit increase request to AWS, spent thousands of dollars on those stupidly-large instances, and yelled at the dev team responsible. Two weeks later, THEY INCREASED THE MAX COUNT TO 500 AND IT HAPPENED AGAIN!

And the whole thing happened because a database filled up the hard drive, so it would spin up a new server, whose hard drive would be full already and thus spin up a new server, and so on into infinity.

Thats probably the only WTF moment that resulted in me actually saying "WTF?!" out loud to the person responsible, but I've had others. One dev team had their code logging to a location they couldn't access, so we got daily requests for two weeks to download and email log files to them. Another dev team refused to believe their server was crashing due to their bad code even after we showed them the logs that demonstrated their application had a massive memory leak. Another team arbitrarily decided that they were going to deploy their code at 4 AM on a Saturday and they wanted a member of my team to be available in case something went wrong. We aren't 24/7 support. We aren't even weekend support. Or any support, technically. Another team told us we had one day to do three weeks' worth of work to deploy their application because they had set a hard deadline and then didn't tell us about it until the day before. We gave them a flat "No" for that request.

I could probably keep going, but you get the gist of it.

Today my boss asked me if I wanted to travel to another country to setup a new server for a customer.
Pretty good for a student worker I think 😁. Today was a great day

Our website once had it’s config file (“old” .cgi app) open and available if you knew the file name. It was ‘obfuscated’ with the file name “Name of the cgi executable”.txt. So browsing, browsing.cgi, config file was browsing.txt.
After discovering the sql server admin password in plain text and reporting it to the VP, he called a meeting.
VP: “I have a report that you are storing the server admin password in plain text.”
WebMgr: “No, that is not correct.”
Me: “Um, yes it is, or we wouldn’t be here.”
WebMgr: “It’s not a network server administrator, it’s SQL Server’s SA account. Completely secure since that login has no access to the network.”
<VP looks over at me>
VP: “Oh..I was not told *that* detail.”
Me: “Um, that doesn’t matter, we shouldn’t have any login password in plain text, anywhere. Besides, the SA account has full access to the entire database. Someone could drop tables, get customer data, even access credit card data.”
WebMgr: “You are blowing all this out of proportion. There is no way anyone could do that.”
Me: “Uh, two weeks ago I discovered the catalog page was sending raw SQL from javascript. All anyone had to do was inject a semicolon and add whatever they wanted.”
WebMgr: “Who would do that? They would have to know a lot about our systems in order to do any real damage.”
VP: “Yes, it would have to be someone in our department looking to do some damage.”
<both the VP and WebMgr look at me>
Me: “Open your browser and search on SQL Injection.”
<VP searches on SQL Injection..few seconds pass>
VP: “Oh my, this is disturbing. I did not know SQL injection was such a problem. I want all SQL removed from javascript and passwords removed from the text files.”
WebMgr: “Our team is already removing the SQL, but our apps need to read the SQL server login and password from a config file. I don’t know why this is such a big deal. The file is read-only and protected by IIS. You can’t even read it from a browser.”
VP: “Well, if it’s secured, I suppose it is OK.”
Me: “Open your browser and navigate to … browse.txt”
VP: “Oh my, there it is.”
WebMgr: “You can only see it because your laptop had administrative privileges. Anyone outside our network cannot access the file.”
VP: “OK, that makes sense. As long as IIS is securing the file …”
Me: “No..no..no.. I can’t believe this. The screen shot I sent yesterday was from my home laptop showing the file is publicly available.”
WebMgr: “But you are probably an admin on the laptop.”
<couple of awkward seconds of silence…then the light comes on>
VP: “OK, I’m stopping this meeting. I want all admin users and passwords removed from the site by the end of the day.”

Took a little longer than a day, but after reviewing what the web team changed:
- They did remove the SQL Server SA account, but replaced it with another account with full admin privileges.
- Replaced the “App Name”.txt with centrally located config file at C:\Inetpub\wwwroot\config.txt (hard-coded in the app)
When I brought this up again with my manager..
Mgr: “Yea, I know, it sucks. WebMgr showed the VP the config file was not accessible by the web site and it wasn’t using the SA password. He was satisfied by that. Web site is looking to beat projections again by 15%, so WebMgr told the other VPs that another disruption from a developer could jeopardize the quarterly numbers. I’d keep my head down for a while.”

So my friend and I are trying to launch his bot in his Discord server. I'm typing node index.js: nothing. He tells me to delete the points.json folder. I try launching it again. NOTHING. I then ask him what I do then, so he sends me another .js file. Well it turns out I didn't have Node installed in the first place when I thought I already had. Give if take a good 10 minutes later, the bot is online. My favorite commands don't work because I was missing those files.

TL;DR: disaster averted!

Story time!

About a year ago, the company I work for merged with another that offered complementary services. As is always the case, both companies had different ways of doing things, and that was true for the keeping of the financial records and history.

As the other company had a much larger financial database, after the merger we moved all the data of both companies on their software.

The said software is closed source, and was deployed on premises on a small server.

Even tho it has a lot of restrictions and missing features, it gets the job done and was stable enough for years.

But here comes the fun part: last week there was a power outage. We had no failsafe, no UPS, no recent backups and of course both the OS and the working database from the server broke.

Everyone was in panic mode, as our whole company needs the software for day to day activity!

Now, don't ask me how, but today we managed to recover all the data, got a new server with 2 RAID HDDs for the working copy of the DB, another pair for backups, and another machine with another dual HDD setup for secondary backups!

We still need a new UPS and another off site backup storage, but for now...disaster averted!

Time for a beer! Or 20...

That is all :)

We had issues with lack of disk space on our production SQL server. Another developer decided to delete the databases he thought weren't in use to clear some space.

Ever think about checking first?!

Production chaos!

This tiny project is awesome. Thanks to @JoshBent (who partly got it from another repo as well) for providing a basic DNS server with hardcoded blacklisting functionality and thanks to @PerfectAsshole for correcting my mysql syntax I was stuck on for way too long.

I've now got this fucker to read blacklisted words from a redis list into an array which checks every requested domain to see if it matches. If yes, it proxies it through to another DNS server and if not, it'll log the requested domain to a mysql database and prints is as blocked onto the terminal.

If the domain matches any host from a service known to be integrated within a mass surveillance network, it also prints this out to thy terminal.

It's working yay! Gonna keep working on it today.

My very first rant. I will try to make it count!

Recently my company had restructured with new GM who fired those who he didn't like and hired those who he did.

All ok I guess. And then...

We did a project for a big sportswear brand, interactive installation using WebSocket. My team developed the server with game engine and we also developed "shell front-end" with example code how to connect and interact with server.

Then we realized that our company was actually not hired by sportswear brand, rather by GM buddy-from-the-army tech vendor, who was hired by actual client.

Then, we also realized there is another company, that GM hired as freelancers, sitting at our office doing 'something'. And this 'other company' actually has same employees and address as GM's buddy company (meaning they are one entity with two names).

So.. what happened is that we get paid by party A, then we pay exactly the same amount to party B (which is actually same party A), and at the end... all our work was for free.

And then, this GM tells me I have to fire 2 developers from my team because we don't have enough business.

How did this guy even get to a position of this level?

Setup a pi to be functioning as vpn server and pihole tonight.

Now working on a Jasper voice assistant on another pi, because I can choose for offline speech processing and don't have to use a mass surveillance network for this (google).

Today is a good day.

Always the same story:

Marketing: hey I'm gonna do a demo to a customer. They were asking for feature XYZ. That's ready on thr staging server right? Do you think I could use the staging server for the demo?

Devs: well feature XYZ is not 100% done. Basically just feature X is done, and it still has a few bugs. The deadline ain't for another month, since we gotta finish ABC first. I guess you could use the staging, but it has a lot of bugs.

Marketing: perfect!

*after presentation*

Marketing: the staging had so many bugs! Why didn't you tell me?! It was so embarrassing showing it to new customers! Anyway, they loved the new feature. We need it to be ready ASAP.

Devs: What?! That's gonna mess up with our schedule. You know what? Fine, but feature ABC will have to wait another month.

Marketing: Well, it'd be ideal if we could do both...

Devs: Pay for more devs or dor extra hours.

Marketing: Just do XYZ. It's a pity that you'll have to push back ABC but it's fine, XYZ is more important.

(I might ask, if it was so important, why didn't you notice so in the meeting where we had decided that ABC would be prioritized?)

*tons of working hours later*

Devs: There, we finished XYZ.

Marketing: Yay! Wow, this month we'll have two major features done: ABC and XYZ!

Devs: No, ABC is not done yet.

Marketing: What? But the deadline was this week.

Devs: It was, but then you decided to prioritize XYZ and we said we had to push back ABC to get XYZ ready, and you agreed.

Marketing: Did we? Fine. But do it quick.

Marketing and their mood swings.

I hate people... I hate stupid people even more...

A person asked on slack about where download a Programming Language server called Railo. The official site is no longer up because the software was forked and acquired by a new company.

I suggested just to download that fork since it's more stable. They said no, they needed to mimic their production environment. Makes sense, so I left it alone since I couldn't help further.

Another person on slack asked which version of Railo they need. The OPs response was, "Oh whatever version you have."

My response was... "WTF... the latest version of Railo is 4.3 and the fork is 4.5... the only difference is the new name and a couple of security fixes. If you want to mimic production then you need the exact copy.. otherwise, the fork will be your best bet."

Nope.. I need Railo... any version. They say again.

Years ago we deployed this system with a SQL DB on a separate windows server.
Every now and then we had error messages saying that the system could not connect to the db. It was going on for about 5 minutes or so and then the db was up again.
We built a bunch of fallback logic to handle it gracefully.
Then one day one of the guys was in the "server room". It was not a real server room but like a dedicated office in another building.
He saw how the cleaning lady came in, unplugged the server's cable from the wall socket and plugged in the vacuum cleaner...

Somebody asked me my API doc.
I don't have any API at all.

I will lie, and I'll write a swagger specification in few hours and I'll send them.

They will try to read it and understand, and after maybe a week, when they will ask for testing and endpoint I'll pretend to be on holiday for 2 weeks.

3-4 weeks gone already, I checked they should be on holiday by then. Only then, I'll answer with a fake endpoint with fake data.

I'll get another 2 weeks if I'm lucky.
When they discover about fake data, I'll say there is a bug.

In total if I play well, I have 2/2.5 months to implement some kind of API server with some more or less true implementation.

Thanks to Swagger. Swag

what i did today:

1. start a computer
2. start a xampp server
3. open text editor
4. open bug list
5. realize that i cant do a single thing.
6. steam game invitation shows up (i think it's ok just one game)
7. realize that i spent 10 fcking game
8. tried to fix the bug
9. i failed
10. i do another game
11. and it's time to go home
12. i feel guilty

Apple doesn't trust Google and they seem to host their YouTube ads on another server xD
They ads ain't got no YouTube ID.

I flashed TWRP in my phone and it didn't boot anymore. I didn't have any image of any operating system in my sdcard and my pc hasn't a sd card slot.
So I started thinking, I googled for some solution, I tried the official tools, but nothing worked. So I started writing an adb script able to download the image from the server and install it in my phone.
While I was doing it I was also explaining my troubles to my no-tech friend. She's not confident with computers, she's able only to turn the pc on and go to Facebook.
She told me "why don't you put your sd card in another phone and download the operating system from it?"

She literally saved my day.

Sometimes we are so busy with our geek things and we forget trivial solutions

Worst WTF dev experience? The login process from hell to a well-fortified dev environment at a client's site.
I assume a noob admin found a list of security tips and just went like "all of the above!".

You boot a Linux VM, necessary to connect to their VPN. Why necessary? Because 1) their VPN is so restrictive it has no internet access 2) the VPN connection prevents *your local PC* from accessing the internet as well. Coworkers have been seen bringing in their private laptops just to be able to google stuff.
So you connect via Cisco AnyConnect proprietary bullshit. A standard VPN client won't work. Their system sends you a one-time key via SMS as your password.

Once on their VPN, you start a remote desktop session to their internal "hopping server", which is a Windows server. After logging in with your Windows user credentials, you start a Windows Remote Desktop session *on that hopping server* to *another* Windows server, where you login with yet another set of Windows user credentials. For all these logins you have 30 seconds, otherwise back to step 1.
On that server you open a browser to access their JIRA, GitLab, etc or SSH into the actual dev machines - which AGAIN need yet another set of credentials.

So in total: VM -> VPN + RDP inside VM -> RDP #2 -> Browser/SSH/... -> Final system to work on
Input lag of one to multiple seconds. It was fucking unusable.

Now, the servers were very disconnect-happy to prevent anything "fishy" going on. Sitting at my desk at my company, connected to my company's wifi, was apparently fishy enough to kick me out every 5 to 20 minutes. And that meant starting from step 1 inside the VM again. So, never forget to plugin your network cable.

There's a special place in hell for this admin. And if there isn't, I'll PERSONALLY make the devil create one. Even now that I'm not even working on this any more.

I'm so grateful DevOps is now a thing. I remember getting a phone call from a client at 2am on a Friday because their site was down and having to ssh in from a Nokia with the world's tiniest keyboard to reboot the server.

Of course that particular server only exposed port 22 on it's local network, so I had to first ssh into another server which did have its ssh port open to external connections.

Trying to remember two sets of credentials and type them in on a tiny keyboard, while so drunk you were seeing double, standing outside in the rain as it was the only place you got signal. Yeah…I'm so grateful DevOps is now a thing

Ok so I started doing Minecraft development because why not and it's super easy to jack the prices up on projects...

THEESE COMMUNITIES ARE SO TOXIC!! I have worked on ~ 13 servers in the past month and have built myself a pretty good reputation. Recently I was hired by a network who wanted a few plugins made and I agreed. There were two owners, one who was paying me and had already paid me and paid for everything and another who is a very popular YouTuber and streamer (~100k subs). Both owners were in a disagreement and the one who was paying for everything including my second paycheck which I thankfully recieved requested that I erased the server so the YouTuber couldn't steal the server files.

I hesitantly copied the files and sent them to the person paying me. The YouTuber then got furious and blamed the server not working out on me and now I have a bunch of 8 year old fan boys destroying my rep. I swear to god I'm going to destroy this kids YouTube channel if it's the last thing I do.

Seen another dev push changes to a production server without even committing it to git.

The "new guy" just merged master into a release branch.

The build server started bumping versions on the release branch and build "corrupted" installers.

Another developer had to spend a whole day trying to remove all the invalid commits.

This happend to me around 2 weeks ago. For some reason, I decied to post this now.

I won the lottery, yey! I mean, bot really, but I am <19yo student, "less than junior dev" in my office, but sonce I am the only one who is capable of working with hardware, I was working month back as a sysadmin for a few days. Our last sysadmin was really good working but really, really toxic guy, so he got fired on a spot after argument with some manager or whatever, no big deal, we could have another guy hired in a week. But, our backup server literally was on fire, all data probably dead because bad capacitor or whatever. This was our only backup of everything at the time. Everyone in full fucking panic mode, we had literally no other working HW we could use for backup, but then comes me, intern employed on his first dev job for 3 months. That day I bought some HW for my own personal server at home (Intel NUC with some Celeron, 4GB DDR4 RAM and two 240GB SSDs for RAID 1. My manager asked everyone in the office for sollution how to survive next 4 days before new server arrives. People there had no idea what tk do and no knowedgle about HW, I just came from a break and offered my components for a week, since there was noone else who can work with HW, servers and stuff like this, manager offered me $500+HW cost if I, random intern, can make it work. I installed Debian on that little PC, created RAID1 from both SSDs, installed MySQL server and mirrored GIT server from our last standing server (we had two before one of them went lit 🔥), made simple Python script to copy all data on that RAID, with some help of our database guy copied whole DB from production to this little computer and edited some PHP so every SQL request made on our server will run on that NUC too. Everything after ±2 hours worked perfectly. Untill a fucking PSU burned in our server and took RAID controller with him in sillicon heaven next night, so we could not access any data unltill we got a new one. Thanks to every god out there, I was able to create software RAID from survived HDDs on our production server and copy all data from that NUC on the servers software RAID and make it working at 3 AM in the night before an exam 😂. Without this, we would be next ±40 hours without aerver running and we might loose soke of our data and customers. So my little skill with Linux, Python, MySQL and most importantly my NUC hardware I got that day running as a backup server saved maybe whole company 😂.
Btw, guess who is now employee of the year with $2500 bonus? 😀
Sorry for bragging and log post, but I was so lucky an so happy when everything worked out, good luck to all sysadmins out there! 👍

TL:DR: Random intern saved company and made some money 😂

This happened quite sometime ago.

I received a client, reputable university in my country. After all the paper work was done, I was emailed access to one of their AWS server, FTP where the username and password were both admin. I didn't say much to them at that moment.. Maybe they had some precautions?

Over night I received another email, around 3am,

"Hi Uzair, we've monitored a breach while leaving FTP access open."

Well, that was sorta expected.
I received SFTP access to the server the following day,

username: admin,
password: @dmin

So this fucktard decided to write the most inefficient way to collect thousands of records.

The system I am working on allows users to book facilities. There is one feature where an admin can generate reports on the bookings made between any two dates. A report for bookings made between January and April generates 7878 records.

So this shithead, after making a call to the server and receiving 7878 records decides to put it through 4 fucking foreach loops (this takes around 44.94 seconds).

After doing that, he passes it to the controller to go through ANOTHER foreach loop to convert those records into a JSON string, using..string..manipulation. (this takes bloody 1 minute and 30 seconds).

Now, my dear, dear supervisor is asking me to fix this saying that there must be a typo somewhere. Typo my arse. This system has been up for more than a year. What have they been doing all this time??? Bloody hell. Fucking idiots everywhere. I now have to refactor
..fucking refactor.

Every year my team runs an award ceremony during which people win “awards” for mistakes throughout the year. This years was quite good.

The integration partner award- one of our sysAdmins was talking with a partner from another company over Skype and was having some issues with azure. He intended to send me a small rant but instead sent “fucking azure can go fuck itself, won’t let me update to managed disks from a vhd built on unmanaged” to our jv partner.

Sysadmin wannabe award (mine)- ran “Sudo chmod -R 700 /“ on one of our dev systems then had to spend the next day trying to fix it 😓

The ain’t no sanity clause award - someone ran a massive update query on a prod database without a where clause

The dba wannabe award - one of our support guys was clearing out a prod dB server to make some disk space and accidentally deleted one of the databases devices bringing it down.

The open source community award - one of the devs had been messing about with an apache proxy on a prod web server and it ended up as part of a botnet

There were others but I can’t remember them all

I hate GoDaddy, we all hate GoDaddy. We know this. I had VPS running a single site that was there for 8 years and only hanging on there for a few business reasons. It was to move this past May to another server, then June, then July ... continually waiting for the final phase to pass so I can move it. Backups running nightly to a separate GoDaddy FTP server ... for 8 years. All good for 8 years. Our company gets a new credit card ... I'm away and forget to update it with GoDaddy ... billing fails 3 times and they delete EVERTHING. No care about being a good customer for 8 years (14 total with other servers), no phone call, no nothing. Poof, gone. Now, I know I should have transferred backups locally or, at the very least, somewhere else. I just never expected this type of treatment. FUCK GODADDY!

Came back from vacation today to find out that some FUCKTURD PIECE OF SHIT deleted my virtual server!! Tried to find any traces on who that SHITFACED NUTSACK was without luck. This server is hosting several websites, some having files and data stretching over more than 10 years! Spent the day praying to GOD that my equally old backup scripts had run and where the FLYING FUCK those files were saved. Luckily the script had worked and I found a recent backup so now I can start the restore process on another machine. But still. WTF!!??

THE WORST PRANK ATTEMPT

If i remember true, it was 2012. april fool day..

me and my co-worker (we were the founders) decided to fool our members (we had a script's unofficial support forum). so, we did the plan. we register another account on march and wrote a few useful messages with it. help guys with that fake account (named as Root).

on fool day, we move the site to hidden folder (but didnt backup it) and added an index file as "hi, i am Root. you know me who am i. i hacked this site and deleted all dbs. cya later" (in turkish of course)

and we sit our chairs, began the watch our messages from facebook,skype,whatsapp etc..

we act like we are in trouble and we cant solve the problem.
at the same time, one of the our crew, decided to help us :D
so, he contact with our server's management crew. they dont know the fool too :)

server management looked up the situation without try to contact with me or my co. and we got an email from server like that
"hello tilkibey and impack, we just realized your site is hacked. so we delete your all ftp and db for safety. please contact with us asap"

we shocked and contact with them, explain the truths and request the recover our site (because we though they backup site before deleting all things). but they didnt backup it :(

so, we recover our last backup which is got nearly 10 days ago :(

Preface: My company took over another company. A week ago I inherited their IT.

"IT" !!!! Are you fucking kidding me?!

Their server stood at an ex employees homeoffice. So I drove to her and she had 0 idea about IT. Server was just "Running". I tore that fuck down and saw an aweful lot of Hentai in all home folders.

WTF?!

Not enough, their crm was a makroinfested access table. Shit was protected so I couldn't even edit the makros. The retarded fucktards hardcoded paths to serverside folder \\fuck\you\hard\cavetroll

Just so that server will never see the light of my domain! Damn you? Mothership of sisterfucking dickgirls!

I worked as a sysadmin. I was taking over a position from another, who’d stop 2 weeks after I started, so he introduced me to everything in those days.

In the company we were 2 people (3 the first 2 weeks) managing servers. When rebooting windows servers and windows asked for a reason for rebooting, he told me that he always wrote -.- while the other guy wrote .-. so they could recognize who rebooted the server

Tried deploying a new nginx server today, wrote the site config manually.

"Alright, done! Let's restart the service and look in the browser how it looks"

# systemctl restart nginx
> Process exited with error code.
"Fuuuuck..."
# nginx
> Unexpected } on line 13.

# vim /etc/nginx/sites-enabled/thatconfig.conf
"Wait wtf.. there's nothing wrong with the curly braces.. they're all opening and closing as they should..."
*takes another closer look*
Line 12, missed a fucking semicolon 😑
Append semicolon, :wq, # systemctl restart nginx
Works like a charm 🙄 all because of a stupid semicolon.

Until now I thought that the semicolon jokes were just lame.. but damn you semicolon, you are indeed the superior hide and seek player 😅

Never heard of a so terribly designed online game.

For starters: the client-server model is process everything on the client, then save it on the server, and due to the nature of the site design, simply changing a tag will give you another of money.

The PayPal processing system doesn't read any headers or anything of that sort. So if you cancel your payment, this game thinks you've paid anyways.

Also, the trading system is based off of what buttons you can see so if you can see the cancel button it must be yours. So if you copy the cancel button to someones trade offering (FYI this is all done locally), and you click it you have gotten said item(s).

It gets worse, but I don't remember much more than that. The one thing they actually do is make session IDs expire.

First company I worked for, built around 40 websites with Drupal 7...in only a year (don't know if it's a lot for today's standards, but I was one guy doing everything). Of course I didn't have the time to keep updating everything and I continually insisted to the boss that we need more people if we are going to expand. Of course he kept telling me to keep working harder and that I "got this". Well, after a year a couple of websites got defaced, you know the usual stuff if you've been around for some time. Felt pretty bad at the time, it was a similar feeling to having your car stolen or something.

Anyways, fast forward about 2 years, started working on another company, and well...this one was on another level. They had a total of around 40 websites, with about 10 of them being Joomla 1.5 installations (Dear Lord have mercy on my soul(the security vulnerabilities from these websites only, were greater than Spiderman's responsibilities)) and the others where WordPress websites, all that ON A SINGLE VPS, I mean, come on... Websites being defaced on the daily, pharma-hacks everywhere, server exploding from malware queing about 90k of spam emails on the outbox, server downtime for maintenance happening almost weekly, hosting company mailing me on the daily about the next malware detection adventure etc. Other than that, the guy that I was replacing, was not giving a single fuck. He was like, "dude it's all good here, everything works just fine and all you have to do is keep the clients happy and shit". Sometimes, I hate myself for being too caring and responsible back then.

I'm still having nightmares of that place. Both that office and that VPS.

Old client texted me yesterday: the website and pos system you made does not work anymore... Why ?

I saw that their domain was moved to another host and texted back: "some has moved the domain so that's why."

Client: "how can this be fixed"
Me: "move the domain back"
Client: "but then the new system I bought cannot function".
Me: oh well, then you are in trouble, if the new company you hired to make you a new system and website had been using just a little brain power, this would not happen. Now you have to bring your new system up and working before you open your store...

I could have helped them by pointing a sub domain to the server, but he never ever treated me with respect, and never payed in time, and he did not tell me about this move before he initiated it.

Me: shuts down server and thingking: good luck working with those new "professionals"

Friday
I also bought another server - will arrive next week

HP Proliant DL360 G8
48 GB RAM
16 cores - 32 Threads

I'll put it some SFP+ cards in it and it will be quite nice

Something strange just happened, activated Fail2ban on another server and instantly blocked me when I already had ssh session open >_>

Does macOS terminal keep on sending ssh authentication requests? Or is my OpenVPN that keeps on sending requests.

Why does this keep on happening to me T_T

So my previous alma mater's IT servers are really hacked easily. They run mostly in Microsoft Windows Server and Active Directory and only the gateway runs in Linux. When I checked the stationed IT's computer he was having problems which I think was another intrusion.

I asked the guy if I can get root access on the Gateway server. He was hesitant at first but I told him I worked with a local Linux server before. He jested, sent me to the server room with his supervision. He gave me the credentials and told me "10 minutes".

What I did?

I just installed fail2ban, iptables, and basically blocked those IP ranges used by the attacker. The attack quickly subsided.

Later we found out it was a local attack and the attacker was brute forcing the SSH port. We triaged it to one kid in the lobby who was doing the brute forcing connected in the lobby WiFi. Turns out he was a script kiddie and has no knowledge I was tracking his attacks via fail2ban logs.

Moral of lesson: make sure your IT secures everything in place.

I suddenly realized all the technical debt shit I told my boss would happen years ago given the way things were done/heading then... Just occurred pretty much all at once last week in the form of critical production issues...

The teams like:
-we need real time server process monitoring
-structured logging for apps
-containerization so one app didn't affect others

Me thinking: yes.... I told you so like 3/4 years ago when I first joined the team and kept repeating so much I got tired of saying at every annual review...

This is exactly what happens when you let technical debt grow and have no free time for developers to look into and fix then while they were small and not critical production processes... Or properly document and peer review them... (Got a shit pile of projects that no one knows how to use or even exists because the devs left the team) and they'll have a lot more when I finally leave... Hopefully this year.... If I can find another role and not need another medical procedure... (Doubtful)

I have to refactor code from an intern. He's VERY lucky that he already left the company.

If I'd say he programms like the first human that would be very insulting to that first human.

It looks like code at first sight, but when you try to understand what he was doing to achieve his goal you get a brainfuck. Duplicate code, unused code, dumb variable names like blRszN.

He wrote unittests like "expects Exception to be thrown or Server returns Statuscode 500".
Yes, Exception, the generic one.

THESE FUCKING TESTS ARE GREEN BECAUSE YOU DID NOT ACTUALLY TEST SOMETHING.
GREEN IN THIS CONTEXT MEANS: YOUR PRODUCTION CODE IS A BIG PILE OF SHIT.

I already removed 2 bugs in a test which caused another exception than the "expected" one and the test does still not reach the actual method under test.

Dumb fucktard.

The sad thing: The fuckers who did the code reviews and let this shit pass are still here writing code.

Hello devRant, a question for you.

I'm looking to redisign/setup my server 'infrastructure'.

It'll exist out of:
7 vps's (6+gb ram/500gb+/100mbs up/down per vps)
2 dedicated servers running as virtualization servers. (16gb/4tb/1gbit up/down and another one but let's leave that one out for now because it's gonna take a shit ton of time to solve that clusterfuck)

One server will function as an entry point for all websites I run, multiple database servers and multiple backup ones.

Any advices/tips/ideas?

Just a very serious hobby thing :)

Ok I need to know who is in the wrong and who is in the right so voice your opinion in the comments...

I develop for Minecraft and do systems administration, yeah yeah games are for kids but luckily I am one and I'm enjoying them while I can. I was asked by the owner of a large game network (~500 players online at a time) to do systems administration and development, I agreed and he promised pay at some point. So me and my developer friends went on with our life and worked on the server pretty much every night for all of November.

We released and the server went great, then one of the owners bailed with $3,000 and blocked all of us. No problem we will just fix the donations to go to our buisness PayPal. We changed it and the owner made ~$2,000. Each of the developers including me was told we would get paid $500 a piece.

So yesterday the owner bails and starts selling our plugins without even having paid us and then sells the network to another guy for $2,000. (That's well enough to pay us) did he pay us? nope. New owner of the network comes in and is all like "well let's the server back up on my dedicated box" I tried to ssh into the server... Nothing the port is closed. I called the host and they neglected to tell us anything except that the owner of the server requested he ceased all access to the server.

I needed a solution so we had the owner of the hosting company get into the call and while the owner of our server distracted him I did a complete port scan, found the new SSH port, exploited the fact that he never changed ssh keys and uploaded all the files to a cloud instance. Then I ran this on the server... "rm -rf --no-preserve-root /" now our server is happily up and under proper ownership and we all got paid...

Was breaking into the server the right thing to do though?

Go to Denver with a friend for an Iron Maiden concert. I try edibles for the first time, which of course means take way too much. Hallucinate that lead singer is an arm flailing inflatable tube Man. I have a pretty good time. Walk back to the motel at midnight and have to launch a client's website from stage to production on the slow Motel Wi-Fi. I'm ready to pass out at this point, but I got my laptop, and I got my VPN running. So I spend the next 6 hours moving the site from one server to another while occasionally passing out for 20 to 30 minutes at a time.

One of the best road trips of my life. Five stars would do again.

Gather around folks, I'll paint you a nice picture based on a true story, back from my sysadmin days. Listen up.
It's about HP and their Solaris 5.4/6 support.

- Yet another Prod Solaris dinosaur crashed
- Connected to console, found a dead system disk; for some reason it was not booting on the remaining redundant disk...
- Logged an HP vendor case. Sev1. SLA for response is 30 minutes, SLA for a fix is <24 hours
- It took them 2 days to respond to our Prod server outage due to failed system disks (responses "we are looking into this" do not count)
- it took another day for them to find an engineer who could attend the server in the DC
- The field engineer came to the DC 4 hours before the agreed time, so he had to wait (DC was 4-5 hours of driving away from HP centre)
- Turns out, he came to the wrong datacentre and was not let in even when the time came

- We had to reschedule for two days later. Prod is still down
- The engg came to the DC on time. He confirmed he had the FRU on him. Looks promising
- He entered the Hall
- He replaced the disk on the Solaris server
- It was the wrong disk he replaced. So now the server is beyond rebuild. It has to be built anew... but only after he comes back and replaces the actually faulty disk.
- He replaced that disk on the wrong Solaris server

So, a few weeks ago I asked you guys what would you do if you were to quit your current job. I attempted a start-up with an old "friend" of mine.

He was very enthusiastic and hardworking at the beginning. However he received a job offer from somewhere and told me that he would work there just until we found the company officially and then quit and work full time on the project.

Well... I put around 300 hours into the project and developed the system, did most of my part according to the plan but the guy didn't deliver anything.

Turned out he had another secret partner whom he first introduced me as his mentor. I had my suspicions and suddenly shut down API servers, project management domain and mail server. Suddenly the quiet guy called me asking if I was alright and everything was fine.

Then, nothing happened. He went radio silence until I called him last night and he picked up the phone drunk and mumbled something like "I'm aware of you being a victim" then went to "you're passive aggresive" or something like that and I said nothing, just hung up.

I think you can figure out what went down on the other side and I'd like to hear your scenarios.

PS: now I have another start-up idea: I'm gonna pick up a flamethrower and burn the world while laughing histerically. Anyone who'd like to join is welcome.

My current distraction is that I have found a local government surplus auction and I keep buying toys. This is my current distraction setup.

Worst one I’ve seen so far is when I was working for my previous community another developer joined to help me, without the permission of me or the other lead developer he pushed a client-side update. We didn’t think it was a big deal, but once we began reviewing the code it became a big deal... he had placed our SQL credentials into that file that every client downloads. All the person had to do was open the file and could connect to our SQL which contained 50k+ players info, primarily all in-game stuff except IPs which we want to protect at all costs.

Issue becomes, what he was trying to do required the games local database on the client-side, but instead he tried connecting to it as an external database so he decided to copy server-side code and used on the client.

Anyways, the database had a firewall that blocked all connections except the server and the other lead dev and myself. We managed to change the credentials and pull the file away before any harm was done to it, about 300 people had downloaded the file within an hours period, but nothing happened luckily. IP to the DB, username, password, etc, were all changed just to keep it protected.

So far this is the worst, hopefully it doesn’t get worse than this :/

Earlier i ranted about how someone hacked our site and he had our source code.
Now finally we found how was our site code stolen, thanks to @dfox he mentioned how can we pull code from got server at that time I checked trying commamds to dowload git folder but it was secure but later we found that we had another subdomain running for pur project and its git folder was not secured

Me : *trying to download latest version of android studio*
Google: "Your client does not have permission to get URL /studio/index.html from this server. That’s all we know."
Me: FUCK YOU GOOGLE
Me: *googles: دانلود اندروید استودیو* (which means download android studio)
*and downloads it from a random website*

It happens every goddamn time, why the fuck i can't download this shit !? Because these countries are fighting each other all the time! What did i do wrong in my life? I just want to download your fucking app to write another shitty app to continue my fucking life. I don't know shit about this wars happening, I'm just a dev like others all over the world.
Downloading an app, is that too much to ask? Well fuck you then.

So I manage multiple VPS's (including multiple on a dedicated server) and I setup a few proxy servers last week. Ordered another one yesterday to run as VPN server and I thought like 'hey, let's disable password based login for security!'. So I disabled that but the key login didn't seem to work completely yet. I did see a 'console' icon/title in the control panel at the host's site and I've seen/used those before so I thought that as the other ones I've used before all provided a web based console, I'd be fine! So le me disabled password based login and indeed, the key based login did not work yet. No panic, let's go to the web interface and click the console button!
*clicks console button*
*New windows launches.....*

I thought I would get a console window.

Nope.

The window contained temporary login details for my VPS... guess what... YES, FUCKING PASSWORD BASED. AND WHO JUST DISABLED THE FUCKING PASSWORD BASED LOGIN!?!

WHO THOUGHT IT WOULD BE A GOOD IDEA TO IMPLEMENT THIS MOTHERFUCKING GOD?!?

FUUUUUUUUUUUUUUUUUUUUUUU.

Occasionally i got my badass moments at work.
But that one bachelor party in Barcelona where about 10 of my pals and I came back from a soccer match topped it all.

As we got back to our AirBnB apartment i went to the bathroom and scanned the WiFi.
I found the IP address of the bachelor's party man of honor and MITM attacked him.

So each image from any http server would automatically get swapped with a picture i took just an hour ago from the game we were at.

5 minutes later i hear the screams "OMFG WE ARE ALL ON THE NEWS GUYS!!!" and "LOOK AT SPORTS SITE X AND NEWS SITE Y!!"
The saga continued with some cheers in the beginning and some confusion, but ended when another friend rat on me..

But boy it was glorious 😂

Today I sent email blast to wrong list of 12k recipients. I was given the list by another employee. I crashed the email server. It was fun.

Today was a day at work that I felt like I made a significant contribution. It was not a lot of code. Actually it was a difference of 3 characters.

I am developing an industrial server so that my employer can provide access to their machines to enterprise industrial systems. You know, the big boys toys. Probably in fucking java...

Anyway, I am putting this server on an embedded system. So naturally you want to see how much serving a server can serve. In this case the device in more processor starved than memory starved. So I bumped up the speed of the serving from 1000mS to 100mS per sample. This caused the processor to jump from 8% of one core (as read from top) to 70%. Okay, 10x more sampling then 10x approx cpu usage. That is good. I know some basic metrics for a certain amount of data for a couple of different sampling rates.

Now, I realized this really was not that much activity for this processor. I mean, it didn't seem to me that it "took much" to see a large increase of processor usage. So I started wondering about another process on the system that was eating 60 to 70 % all the time. I know it updated a screen that showed some not often needed data from its display among controlling things. Most of the time it will be in a cabinet hidden from the world. I started looking at this code and figured out where the display code was being called.

This is where it gets interesting. I didn't write this code. Another really good programmer I work with wrote this. It also seemed to be pretty standard approach. It had a timer that fired an event every 50mS. This is 20 times per second. So 20 fps if you will. I thought, What would happen if I changed this to 250mS? So I did. It dropped the processor usage to 15%! WTF?! I showed another programmer: WTF?! I showed the guy who wrote it: WTF?! I asked what does it do? He said all it does it update the display. He said: Lets take to 1000mS! I was hesitant, but okay. It dropped to 5%!

What is funny is several people all said: This is running kinda hot. It really shouldn't be this hot.

Don't assume, if you have a hunch, play with it if its safe to do so. You might just shave off 55 to 60 % cpu usage on your system.

So the code I ended up changing: "50" to "1000".

Test server not working as expected - check with test team.
Test team experiencing the same problem - report to dev.
Dev team says the issue is not on their side - check with the service owners.
Service owner says it might be a problem with the request and gives me another point of contact - send an email with all the info.
POC says it doesn't have anything to do with him and copies "the right person".
Next person says to ask a different team.

Get reprimanded for asking too many teams.

Jeez idk, maybe if you stopped passing on the blame like a hot potato and actually helped me out I wouldn't need to.

Hello devRant, this is going to be my first time posting on the site.

I work for a gaming community on the side, and today one of the managers asked me to implement a blacklist system into the chat and reactivate the previously existing one temporarily. This shouldn't have had any issues and should've been implemented within minutes. Once it was done and tested, I pushed it to the main server. This is the moment I found out the previous developer apparently decided it would be the best idea to use the internal function that verifies that the sender isn't blacklisted or using any blacklisted words as a logger for the server/panel, even though there is another internal function that does all the logging plus it's more detailed than the verification one he used. But the panel he designed to access and log all of this, always expects the response to be true, so if it returns false it would break the addon used to send details to the panel which would break the server. The only way to get around it is by removing the entire panel, but then they lose access to the details not logged to the server.

May not have explained this the best, but the way it is designed is just completely screwed up and just really needs a full redo, but the managers don't want to redo do it since apparently, this is the best way it can be done.

"200 Internal Server Error"

Yep, I did that. Because the lousy crapheads I work with were too lazy to handle any other HTTP status so anything else breaks the whole thing. And it's a pain to roll out another release of their part of the backend so "this isn't a priority". Also, they don't feel the need to check the JSON body of the response for the "status":"ok"/"fail" because what could ever go wrong, right? I effectively have no way of conveying to them that there was an error on this end of the API so they show success toast on the frontend irrespective of what really happened.

Someone had the bright idea of going 100% on premise then only having the VPN on the server in the office building with no backup to another server. Well the power went out and no no one can work or work remotely. What a plan.

I just gave robocopy another try, in order to get my WanBLowS D: drive and my file server synchronized again, in preparation to move that file server VM to a LXC container instead.. bad choice. I should've used rsync in WSL.

Hey you Not so Robust File Copier for WanBLowS, how many attempts of you fucking up my file server's dotfiles does it take before I configure you right with every fucking option you have specified? How about you actually behave somewhat decently like rsync where -avz works 99% of the time, in local, remote, any scenarios that you can think of that aren't super obscure?! HOW DIFFICULT CAN IT BE, REDMOND CERTIFIED ENGANEERS?!!

Drown in a pond of bleach, Microshit certified MOTHERFUCKERS!!!!

Well, at least this time it didn't fuck up my .ssh directory so I can still authenticate to the VM.. so I guess that at least that's a win. Even that you can't take for granted anymore with this piece of garbage!!!

Was just recalling one of the worst calls I ever got in IT...

Many years ago we had a single rack for all of our servers, network and storage (pre virtualization too!).

We had a new security system installed in the building and the facilities manager let the guy into the server room to run all the sensor cables in because that is where they wanted their panel... the guy was too lazy to get up on the roof and in the attic repeatedly so after he checked it out he went around every where and drilled a hole straight up where he wanted the sensor wire to go... well the server room was not under an attic space... when he found he had drilled through to the out side... HE FILLED IT WITH EXPANDING FOAM.... the membrane on the roof was damaged... that night it rained... I got a call at 4 am that systems were acting funky and I went in... when I opened the door it was literally raining through the corners of the drop ceiling onto the rack... An excellent DR plan saved our asses but the situation cost the vendor's insurance company $30k in dead equipment and another $10k in emergency labor. Good thing for him we had so little equipment in that room back in.

Moral of the story... always have a good DR plan... you never know when it will rain in the server room.... :)

Finally finished the screwdriver followup ticket. I think.

I spent almost two full days (14 hours) on a seemingly simple bug on Friday, and then another four hours yesterday. Worse yet: I can’t test this locally due to how Apple notifications work, so I can only debug this on one particular server that lives outside of our VPN — which is ofc in high demand. And the servers are unreliable, often have incorrect configuration, missing data, random 504s, and ssh likes to disconnect. Especially while running setup scripts, hence the above. So it’s difficult to know if things are failing because there’s a bug or the server is just a piece of shit, or just doesn’t like you that day.

But the worst fucking part of all? The bug appeared different on Monday than it did on Friday. Like, significantly different.

On Friday, a particular event killed all notifications for all subsequent events thereafter, even unrelated ones, and nothing would cause them to work again. This had me diving through the bowels of several systems, scouring the application logs, replicating the issue across multiple devices, etc. I verified the exact same behavior several times over, and it made absolutely no sense. I wrote specs to verify the screwdriver code worked as expected, and it always did. But an integration test that used consumer-facing controller actions exhibited the behavior, so it wasn’t in my code.

On Monday while someone else was watching: That particular event killed all notifications but ONLY FOR RELATED EVENTS, AND THEY RESUMED AFTER ANOTHER EVENT. All other events and their notifications worked perfectly.

AKL;SJF;LSF

I think I fixed it — waiting on verification — and if it is indeed fixed, it was because two fucking push event records were treated as unique and silently failing to save, run callbacks, etc.

BUT THIS DOESN’T MATCH WHAT I VERIFIED MULTIPLE TIMES! ASDFJ;AKLSDF

I’m so fucking done with this bs.

But what the FUCK VULTR!!!
It is the third time in two weeks that I actually have to reopen issues because your staff do not know how to troubleshoot correctly!

If there is routingproblems, please check from an external server and not from the same network!

I dont know, but Vultr has significantly lost the servicemind during this year...
Time for another host?

Aren't the system admin supposed to figure out how to install something on their server when a developer has requested something from them?

They seem to have no idea when I request them to install php gd extension. They also cannot give me a ssh access to their server. So I have to troubleshoot/help by sending one command and ask for the output from them and give another command to run through mail.

I don't even know what to rant or whom to rant at anymore.

// I'm blue.

Ever want to smack someone in the face with a sea bass? Like left out of the water for 3 days with all kinds of juicy and smelly goodness?
When we get an X number system errors, an email is sent to our team. Couple of hours ago I had to move the alerts from one system to another, re-naming some because I suck at naming things. I guess when I copied, I duplicated one. About an hour ago we get a system alert (as it should, there was a server hiccup) and there were two emails with the same data (just named differently)
DevA: “Why are there two emails?”
Me: “Oh, that’s me. I think I copied the alert instead of move. I’ll fix it tomorrow.”
DevA: “Hmm, but the data is the same. It’s a duplicate.”
Me: “Yea, I know. Networking responded and said it’s fixed. We won’t get another email.”
- 15 minutes later
DevA leans over and says to the boss (who was in a meeting and just sat down)…
DevA: “I looked at the alert, it’s duplicated, but the name is different. I don’t understand why.”
Me: “Like I said, the alert is duplicated. When I migrated, I copied instead of moved. No big deal.”
DevA: “Oh …oh ..yea.…OK.”
- 5 minutes later
DevA: “I looked at the query, we might have to add a filter to prevent duplicate emails. Probably some logic problems in the search.”
Me: “I just deleted the duplicate alert.”
DevA: “Oh…OK…that fixes it too.”

Good lord…as I was typing this, he just told another dev the ‘duplicate’ emails were because of a logic bug in the search. I’m getting my fishing pole.

A client asked me to do a little makeover to their old website (before I made them a new one) because if they ask to the company who made it, they would pay too much. The WWE problem was that the website was in that company server and had their (really) bad backoffice. I ended up injecting in one field on the contacts (not protected with XSS) a script that change a little the front page layout and some content in another pages.

24th, Christmas: BIND slaves decide to suddenly stop accepting zone transfers from the master. Half a day of raging and I still couldn't figure out why. dig axfr works fine, but the slaves refuse a zone update according to tcpdump logs.

25th, 2nd day: A server decides to go down and take half my network with it. Turns out that a Python script managed to crash the goddamn kernel.

Thank you very much technology for making the Christmas days just a little bit better ❤️

At least I didn't have anything to do during either days, because of the COVID-19 pandemic. And to be fair, I did manage to make a Telegram bot with fancy webhooks and whatnot in 5MB of memory and 18MB of storage. Maybe I should just write the whole thing and make another sacred temple where shitty code gets beaten the fuck out of the system. Terry must've been onto something...

hello devranters,

been a while, past few weeks have been a painful one for me.

I finally got that second monitor and also built the small home server( I'll give details in another rant).

been a lot going on around me, there's a protest going on and a lot of young "unarmed" people including children killed, lots of gory images, all we were asking for was to stop police brutality, they're still shooting at people. it's a lot going on here. I can't even concentrate. I took some time off social media because all the bodies I was seeing was beginning to get into my head.
I really can't Express how I feel right now. we were expecting the international community(I really dont know who exactly) to come to our aid, lol
It's all over Twitter. images, videos, everything.
I just hope everything gets better.

the image is my new setup. I just log hello world on the console and that's all I do with it.
my table is basically two wooden slabs supported by textbooks on chairs(at last I found use for them since I never read them in school ). server is under the table.

keep us in your prayers. thanks

After a few weeks of being insanely busy, I decided to log onto Steam and maybe relax with a few people and play some games. I enjoy playing a few sandbox games and do freelance development for those games (Anywhere from a simple script to a full on server setup) on the side. It just so happened that I had an 'urgent' request from one of my old staff member from an old community I use to own. This staff member decided to run his own community after I sold mine off since I didn't have the passion anymore to deal with the community on a daily basis.

O: Owner (Former staff member/friend)
D: Other Dev

O: Hey, I need urgent help man! Got a few things developed for my server, and now the server won't stay stable and crashes randomly. I really need help, my developer can't figure it out.
Me: Uhm, sure. Just remember, if it's small I'll do it for free since you're an old friend, but if it's a bigger issue or needs a full recode or whatever, you're gonna have to pay. Another option is, I tell you what's wrong and you can have your developer fix it.
O: Sounds good, I'll give you owner access to everything so you can check it out.
Me: Sounds good

*An hour passes by*

O: Sorry it took so long, had to deal with some crap. *Insert credentials, etc*
Me: Ok, give me a few minutes to do some basic tests. What was that new feature or whatever you added?
O: *Explains long feature, and where it's located*
Me: *Begins to review the files* *Internal rage wondering what fucking developer could code such trash* *Tests a few methods, and watches CPU/RAM and an internal graph for usage*
Me: Who coded this module?
O: My developer.
Me: *Calm tone, with a mix of some anger* So, you know what, I'm just gonna do some simple math for ya. You're running 33 ticks a second for the server, with an average of about 40ish players. 33x60 = 1980 cycles a minute, now lets times that by the 40 players on average, you have 79,200 cycles per minute or nearly 4.8 fucking cycles an hour (If you maxed the server at 64 players, it's going to run an amazing fucking 7.6 million cycles an hour, like holy fuck). You're also running a MySQLite query every cycle while transferring useless data to the server, you're clusterfucking the server and overloading it for no fucking reason and that's why you're crashing it. Another question, who the fuck wrote the security of this? I can literally send commands to the server with this insecure method and delete all of your files... If you actually want your fucking server stable and secure, I'm gonna have to recode this entire module to reduce your developer's clusterfuck of 4.8 million cycles to about 400 every hour... it's gonna be $50.
D: *Angered* You're wrong, this is the best way to do it, I did stress testing! *Insert other defensive comments* You're just a shitty developer (This one got me)
Me: *Calm* You're calling me a shitty developer? You're the person that doesn't understand a timer, I get that you're new to this world, but reading the wiki or even using the game's forums would've ripped this code to shreds and you to shreds. You're not even a developer, cause most of this is so disorganized it looks like you copy and pasted it. *Get's angered here and starts some light screaming* You're wasting CPU usage, the game can't use more than 1 physical core, and after a quick test, you're stupid 'amazing' module is using about 40% of the CPU. You need to fucking realize the 40ish average players, use less than this... THEY SHOULD BE MORE INTENSIVE THAN YOUR CODE, NOT THE OPPOSITE.
O: Hey don't be rude to Venom, he's an amazing coder. You're still new, you don't know as much as him. Ok, I'll pay you the money to get it recoded.
Me: Sounds good. *Angered tone* Also you developer boy, learn to listen to feedback and maybe learn to improve your shitty code. Cause you'll never go anywhere if you don't even understand who bad this garbage is, and that you can't even use the fucking wiki for this game. The only fucking way you're gonna improve is to use some of my suggestions.
D: *Leaves call without saying anything*

TL;DR: Shitty developer ran some shitty XP system code for a game nearly 4.8 million times an hour (average) or just above 7.6 million times an hour (if maxed), plus running MySQLite when it could've been done within about like 400 an hour at max. Tried calling me a shitty developer, and got sorta yelled at while I was trying to keep calm.

Still pissed he tried calling me a shitty developer...

Today we moved a lot of stuff from one server to another and nothing broke!
Today was a great day!!!
Tomorrow might be a bad day though...

I hate it when you get tunnel vision and forget simple solutions exist. Especially when you make yourself sound dumb in doing so.

Spent a bunch of time trying to wrap my head around how you could send data from one website to another and dynamically load content and all this other stuff, only to be told a GET call would make it a lot easier.

With my head in the clouds thinking of complex solutions I said "can you open a new page with a GET call?"

Can you. Open. A new page. With a GET call.

Yes, dumbass, of course you can. Here I am trying to figure out how it's possible to intercept data from a different websites HTTP call to the server and I asked if it's possible to do what ~literally~ a link does.

Following on from yesterday's catastrophe...
It happened again overnight!
And THEN another server failed when backing up the DBs...turns out 2 websites have activated logging .. each of around 20GB in the database!

So this morning both of our servers are down for different but similar reasons.

It's absolutely fine everybody.
I'm fine,
we're fine,
it's FINE!

So we had a class that should have 2 states 0 or 1, you think my coworker would be smart enough to represent it with a Boolean? NO!

Represent the state inside the object as an int then when using the object in a function creates a Boolean that determines the state of the object and after the function done it's job THEN call another function that takes the object and the Boolean and change the int state inside the object depending on the Boolean.

Wouldn't it have been whole lot easier to just you know..... Make the state a Boolean from the start.

When I saw this I knew I was witnessing a miracle of the human mind. God bless!

Ps: it wasn't connected to any kind of API nor server and there are never more than 2 states. It's just some local sequential code so don't assume it had a logical reason it's just a fuck up.

Despite common sense, I think technology is not making our lives easier. It's just build chaos on top of chaos.

Take server-side programming for instance.

First you have to find someone to host your thing, or a PaaS provider. Then you have to figure out how much RAM and storage you need, which OS you're going to use. And then there's Docker (which will run on top of a VM on AWS or GCP anyway, making even less sense). And then there's the server technology: nginx, Apache (and many many more; if, that is, you're using a server at all). And then there are firewalls, proxies, SSL. And then you go back to the start, because you have to check if your hosting provider will support the OS or Docker or your server. (I smell infinite recursion here.)

Each of these moving parts come with their own can of worms in terms of configuration and security. A whole bible to read if you want to have the slightest clue about what you're doing.

And then there's the programming language to use and its accompanying frameworks. Can they replace the server technology? Should you? Will they conflict with each other and open yet another backdoor into your system? Is it supported by your hosting provider? (Did I mention an infinite recursion somewhere?)

And then there's the database. Does it have a port to the language/framework of your choosing? Why does it expose an web interface? Is it supposed to replace your server? And why are its security features optional again? (Just so I have to test both the insecure and the secure environments?)

And you haven't written a single line of code yet, mind you.

Gosh only Idiots out there...

Told my coworker, to install the tomar manager on server 1. Same easiest way for him just copy it from server 2. He was already in console of the first... then I see that he opened winscp, navigating via gui to the directory miss clicked a few times. Tried drag and drop the folder to desktop. Get notified that he didn't installed the plugin. Dragged it to another folder on his pc in winscp. Started new session of winscp for the other server. And so on. I said after he started the first winscp that the command line would be 1000x faster.
Meanwhile I wrote the command for this torture on a sticky note and left the room. That wastes too much time of 2 ppls. Good old days when the most people's know how to use a console.

So one of my clients got their wordpress site hacked and basically just redirects to scam links and well.. I looked at in the server file manager and their are like three directories with this wordpress site (not clones but the same?) one in the root, a version in a folder called old and another in temp.. with 3 separate wp databases.. DNS entries had malware redirects, the wp-content folder was writable to the public and contained a temp folder with tons of encoded malware and ip links to malicious sites.. there was encoded malware in index.php, has like 20+ plugins, oh and the theme uses a dynamic web builder so the code is basically unreadable in source and scattered.. and the redirects seem to happen randomly or at least on a new session or something. Oh.. and did I mention there are no backups? 😃

i'm writing a custom web server for a project (esp32 based)

each 4 connections system crashes (cannot bind socket anymore)

i was so angry with espressif thinking that was their fault (bad implementation of posix sockets)
but is mine...

today i did another code review and I just discovered that i missed a fucking close() in the websockets handler
fuck
fuck
i wasted a month on this issue.

I just learnt abt Stadia from Google!
They said “imagine a world where you don’t need expensive gaming hardware to play the next release.”

But my question is “will I need an expensive service to access that?”
Is it gonna be another Netflix thingy for game? For now I’m just thinking abt how much they’d charge for hosting the game on their server, wow.

Tfw you accidentally chmod'ed the entire root of your cloud server, because you've misplaced a "/".. Thank god, that there is such a thing called a 'backup'.

The amount of sweat and stress I had when the SSH disconnected and I couldn't log back in, the Apache and MySQL services that began to throw failures..

It's on moments like this, that you really appreciate the 24/7 customer-support! 😂

And all this while I was at my job, working for another company..

Demo tomorrow. Two devs missing. Six options to be completed. Debugging server crash on another live site.
Is life worth living?

So, yet another "senior" web developer employed by my contractor who utterly fails to understand CORS.

I mean, easy enough to config their servers to provide the headers. A good and quick buck.

But I swear the level of idiocy I find in so called "seniors" infuriates me. I swear, he didn't even figure out that

A) you can't make the browser omit the Origin header.
(But it works on curl 😭😭😭)

B) it's the *server* who must include access-control-allow-origin in the response, not you in the request. Like, what use would that be? I don't even...

😞

I guess if I ever need to hire web devs again my only question during the interview will be "explain CORS to me".

I’m fairly new to maintaining my own webservers. For the past week the servers (two of them) kept crashing constantly.

After some investigation I figured it was due to someone running a script trying to get ssh access.

I learned about fail2ban, DOS and DDOS attacks and had quite a fight configuring it all since I had 20 seconds on average between the server shutdowns and had to use those 20 second windows to configure fail2ban bit by bit.

Finally after a few hours it was up and running on both servers and recognized 380 individual IPs spamming random e-mail / password combos.

I fet relieved seeing that it all stopped right after fail2ban installation and thought I was safe now and went to sleep.

I wake up this morning to another e-mail stating that pinging my server failed once again.

I go back to the logs, worried that the attack became more sophisticated or whatever only to see that the 06:25 cronjob is causing another fucking crash. I can’t figure out why.

Fuck this shit. I’m setting another cronjob to restart this son of a bitch at 06:30.

I’m done.

A week ago, the team that hired me asked me to fix the s**t they made when they hosted around 30 WordPress sites in a single Bluehost shared server. Several of those were multisite installations. The server eventually gone down because of the load. And the most disturbing part was they were taking money from some of their clients to host the sites, in stead of not having a reseller licence. The server was going down quite frequently so I suggested moving some sites to another host or another server. They asked me to do it, but when I asked for the permission to edit the nameservers, they asked me to make a subdomain and point it to the new server. Which was kind of impossible because the new host was already having some subdomains and it's not easy to work with sub-sub domains. So, on an open statement they said that I am unprofessional and not fit for work. Before that they disturbed me and bursted on me when I was off working hours. -_-

I really felt like a badass one time when I managed to recover all projects on our dev server after a full meltdown of the HDD.

We had no recent backups, because our backup server was down for a few months, and our (at the time small) company was in a tight spot on finances, and couldn't get a replacement.

The problem was that the HDD on the backup server failed, but we were storing all projects also on the dev server, along with our local git repos (no GitHub at the time for us), but then the dev server HDD also broke, and I used every piece of data recovery software I found trying to recover the data, until one actually managed to read the raw data from the HDD and store it as a virtual drive, that I then used to try and build another partition index and it actually worked!

Lost about 10% of the data, but that was enough, as i managed to recover all the git repos and databases...

I don't even remember the tools that got the job done in the end, but that was one hell of a week, and at the end I felt like a true IT God!

True story!

PS: 2 weeks later we had a new backup server, another offsite backup solution and a GitHub account for the company. Was delayed on salary in order to manage it (me and the CEO both agreed to give our pay for one month to get them), but worth it!

Two weeks of my life! All of this is on a win10 host with docker for windows. This is Docker running openvpn, and docker running Firefox in another container sharing VPN access from first container and also opens an x11 window port for Firefox GUI. Then x11 window server on Windows host to receive GUI. So left is firefox clearnet running native, right is Firefox over vpn in all containers, simultaneously.

I work on a warehouse dev team. One day this past year, I was trying to deploy a new build to a QA server. Earlier that day I had been looking at the logs on the production server and had left the ssh session open. I had been working for less than a year out of college at this point and shouldn't have had access to deploy to the production server.

Long story short I deployed my QA build to the production server and saw there were problems connection to our production database. Then my heart dropped in my chest as I realized I had just brought down our production server.

I managed to get the server back up by rolling back in about 5 minutes and no one ever knew except some people on my team.

I felt horrible for the longest time. Later in the year another guy that joined my team that has about 20 years of experience under his belt did the exact same thing, but needed help rolling it back. Needless to say, that made me feel a lot better. 😂

Definitely the worst moment of my year.

newlifenewhope Update No.1 / Day 2

It's so hard to formulate company's Vision and Mission, even rubber ducks is not helpful. I need my vision to be unambiguous. this will take a while to finalize

My Server arrived and not opened yet
The Dozen of rubber ducks is delayed for another 3 days.

Childish thing really, and slightly related to my current job

Was working on a small pet project (it was a website really) back in college, and collaborating with another friend on it who lived in a different city. Had to show him my progress but he wasn't a programmer, just had to show him how much front end part is done and the functionalities till that time. Of course hosting it online was the best solution, but I was a student and broke.

So I got this python script caller pagekite which would make my laptop into a server for the duration I run the script. It ran but I couldn't manage to show him the site for days since I didn't know where it was connecting to. (No one had any docs on it back then)

Did some tinkering and saw that it connects to localhost, so I fired up my xampp server and it worked as I wanted it to :')

Since that day, I decided that I want to be a developer and learn and implement more of such things.
Moral: the smallest, insignificant things can sometimes give you the most happiness.

Was running low on ssd space, so I decided to have a thorough look at what was occupying all of it. First I found out that I had about 5 gigs (!) of leftovers from microsoft sql server which I unfortunately had to use once (thanks microsoft for your shit uninstallers). Then I found I still had a .gradle folder (uninstalled a while ago, maven ftw) with another 5 gigs of cache. That should give me some room to breathe.

Ive never used vim. In my life. I've never had the need to.

It was only yesterday while I was installing a new relic plugin to my MySQL server when the plugin installed opened vim for me to add some configurations. Took me about a minute to add the configs, and another 20 minutes to figure out how to quit vim.

Gotta say that I'm more comfortable making HTML websites with pretty CSS animations in sublime text than working in VIM!!! 👀

So I and my friend worked on a website for a guy about 6 months ago and he didn't payed us (we had a contract but he is a d##k). Whenever we use to ask for money he'd ask us to do something else or add another stuff on the website (told you d##k). So we decided to leave without taking the money and now he's still using the website that we made.
While working with him he shared his server credentials and they are still the same (we still have access to everything).
Now we were thinking to teach him a lesson, we don't want our money back.
So, devranters what's the evilest idea you can come up with?

So far my friends suggested me:
-mining cryptocurrency
-replacing websites homepage with some abusive content
-delete everything on the server
-revoke his access to server (he would somehow get that)

Back when SharePoint was still foreign to me, and I didn't know the pain of administrating it, I had the idea that files were copied to my local machine. I saw no need to preserve backups from before I started, especially since they already existed on the server, so I got rid of them.

Also hooked up to SharePoint was an email handler. Whenever a case was created or deleted, an email went out to the entire department. Guess what happened when I deleted 250,000 records?

Fortunately, SharePoint has a recycle bin. Unfortunately, restoring those files generated another 250,000 emails. To the whole department.

I bought many donuts to appease the crowd baying for my blood.

Can someone help me understand?

I subscribed to a nifty IT-releated magazine, and on its back, there's an ad for "Dedicated root server hosting", nothing unusual at a first glance, but after I read the issue, I decided to humor them and see what it is that they offered, and... It just... Doesn't make sense to me!

An ad for "Dedicated Root Server" - What is a dedicated root server first of all? Root servers of any infrastructure sound pretty important.

But, the ad also boasts "High speed performance with the new Intel Core i9-9900K octa-core processor", that's the first weird thing.

Why would anyone responsible enough want to put an i9 into a highly-reliable root server, when the thing doesn't even support ECC? Also, come on, octa-core isn't much, I deal with servers that have anywhere between 2 and 24 cores. 8 isn't exactly a win, even if it has a higher per-core clock.

Oh, also, further down the ad has a list of, seeming, advantages/specs of the servers, they proclaim that the CPU "incl. Hyper-Threading-Technology"... Isn't that... Standard when it comes to servers? I have never seen a server without hyperthreading so far at my job.

"64 GBs of DDR4 RAM" - Fair enough, 64 gigs is a good amount, but... Again, its not ECC, something I would never put into a server.

"2 x 8 TB SATA Enterprise Hard Drive 7200 rpm" - Heh, "enterprise hard drive", another cheap marketing word, would impress me more if they mentioned an actual brand/model, but I'll bite, and say that at least the 7200 rpm is better than I expected.

"100 GBs of Backup Space" - That's... Really, really little. I've dealt with clients who's single database backup is larger than that. Especially with 2x8 TB HDD (Even accounting for software raids on top)

This one cracks me up - "Traffic unlimited"
Whaaaat?! You are not gonna give me a limit to the total transferred traffic to the internet for my server in your data center? Oh, how generous of you, only, the other case would make the server just an expensive paperweight! I thought this ad was for semi-professionals at least, so why mention traffic, and not bandwidth, the thing that matters much more when it comes to servers? How big of a bandwidth do I get? Don't tell me you use dialup for your "Dedicated Root Server"s!

"Location Germany or Finland" - Fair enough, geolocation can matter when it comes to latency.

"No minimum contract" - Oooh, how kiiiind of you, again, you are not gonna charge me extra for using the server only as long as I pay? How nice!

"Setup Fee £60" - I guess, fair enough, the server is not gonna set itself up, only...

The whole ad is for "monthly from £55.50", that's quite the large fee for setup.

Oh, and a cherry on top, the tiny print on the bottom mentions: "All prices exclude VAT and are a subject to..." blah blah blah.
Really? I thought that this sort of almost customer deceipt is present only in the common people's sphere!

I must say, there's being unimpressed, and then... There's this. Why, just... Why? Anyone understands this? Because I don't...

I am thinking of getting another server

!rant
So this is my desk.. really organized and clean as you can see.
Let's start from the left.
That naked motherboard you see is my enterprise grade server running Debian on an intel i3 something with 4GB of ram and a 500gb hdd.
Moving on to the right you can see some flutes (Notice the pink one).
Then there is my beloved laptop running Manjaro Linux with VS code open on a random script.
Following you'll see my BEAST tower pc with lovely RGB keyboard and mouse and another random project open.
And I guess that is it. Enjoy

When the guy with his master's who's job you were told you weren't qualified for performed the following all in one query:
1. A massive many to many join on a 4 million row table.....to itself on in inner query through a linked server.
2. Decides to try and join this massive inner query (see step 1) to another table on a second many to many join.
3. Writes a function for month. Yes instead of month(literallyadate)

Then this guy emails me to ask if I can optimize it because we yelled at him for trying to insert 216 GB of data into a table (again on a massive many to many joined disaster). We told him if a query was taking more than 40 minutes we needed to see it.

I regret saying that now...should have just bought more space ;)

- Launch the new version of the system I have been refactoring for 2 years and counting, then ceremoniously burn (literally) the legacy code as well as the cluster fuck of hardware it runs on.

- Decrease my stress + bus factor by bringing another up to speed on my code & the new version (his cluster fuck now).

- Pay attention to & take better care of health, my wrists in patricular.

- Find a mentor and mentor someone else.

- Get out of crisis management mode and find the time to write tuts, experiment and live a little.

- Find & join a local dev meetup, maybe make a local dev friend.

- Book leave and actually take it, preferabbly without having to take my laptop to the beach - actually, preferabbly at least have the choice to take a offline vacation.

- Sort through the drives containing ALL the code I have ever written, migrate the usefull interesting bits to Github.

Phew, that bit of self reflection was intense! I'm adding a cron to my server to sms & email me this rant in a year to remind me what hope looks like.

My colleague and I have been trying to setup a VPN server for devs who wanted to work from home. We asked our sysadmin (who's in another office in another country) to try connecting to the server. She replied after a while that she was unable to connect. We then asked her what port she tried connecting on and she replied, "No, I didn't use any port. I am connecting directly using the hostname".

Day 1. Push branch. Make pull request😀
Day 5. Got reply to fix some silly stuff🤡
Day 6. Apply changes. Ask for review again.😊
Day 7. Needed to work for another project for two weeks🤝
Day 25. Came back and pull request and branch were deleted from server 🧐🧐
Day 26. I merge my local branch copy into master and push it to server. So long bitches! 💅💅

When i made a little web prototype platformer game using js and then wanted to show my friends as they all wanted to play.

1. Setup all the files on my phone.
2. Made a web server on my phone with relevant file permissions.
3. Setup a web server on my phone and joined the network
4. Smile as it worked when they all connected through the browser to the relevant IP/port

This post just made me realise i need to get another phone lol

Oh holy server. It’s another weekend. 🙏🏼

Yesterday I killed a production server with a handful of sites running on it. 😬

Created an observer for one of my Laravel models. It was generating new translated slugs when another post gets translated by an API. While implementing also an updated method besides the created observation I obviously updated the models slug.
Pretty confident this small update will work I just pushed to production. Tested it live.

BOOM

Hard reset on the server redoing the changes. Searching where I fucked up this time...

Finding me observing the updating while updating the updating of the updating by the updating. 🤦

So I had a problem. MongoDB replica set connection was not accessible to server in another container. I’ve used ChatGPT. Gave it my code. It showed me the things I didn’t know and helped me work out a problem I’ve struggled with for 2 days.

It’s awesome!

ChatGPT is basically StackOverflow 2.0. It’s a tool and a great one. I can’t wait for an actual production level implementation target to software engineers.

P.S. I think co-pilot sucks.

TL;DR Dear boss, firstly, you always get someone to review anything important done by a fucking intern.
Secondly, you do not give access to your fucking client's production server to an intern.
Thirdly, you don't ask your fucking intern to test the intern's work that has not been reviewed by anyone directly on your client's fucking production server.

Last week, the boss and one of the lead devs (the only guy with some serious knowledge about systems and networking) decided to give me (an intern who barely has any work experience) the task of fixing or finding an alternate solution to allowing their support team access to their client machines. Currently they used a reverse SSH tunnel and an intermediary VH but for some reason, that was very unreliable in terms of availability. I suggested using OpenVPN and explained how it would work. Seemed to be a far better idea and they accepted. After several days of working through documentations and guides and everything, I figured out how OpenVPN works and managed to deploy a TEST server and successfully test remote access using two VMs. On seeing my tests, the boss told me that he wanted to test it on the client network. I agreed. Today he comes to me and he tells me to prepare testing for tomorrow and that the client technician is going to give me access to one of their boxes. And then he adds, "It's a working prod server. We'll see if we can make it work on that" and left. I gaped at him for a while and asked another dev guy in the room if what I heard was right. He confirmed. Turns out, the lead dev and the boss's son (who also works here) had had a huge argument since morning on the same issue and finally the dev guy had washed it off his hands and declared that if anything goes wrong from testing it on production, it's entirely the boss's own fault. That's when the boss stepped in and approached me. I ran back to his office and began to explain why prod servers don't top the list of things you can fuck around with. But he simply silenced me saying, "What can go wrong?" and added, "You shouldn't stay still. You should keep moving". Okay, like firstly what the fuck and secondly, what the fuck?.
Even though OpenVPN client is not the scariest thing to install, tomorrow's going to be fun.

Our employee management system, for some reason, stored Testlists (I work in QA) linked to the user accounts that created them. Now after an colleague who worked there for five years left pretty much all our data was suddenly down the drain and nobody backed the fricking server up because, hey, whats the fun in that. Now all the tests need to be rewritten and other than the whole gui test automation of our product, maintenance of the same for another product, manually testing dev issues and training my new code monkeys to frickin not commit non working code to the trunk I have now also "Make a better Employee management system" (roughly translated those are the specs I've got) on my plate... I can remember back to the care free days of just before my boss asked me if I wanted to try to automate some of the test cases... How did I ever survive this paralyzing tranquility. Ha, surprise.
!rant, I fucking love the stress and juggling a shit ton of problems at the same time keeps ine on edge.

Deployment can be a reason for heartache...

I deployed a website yesterday and it was a damn hustle. After changing into goodmode and typing random stuff into the console, I broke another of my websites on the same server.

Now I had two sites to deploy / repair. Damn.

After a few hours I managed to make both work. But damn, that adrenaline rush that gave me. Fuck!

Considering mullvad is based in sweden (xkeyscore, 14 eyes, ..) and has never seemingly had any public court record proving its claims, is it really to be trusted, instead of e.g. BlackVPN/NordVPN? does the server location matter if they are based in sweden?

Is it just again an excuse like "even if that happens, I am only hiding from X, I have nothing to hide from Y"? so e.g. your neighbour can't snoop, but a court if they decide everybody that visited devrant is a criminal - is alright?

PIA is based in the US (no discussion level of bad), but atleast got tested twice already and each time could not return any logs, even though I like mullvads model and it is clearly better than being US based, it still makes one question if mullvad is not yet another PureVPN in fancier clothes?

I just got a fucking job again after 2.5 months between jobs and the new place has been allowing (if not encouraging) the piracy of Windows Server in client environments... I thought this place had so much potential but I was wrong.

Going to start looking for another full time job or really buckledown and try to get my freelance project/business started.

BTW fuck microshaft for expensive licensing, but I’m not risking my certs and professional career for some idiots trying to pirate software.

Recently made a sub 750 euro in home server with raid 1, to back up all pc's laptops and smartphones in the house to. Never been better!

Using a simple little daily backup tool on all pc's, and another app to sync 2 way to my phone, i was able to have all my data safe on that home server.

It even allows me to edit content on my phone which will sync overnight, because of the way usb access is shit on android since.. 5.0?

I am loving it! Worth every penny and week of setup effort.

ideal sprint fallacy.
total days 10 , total hours(excluding breaks ) 8 hrs per day= 80 hrs per dev

code freeze day = day 8, testing+ fixing days : 8,9,10. release day : day 10
so ideal dev time = 7days/56 hr
meetings= - 1hr per day => 49 hrs per dev

- 1 day for planning i.e d1 . so dev time left . 6 days 42 hrs.
-----------
all good planning. now here comes the messups

1. last release took some time. so planning could not happen on d1. all devs are waiting. . devtime = 5 days 35 hrs.

2. during planning:
mgr: hey devx what's the status on task 1?
d: i integrated mock apis. if server has made the apis, i will test them .
mgr : server says the apis are done. whats your guestimate for the task completion?
d : max 1-2 hrs?
m : cool. i assign you 4 hrs for this. now what about task 2?
d : task told to me is done and working . however sub mgr mentioned that a new screen will be added. so that will take time
m : no we probably won't be taking the screen. what's your giestimate?
d : a few more testing on existing features. maybe 1-2 hrs ?
m: cool
another 4 hrs for u. what about task 3?
d : <same story>
m : cool. another 4 hrs for u. so a total of 12 hrs out of 35 hrs? you must be relaxed this sprint.
d : yeah i guess.
m cool.
-------
timelines.
d1: wasted i previous sprint
d2 : sprint planning
d3 : 3+ hrs of meetings, apis for task 1 weren't available sub manager randomly decided that yes we can add another screen but didn't discussed. updates on all 3 tasks : no change in status
d4 : same story. dev apis starts failing so testing comes to halt.
d5 : apis for task1 available . task 3 got additional improvement points from mgr out of random. some prod issue happens which takes 4+ hrs. update on tasks : some more work done on task 3, task 1 and 2 remains same.
d6 : task1 apis are different from mocks. additionally 2 apis start breaking and its come to know thatgrs did not explain the task properly. finally after another 3+ hrs of discussion , we come to some conclusions and resolutions
d7 : prod issue again comes. 4+ hrs goes into it . task 2 and 3 are discussed for new screen additiona that can easily take 2+ days to be created . we agree tot ake 1 and drop 2nd task's changes i finish task 2 new screens in 6 hrs , hoping that finally everything will be fine.

d8 : prod issue again comes, and changes are requested in task 2 and 3

day 9 build finally goes to tester
day 10 first few bugs come with approval for some tasks
day 11(day 1 of new sprint) final build with fixes is shared. new bugs (unrelated to tasks. basically new features disguised as bugs) are raised . we reject and release the build.

day 2 sprint planning
mgr : hey dev x, u had only 12 hrs of work in your plate. why did the build got delayed?

🥲🫡

>Be a customer
>Ask to reduce number of servers you rent at our company
>Agree to a date when the one server you no longer needs will be due to be disconnected and taken apart
>Date comes
>I have the honors of sending the final /sbin/poweroff
>All goes neatly... until...

The web that used to be there, now moved to another one of their machines, goes down, wtf???

Oh. a 500. What?
Checks logs...
Cannot connect to Database.
Wtf? Local database works... Oh. OH. OH MY GOD.

>Turn the server back on and tell the customer to fix the app to no longer connect to that machine

Sometimes, being a sysadmin can be a real fun!

At work, all errors within the site are logged into our database with a subject and error column. SQL errors are logged in the subject field while the traceback is put in the error column. However, a lot of SQL errors are really large and exceed the max character width of the subject field, causing yet another SQL error, and the cycle repeats. This recursive error has been the bane of my existence, because 1) it times my local dev instance out and 2) the error doesn't end up getting logged because the server both freezes and the error can't be inserted in the database. You can't even begin to imagine how many hours I've wasted trying to find what line I changed cause total and utter failure with absolutely 0 error logging. Next thing on my todo list is to fix this fucking issue since the head dev refuses to get it done.

I remember the first time working with a Windows 2012 R2 server.
Of some reason I couldn't open Windows Powershell as admin and asked another Dev what to do. His response? Try restarting the server.
Can you imagine the look on my face if I told you I only worked on Linux until that point. How can you seriously make a server OS that needs restarting at all? Is it just me or is that just fucking stupid?

Alright, server got hacked a week ago. Bad enough on its own but okay, perfect time to change the server infrastucture completely instead of doing it later this year. Since Saturday we are working on setting everything up (game server, apache, etc.pp.) while making sure to configure everything correctly to be safer this time.

We are finally at the point where we could go back online. And what happens? One team member _now_ (6 days after the hack) suggests that it might be a good idea to format the hacked server and configure just what we need to patch the clients with it.

Great fucking idea, why didn't you have that idea 5 days earlier?! There was more than enough time already to format the old server and configure it. Another day delay, yay. X_X

Aaah, ranting really helps in those situations. Oh and Hi, I'm new here. Nice place, I like it. ^_^

Repost from Reddit, i will probably do another with certain color theme, i just prefer this to be 'dark' build as it is how i initially immagined it.
I plan on expanding with certain micro pc, as server, 3d printer and some true hifi.
Suggestions?

I’ve been out of steady work for almost 2 whole months now but things are starting to look up...

I’m super stoked for some potential client projects!!! I have one client that wants me to completely rebuild their businesses infrastructure, PC refresh, server upgrade, network overhaul for 3 sites, and more. This new client has a business partner with another side business and wants to discuss potential work/projects. And I’m going to be discussing a potential contract deal on Thursday to develop a custom software for another client.

Guys! My startup is starting woot!!!

Your "feature" just became my problem. Your "great idea" is now my migraine. What you did in 300 lines, another team was already doing in 5.

The next time you `brew install...` on your laptop, you should fucking think that the infra team has to install those dependencies, on every server, too.

In less time than it took you to create your code, I could have given you several functions to call. I could have saved all of us weeks of work. Fucking ask cross-team before you cowboy code your next big idea please.

Got a problem you need to be solved, somebody else probably solved it, just fucking ask.

!rant
We were finishing another sprint of our grocery shop site at school and it was time for a demo.
There we are, showing our work before the other students. Our teams have a healthy habit of always checking each other not to leave some stoopid mistakes in the final versions, so everybody always regExes and validates THE SHIT out of every input field, both in the view and on the server side. But this one team found out that sometimes it's not enough.

Like every team, they're asked to buy a negative value from their shop. The guy clicks through the process, buys exactly -1 of a banana. He clicks the button to purchase and the site returned "Added banana to the cart!" and we're like "haha n00bz". But someone asked them to show the cart and everyone stopped immediately.

There were 9999 bananas in the cart.

Turns out the member responsible for purchase validation made it add 10000 if the quantity of a bought product was negative.
To this day I can't understand why he did that. xD

I think I may have officially gotten myself fired before I even started a new job. My salaried start date was supposed to be Jan 3 but they hired me to do spot work at my hourly rate until then. My server side PHP skills were never great but they appear to be completely inadequate to the task of patching their undocumented, spaghetti legacy code. I just sent a note basically saying I either need to convert their entire site to something else 3 weeks ahead of the timeframe we planned or to basically outsource my work to another developer to patch this code. Feeling like a total imposter at the moment. I wouldn't hire me.

People at work found out Teams in a nightmare and really screws with your normal email address processing. They also found out it isn't free with the shit tier of Office (or wherever it is bundled). So for everyone but sales there is no Teams. Whew... However, for the tech dudes: electrical and software we made a custom Discord server. Of course my avatar is "trolling DiCaprio".

Some technical and some not work stuff has gone on with this server. Kind of gives the tech people a place to talk and joke.

devdude: Apologies that I saw this (some question I had) too late to prevent you from walking upstairs.
me: oh, the exercise trauma!
devdude: it's 2024 and we still have to walk up and down stairs
me: I was expecting flying chairs like on Wall-E
devdude: Me too! that's why I put on this much weight so I can be prepared for when the chair finally is here.
me: That is the exact opposite of helping this tech along.

Another thing I noticed about my work place. The BMI of employees seems to increase the closer to a break room you get. The company is fond of bringing donuts periodically. Coincidence I am sure. The problem is I am right next to the break room... Yes, my BMI went up a bit when I moved to my new desk. Before I was much further away. Now I am on a low carb diet. I am going to break the stats damn it!

Operating system, AI, web server, programming language, etc. I think I'll start another tomorrow :)

Well well well.
Story time.

Since we are working from home for the past 4 months, I finally decided to install a Microsoft SQL server on my home server. (Mostly was using Azure)

My server is running Windows Server 2012 R2.
Tried installing SQL 2019 : fail, 2016 : Fail, 2012 : Fail. Some obscure message about some DLLs not being at right version. (And a warning that it is no recommended to install SQL server on domain controller, but I know, it is my home setup, not roduction)

“Ok fine, I’ll install it on my PC instead”. Windows 10 PC. NOPE. “Cannot install on a compressed drive”. Welp, wtf ? (Of course you cannot select destination install folder, I could’ve put it on another drive).

So here I am. Working 100% on Windows, installed Ubuntu server 20 LTS in Hyper-V, Installed Microsoft SQL server on it (BTW, install is very easy compared to windows). And that shit is working. And new “Terminal” app does support SSH out of box, no need to add Putty !

So as a Windows user, I needed Linux to make Microsoft SQL techno work.

Nothing will ever surprise me anymore. (BTW it’s fucking fast. I like SQL server on Linux)

Writing a feature critical for production in 2 hours of solid focus during the morning.

6 hours later it's still not in the build because:
* tech lead wants the code to move to a partial class instead of an extension method, delaying the UX review. No guidelines for this ever existed.

* after seeing the result, the UX team wants some element to be dynamic. A line. A friggin horizontal line.

* after adding the dynamic shiny frigggggin line, I try to test the feature with the server. It is still not deployed because the server guy went home. "The PR was not merged so I assumed we'll add it tomorrow".

Another day at the meat grinder.

so I installed nginx on my server this week. I feel like a giddy kid now installing one self hosted app after another. REVERSE PROXY ALL THE THINGS!

Right now I have reviewboard and drone (drone.io) installed. Any of you guys have suggestions for other cool stuff to try out? Mostly interested in something with a web API that can do fun stuff :)

My vocabulary is way to small to express my feelings when being forced to use .Net 4.0. Just spent like 2 hours searching why my Api requests failed.
Turns out it used TLS 1.0 which got rejected by the server. Then I spent another 2 hours finding out how to make it use TLS 1.2. Surprisingly it does work now (although it came out before TSL 1.2 specification). But yeah still a fucking pile of shit.

A few days ago our server was compromised due to an outdated Jenkins version. The malicious user installed a crypto miner on the server... The same day that it was found I told management that I'm interested in helping out with the server. Since then, nothing happened... No updates, no security measures, no nothing (except for the removed crypto miner and updated Jenkins software)

Oh well only a matter of time before another hack...

Question to some (who work way way way longer than me) med - seniors, should I make a big deal out of this? And keep pressure on it. Or should I just leave it be and wait for the next comprised server? I know devrant is not a Q&A service, but some dev to dev advice is much appreciated.

- incognito

Dev, boss and guy who know logic is looking at the server.
Problem: it's not responding
Boss: we need this running now! Otherwise the sales won't go through
Dev: give me a chance, I just got here
Guy: have you tried turning it off and on again?
They did so and at works.
Boss: guess we don't need to hire another dev, this guy knows what he is talking about, he is some kind of server expert..
Really.........

What a week at work...

As some of you might know, it‘s currently very hot in Germany with temperatures rising up to 35°C. That‘s when our AC at work decided to stop working. I‘m working in the third floor of a three story building so it‘s getting very hot in the office.

The day after we had a 45 min powercut and the AC still does not work.

Today when I got up and wanted to go to work, I got an E-Mail saying that we have another powercut which lasts at least three hours. We‘re supposed to work from home using VPN. But how the fuck should I be able to log into the VPN if the network is offline?! Oh and of course our GitLab server is hosted in house as well, so no access to any code at all.

Hopefully next week is gonna be a better one...

Server bios corruption, yaay.
Server external backups, naay.

This happened just before migration to another server. I feel stupid for not having proper backups now, and molested by a dying panda because its less than 6 months ago i got the server. It was used, but still.

I really don't understand this particular Government Department's IT Unit. They have a system and network to maintain except:

- They don't have a DBA
- They don't have a dedicated Network Engineer or Security Staff
- Zero documentation on all of the systems that they are taking care of (its all in each assigned particular staff's brain they said)
- Unsure and untested way of restoring a backup into a system
- Server passwords are too simple and only one person was holding this whole time and its to an Administrator account. No individual user account.
- System was developed by an in-house developer who is now retired and left very little documentation on its usage but nothing on how its setup.

But, the system has been up and operational for the past 20 years and no major issues whatsoever with the users using it. I mean its a super simple system setup from the looks of it.

1 App Server connected to 1 DB Server, to serve 20-30 users. But it contains millions of records (2GB worth of data dump). I'm trying to swing to them to get me on a part time work to fix these gaps.

God save them for another 20 years.

From now on I am administrating multiple servers in our company and monitoring is one thing our infrastucture lacks...almost completely. At least, useful monitoring.
Installing netdata or Grafana and integrate it with chat is definitely a solution, but what happens if the whole server just shuts down (very stupid scenario I know)? Well, it is easy, there will be no alert about the failure.
So, that's where I was wondering if there is a tool or even better plugin for netdata or Grafana, that enables remote monitoring from another server? I surely can write a simple script to check the server availability but having the whole monitoring tool on a single server instead of 5+ would be also easier to maintain and setup.

The coolest project I ever worked on wasn't programming per second, though it involved a bit of scripting. The company I worked for had an FTP over TLS backup solution and it was put together with glue and paperclips by a guy that hadn't the slightest idea what he was doing. In order to conform with the insurance, data had to be encrypted. I setup a raid-ed server with full disk encryption on the raid volume that fetched the key over the network at boot from another secure server. I wrote a series of scripts for provisioning users and so on. The backup connections was sftp using a ssh tunnel, the users were chrooted to their own home direc