Chatting on Slack with a junior dev:

[Junior Dev] How do I get that file from the server ?
[Me] ssh into it and then use scp

I see Junior across the room, literally saying "shshh..." at the computer.

Packed my stuff and quit that day.

I worked with a good dev at one of my previous jobs, but one of his faults was that he was a bit scattered and would sometimes forget things.

The story goes that one day we had this massive bug on our web app and we had a large portion of our dev team trying to figure it out. We thought we narrowed down the issue to a very specific part of the code, but something weird happened. No matter how often we looked at the piece of code where we all knew the problem had to be, no one could see any problem with it. And there want anything close to explaining how we could be seeing the issue we were in production.

We spent hours going through this. It was driving everyone crazy. All of a sudden, my co-worker (one referenced above) gasps “oh shit.” And we’re all like, what’s up? He proceeds to tell us that he thinks he might have been testing a line of code on one of our prod servers and left it in there by accident and never committed it into the actual codebase. Just to explain this - we had a great deploy process at this company but every so often a dev would need to test something quickly on a prod machine so we’d allow it as long as they did it and removed it quickly. It was meant for being for a select few tasks that required a prod server and was just going to be a single line to test something. Bad practice, but was fine because everyone had been extremely careful with it.

Until this guy came along. After he said he thought he might have left a line change in the code on a prod server, we had to manually go in to 12 web servers and check. Eventually, we found the one that had the change and finally, the issue at hand made sense. We never thought for a second that the committed code in the git repo that we were looking at would be inaccurate.

Needless to say, he was never allowed to touch code on a prod server ever again.

Manager: We need to setup the security in the Mexico server

Dev: You mean that 3rd party firewall add on?

Manager: Yes

Dev: And set up the billing on the Mexico account?

Manager: Yes

Dev: lol, sure thing I’ll create the ticket

Manager: What’s so funny?

Dev: Nothing

Ticket: Build wall and get Mexico to pay for it.

>Building advertising platform
>Life is good
>Fire up the dev server and open in browser
>No assets
>What the fuck
>Check code and config for like an hour it's all good
>What the fuck
>Try incognito browser, it works.
>What the fuck
>Oh yeah. Adblock. Fuck.

I use a lot of dev tools, but one of my favorites is the Linux screen utility. It's awesome to be able to keep shit running on servers whether my laptop stays connected tot he server or not. It's great for jobs that take a long time, can't get interrupted, etc.

If you haven't used screen definitely give it a try!

So this happened today.

Client: hey I sent this ticket, what's the status/have you located the issue?
Me: well, it says it quite obviously in the error message...? (i actually said that, toned down afterwards a little)
Client: where's the error message then?
Me: 5th line....? It's literally there in plain english?
Client: ok so what does it mean?
Me:..............? "marked as spam by the receiving server"?!
Client: yeah ok but what does that mean?

😐

Thing to keep in mind: they're a web dev/email solutions company.

😐😩

Maintain your LinkedIn, write little articles about implementations on a tech blog, check issues on popular github projects and make PRs, create a portfolio website. Register as a company and do some freelance work, even if it's just a cheap website for your grandma's knitting club.

Do the tour/tutorial of every popular language/framework. Learn the basics of react/vue as a backend dev, learn some sql as a frontend dev. Set up a vps server at DO or AWS, host a few small services. Fullstack is bullshit, but communication is key in development, which means you need to know about the whole playing field.

Recruiters can be useful, but knowing developers in your area is even more valuable. So especially if you're unemployed, go to hackathons, conferences and meetups.

So at work today my coworker overlooked my laptop running Linux with i3.

Coworker: How do you live with this?

Me: What do you mean? This is customized to work with Git and my IDE efficiently while I do dev ops with my server.

Coworker: Your mouse barely works and you operate this thing totally on keyboard shortcuts. Linux will never be a serious platform.

Me: I'm not saying you or anyone at work has to use this, I built an environment to suite my needs. Same as anyone. I thought you liked consumer choice?

Needless to say we didn't get much further beyond him thinking I was nuts for configuring my server in the cli. I swear I don't understand why I try to explain anymore. 😡

Guy: dot net dev (C#) on windows. (desktop + server)
Team(not his team, he just happened to sit next to us): php/frontend devs and Linux (server) people.

Team: starting a new project! We'll have to see what framework to use and what server :D
Guy: i know it's none of my business...... but I'd recommend dot net and windows server!
Me: respectfully, that hardly makes sense, you know our skillset/field... i understand that it works for you but it doesn't really for us :).
Next to that we'd rather not use windows for security reasons.

It's fine if that happens once.

When it happened for the 1748472823'th time, I had a real hard time controlling myself.

My non-dev boyfriend installed Python via the command line and ran a server to render a map of Pokémon in his neighborhood. I don't know whether to be impressed or scared 😂

Manager: THE SERVER IS DOWN THE SERVER IS DOWN!!!!

Dev: Ok I’ll look into it

*5 mins later

Dev: Wow these are really strange logs, it’s like config values are being changed all over the place while I’m looking at it

Manager: Well I figured while you were looking into it I’d go i to the server settings and change everything I could find in order to try and get the server back up again. Two sets of hands are better than one, Is it up yet???

Dev: …No.

Manager: I THOUGT YOU SAID YOU’D LOOK INTO THIS. I NEED ANSWERS NOW. WHAT IS TAKING SO LONG?!?!?

Dev: …

How my birthday is going so far:
1. Dropped coffee on my shirt
2. Client is wondering why we removed a feature that he wants yet asked us to remove a week ago
3. Dev server is no longer working
4. Internet connection is shit
5. My VPN keeps on disconnecting (see #4)

Is this day going to end or what?

My biggest dev blunder. I haven't told a single soul about this, until now.

👻👻👻👻👻👻

So, I was working as a full stack dev at a small consulting company. By this time I had about 3 years of experience and started to get pretty comfortable with my tools and the systems I worked with.

I was the person in charge of a system dealing with interactions between people in different roles. Some of this data could be sensitive in nature and users had a legal right to have data permanently removed from our system. In this case it meant remoting into the production database server and manually issuing DELETE statements against the db. Ugh.

As soon as my brain finishes processing the request to venture into that binary minefield and perform rocket surgery on that cursed database my sympathetic nervous system goes into high alert, palms sweaty. Mom's spaghetti.

Alright. Let's do this the safe way. I write the statements needed and do a test run on my machine. Works like a charm 😎

Time to get this over with. I remote into the server. I paste the code into Microsoft SQL Server Management Studio. I read through the code again and again and again. It's solid. I hit run.

....

Wait. I ran it?

....

With the IDs from my local run?

...

I stare at the confirmation message: "Nice job dude, you just deleted some stuff. Cool. See ya. - Your old pal SQL Server".

What did I just delete? What ramifications will this have? Am I sweating? My life is over. Fuck! Think, think, think.

You're a professional. Handle it like one, goddammit.

I think about doing a rollback but the server dudes are even more incompetent than me and we'd lose all the transactions that occurred after my little slip. No, that won't fly.

I do the only sensible thing: I run the statements again with the correct IDs, disconnect my remote session, and BOTTLE THAT SHIT UP FOREVER.

I tell no one. The next few days I await some kind of bug report or maybe a SWAT team. Days pass. Nothing. My anxiety slowly dissipates. That fateful day fades into oblivion and I feel confident my secret will die with me. Cool ¯\_(ツ)_/¯

I think I've shown in my past rants and comments that I'm pretty experienced. Looking back though, I was really fucking stupid. Since I haven't posted a rant yet on the weekly topics, I figure I would share this humbling little gem.

Way back in the ancient era known as 2009, I was working my first desk job as a "web designer". Apparently the owner of this company didn't know the difference between "designer", which I'm not, and "developer", which I am, nor the responsibilities of each role.

It was a shitty job paying $12/hour. It was such a nightmare to work at. I guess the silver lining is that this company now no longer exists as it was because of my mistake, but it was definitely a learning experience I hold in high regard even today. Okay, enough filler...

I was told to wipe the Dev server in order to start fresh and set up an entirely new distro of Linux. I was to swap out the drives with whatever was available from the non-production machines, set up the RAID 5 array and route it through the router and firewall, as we needed to bring this Dev server online to allow clients to monitor the work. I had no idea what any of this meant, but I was expected to learn it that day because the next day I would be commencing with the task.

Astonishingly, I managed to set up the server and everything worked great! I got a pat on the back and the boss offered me a 4 day weekend with pay to get some R&R. I decided to take the time to go camping. I let him know I would be out of town and possibly unreachable because of cell service, to which he said no problem.

Tuesday afternoon I walked into work and noticed two of the field techs messing with the Dev server I built. One was holding a drive while the other was holding a clipboard. I was immediately called into the boss's office.

He told me the drives on the production server failed during the weekend, resulting in the loss of the data. He then asked me where I got the drives from for the Dev server upgrade. I told him that they came from one of the inactive systems on the shelf. What he told me next through the deafening screams rendered me speechless.

I had gutted the drives from our backup server that was just set up the week prior. Every Friday at midnight, it would turn on through a remote power switch on a schedule, then the system would boot and proceed to copy over the production server's files into an archive for that night and shutdown when it completed. Well, that last Friday night/Saturday morning, the machine kicked on, but guess what didn't happen? The files weren't copied. Not only were they not copied, but the existing files that got backed up previously we're gone. Why? Because I wiped those drives when I put them into the Dev server.

I would up quitting because the conversation was very hostile and I couldn't deal with it. The next week, I was served with a suit for damages to this company. Long story short, the employer was found in the wrong from emails I saved of him giving me the task and not once stating that machine was excluded in the inactive machines I could salvage drives from. The company sued me because they were being sued by a client, whose entire company presence was hosted by us and we lost the data. In total just shy of 1TB of data was lost, all because of my mistake. The company filed for bankruptcy as a result of the lawsuit against them and someone bought the company name and location, putting my boss and its employees out of a job.

If there's one lesson I have learned that I take with the utmost respect to even this day, it's this: Know your infrastructure front to back before you change it, especially when it comes to data.

First day at new web dev job:
Me: what IDE do you guys use?
Coworker: Notepad/notepad++
M: Okay... Any version control?
C: Oh we don't need it. We just update the server pages.
*Boss walks in*
Boss: Heres the project for you to do just need you to rebrand this web app we made for client A for client B just need to change some scripts. Should only take afew hours.

I take a look. No comments. Not formatted. Missing braces and brackets. Semicolons at odd places and missing at others. 7802 lines of code...

Dev: Hey our current server is starting to chug a bit. Can I get approved for $1200 additional spend to double the speed?

Manager: *Sharp inhale*. We need this project to cost as little as possible, we really can’t justify spending any additional money for any reason right now.

*2 days later*

Manager: YOU ARE APPROVED FOR $100,000 TO IMMEDIATELY IMPLEMENT SOMETHING RELATED TO NFTs IN ANY OF OUR APPS. THE BUSINESS NEEDS TO EXPAND INTO THE METAVERSE ASAP IMMEDIATELY. I NEED AN ETA BY EOD AS TO WHEN THIS CAN BE ROLLED OUT.

Dev: …

Boss: are you sure errors are logged on the server?
Dev: yes 100% sure

Every time I hear a developer say "works for me", I'm gonna hide a service of mine that they use behind the VPN.

Dev: "oy mate, this server is down"
Me (with VPN connection): "sorry mate, works for me"
Dev: "but here, check this out, it's down!!"
Me: "mate, check your network connection. You must have a shitty network connection."
Dev: -_-
Me: "Maybe shitty hardware? Driver issues on your network card? 🤭"

Because you know, we sysadmins can do that too 😉

!rant
!!git

Who here uses `master` for development?

My boss (api guy) tried to convince me that was normal practice. I gently told him that it sounded crazy and very very bad.

Here's the dev path I'm enforcing on my repos:

(feature branches) -> dev -> qa* -> master -> production*

*: the build server auto-pulls from these branches, and pushes any passing builds to staging/production.

Everyone works on their own feature branches, and when they're happy with their work, they merge it into `dev`. `dev`, therefore, is for feature integration testing. After everything is working well on `dev`, it gets merged into `qa` for the testers to fawn over and beat with sticks. Anything that passes QA gets merged into `master`, where it sits until we're ready to release it. When that time comes (it's usually right away, but not always), `master` gets merged into `production`.

This way, `master` is always stable and contains the newest code, so it's perfect for forking/etc. Is this standard practice, or should I be doing something different?

Also, api guy encourages something he calls "running a racetrack" -- each dev has their own branch (their initials) and they push to that throughout the day. everyone else pulls from it regularly and pushes to their own branch. When anyone's happy with their code, they push from their (updated) branch to `qa` (I insisted on `dev` instead.)

Supposedly this drastically reduces the number of merge conflicts when pushing to an upstream branch due to having a more recent ancestor node?

I don't quite follow that, but it seems to me that merging/pushing throughout the day would just make them happen sooner? idk.

What are your thoughts?

"Arch Linux is actually not that difficult".

I ssh'ed into my home server yesterday.

I was greeted by a message from an ext3 disk about needing fsck. Fine, "I haven't been in here for a while, might as well do some maintenance". fsck /dev/sda6, let's go!

This nicely "repaired" the sshd service (i.e. cleared the sectors), I cursed at myself for pressing enter at "repair (y)" right before the connection broke.

So I connected a display and keyboard... ok so let's just pacman -Sy sshd or whatever. We can do this! Just check the wiki, shouldn't be that hard!

Wait... pacman has not run since 2010? WAIT IT'S ACTUAL UPTIME IS 9 YEARS??? I guess we know why I'm a DB admin and not devops...

Hmm all the mirrors give timeouts? Oh. The i686 processor architecture isn't even supported anymore...?

4 hours, 11 glasses of cognac, 73 Arch32 wiki/forum pages, 2 attempts at compiling glibc, and 4 kernel panics later: "I think I'll buy a new server".

You think a junior dev pushing his code onto a production server is bad? Wait till you have that admin who is illegally mining Bitcoin on your production server. 😂

I went for a Cyber Security conference today with one of managers and this was one of the life experiences some of the speakers shared.

Having a server with a lower spec CPU and RAM than dev machines, to ensure that if it works on the dev machine it will always run perfectly on the server.

This means we can avoid any debugging tools or techniques (including console logs), because “it will be perfect and it’s not necessary”.

More sysadmin focused but y’all get this stuff and I need a rant.

TLDR: Got the wrong internship.

Start working as a sysadmin/dev intern/man-of-many-hats at a small finance company (I’m still in school). Day 1: “Oh new IT guy? Just grab a PC from an empty cubicle and here’s a flash drive with Fedora, go ahead and manually install your operating system. Oh shit also your desktop has 2g of ram, a core2 duo, and we scavenged your hard drive for another dev so just go find one in the server room. And also your monitor is broken so just take one from another cubicle.”

Am shown our server room and see that someone is storing random personal shit in there (golf clubs propped against the server racks with heads mixed into the cabling, etc.). Ask why the golf clubs etc. are mixed in with the cabling and server racks and am given the silent treatment. Learn later that my boss is the owners son, and he is storing his personal stuff in our server room.

Do desktop support for end users. Another manager asks for her employees to receive copies of office 2010 (they’re running 2003 an 2007). Ask boss about licensing plans in place and upgrade schedules, he says he’ll get back to me. I explain to other manager we are working on a licensing scheme and I will keep her informed.

Next day other manager tells me (*the intern*) that she spoke with a rich business friend whose company uses fake/cracked license keys and we should do the same to keep costs down. I nod and smile. IT manager tells me we have no upgrade schedule or licensing agreement. I suggest purchasing an Office 365 subscription. Boss says $150 a year per employee is too expensive (Company pulls good money, has ~25 employees, owner is just cheap) I suggest freeware alternatives. Other manager refuses to use anything other than office 2010 as that is what she is familiar with. Boss refuses to spend any money on license keys. Learn other manager is owners wife and mother of my boss. Stalemate. No upgrades happen.

Company is running an active directory Windows Server 2003 instance that needs upgrading. I suggest 2012R2. Boss says “sure”. I ask how he will purchase the license key and he tells me he won’t.

I suggest running an Ubuntu server with LDAP functionality instead with the understanding that this will add IT employee hours for maintenance. Bosses eyes glaze over at the mention of Linux. The upgrade is put off.

Start cleaning out server room of the personal junk, labeling server racks and cables, and creating a network map. Boss asks what I’m doing. I show him the organized side of the server room and he says “okay but don’t do any more”.

... *sigh* ...

CIO: what kind of web server do you want for your dev environment? WordPress?
Me: Uhm, Linux centos running apa-
CIO: whoa that's dangerous you need to think of the people who are going to support this.
Me: right...
CIO: we're going to pick something and stick with it.

FML company is just starting to do in house dev. CIO is heavily involved and knows more than I do... My life is a Dilbert comic strip

Me: Well, it's time to make a new app!
* opens up VS Code *
* opens folder selection dialog *
* creates a new folder called "notes app" *
* yarn inits that folder *
* installs react and react-dom *
* installs webpack, webpack-cli, babel-core, babel-loader, babel-preset-env, babel-preset-react, style-loader, css-loader, file-loader, html-webpack-plugin and clean-webpack-plugin as a dev dependency (install is pending) *
* copies a webpack config from some other project *
* creates a babelrc file *
* copies a yarn script called "build:dev" which would launch webpack *
* dev dependencies installed *
* tries to save *
* vscode doesn't save because files differ *
* tries to copy dev dependencies *
* fail *
* tries again *
* saves *
* writes bare-bones index.jsx *
* yarn build:dev *
* opens build/index.html in firefox *
* gets satisfaction *
* writes bare-bones App.jsx which is a react component but it's an entire app *
* yarn build:dev *
* opens build/index.html in firefox *
* gets satisfaction *
-- trim --
* walks out of his room to his mom's room where's sbc is located *
* grandma plays solitare on laptop *
* i ask grandma for a laptop *
* grandma gives me laptop *
* glues all components into App.jsx *
* yarn start:dev (magic of webpack-dev-server) *
* opens localhost:8080 in firefox *
* searches how to update a component prop *
* nothing found *
* registers on devrant and verifies his email *
* writes this rant *

My computer science teacher won't stop developing on the production server 😭 he switches the branch on the production machine to dev all the time and merges broken code into master. Kill me

So apparently two "senior" "laravel-engineers" spent a total billed 35 hours trying to figure out a "critical bug" which "doesn't happen locally".

I went to the dev-console, saw it is generating http urls (fronted by cloudflare https, running on http server-side) and fixed that in maybe ~15 minutes, fucking morons.

Yesterday, the whole dev team went out to lunch and we ate a lot and we drank a lot until we all got drunk! Since we are paid for the number of hours we work, we all decided to go back to the office and work.

WE WERE DRUNK WORKING! Drunk coding ftw

Result: The server is down right now because someone fucked it up and I think i ruined my code yesterday because I wasn't really myself. The whole team was crazy as fuck. One of us just came back from Poland so we were drunk and high from all the polish chocolate we ate.

I hope they fix the server so I can check what kind of bullshit the drunken me did yesterday o.O

I found a cool project on GitHub. I forked it and added a simple dev server with the intent of making it more accessible which could lead to more activity = improved project. I created a PR with small concise commits with very informative messages.

The guy who owns the project comments and says "I don't want your dev server, I have an apache instance locally on my computer". I tell him "Ok sure, but wouldn't it be nice if everyone else also had a nice dev server which can be started with a single command?", and other people join the PR and agree with me that we should make it available for everyone.

But the fucking idiot doesn't care, "No, I prefer to use my apache server". YOU FUCKING ASS WIPE, why do you even put it up on GitHub if you don't want contributions to make your project better and more available? I saw other open PRs where he basically did the same thing, left a snarky comment without merging it. What a fucking tool. Worst spent time ever.

FUCK YOU

The server for an internal application was acting up, but everything was working fine locally. After an 18 hour day, I just rerouted the traffic to my dev machine and left it on overnight so I could go home.

A piece of me hates myself and also loves myself for that one.

This just in everyone...

Android Dev: *sent and email to network admin* can you please unblock github for a few mins.

Network admin: *Replied* Can you take a screenshot whats the error your getting.

Android Dev: *Replied with screenshot* "Failed to load resource: the server responded with a status of 503 (Service Unavailable)"

Network admin: that is a known issue. *Replied with Wordpress Links.

Android Dev: why is github working outside our network then?

Network admin: there must be a problem with your code that needs to be tweaked.

Team: *FACE PALM*

Senior Dev: "-bleep- I hate Javascript. It is such a pain to have to debug in Chrome"
Mgr: "Why are you 'having to' debug in Chrome?"
-in an almost 'you didn't know?' condensing tone -
Senior Dev: "Because you can't debug Javascript in Visual Studio."
Me: "Umm...pretty sure you can."
Senior Dev: "No, its impossible. I have to make a simple change in Visual Studio, save it, deploy all the files to the server, restart IIS, open up Chrome and use it's developer tools to find bugs. -bleep- Javascript sucks sooo bad."
-I do a quick search on stackoverflow-
Me: "No, I'm looking right at it on stackoverflow. You can debug Javascript in visual studio just like anything else."
- Mgr looks over and smiles, not trying to laugh -
Senior Dev: "Hey, did you watch that scene in Stranger Things...man thats a good show ..."
- other devs jump in to comment about the show, completely dismissing the VS/Javascript conversation -

Not sure WTF just happened.

Dev gets hold of me, says my service is down in QA. Works if he hits it locally, works via Postman, but via the QA app server it gives a 401.

I’m like, look, if it works everywhere else, there’s something wrong on your side in QA.

He insists, no, I must help him, and begins CCing all the managers telling them this system has been down for days.

So I eventually climb into his system, check the credentials they’re using in the QA environment, and sure enough, the password is wrong.

Project Manager: Hey Gid, we need to start migrating project-A to the new Server.

Me: Okay, I will inform Dev-Q.

Project Manager: Please do and treat as top priority!

Me: Hey Dev-Q, we need to migrate project-A to the new Server and we need to get it done asap.

Dev-Q: But I'm currently working on some critical bug XYZ which PM wants fixed before COB.

Me: I dunno maybe you want to speak with him.

Dev-Q: I was told to...

Project Manager: Yes! we need that done right away.

Dev-Q: What about the critical...

Project Manager: No! treat this as top priority the client just called.

Dev-Q: Okay.

Me: Any update yet?

Dev-Q: Yep but it seems like the database is quite large and the migration may take a while.

Me: Okay take your time.

Dev-Q: {hours later} Pheww done! All files and database migrated successfully.

Project Manager: Good good. So the critical bug XYZ was also completed and migrated to the new server right?

Dev-Q:

Worst WTF dev experience? The login process from hell to a well-fortified dev environment at a client's site.
I assume a noob admin found a list of security tips and just went like "all of the above!".

You boot a Linux VM, necessary to connect to their VPN. Why necessary? Because 1) their VPN is so restrictive it has no internet access 2) the VPN connection prevents *your local PC* from accessing the internet as well. Coworkers have been seen bringing in their private laptops just to be able to google stuff.
So you connect via Cisco AnyConnect proprietary bullshit. A standard VPN client won't work. Their system sends you a one-time key via SMS as your password.

Once on their VPN, you start a remote desktop session to their internal "hopping server", which is a Windows server. After logging in with your Windows user credentials, you start a Windows Remote Desktop session *on that hopping server* to *another* Windows server, where you login with yet another set of Windows user credentials. For all these logins you have 30 seconds, otherwise back to step 1.
On that server you open a browser to access their JIRA, GitLab, etc or SSH into the actual dev machines - which AGAIN need yet another set of credentials.

So in total: VM -> VPN + RDP inside VM -> RDP #2 -> Browser/SSH/... -> Final system to work on
Input lag of one to multiple seconds. It was fucking unusable.

Now, the servers were very disconnect-happy to prevent anything "fishy" going on. Sitting at my desk at my company, connected to my company's wifi, was apparently fishy enough to kick me out every 5 to 20 minutes. And that meant starting from step 1 inside the VM again. So, never forget to plugin your network cable.

There's a special place in hell for this admin. And if there isn't, I'll PERSONALLY make the devil create one. Even now that I'm not even working on this any more.

Dev checked in code (I suspect purposely not inviting me on the code review invite) saying he "fixed" the authentication bug in the web service.

Um no, like I told you last week, the authentication error is because the load balancer wasn't passing the user's authentication to IIS.

If I didn't overhear him telling a user "Still getting the error? I don't know, we might have to re-write that service", he might have gotten away with it.

Me: "Wait, that doesn't sound right. If I hit the server directly, authentication works. Its an issue with the load balancer, not the service"
Dev: "Admin said the load balancer is fine and it has to be the service."
Me: "I don't buy it. IIS is returning the authentication error, not the service."
Dev: "I added exception handling and nothing is being logged. Must be something in the service configuration."
Me: "No, IIS performs the authentication, not the service. I explained that last week, remember?"
Dev: "Oh yea. What changes do we need to make to the service?"
<my blood pressure starts to spike>
Me: "None. Give me a sec.."
<we have other apps on the same server farm that work just fine, so I re-configure the service pool settings to match theirs>
Me: "See, now going through the load balancer, the service works fine. For some reason, the admin had our service set up differently."
Dev: "OK, I'll let the users know the service is fixed."
Me: "Service was never broke and I'm not leaving it in its current state. In the morning I'll talk to the admin and see what he can do to fix."

The riskiest dev choice...

How about "The riskiest thing you've done as a dev"? I have a great entry for that. and I suppose it was my choice to build the feature afterall.

I was working on an instance of a small MMO at a game company I worked for. The MMO boasted multiple servers, each of them a vastly different take on the base game. We could use, extend, or outright replace anything we wanted to, leading to everything from Zelda to pokemon to an RP haven to a top-down futuristic counterstrike. The server in this particular instance was a fantasy RPG, and I was building it a new leveling and experience system with most of the trimmings. (Talents, feats/perks, etc. were in a future update.)

A bit of background, first: the game's dev setup did not have the now-standard dev/staging/prod servers; everything ran on prod, devs worked on prod, players connected and played on prod, etc. Worse yet, there was no backup system implemented -- or not really. The CTO was really the only person with sufficient access. The techy CEO did as well, but he rarely dealt with anything technical except server hardware, occasionally. And usually just to troll/punish us devs (as in "Oops ! I pulled the cat5 ! ;)"). Neither of them were the most reliable of people, either. The CTO would occasionally remote in and make backups of each server -- we assumed whenever he happened to think of it -- and would also occasionally do it when asked, but it could take him a week, sometimes even up to a month to get around to it. So the backups were only really useful for retreiving lost code and assets, not so much for player data.

The lack of reliable backups and the lack of proper testing grounds (among the plethora of other issues at the company) made for an absolutely terrible dev setup, but that's just how it was, and that's what we dealt with. We were game devs, afterall. Terrible or not, we got to make games! What more could you ask for!? It was amazing and terrible and wonderful and the worst thing ever, all at the same time. (and no, I'm not sharing the company name, but it isn't EA or Nexon, surprisingly 😅)

Anyway, back to the story! My new leveling system also needed to migrate players' existing data, so... you can see where this is going.

I did as much testing and inspection of my code as I could, copied it from a personal dev script to the server's xp system, ... and debated if I really wanted to click [Apply]. Every time I considered it, I went back to check another part or do yet more testing. I ended up taking like 40 minutes to finally click it.

And when I did... that was the scariest button press of my life. And the scariest three seconds' wait afterwards. That one click could have ruined every single player's account, permanently lost us players ...

After applying it, I immediately checked my character to see if she was broken, checked the account data for corruption or botched flags, checked for broken interactions with the other systems....

Everything ended up working out perfectly, and the players loved all of the new features. They had no idea what went into building them, and certainly had no idea of what went into applying them, or what could have gone wrong -- which is probably a good thing.

Looking back, that entire environment was so fragile, it's a wonder things didn't go horribly wrong all the time. Really, they almost never did. Apocalypses did happen, but were exceedingly rare, and were ususally fixed quickly. I guess we were all super careful simply because everything was so fragile? or the decent devs were, at least. We never trusted the lessers with access 😅 at least on the main servers where it mattered. Some of the smaller servers... well, we never really cared about those.

But I'm honestly more surprised to realize I've never had nightmares of that button click. It was certainly terrifying enough.

But yay! Complete system overhaul and migration of stored and realtime player data! on prod! With no issues! And lots of happy players! Woooooo!

Thinking back on it makes me happy 😊

Taking "fixing a bug in your head once you walk away from the machine" to a new level.

Fixed a bug, checked it in. Happy.
Go to a meeting 5 minutes later. 10 minutes into the meeting have the sudden realisation that the bug fix was wrong and while it would fix the issue it would break something else.

Anxiously sit there for 50 more minutes not really paying attention because all I can think about is that sucker being auto deployed to our Dev server.

Managed to fix it and get it committed without anyone noticing but FML.

Seen another dev push changes to a production server without even committing it to git.

This happend to me around 2 weeks ago. For some reason, I decied to post this now.

I won the lottery, yey! I mean, bot really, but I am <19yo student, "less than junior dev" in my office, but sonce I am the only one who is capable of working with hardware, I was working month back as a sysadmin for a few days. Our last sysadmin was really good working but really, really toxic guy, so he got fired on a spot after argument with some manager or whatever, no big deal, we could have another guy hired in a week. But, our backup server literally was on fire, all data probably dead because bad capacitor or whatever. This was our only backup of everything at the time. Everyone in full fucking panic mode, we had literally no other working HW we could use for backup, but then comes me, intern employed on his first dev job for 3 months. That day I bought some HW for my own personal server at home (Intel NUC with some Celeron, 4GB DDR4 RAM and two 240GB SSDs for RAID 1. My manager asked everyone in the office for sollution how to survive next 4 days before new server arrives. People there had no idea what tk do and no knowedgle about HW, I just came from a break and offered my components for a week, since there was noone else who can work with HW, servers and stuff like this, manager offered me $500+HW cost if I, random intern, can make it work. I installed Debian on that little PC, created RAID1 from both SSDs, installed MySQL server and mirrored GIT server from our last standing server (we had two before one of them went lit 🔥), made simple Python script to copy all data on that RAID, with some help of our database guy copied whole DB from production to this little computer and edited some PHP so every SQL request made on our server will run on that NUC too. Everything after ±2 hours worked perfectly. Untill a fucking PSU burned in our server and took RAID controller with him in sillicon heaven next night, so we could not access any data unltill we got a new one. Thanks to every god out there, I was able to create software RAID from survived HDDs on our production server and copy all data from that NUC on the servers software RAID and make it working at 3 AM in the night before an exam 😂. Without this, we would be next ±40 hours without aerver running and we might loose soke of our data and customers. So my little skill with Linux, Python, MySQL and most importantly my NUC hardware I got that day running as a backup server saved maybe whole company 😂.
Btw, guess who is now employee of the year with $2500 bonus? 😀
Sorry for bragging and log post, but I was so lucky an so happy when everything worked out, good luck to all sysadmins out there! 👍

TL:DR: Random intern saved company and made some money 😂

So, I told my new newbies intern; "Please keep remind yourself, not accidently type 'rm -rf /' as root. You don't want to know what going to happen". He nod.

Next day, in the dev server, "Kernal Panic". Human, full of curiosity.

Today 🙄
This dev goes "I connect using plain FTP over a VPN to update why can't you!?"

Because it's unsecure you fucking idiot.

His FTP server can't even do secure connections. Some how.

Guess I have a new site to take over 😏

Pm: '[...] So when will you set up the server its urgent'

Me: 'look buddy i applied here as a java dev, ended up doing fronted. Fine, i like js, i can do that. But then dont fucking ping me daily with server stuff what has been overcomplicated and i got no idea about, theres a backend team, ask them.'

Pm: 'you said you would do the frontend'

Me: 'yeah that doesnt contradict a word i said'

Pm: 'you took responsibility, fix it'

Me: '?????? THATS WHAT IM SAYING YOU FUCKFACE COCKSUCKER ASSHOLE THAT I DIDNT AND I WONT, FUCK A HORSE'

I know a lot of you already know/do this, but to those of you who don't - I know it has saved me more than once:

If you use SSMS (SQL Server Management Studio) - when connecting to a server, before you hit the "connect" button, under options, choose a color to represent which server you are connecting to. I personally use stoplight colors: red = prod, yellow = QA, green = dev....this way, it helps you realize what server you are connected to, minimizing our dreaded "oh shit" moments....lol

Hopefully this helps someone ☺

Lead dev: Hey boss, you really do like Python right?

Me: No

Lead dev: Well it's cuz I was think....wait what? WTF do you mean no, you have automated a fuckload of BS with Python and we are still using it, why tf would you use Python if you don't like it?

Me: I like it enough for the automation scripts that we have and for parsing documents or generating glue scripts, its already installed in every server that we have, so testing bs in dev and then using them in prod is cake, it doesn't mean I LOVE python, I like it for what we use it.

Lead dev: Well ain't already bash and perl installed as well?

Me: Do you know bash and or perl?

Lead dev: No, don't you?....

Me: No......

L Dev: (using a Jim Carrey impersonation) WELLL ALLRIGTHY THEN! What is the other language that you used for X project?

Me: Clojure, do you remember that one?

* he said paren paren paren paren yes paren i space paren do close paren close paren etc etc
L Dev: (((((((yes (i (do)))))))) and nevermind, I'll get back to working more with Python

Me: das what I fucking thought esse

When I was in college, our email was on a Unix server. We would login via serial connection or telnet over the network, and get a korn shell. The server was poorly secured. Everyone's login device was world writable. So people would just see who was online, see the username of someone they wanted to mess with, take note of the pts(network) or tty(serial) device their connection used, and cat ASCII penises to it.

cat animated_dong.txt > /dev/pts/4

It was a simpler time.

this happens way too often in our company

PM: did you made that change I requested?
me: yeah, its on the live server now, why?
PM: I cant see it...
me: *wtf, I specially forced the JS to reload to eliminate problems with cache* could you send me a screenshot?
PM: *sends screenshot*
me: I dont get it... I can see the changes in my browser *dev feeling intensifies* ... refresh the site and try again
PM: oh... suddenly its there, ...anyway, thanks! it looks great!
me: *facepalm*

turns out our managers just dont refresh websites, they want changes to take place immediately

Got to a client, we are taking over their software dev and IT.

"you're not touching my code ". Listen fuck twit, a robocopy script is not fucking code.
"I wrote a 3500 line code for this company" no fuck twit, you wrote a whole lot of fucking gibberish that looks like someone shat out BASH and it met html along the way. It doesn't compile, it doesn't run, it's a fucking dormant file. You charged people for shit all.

Setting up exchange is also not a big whoop.

Moving them over to CentOS server (he had them on XP still), and writing enough code to qualm my frustration at people.

Unaware that this had been occurring for while, DBA manager walks into our cube area:

DBAMgr-Scott: "DBA-Kelly told me you still having problems connecting to the new staging servers?"
Dev-Carl: "Yea, still getting access denied. Same problem we've been having for a couple of weeks"
DBAMgr-Scott: "Damn it, I hate you. I got to have Kelly working with data warehouse project. I guess I've got to start working on fixing this problem."
Dev-Carl: "Ha ha..sorry. I've checked everything. Its definitely something on the sql server side."
DBAMgr-Scott: "I guess my day is shot. I've got to talk to the network admin, when I get back, lets put our heads together and figure this out."
<Scott leaves>
Me: "A permissions issue on staging? All my stuff is working fine and been working fine for a long while."
Dev-Carl: "Yea, there is nothing different about any of the other environments."
Me: "That doesn't sound right. What's the error?"
Dev-Carl: "Permissions"
Me: "No, the actual exception, never mind, I'll look it up in Splunk."
<in about 30 seconds, I find the actual exception, Win32Exception: Access is denied in OpenSqlFileStream, a little google-fu and .. >
Me: "Is the service using Windows authentication or SQL authentication?"
Dev-Carl: "SQL authentication."
Me: "Switch it to windows authentication"
<Dev-Carl changes authentication...service works like a charm>
Dev-Carl: "OMG, it worked! We've been working on this problem for almost two weeks and it only took you 30 seconds."
Me: "Now that it works, and the service had been working, what changed?"
Dev-Carl: "Oh..look at that, Dev-Jake changed the connection string two weeks ago. Weird. Thanks for your help."
<My brain is screaming "YOU NEVER THOUGHT TO LOOK FOR WHAT CHANGED!!!"
Me: "I'm happy I could help."

99% of our server-side code is Python and PHP (legacy applications).

Asked a junior dev to make a small update to a PHP site so we could have it run some cleanup server side. Plenty of existing PHP code to look at and piece something together. Should be 50 lines max.

Did he use the existing PHP code to do this task? Nope. Did he at least use Python? Nope.

Node.js

His response?

"I couldn't figure it out and Node.js seemed to have good support for mongo so I used that instead."

We have 0 lines of server side javascript. Never had node installed. Literally none of the devs use node here. Not only is this completely outside of our tech stack, but he had to take the time to learn Node and JS just because he thought it was easier.

Much would of rather he put in twice as much time to learn the tools of our stack.

developer: hey, want to hear a joke?

manager: sure

dev: what did the developer say to their manager after doing flaming shots in the server room and accidentally setting fire to all their systems?

manager: i don't know, what did they say?

dev: "hey, want to hear a joke?"

Im going to fucking murder the QA team if they don't stop sender bullshit issues!

QA: hey dev, there is an error with attached files.
Me: okay what's the issue?
QA: it's just a random file that gets attached. Can you fix it by the end of the sprint(tomorrow)?
Me: I need to investigate it a bit before I can tell you how long it will take, how can I reproduce it?
QA: idk, it was just there.
*several hours of testing later*
Me: I've tried to cause the 'issue' on my local server, the test server and the live server. But I haven't seen it and I have no clue what could cause it.
*30min. before I go home*
QA: dev you have to fix it before you go home! Because we have some other important issues you have to fix tomorrow!

FUCK YOU AND YOUR IMAGINARY ISSUES I'M GOING HOME

Making an infinite loop of cat /dev/urandom for anyone trying to login to server via ssh as an April Fool...
But on prod server instead of dev server...

Fellow dev complaining about server.

Me: My servers could handle the 10x traffic. If they weren't busy apologising for your shit codebase !

*First Rant btw*

Me (front dev): We should use nodejs for this project...
CTO (back): Nah... Fu*k Node! Php ftw
Now: we have a php framework with a nodejs server for the front end

Three days I'm not there, three bloody days and the 2nd sys admin has managed to:

Destroy the auto load balancing that I have scripted and implemented

Fuck up the backups to the cloud, and to a tape drive.

Overload a physical server (in pic, #1 and #2 are hypervisors). This is just one smaller site, I would hate to know what he has done elsewhere.

This is why I'm moving into dev.

Junior dev requests for sudo access on a server instance for some package installation, gets it, figures out how to open the root shell - never goes back. They do everything on root.

Fast forward to production deployment time, their application won't run without elevated privileges. Sysadmin asks why does the application require elevated privileges. Dev answers, "Because I set it up with root" :facepalm:

Sometimes, when I write scripts to scan random IP addresses for an unsecure VNC server or develop my own NES emulator and someone asks me "Whatcha doin'?", it's just easier to lie rather than start explaining, so I reply:
- Nothing... Just some web dev.

Assigned to a new project team..

Using git, in a creative way. So.. "master" is "dev" branch, usually. Everyone can push their branch to dev server .. so it's "dynamic for us". Production branch is whatever, as long as the branch has the release version. Sometimes, the release comes from "master".. that mean "dev" in normal geek..

That's just Git. The source code is a saturated spagetti of Entity framework and Caliburn. It is littered with antipatterns, especially basebean. Holy Christmas and Easter that baseclass do a lot of stuff that has no place as a base class ..

Fucking frameworks, I'm gonna start to evangelize frameworks as the no1 antipattern.

MS SQL as the main DB, but is dumped to json FILES through a scheduled task to increase read performance on web.

There is a soap endpoint to expose the json files, fml..

I am assuming I was placed here to improve stuff, I have never in my life seen anything like this before.

There is a special place in hell for this repository

I find it amusing that if you tell an SMTP server "quit", it responds "Bye" before closing the connection...

It's the little things in dev life...

Really just an average week.
Just feel I need a bit of venting. (:

@meet: (monday)
- mgr: we need video transcoding and VOD ASAP.
- dev: on what server? It's expensive, especially without a GPU.
- mgr: prod is beefy. Put it there.
- dev: everything else is gonna crawl then.
- mgr: you have till the end of this week.

@demo (Friday)
- dev: k, it's ready.
- mgr: Why is everything slow??!
- dev: transcoding. Expensive.
- mgr: Why do we transcode? Never said I wanted transcode!
Can't we upload to YT?
- dev: ...yes. But will then each customer that wants VOD will need to setup YT studio and provide an endpoint and stream key.
- mgr: OK. But we're now behind schedule because of this and the customers will not be pleased.
- dev: oh, didn't know we're into gaming.
- mgr: ???
- dev: nvm, see you Monday.

...
Later Friday evening
...
*ding* mgr has added 5 new tasks to your list.
*ding* mgr subtracted 30 points from you.
reason: deadline over due.

Ya ya, the usual shenanigans.
Time to mute for the weekend.

Greatest thing just happened.

Get a ticket about orders not being processed in our webshop. Angry customer. Critical!!!!

Starts troubleshooting. Nothing has changed in the code recently, was working just fine yesterday. Works locally and on test server. Hmmm...

Take a chance. Writes back to customer: “there! Try to place an order again” without changing anything.

5 minutes I get back “awesome! Everything works again and all previous orders have appeared. Good work!”.

Happy customer. Happy dev :)
Fin

The senior dev is mentoring our new recruit.
😨 I know, my face too.

When the newbie asked how to deploy, senior dev says, "Well, we copy and paste this folder from your local box to the server you need to deploy it on. Much better than that git shit, you have so much more control!"

😭

Recently I fucked up my laptop's rootfs USB stick again by tugging on it with some wire.. I think it got detached during runtime. Doesn't boot anymore.

So I attached it to my server to chroot into it and see what's wrong..
# cryptsetup luksOpen /dev/sdf2 cryptroot
> Unlocks without errors.
# btrfsck /dev/mapper/cryptroot
> Nothing wrong.
# mount /dev/mapper/cryptroot /mnt
> Mounts just fine.
# chroot /mnt (some other filesystems like /proc, /sys, and /dev were mounted first but meh)
> Enters chroot just fine.
# pacman -Syu
> Upgrades just fine.
# su condor
> Switches user just fine.
$ vim -p some files
> Enters the editor just fine.

Mounted it again to my laptop and try to boot, because it clearly seems like everything is just fine..
> Not gonna boot up. You can unlock your cryptroot and then I'll just fucking stall without saying shit.

MotherFFFFUUUUCCKKKEERRRRRRR!!!!!!! Fuck you HP for making such horrible USB connectors, and fuck you Arch for not giving something more verbose related to the issue, so that I can actually know what's wrong with you, and fucking FIX IT!!! Fucking pieces of junk! Do I really have to build my own PC and build my own LFS, just to have something halfway decent?!

devCraft {
Closing the minecraft server for a little while!
I'll be adding mods, writing up a perms file, and hosting the pack on git! I'll post a rant with the repo link.
One of our lovely ranters offered a VM to host the server on, so the ip is probably gonna change as well! (i also gotta make an arch bootable USB, and running the server would slow that down lol)
i'll notify you all once it's donevia a rant, like i said. until then, formulate plans, and suggest some developer-related mods for me to add in! (must be 1.7.10)
Currently planned mods are:
- ComputerCraft
- Applied Energistics
- Buildcraft
- Project Red
and a few from whatever you guys suggest. see you then!
}

21:30, sysadmin, chatting with my colleagues, when one posts a screenshot of a message he just received from a dev :
"Hello, sorry for bothering you this late but we have a demo tomorrow morning and the app is completely stalled. It fails with the message 'cannot write to <file>, no space left on device'"

I say "I bet that they somehow managed to make their DB grow uncontrollably".

Colleague asks which server hosts the app, Dev answers "one of ours", then adds after a few seconds "wait, do you need the IP of the server? Dev2 should be able to provide it", before finally adding "we use a scheme in <other project> DB actually".

Finally, Dev2 declares that the bug is solved: "There was a loop that caused a DB view to grow constantly and it filled the filesystem".

Me: "Called it".

They cleaned the view: 41GB freed.

After 'Dev' deployed a service using Azure ServiceBus, a particular queue/client was receiving errors.

Dev: "Looking at the logs, client is getting faulted."
Me: 'What is the error being logged?'
Dev: 'Client is faulted'
Me: 'No, that is our error when the client is either unable to connect or there is an exception in the middle of sending a message. What is the exception from Azure?'
Dev: 'Client is faulted. That's it. I'm going to have to re-engineer the code to implement a retry policy.'
<OK, I smell someone cooking up some solution finding, so I dig into the logs a little further>
Me: "Looks like an invalid connection string. The actual exception being thrown and logged is from the Azure client connection string builder. The value cannot be null."
Dev: "No, I'm looking right at the connection string in the config. Looks fine."
Me: "Looks correct on your machine, but what is actually being deployed to the server?"
<I could tell he was getting agitated>
<Dev clicks around, about 10 min. later>
Dev: "Aha!..I found it. The connection string in the config on the main branch is wrong, in fact, the entry is missing."
<dev fixes, re-deploys, life is good, I document the error and the root cause>
Boss: "Great job Dev."

*sigh* ..go teamwork?

Created a batch script to write some filenames to a text file using a loop.

Missed out the echo command, the script tries to open 100+ zip files on a production/potato server (I feel like prodtato should be a word).

Server cries and crashes
Dev cries and crashes

How do I know when I’m ready for a junior dev job? Currently I’m very familiar with syntax rules, data structure, I even created my first p2p app and I’m feeling more comfortable with server side programming and handling dependencies.

But I can’t like code blindfolded and I’m very slow and have to think very deeply and concentrate very hard before doing any minor thing.

How do I know??

Ended in an UPDATE without WHERE query to a core table in a Drupal project (in a dev database stored in the same server that production database was)

I really, really, fucking god damn it REALLY need to move a legacy project from the grave yard server and get it in git, and then build a dev environment for it, so I can stop making incredibly volatile changes direct to PROD (backend, frontend and DB all at once and then test it while it’s live and being used, but fuck me if I can be bothered digging through a 10GB code base and attempting to make it work in a multi-environment setup when it’s going to be a long trip down the error logs until it works again 😱🔫

Fuck spent like 2 days just to figure out dev build packages since they doesn’t allow nuget restore on dev server

Our dev server runs php 7.4

Our production servers run php 7.2

"Just link your ide to the prod server and dev like that"

Me: Hey sys admin guys (group chat), our servers are in a complete different ip range. (192.168.x.x)
Sys admin: acknowledged
Sys admin: *installs tools on our server*
Sys admin: hey dev, please open up your ports
Me: why?
Sys admin: our tools can't talk from your server
Me: ok, what ip range
Sys admin: (10.0.x.x)
Me: 😑😑😑

I think my server got hacked, yesterday I made a new server on scaleway for the sake of testing I made a user called dev, with password dev. Forgot to change password before I went to bed.

Logged in today to find that load is 5x.x and this (image) in my crontab

Note to self: You are a disgrace, who the hell uses 'dev' as password for ssh on port 22 -_-

Creative dev compliments? Wow, that shows what type of person I am. I only know creative dev insults. Okay let me give it a go!

Wow, I'm gonna star that on GitHub!

Good work! I'm gonna put a shortcut to that right here on my desktop (in the same tone as "I'm gonna hang that on the fridge")

You're so meticulous that the unit tests are to make sure the computer works correctly, not your code.

If you were a web server, you'd always return 200/OK.

Your parents must have compiled you with -Ofast

!dev

The school I went to didn't have PCs when I first joined (had some RISC OS machines instead). They got Windows 95 PCs eventually and networked them. I had no experience with networking before this, but had a PC at home. We all had mapped drives to resources on the server. The PCs were pretty locked down - no "Run" command etc.

Anyway, one day the head of IT came in to one of the lessons and asked me "how I did it".

What had I done? Well, clearly he had seen something I'd taught one of my friends. I wrote it down for him.

1. Right-click the desktop
2. New shortcut
3. \\nameofserver
4. OK

Such hax, being able to see the file shares on the server.

Shortly after this, all computer areas had signs saying "no shortcuts allowed"...

Me:
*spins up Vscode*
*spins up postman*
*opens terminal, spins up docker*
*spins up frontend dev server*
*spins up backend dev server*
*opens new chrome window with not less than 20 tabs*

My 4year old 8gb RAM Core-i3 PC (barely fighting for her life):

Manager: Explain “Kooburrnehteez” to me.

Dev: Well when a mommy server and a daddy server love each other very much…

Test server not working as expected - check with test team.
Test team experiencing the same problem - report to dev.
Dev team says the issue is not on their side - check with the service owners.
Service owner says it might be a problem with the request and gives me another point of contact - send an email with all the info.
POC says it doesn't have anything to do with him and copies "the right person".
Next person says to ask a different team.

Get reprimanded for asking too many teams.

Jeez idk, maybe if you stopped passing on the blame like a hot potato and actually helped me out I wouldn't need to.

One of the dumbest things I've done as a dev... I was a front-end/php guy that got hired into a asp.net company. My boss told me to take down a section of the website. I hopped on the server and started looking at files and saved a config file. 2 minutes later I had 3 devs in my office asking what I'd done that took down our entire network.

Thankfully my boss laughed it after they republished the site, but that was my intro into asp.net development.

Me, being a lowly junior dev, had the honor of being in a same group chat with a big corporation devOps team.
Finally ready to play with the big boys!!

*opens chat*
DevOps 1: "so we need to remove the CSS cache from our clients computers."
DevOps 2: "ok, well... just delete the server cache"

*watching in awe as they all try to figure out why it's not working*

This continued on for a while...
Until my boss had enough laughs and giggles and put an end to this stupidity :D

I have worked with a handful of very green devs in the last 10 years. A common theme has emerged.

They don't heed any of my advice.

An exercise to the reader:

If you have a Windows machine, but need to work in a Linux environment, what would be your first instinct how to proceed?

In this exercise, you are as green as it gets. You have very little professional development experience, let alone server admin experience. And your lead dev has suggested setting up a VM.

1. Set up a Linux VM
2. Use a live CD or set up a dual boot system
3. Pay for a cloud server and set it up from scratch

I have no idea how this person intends to get any work done on a remote, terminal only, Linux server. That is if I can even get their environment into a sane configuration.

When you name your function list and it works on your local server but causes syntax error on the dev server

Those darn luddites!

!Rant

My boss just gave me a task "deploy this project today".
1. Made by a junior dev, and I have to take responsibility to upload it in this short time (no blame to jun dev, she's new and need to practice)
2. In a crm that I personally never used before, I need to study it and adapt at least the paths
3. Task was given at 13.48 during lunch pause.
But
But
The best is yet to come
Wait

.
.
.
.
.
.
.
.
.
.
Ready?
4. He doesn't know the server where to deploy it!
No one knows.
The IT doesn't answer, but still, I have to deploy a project TODAY (2h and 47 minutes left eob).

Has anyone had to hack into a server so they can set up their work,

Literally have a client asking me to do this change to their site and they are asking the dev who is controlling it but they just not letting me access ..

Long night ahead 🙄 got to add a ftp account ... All I need to do is add small lines of code for tracking, but this guy doesn't want to let me on it cause I'm slowly taking over his work... if he did his job right it's not like it would be happening anyway

Fucking shit, this university's website is so damn slow! Basically Every Semester, every student need to enroll to certain classes in University Website.

But the Infrastructure is not enough to handle such a big amount of students, we have approx. 7000 students enrolling at the same fucking time.

And here i am can't enroll to any class at all this semester. Fuck such a waste of time. This always been a thing since they digitalize enrolling system.

I don't want this to happen again. The student always be a victim since they cannot handle the request. Now, as a dev, i want to propose something better to optimize the server, i have some connection to pass some bureaucracy. I am going to do some brainstorming and I will need some solution.

Here some data i gather when i am mad from my univ infrastructure division :
1. The Server is a simple Local Server Forwarded to the Internet.
2. The Server use Windows Server 2007.
4. Web Server Using Microsoft IIS
3. The Website built using ASP.NET
4. The connection is not SSL encrypted (yes its fucking use the http)
5. Hardware Spec (not confirmed officialy, i got this information from my professor) :
- Core i5 4460
- 4 GB Ram
- 1 Gbit NIC

I will summon some expert here and i hope want to help me(us all) out.

I just love how liferay keeps finding ways to surprise me...

Customer: need to fix this security issue
me: ok
me: fixed. Testing locally. Works 100%.
Me: testing on dev server. Works 100%
qa: testing on dev server. Works 100%
me: all good. Deploying to preprod
customer: it doesn't work
me: testing preprod - it doesn't work.
Me: scp whole app to local machine. App works 100%.
Me: preview loaded liferay properties in preprod via liferay adm panel. All props loaded ok.
Me: attach jdb to preprod's liferay to see what props are loaded. Only defaults are used [custom props not loaded according to jdb]

me: is there some quantum mechanics involved..? Liferay managed to both load and not to load props at the same time and the state only changes as it's observed...

True story
Was free at work so applied for a stretch project.
PM: you know mobile dev?
Me: yes, built a few apps
PM: good, we have an old app we want you to take the JAR change everything but the look of it and make it a new hybrid app with the required features.
Me: *kill me please*
PM: and use WordPress as the server.
Me: ...........

Accepted the challenge. Did the entire app in ionic and build a server for it with SpringBoot. Client loves it PM is still doesn't know😂

Just saw a role advertised for a front end developer. Skills required amongst other things·
· Integrating with middle-tier microservices such as NodeJS

· .NET Core (2.1+), C# 7.0+ and JAVA

· SQL Server, T-SQL, MySQL

· Azure Dev-ops

There are other standard and expected front end requirements but want someone with 4+ years experience

Salary £19,000 - less than two thirds of the national average salary for non UK folks.

Applications: 0

Hmm I wonder why

Him: Why choose 200 over 500, minimalist?
Me: Why choose an internal server error over a successful request? Less is more.

Today I spent 25 minutes explaining to this dev that there’s no way that a http request can access directly to an ftp server and retrieve a file.

A dev with 4 years of experience...

I got an interview with a big multinational software company as a senior dev - the kind of place I never thought I would be privileged or knowledgeable enough to work for and wasnt expecting to get In to...

I aced it. They gave me an offer but - FOR DEVOPS 😬

basically my skills fit in perfectly with the server/ scaling issues they have and are far more valuable there. I know they do, I also know I can fix the issue and will have alot of fun coding it - I just dont think I want to monitor it or anything else.

I mean I do devops stuff all the time in aid of anything I code but their stack is a full time job- im scared that once the toolchain is automated ill be pulled towards sys admin like duties and lose touch with my craft... what do you guys think? Anyone shifted from dev to devops?

I could not do anything else. Physics, Chemistry, Biology didn't interest me.

Tried competitve programming, failed miserably.
Tried app dev and server dev, they surprise me with their awesomeness everyday :D

So I've got a few stories, but the first one goes like this:

>> Spring semester last year
>> Likely have the best grade in a front end web dev class
>> Has a file on a server that we used for this class.
>> Copies HTML file to desktop from server
>> Opens desktop file in VS (or VSCode, I forgot lol)
>> Opens server HTML file in chrome
>> Edits the file
>> Browser doesn't change.
>> "What to fuck?"
>> Clears cache
>> Doesn't change anything
>> Closes and reopens Chrome
>> Still no change
>> 'Yo what the fuck???"
>> Calls professor over to get help
>> Explains problem
>> Instantly realize I'm a dip shit and open the desktop HTML file in Chrome and see the changes I made

I salute all server admins here. I might never understand how you guys get through with all those terminals and debugging and greb and runlevel and all these weird things.

I spent two weeks trying to set up a dev server on CentOS installed on a VM. Just configuring the server took hours of trying to figure out what goes where and in the end I realized that the only thing I did wrong was the runlevel! Which I found out today is actually a thing!!!!

I thank you all for existing. Without you, us web developers would go crazy!

A lot of PM-rants (for good reason) but how about a reversed one:

Lead Dev: "Yes, but do we really need a dev and staging server? Can't we just try it out and see if it works?"

Dev: The server is completely down right now. Nobody can access the application, we need to divert some resources to horizontally scaling our app.

Manager: Hm, this was not in the schedule. I need to consult senior leadership on what to do about this. I don’t want to be held accountable for making a decision on this complex and highly nuanced situation.

Dev: No need, I have a solution. Just need a week to build/test/deploy the ability to horizontally scale.

Manager: But that will cause delays to new features.

Dev: New features don’t matter if the app can’t even load.

Manager: Ok you can implement your solution but it can’t take any time. I need those new features out.

Dev: ???????

Conversation between a developer and admin team to change configuration in higher environment.
Dev: Please change these config parameters.
Admin: What is environment url
Dev: 😑. Gives the url
Admin: How to change?
Dev: 😑. Gives all the details
Admin: I am unable to perform as server is slow.
Dev: Whom will you report the slowness of server as you are the admin?😵
Admin shocked developer rocked

remember when our whole dev team was drunk but we went back to work anyway? turns out, someone rm -rf a whole folder of js scripts!! it took us some time before we could find the back up files which were in the old server (that crashed) because recently we changed servers (so we were transferring files) and we are now using amazon.

our boss got so pissed this morning but oh well, it was his idea to drink and feast in the first place!

...He hired a shit dev who did the same work in 3 times less than what I asked for.

He's now back crying to fix his Fuck up.

You ask how I know he is shit. He SSH-ed into the server. Worked directly off the production files. Worst of all, he installed phpmyadmin, changed the db structure without even writing a fucking migration !!!

How the hell am I supposed to know what he changed!! It's gonna be a long night 😥

I've been thinking about how to answer this for a while, but I'll approach it from a different angle. The time I (nearly) lost faith in my dev future wasn't because of a technology, bad programming language or an external influence. It was *me*.

The first job I had after the PhD, I was (in the first couple of weeks) tasked with updating various packages on a live Redhat server. "No problem", I thought, "I've done this before many a time on Debian, easy as pie!"

Long story short, I ended up practically bricking the server because I mistyped and uninstalled something I shouldn't have, didn't understand a piece of configuration, then tried to bodge it back and cocked things up further. Couldn't even log in via SSH, the hosting company had to be called, a serial connection set up, etc.

To say I was mortified, embarrassed and had my pride dented would be a massive understatement. I seriously thought I'd get fired on the spot, and that I should perhaps change careers to something where I couldn't cock things up as much.

...but you can't think like that, otherwise the world leaves you behind. So I picked myself up, apologised profusely, took some relevant training, double checked everything I was doing on that server in future and got back to work. After a few months of "proving myself", it was then seen as nothing more than a rather amusing story, and I became a senior dev there a couple of years later.

I really felt like a badass one time when I managed to recover all projects on our dev server after a full meltdown of the HDD.

We had no recent backups, because our backup server was down for a few months, and our (at the time small) company was in a tight spot on finances, and couldn't get a replacement.

The problem was that the HDD on the backup server failed, but we were storing all projects also on the dev server, along with our local git repos (no GitHub at the time for us), but then the dev server HDD also broke, and I used every piece of data recovery software I found trying to recover the data, until one actually managed to read the raw data from the HDD and store it as a virtual drive, that I then used to try and build another partition index and it actually worked!

Lost about 10% of the data, but that was enough, as i managed to recover all the git repos and databases...

I don't even remember the tools that got the job done in the end, but that was one hell of a week, and at the end I felt like a true IT God!

True story!

PS: 2 weeks later we had a new backup server, another offsite backup solution and a GitHub account for the company. Was delayed on salary in order to manage it (me and the CEO both agreed to give our pay for one month to get them), but worth it!

PM: oh hey you guys are back early, did you figure out that bug yet?
Dev: server’s haunted
PM: …what?
Dev: *loading pistol and stepping back into server room* server’s haunted

In one of my teams there was this non-IT girl.
One morning, she asks out loud:
G - Can I run a Json?
Me - Wait! What are you trying to do?
G - I need to deploy my changes into the Dev server.

Suddently I realized what she meant.
Me- It's Jenkins! Not Json. :D

I work on a warehouse dev team. One day this past year, I was trying to deploy a new build to a QA server. Earlier that day I had been looking at the logs on the production server and had left the ssh session open. I had been working for less than a year out of college at this point and shouldn't have had access to deploy to the production server.

Long story short I deployed my QA build to the production server and saw there were problems connection to our production database. Then my heart dropped in my chest as I realized I had just brought down our production server.

I managed to get the server back up by rolling back in about 5 minutes and no one ever knew except some people on my team.

I felt horrible for the longest time. Later in the year another guy that joined my team that has about 20 years of experience under his belt did the exact same thing, but needed help rolling it back. Needless to say, that made me feel a lot better. 😂

Definitely the worst moment of my year.

I'm a web dev who decided to take a shot at mobile development (My first mobile app mind you). I'm writing a mobile app and one of it's features is communicating to my server via websockets.

So I write the code, click to send the data and my server doesn't receive it. Fuck. I check why. I log everything. Nothing.
I spend several of hours and I'm exhausted by this point so i call one of our mobile developers to help me. Turns out my emulator didn't have a WiFi module. FUCK.

Alright so I compile it to an apk and install it on my phone. I popped open a terminal and started my local node server.

I click on the mobile app...

NOTHING LOGS. FUUUUUUCK.

And this is the best part.
Apparently I deleted the console.dir call from my server that executed when it received some data from an emitter.

I only thought of this last night at 2am so I got up and checked. Yep. Kill me.

Our smart and very professional sales guy strikes again,

I had to do some research on if I could print a pdf file directly from the server (be it php / nodejs)

When I told him I had found a solution he said, good job and went away, I was like...hmm k..

A few days later he came to my senior being mad that the project wasn't done in time.. And we were like.. Dude... What project!?

Apparently he made a deal to have a working demo in two weeks, but we (our dev team) never got that message...

So this one day I'm at work and the manager peaks into my office and just says "can you check that platform X is building, pretty much done just a basic bug check" (this bloke had negative 1 technical experience)

I'm not sure what he means, the whole thing is built in Java and I know nothing about that...

I log in the platform on dev server, sure enough it seems to work, charts are drawn, no errors, then I try to log out; this button does nothing...

I don't bother telling the manager, I just go to the dev who's a friend of mine and tell him about it.

A week later, manager jumps in the room all excited "we're launching this product tomorrow, mind checking again?"

Sure, I log in, ... There's no log out button, it's gone... I ask the dev.

"Yeah I fixed it, it's gone now!"

My productivity has gradually increased untill now by using:

Linux-server+VSCode (+Git+Terminal)+tmux+tmux-resurrect

Any further suggestion on dev tooling setup would be appreciated.

I primarily work on DevOps projects - bash scripts, linux server apps, containers, kubernetes,

So I was changing some CSS, but the changes weren't showing.

Was it being cached? Nope
Was the selector wrong? Nope
Well it was the right file yeah? Yup

So after like 10min of scratching my head, restarting the server, etc it turns out I was checking prod instead of dev.

This isn't even the first time this has happened 😑

Guys just remember to keep your dev tab and your prod tab away from each other, like way away.

Well paid java dev. But the HW/SW-Stack is awful.

Monitor: single 1600x1024
5yr old notebook, old i5, magnetic hdd
Forced to use windows 7
No maven server
No CI server
SVN but no git
Eclipse, no intelliJ
No sonar server

There are days where I just can't take it anymore.

Was hired as a mobile dev / mobile web dev, was told my main project would be integrating a tool with phonegap.

... 13 months later I was never asked to download phonegap and was now writing server side java. Surprisingly I left that job

I'm fucking frustrated.

Almost Every project, almost every task I did in the past 6 months has been a failure or partly done. Even the most trivial of tasks take me hours to complete, after immense googling and copypasting.

I know that I'm a junior with less than a year of dev experience but it feels I'm traversing through hell itself. I truly love to program, have tremendous passion and want to be a professional dev but it seems destiny itself wants me to keep doing what I do best but hate(Sysadmining).

When will this nightmare end? When will I be able to accomplish anything I need with code with so much ease, like my dev friends do? How many more courses, bootcamps should I fucking attend and how many more tutorials to watch? When will be able to work at nights without falling asleep? When will I have a fucking dev job and freelance projects instead of being a goddamn server-managing monkey?

FUCKING MICROSOFT IIS SHIT.
I'm a .NET dev since 13 years and EVERY FUCKING TIME STUPID IIS MOTHERFUCKER AND STUPID WINDOWS SERVER have a different problem setting up because of some permission.
You can't never get a site up in IIS without loosing time and patience having weird 400/500.x errors because every fucking machine have to set up some tweaky and hidden permissions.
I have 2 identical fucking win servers and deploying a .NET core applications and on one works (test server) and obviously, on the production server it gives troubles.

FUCK YOU MICROSOFT FUCK YOU I would take the IIS devs personally here and whip them to death until they don't resolve the fucking thing

fuck this!
spent an hour trying get my website working (on a raspberry) ... no errors, dev tools gave nothing, php gives nothing mySQL related... weird.. debugged my code for an hour when it me... db on my pc for testing is not the same one as the "production" server. i am so fucking stupid... i need some sleep

If an http request can't perform the requested operation, should server send 500 error code? Or 200 with status and status message in response?

Isn't 500 used only for unhandled exceptions on server side?

TL;DR Dear boss, firstly, you always get someone to review anything important done by a fucking intern.
Secondly, you do not give access to your fucking client's production server to an intern.
Thirdly, you don't ask your fucking intern to test the intern's work that has not been reviewed by anyone directly on your client's fucking production server.

Last week, the boss and one of the lead devs (the only guy with some serious knowledge about systems and networking) decided to give me (an intern who barely has any work experience) the task of fixing or finding an alternate solution to allowing their support team access to their client machines. Currently they used a reverse SSH tunnel and an intermediary VH but for some reason, that was very unreliable in terms of availability. I suggested using OpenVPN and explained how it would work. Seemed to be a far better idea and they accepted. After several days of working through documentations and guides and everything, I figured out how OpenVPN works and managed to deploy a TEST server and successfully test remote access using two VMs. On seeing my tests, the boss told me that he wanted to test it on the client network. I agreed. Today he comes to me and he tells me to prepare testing for tomorrow and that the client technician is going to give me access to one of their boxes. And then he adds, "It's a working prod server. We'll see if we can make it work on that" and left. I gaped at him for a while and asked another dev guy in the room if what I heard was right. He confirmed. Turns out, the lead dev and the boss's son (who also works here) had had a huge argument since morning on the same issue and finally the dev guy had washed it off his hands and declared that if anything goes wrong from testing it on production, it's entirely the boss's own fault. That's when the boss stepped in and approached me. I ran back to his office and began to explain why prod servers don't top the list of things you can fuck around with. But he simply silenced me saying, "What can go wrong?" and added, "You shouldn't stay still. You should keep moving". Okay, like firstly what the fuck and secondly, what the fuck?.
Even though OpenVPN client is not the scariest thing to install, tomorrow's going to be fun.

Hiring a third party to help us with something...
Third party: yeah okay, we know what we need. Can we get access to your git repo
Me: sure, I'll make sure you'll get it
(To the admins): hey can you get them access to our git server?
Admins: did they sign the personal data processing contract?
Me: oh they won't work with any personal data. It's a dev server and they only need access to the source code. And the usual contracts and NDAs are already done
Admins: well we still need the other one.

... Sure. Why not. Just delays the start of the process for... Like a week and a half until that useless bit of paper has passed through all the necessary departments. Not like time's an issue. Right?

>Gets assigned to this private Game server's project
>Boss wants me to improve the anti advertisement chat
> k
> Looks at old code
> Code is replacing unicode characters to latin ones that look similar which are being used for advertising
> lol'd who tf developed this
> regex101, building a regex query with endless of possibilities (would look something like this) /((L|\|_|I_)(O|0|\[\]|\(\))(L|\|_|_))/gi to detect lol
> Adds alot of similar looking unicode characters to assure that it will find something
> Works really well in the dev version
> Server open hour
> 30 players
> All chat at the same time
> CPU 100%
> BOSS NEVER TOLD ME TO MAKE IT EFFICIENT

Thought I was on a Dev SQL server. Dropped a primary user. Turns out I was on prod.Oops..

Follow-up on yesterday's rant.

Boss hired dev-ops team to restore data from the broken server image. They said it may be completely impossible. Which will probably make clients want to kill us considering it's an government agency.

I wonder how large the contractual penalties will be considering they are almost completely incapable of working without the system and it's going to be few days until anything will be fixed.

Oh well, since no one bothers me it seems it's currently not my problem and I'm free to code ¯\_(ツ)_/¯

This is the last time Microsoft! I'm getting my old Arch image out and removing you from my life forever! Never again will my linux distro randomly uninstall itself without telling me in the middle of implementing new components and crash my development server. Never again will I have to deal with an update that refuses to STFU and go away until I, ME NOT YOU MICROSOFT, decides it's a good time to run the update. No more lack of customization and poor support of common dev tools. I'M DONE WITH YOU, WE NEED TO SEE OTHER PEOPLE.

My first gig straight out of uni was on a project where my role required fixing an FTP client to upload files from a clients machine to our server. It was a spreadsheet. VBA. An excel spreadsheet. FTP client. A spreadsheet.

Me: "why are we using VBA for this?"
PM (was a dev 10+ years ago and started the basics of this 'ftp client'): "VBA is great." *Stern face*

Brilliant

Current workload as dev lead:
- 1% actual development
- 2,5% waiting for SaaS to load
- 2,5% cursing company server network connectivity issues
- 5% switching VPNs
- 7,5% pkg management & deploys
- 10% writing JIRA and support tickets
- 12,5% filling in timesheets
- 15% coaching & reviewing a bot coworker
- 19% doing 2FA, refreshing expired passwords
- give up and spend the remaining 25% doing something meaningful

Just got into office and saw a group email about a recently released feature.

Apparently, it creates a lot of connections which it never closes. This could hang the server since it can only handle a large but finite amount of active connections.

Well the dev said: I will optimize later

My thoughts: later = min(never, production blows up)

In a meeting, we made some pretty major adjustments to a dev server. We broke it. Under pressure, I cracked and I randomly forgot the syntax for a sql update statement that might’ve at least got it back semi-online. To be fair, it’s been several years since I’ve needed any sql. Thank goodness I grabbed a snapshot as soon as I realized things were about to get edited. Saved. 😁

Develop an application on Windows that will be run on a Linux Server. Strange bugs occur on server but not in dev environment.

I remember the first time working with a Windows 2012 R2 server.
Of some reason I couldn't open Windows Powershell as admin and asked another Dev what to do. His response? Try restarting the server.
Can you imagine the look on my face if I told you I only worked on Linux until that point. How can you seriously make a server OS that needs restarting at all? Is it just me or is that just fucking stupid?

I was a c# - sql server dev in every other project I was in.. now in this new project I'm doing only sql server, checking on procedures and data, to calculate risk indicators for financial instruments. I'm away from home 5/7 days a week 😭 I miss C# . I'm so sad 😭 😭

Next weeks rant theme should be worst dev day.
Any how today I fucked up at a whole new level. First ran a script thinking I am deleting my local dev environment.

*An Eternity later*

Fuuuuuuuuuuuuuuckkkkkkk

That dreaded script ran on main dev server and fucked up the server used by a team of 15+ teammates. Dead.

At work, all errors within the site are logged into our database with a subject and error column. SQL errors are logged in the subject field while the traceback is put in the error column. However, a lot of SQL errors are really large and exceed the max character width of the subject field, causing yet another SQL error, and the cycle repeats. This recursive error has been the bane of my existence, because 1) it times my local dev instance out and 2) the error doesn't end up getting logged because the server both freezes and the error can't be inserted in the database. You can't even begin to imagine how many hours I've wasted trying to find what line I changed cause total and utter failure with absolutely 0 error logging. Next thing on my todo list is to fix this fucking issue since the head dev refuses to get it done.

<opinion>
You may be a prod ninja but I believe that every dev should have a decent level of exposure with a low level language(s). Sure you can make an HTTP server, do a sentimental analysis, topic modeling, set up multinode clusters, write ORM queries from dbs and all sorts of awesome stuffs with Python/Ruby/PHP/JS/GO etc but none of them teaches you what happens at kernel level. Things like memory leaks, threading, multiprocessing, memory allocations etc can only be better learnt from a low level language.
</opinion>

P.S. Not a C/C++ fanboy. I'm a python dev 😄

Just spent 8 hours debugging an issue that had been sitting in my issue queue for a week. WooCommerce was claiming transactions were failing, but going through in the background anyway.

Can’t share any more specifics here, but the gist of it is that the server the code was developed on was set to PHP version 7.1, while the server the dev site was on was set to version 5.6 (which I didn’t even think was still installed, never mind the default...).

So yeah, fun times over a trivial fix.

What the best database solution for web and smartphones dev?
Is mysql the good choice?
I’m an “old” dev with old usage, php-mysql-JavaScript.
Is it a 2018 solution or am i a dinosaur?
All data will be stored on server side. Web and smartphone app as client.
Thanks for your experience sharing.

A few days ago our server was compromised due to an outdated Jenkins version. The malicious user installed a crypto miner on the server... The same day that it was found I told management that I'm interested in helping out with the server. Since then, nothing happened... No updates, no security measures, no nothing (except for the removed crypto miner and updated Jenkins software)

Oh well only a matter of time before another hack...

Question to some (who work way way way longer than me) med - seniors, should I make a big deal out of this? And keep pressure on it. Or should I just leave it be and wait for the next comprised server? I know devrant is not a Q&A service, but some dev to dev advice is much appreciated.

- incognito

Not a fight I was involved in but one I observed. A junior dev on my team and a server ops guy had major personality conflicts. One day the server ops guy had enough and physically went after the junior dev. I was senior but still pretty new to my own career and had no idea how to handle it. Server guy got fired soon after. I was glad I didn’t have firing power and that he didn’t even report to me anyways.

Some """friend""" of mine asked me help because he has trouble installing WAMP on a Linux distro.

Me: why don't you try to install a LAMP server instead, you are on Linux, you can do whatever you want.

Him: LAMP is not a thing, this doesn't exist, there's no package called like that.

Me: *sends a tutorial to setup a web dev server*

Him: You are so unhelpful, why are you disrespectful like that ?!?

Me: *having some murder thoughts*

Why do I still reply to these people ?

TL;DR: When picking vendors to outsource work to, vet them really well.

Backstory:
Got a large redesign project that involves rebuilding a website's main navigation (accessibility reasons).
Project is too big just for our dev team to handle with our workload so we got to bring a 3rd party vendor to help us. We do this often so no big deal.
But, this time the twist was Senior Management already had retained hours with a dev shop so they want us to use them for project. Okay...

It begins:
Have our scope / discovery meeting about the changes and our expected DevOps workflow.

Devs work Local and push changes to our Github, that kicks off the build and we test on Dev, then it goes to Staging for more testing & PM review. Once ready we can push to prod, or whenever needed. All is agreed, everyone was happy.

Emailed the vendors' project manager to ask for their devs Github accounts so we can add them to the project. Got no reply for 3 days.

4th day, I get back "Who sets up the Github accounts?"

fuck me. they've never used Github before but in our scope meeting 4 days ago you said Github was fine...??

Whatever, fuck it. I'll make the accounts and add them.

Added 4 devs to the repo and setup new branch. 40min later get an email that they can't setup dev environment now, the dev doesn't know how to setup our CMS locally, "not working for some reason."

So, they ask for permission to develop on our STAGING server.. "because it's already setup"... they want to actively dev on our staging where we get PM/Senior Management approvals?

We have dev, staging, production instances and you want to dev in staging, not dev?... nay nay good sir.

This is whom senior management wants us to use, already paid for via retainer no less. They are a major dev shop and they're useless...

😢😭

Cant wait for today's progress checkup meeting. 😐😐

/rant

This Christmas I transitioned into a new job. At the old job I was the only kubernetes-guy, so since they no longer have any developers who are confident working with that, they decided to go with LAMP-stack.

The data from dev-kubernetes-server was backed up by some guy and moved to an offsite-server, or so they told me. Turns out, he had backed up the kubectl-config-file, and not the databases. Now everything is wiped. Sure glad we still have that config-file!

Of course, since that was only our dev-server, there was nothing too important there, except for all the documentation. The only other backup? On my laptop, which I turned in to them, and is now wiped and used by one of the sales-guys.

Now I’m being called in at least twice a day, since I was their goto-guy for almost anything backend-related. Feels great, after they spent a couple of months attempting to rewrite everything in pure PHP (with a strict no-dependency policy for some reason). Fml.

My biggest challenge has been moving away from an unmaintainable Java/Tomcat/Spring Security application server to a Node.js/Express application server. That handles single sign on and two factor authentication. In 2 weeks.

I'm a front end dev. I'm sure it's fine 😓

One thing I truly fucking dislike about the development life is knowing about server administration. I think that the mental hurdle that is to develop a huge application, make a stable dev environment, learn all the tools, tricks, techniques, modern standards, processes whatever, detailing software engineering are way tf too much to also handle server admin shit.

We don't have anyone at work that deals with that, and as such my devs need to know how to do entire series of maintenance shit that just takes time and effort plus hours of notetaking and study. I mean I get it, they should know their way around a linux environment enough to troubleshoot issues that are related to the os when working with some tools, but fuuuuuuuck me man, setting up a server, even for the holy grail of easy (standard lamp stack) takes way tf too much.

Wish we could have a dedicated server admin in the team.

I know where my faults are, setting up servers is something that I know but just can't be assed with in terms of keeping up, I wish we had a devops dedicated server admin deployment guru cuz I really cannot stand losing hours doing this shit.

It also diminishes good s admins in value, "weLl ThE deVs caN do It" YEAH BITCH but wouldn't it be nice to have an expert concentrating on JUST THAT?

FUCK man

Dev, boss and guy who know logic is looking at the server.
Problem: it's not responding
Boss: we need this running now! Otherwise the sales won't go through
Dev: give me a chance, I just got here
Guy: have you tried turning it off and on again?
They did so and at works.
Boss: guess we don't need to hire another dev, this guy knows what he is talking about, he is some kind of server expert..
Really.........

i wrote a website, a server in go, a small os in c, a game in js, a game and server and web scraper and other desktop apps in java, mobile apps with flutter, a website with php also, implemented aes in go, wrote a parser in java. done sysadmin stuff on my vps and pihole/openvpn/nextcloud on my rpi. learn about c vulnerabilities and used metasploit. attempted to write an interpreted language. did some led displays with arduino. currently learning tensorflow.

i have never...
- written a driver
- made a game with a game engine
- created a file encoding
- implemented an oauth2 server
- made an api
- worked with vr

what am i missing? i want to be a very well rounded dev.

Very tempted to buy a Web server from ebay and set it up as a home dev server.

when a dev fixes a memory leek issue but rebooting the server, and when ask why the production application crashes he casualty replies "I don't know but I restarted the server and its fine now..."

every day I see full stack here and there...
full stack is not only db and code, but also "every step the bit goes through " from end user's screen/input to server and back to him
whether is an app or service, end user is only an example.

it's about knowing how the language behaves, how the server interprets and replies to requests, protocols, even how to do every single configuration on the systems you are using, and in my point of view that includes hardware.

pretty much that...

I get sic when I see on a resume claiming "I'm a full stack dev" and there's nothing on it saying that the guy knows at least to change a light bulb... lol
Even worse, when I see job offers asking for "Full stack Dev, with no experience" ...

that's not possible without experience ! sorry

Dev: Please let us know when will be a good time to make a DNS switch for launching the site?
Client: ok
...
....
Client: one hour later, hey I did a DNS switch to your server but the site is not there?!
Dev: (Fuck me running!!) fuck fuck fuck fuck and fuck my life!

*Dev is non-native english speaker

Dev: we need the VPN ip.
Me: the server ip or the connected device ip.
Dev: the server.
Me: gets the ip.
Dev: this doesn't work, is this the VPN ip ?
Me: Gives the device ip. Works.
Dev: OK. Works now.

Could have just asked for the client IP in the first place but s/he didn't know how to.

I have been trying to freelance for people who don't speak english as a first language and getting the Requirements is the hardest part of the job. 😫 .

P.S. Suggestions needed from remote freelancers. What's your workflow like.

How would you feel if you can't solve your code? I'm almost a week stuck on a server problem. I applied for frontend dev position, it's okay for me to do this kind of stuff but the fact that I can't solve it for so long it makes me feel down :(

I have reinstalled node js 3 times to run JS scripts and "npm run dev" is not starting up the server. This is quite frustrating.

!Dev

I soon will start working from home. meaning I won't have to waste 4 hours a day busing back and forth from the office.

I think with that free time I might start trying to organize my life, I have hundreds of 'read later' and 'useful' bookmarks across 3 computers, too many browsers, and countless bookmark folders. I also have notes in 7 channels on a private discord server.

I think with the free time I might actually clean up that massive mess and using my server I didn't have last time I worked from home, actually figure out a method that'll help keep it from getting this bad again.

Finally made my node production server stable enough that I could focus on writing tests*. I start by setting up docker, mocking cognito, preparing the database and everything. Reading up on Node test suites and following a short tut to set up my first unit test. Didn't go smoothly, but it's local and there are no deadlines so who cares. 4 days later, first assert.equal(1+1, 2) passes and I'm happy.

I start writing all sorts of tests, installing everything required into "devDependancies," and getting the joy of having some tests pass on first try with all asserts set up, feels good!

I decide to make a small update to production, so I add a test, run and see it fail, implement the feature, re-run and, it passes!

I push the feature to develop, test it, and it works as intended. Merge that to master and subsequently to one of my ec2 production servers**, and lo and behold, production server is on a bootloop claiming it "Cannot find module `graphql`". But how? I didn't change any production dependencies, and my package lock json is committed so wth?

I google the issue, but can't find anything relevant. The only thing that I could guess was that some dependencies (including graphql) were referenced*** in both, prod and dev, and were omitted when installed on a prod NODE_ENV, but googling that specific issue yielded no results, and I would have thought npm would be clever enough to see that and would always install those dependencies (spoiler: it didn't for me).

With reduced production capacity (having one server down) I decided to npm uninstall all dev dependencies anyway and see what happens. Aaaaand it works.....

So now I have a working production server, but broken local tests, and I'm not sure why npm is behaving like this...

* Yes I see the irony.
** No staging because $$$, also this is a personal project.
*** I am not directly referencing the same thing twice, it's probably a subdependency somewhere.

Start mongo, start my node server, start the app, quick "5mins" look on dev rant started 30mins ago!

As a junior dev accidentally running rm -rf / on a production server

Story time;

Major project, multi million budget, huge business and IT coordination, board level status updates, meeting started back in March 2018 for a Go Live of Aug 2019.

Based on draft requirements (and experience) I request the test environment be built for half of the work. Turns out that no one told Server Eng and they are out of space in both dev and prod until Q2 of 2019. We went from Green to Red because a Service Request.

Second night of my life wasted trying to get fucking Webpack to work. I just want a goddamn working dev server. I just want it to auto-refresh my browser, and yet fuck all happens if I use the "--hot" option for "webpack serve" in my npm start script (as described in the extremely terse and overwhelming documentation). How do the people from create-react-app configure this cancer? Jesus fucking Christ, this is all so overwhelming and extremely confusing, maybe I should quit web development altogether.

Oh my damn god,
I just found the remote-server plugin for vsCode and holy fuck balls, it's the one thing I've wanted from vsCode since ever.

When you dev in VMs and are forever working with remote code using SFTP and git on a remote servers, it's a pain in the ass.
But this, god damn this solves all of that, and with connection specific plugins, I think I just came.

when the dev server is so locked down you can't even recycle the app pools

I upgraded a Linux server one time and data that was serialized in yaml stopped being parsed properly.

It turns out the libyaml people decided to change how hashes were handled, which made any previous hashes come back as blank.

A whole database of valid data in dev was coming back invalid in prod. It was maddening.

It took a day to figure out the problem and how to update the data to the new format in rails.

I now serialize in json.