My team handles infrastructure deployment and automation in the cloud for our company, so we don't exactly develop applications ourselves, but we're responsible for building deployment pipelines, provisioning cloud resources, automating their deployments, etc.

I've ranted about this before, but it fits the weekly rant so I'll do it again.

Someone deployed an autoscaling application into our production AWS account, but they set the maximum instance count to 300. The account limit was less than that. So, of course, their application gets stuck and starts scaling out infinitely. Two hundred new servers spun up in an hour before hitting the limit and then throwing errors all over the place. They send me a ticket and I login to AWS to investigate. Not only have they broken their own application, but they've also made it impossible to deploy anything else into prod. Every other autoscaling group is now unable to scale out at all. We had to submit an emergency limit increase request to AWS, spent thousands of dollars on those stupidly-large instances, and yelled at the dev team responsible. Two weeks later, THEY INCREASED THE MAX COUNT TO 500 AND IT HAPPENED AGAIN!

And the whole thing happened because a database filled up the hard drive, so it would spin up a new server, whose hard drive would be full already and thus spin up a new server, and so on into infinity.

Thats probably the only WTF moment that resulted in me actually saying "WTF?!" out loud to the person responsible, but I've had others. One dev team had their code logging to a location they couldn't access, so we got daily requests for two weeks to download and email log files to them. Another dev team refused to believe their server was crashing due to their bad code even after we showed them the logs that demonstrated their application had a massive memory leak. Another team arbitrarily decided that they were going to deploy their code at 4 AM on a Saturday and they wanted a member of my team to be available in case something went wrong. We aren't 24/7 support. We aren't even weekend support. Or any support, technically. Another team told us we had one day to do three weeks' worth of work to deploy their application because they had set a hard deadline and then didn't tell us about it until the day before. We gave them a flat "No" for that request.

I could probably keep going, but you get the gist of it.

Forgive me father, for I have sinned.  Alot actually, but I'm here for technical sins.  Okay, a particular series of technical sins.  Sit your ass back down padre, you signed up for this shit.  Where was I?  Right, it has been 11429 days since my last confession.  May this serve as equal parts rant, confession, and record for the poor SOB who comes after me.

Ended up in a job where everything was done manually or controlled by rickety Access "apps".  Many manhours were wasted on sitting and waiting for the main system to spit out a query download so it could be parsed by hand or loaded into one of the aforementioned apps that had a nasty habit of locking up the aged hardware that we were allowed.  Updates to the system were done through and awful utility that tended to cut out silently, fail loudly and randomly, or post data horrifically wrong.

Fuck that noise.  Floated the idea of automating downloads and uploads to bossman.  This is where I learned that the main system had no SQL socket by default, but the vendor managing the system could provide one for an obscene amount of money.  There was no buy in from above, not worth the price.

Automated it anyway.  Main system had a free form entry field, ostensibly for handwriting SELECT queries.  Using Python, AutoHotkey, and glorified copy-pasting, it worked after a fashion.  Showed the time saved by not having to do downloads manually.  Got us the buy in we needed, bigwigs get negotiating with the vendor, told to start developing something based on some docs from the vendor.  Keep the hacky solution running as team loves not having to waste time on downloads.

Found SQLi vulnerability in the above free form query system, brought it up to bossman to bring up the chain.  Vulnerability still there months later.  Test using it for automated updates.  Works and is magnitudes more stable than update utility.  Bring it up again and show the time we can save exploiting it.  Decision made to use it while it exists, saves more time.  Team happier, able to actual develop solutions uninterrupted now.  Using Python, AutoHotkey, glorified copy-pasting, and SQLi in the course of day to day business critical work.  Ugliest hacky thing I've ever caused to exist.

Flash forward 6 years.  Automation system now in heavy use acrossed two companies.  Handles all automatic downloads for several departments, 1 million+ discrete updates daily with alot of room for expansion, stuff runs 24/7 on schedule, most former Access apps now gone and written sanely and managed by the automation system.  Its on real hardware with real databases and security behind it.

It is still using AutoHotkey, copy-paste, and SQLi to interface with the main system.  There never was and never will be a SQL socket.  Keep this hellbeast I've spawned chugging along.

I've pointed out how many ways this can all go pearshaped.  I've pointed out that one day the vendor will get their shit together they'll come in post system update and nothing will work anymore.  I've pointed out the danger in continuing to use the system with such a glaring SQLi vulnerability.

Noone cares.  Won't be my problem soon enough.

In no particular order:
Fuck management for not fighting for a good system interface
Fuck the vendor for A) not having a SQL socket and B) leaving the SQLi vulnerability there this long
Fuck me for bringing this thing into existence

From my work -as an IT consultant in one of the big 4- I can now show you my masterpiece

INSIGHTS FROM THE DAILY LIFE OF A FUNCTIONAL ANALIST IN A BIG 4 -I'M NOT A FUNCTIONAL ANALYST BUT THAT'S WHAT THEY DO-

- 10:30, enter the office. By contract you should be there at 9:00 but nobody gives a shit
- First task of the day: prepare the power point for the client. DURATION: 15 minutes to actually make the powerpoint, 45 minutes to search all the possible synonyms of RESILIENCE BIG DATA AGILE INTELLIGENT AUTOMATION MACHINE LEARNING SHIT PISS CUM, 1 hour to actually present the document.
- 12:30: Sniff the powder left by the chalks on the blackboards. Duration: 30 minutes, that's a lot of chalk you need to snort.

13:00, LUNCH TIME. You get back to work not one minute sooner than 15.00

- 15:00, conference with the HR. You need to carefully analyze the quantity and quality of the farts emitted in the office for 2 hours at least
- 17:00 conference call, a project you were assigned to half a day ago has a server down.
The client sent two managers, three senior Java developers, the CEO, 5 employees -they know logs and mails from the last 5 months line by line-, 4 lawyers and a beheading teacher from ISIS.
On your side there are 3 external ucraininans for the maintenance, successors of the 3 (already dead) developers who put the process in place 4 years ago according to God knows which specifications. They don't understand a word of what is being said.
Then there's the assistant of the assistant of a manager from another project that has nothing to do with this one, a feces officer, a sys admin who is going to watch porn for the whole conference call and won't listen a word, two interns to make up a number and look like you're prepared. Current objective: survive. Duration: 2 hours and a half.
- 19:30, snort some more chalk for half an hour, preparing for the mail in which you explain the associate partner how because of the aforementioned conference call we're going to lose a maintenance contract worth 20 grands per month (and a law proceeding worth a number of dollars you can't even read) and you have no idea how could this happen
- 20:00, timesheet! Compile the weekly report, write what you did and how long did it take for each task. You are allowed to compile 8 hours per day, you worked at least 11 but nobody gives a shit. Duration: 30 minutes
- 20:30, update your consultant! Training course, "tasting cum and presenting its organoleptic properties to a client". Bearing with your job: none at all. Duration: 90 minutes, then there's half an hour of evaluating test where you'll copy the answers from a sheet given to you by a colleague who left 6 months ago.
- 22:30, CHANCE CARD! You have a new mail from the HR: you asked for a refund for a 3$ sandwich, but the receipt isn't there and they realized it with a 9 months delay. You need to find that wicked piece of paper. DURATION: 30 minutes. The receipt most likely doesn't even exist anymore and will be taken directly from your next salary.
- 23:00 you receive a message on Teams. It's the intern. It's very late but you're online and have to answer. There's an exception on a process which have been running for 6 years with no problems and nobody ever touches. The intern doesn't know what to do, but you wrote the specifications for the thing, 6 years ago, and everything MUST run tonight. You are not a technician and have no fucking clue about anyhing at all. 30 minutes to make sure it's something on our side and not on the client side, and in all that the intern is as useful as a confetto to wipe your ass. Once you're sure it's something on our side you need to search for the senior dev who received the maintenance of the project, call him and solve the problem.
It turns out a file in a shared folder nobody ever touches was unreachable 'cause one of your libraries left it open during the last run and Excel shown a warning modal while opening it; your project didn't like this last thing one bit. It takes 90 minutes to find the root of the problem, you solve it by rebooting one of your machines. It's 01:00.

You shower, watch yourself on the mirror and search for the line where your forehead ends and your hair starts. It got a little bit back from yesterday; the change can't be seen with the naked eye but you know it's there.

You cry yourself to sleep. Tomorrow is another day, but it's going to be exactly like today.

Most unrealistic deadline I've had?

The client gives us a project with A 3-month deadline. All good, at first.

3 days later they told us that they're cutting out the funding and they gave us two options: Either stop the project right away and get paid for the work so far, or somehow finish the project in 7 days and get paid for the 10 days worth of work.

My idiot boss chose the second.

Saturdays and Sundays were declared working days, everyone screaming at each other, devs running around scrambling code to make the webpage render and keep it responsive.

Forget unit or automation testing, we only did not null and undefined testing and submitted the project.

Yeah, I hauled my ass out of the company just after that.

My old job was great. I was writing automation software for one of the world's biggest storage deployments, and there was always a new challenge. But over time, I was asked to lend a hand with the tedious task of corresponding with procurement vendors and on-site technicians. At first it was one site, then it was two, and then it was an entire region of the US, spread across two time zones I'm not in.

I hated that work, and I found that I didn't have time anymore for software development, because of the time commitment the logistics work was. I was never hired to do logistics work, I was never trained, never qualified, and as I said, I hated it. I agreed to it to temporarily help out a weakness due to a shortage in staffing. But it never got taken off my plate, except for a short stint toward the end, just before I was placed on a PIP, because surprise surprise-- I'm bad at logistics.

About halfway through the PIP, I told my boss I wasn't doing it anymore. I said he could either put me back on software development or let me go, if ticket-monkeying and phone calls is the direction the wind is blowing for our team. I told him I had no intention of resigning, as you are not eligible for unemployment or severance if you resign, so their choice was to let me go. I'm told by people who are still there that everybody on the team is a ticket-jockey button-pusher now. Bleh.

My wife and I sold our old condo in Kansas City earlier in the summer, so we had about a year's worth of cushion, which was why I was willing to be let go. I was profoundly unhappy in my work, and it was bleeding through to my relationship with my wife and kids. So I took advantage of the time between jobs by spending more time with my family and just generally becoming a happier person again.

Meanwhile, I was in no desperate hurry to find a new job, so I got on linkedin, and had no more than two irons in the fire at a time. After just over two months I got an offer for a better job than before, which I accepted. There wasn't anything remarkable about that process though-- it's just something I've gone through recently.

Not really a hack but still worth telling:

I was working in the QA team for a big project. I tried to do some automation when I realized some radio button behaved weird... out of curiosity I checked the source and saw that there was a hidden option for a unimplemented payment option.

I was like: Let’s see how the system behaves if I just submit that form with that hidden value...

Well I was very surprised when I received the email that my order has been processed successfully.

During the investigation we found out that this bug was in prod for over two years. And it requires a one liner executed in the browsers console to skip the payment.

It was kind of a big deal and although I was (and am) still a trainee (in apprenticeship) I got invited to meet up with the client and the bosses.

It was kind of a door opener! After that they trusted me more. I have more responsibility, more interesting tasks and more client contact ever since.

To make a long story short:
Validate everything on the server side ;-)

Well it's a bit long but worth reading, two crazy stories in one rant:

So there are 2 things to consider as being my first job. If entrepreneurship counts, when I was 16 my developer friend and I created a small local music magazine website. We had 2 editors and 12 writers, all music enthusiasts of more or less our age. We used a CMS to let them add the content. We used a non-profit organization mentorship and got us a mentor which already had his exit, and was close to his next one. The guy was purely a genius, he taught us all about business plans, advertising, SEO, no-pay model for the young journalists (we promised to give formal journalist certificates and salary when the site grows up)
We hired a designer, we hired a flash expert to make some advertising campaigns and started filling the site with content.

Due to our programming enthusiasm we added to the raw CMS some really cool automation: We scanned our country's radio charts each week using a cron job and the charts' RSS, made a bot to search the songs on youtube and posted the first search result as an embedded video using some reg-exps. This was one of the most fun coding times I've had. Doing these crazy stuff with none to little prior knowledge really proved me I can do anything with the power of will.

Then my partner travelled to work in an internship in the Netherlands and I was too lazy to continue it on my own and it closed, not so surprisingly for a 16 years old slacker boy.

Then the mentor offered my real first job. He had a huge forum (14GB of historical SQL) but it was dying, the CMS version was very old and he wanted me to upgrade it to the latest. It didn't seem hard at first, because there were very clear instructions in the CMS website on how to do that. However, the automation upgrade scripts didn't work well because the forum owners added some raw code (not MVC plugins but bad undocumented code) and some columns to the SQL tables. I didn't give up and decided to migrate between the versions without the scripts. I opened a new CMS and started learning by heart all of the database columns so I can make a script to migrate between the versions. The first tests ran forever because processing 14GB of data on a single home computer is not a task meant to be done. I didn't give up. I made an old forum and compared the table structures and code with my mentor's. I think I didn't exhaustively finish this solution, the task was too big on my shoulders and eventually I gave up. I still owe thanks for that mentor for teaching me how to bare with seemingly (and practically) impossible tasks, for learning not to fear from being a leader and an entrepreneur and also for paying me in time even though I didn't deliver anything 😂

What do you think about my language choice set for the future (knowing I want to work as a software and app developer) ? Anything to add / remove ?

- C++: Fast and well-documented, so I think it's a standard even in the next decades to come

- Java: Although I think that this language will more likely die in the next decade, I'll maybe keep this language because some dinosaurs enterprises still rely on it. Ah and mainly because it's still widely used in Android apps programming. For now.

Talking about Android, does learning Kotlin worth it ?

- Python: Will mainly use it for automation and prototyping, but nothing more, as it seems not to be widely use in the software development field (or it is ?). I'll also keep it for hobbies, however.

- Rust: This language seems to be a rising star in the industry since it is very clean, classic, as fast as C / C++ while introducing more safety. However I'll wait a bit for this one since it requires more complicated and abstract knowledge I do not have yet.

- Javascript (or more particularly JSX): Hurts to say I'll keep it, even more than Java. I'd let it in the web development hell I won't step in if it was not used in webapps / cross-platform mobile applications. And since this kind of stuff looks trendy, I don't think I can avoid it. Plus, I liked working with React Native. Sorry.

- C#: Seems to be a must when working on Windows software interfaces, so guess I'll have to learn this one. Will do so gladly, it looks better than Java

So fucking exhausting when you have to change many things in a codebase in a repetitive manner with search and replace and it's not worth the time to write a rool to automate it. One of those special cases where you wish search and replace would support some kind of templating or be variable depending on the search result

Woke up yesterday morning from a dream where I was explaining what needed to be done to upgrade a Drupal 6 site. It hasn't been supported officially for years and I was explaining how there isn't a decent port of the main module we use Audio. And even the guy I was explaining this too seemed somewhat exasperated. So yeah, this is reality.

I could probably write a real upgrade path for the Drupal module and take all of our content into a new version of Drupal. But it would involve a fair amount of learning and outdated syntax and then learning Drupal 8. This would be all volunteer and take away from my time working on my other open source radio automation project.

All the while I've been learning Ruby on Rails for a class and I could just upgrade the app right out of Drupal, but this would require me to support the site into perpetuity. Which I already more or less do.

Drupal at this point is like an ex- girlfriend who I've grown away from, we did cool things but always got into fights about stupid things. Now I have to revisit my past mistakes and decide what to clean up and what to take into the future. I'm a better programmer now but I'm still not sure if it is worth my time to rekindle my romance with Drupal or it would just distract me from my current pursuits. Anyone who has been through the transition of a Drupal site from one major version to the next should feel my pain. At least it's not Word Press.