Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
-
Tested on local work perfectly,
Push to production
Restart service
CPU went up to 100%, process alive but all services down.
Panic.
Try to think of what would be the cause.
With no hope, i restarted the service again.
Everything works as expected.
I still dont know what happened. :/7 -
I recently joined the dark side - an agile consulting company (why and how is a long story). The first client I was assigned to was an international bank. The client wanted a web portal, that was at its core, just a massive web form for their users to perform data entry.
My company pitched and won the project even though they didn't have a single developer on their bench. The entire project team (including myself) was fast tracked through interviews and hired very rapidly so that they could staff the project (a fact I found out months later).
Although I had ~8 years of systems programming experience, my entire web development experience amounted to 12 weeks (a part time web dev course) just before I got hired.
I introduce to you, my team ...
Scrum Master. 12 years experience on paper.
Rote memorised the agile manifesto and scrum textbooks. He constantly went “We should do X instead of (practical thing) Y, because X is the agile way.” Easily pressured by the client to include ridiculous (real time chat in a form filling webpage), and sometimes near impossible features (undo at the keystroke level). He would just nag at the devs until someone mumbled ‘yes' just so that he would stfu and go away.
UX Designer. 3 years experience on paper ... as business analyst.
Zero professional experience in UX. Can’t use design tools like AI / photoshop. All he has is 10 weeks of UX bootcamp and a massive chip on his shoulder. The client wanted a web form, he designed a monstrosity that included several custom components that just HAD to be put in, because UX. When we asked for clarification the reply was a usually condescending “you guys don’t understand UX, just do <insert unhandled edge case>, this is intended."
Developer - PHD in his first job.
Invents programming puzzles to solve where there are none. The user story asked for a upload file button. He implemented a queue system that made use of custom metadata to detect file extensions, file size, and other attributes, so that he could determine which file to synchronously upload first.
Developer - Bootlicker. 5 years experience on paper.
He tried to ingratiate himself with the management from day 1. He also writes code I would fire interns and fail students for. His very first PR corrupted the database. The most recent one didn’t even compile.
Developer - Millennial fratboy with a business degree. 8 years experience on paper.
His entire knowledge of programming amounted to a single data structures class he took on Coursera. Claims that’s all he needs. His PRs was a single 4000+ line files, of which 3500+ failed the linter, had numerous bugs / console warnings / compile warnings, and implemented 60% of functionality requested in the user story. Also forget about getting his attention whenever one of the pretty secretaries walked by. He would leap out of his seat and waltz off to flirt.
Developer - Brooding loner. 6 years experience on paper.
His code works. It runs, in exponential time. Simply ignores you when you attempt to ask.
Developer - Agile fullstack developer extraordinaire. 8 years experience on paper.
Insists on doing the absolute minimum required in the user story, because more would be a waste. Does not believe in thinking ahead for edge conditions because it isn’t in the story. Every single PR is a hack around existing code. Sometimes he hacks a hack that was initially hacked by him. No one understands the components he maintains.
Developer - Team lead. 10 years of programming experience on paper.
Writes spaghetti code with if/else blocks nested 6 levels deep. When asked "how does this work ?”, the answer “I don’t know the details, but hey it works!”. Assigned as the team lead as he had the most experience on paper. Tries organise technical discussions during which he speaks absolute gibberish that either make no sense, or are complete misunderstandings of how our system actually works.
The last 2 guys are actually highly regarded by my company and are several pay grades above me. The rest were hired because my company was desperate to staff the project.
There are a 3 more guys I didn’t mention. The 4 of us literally carried the project. The codebase is ugly as hell because the others merge in each others crap. We have no unit tests, and It’s near impossible to start because of the quality of the code. But this junk works, and was deployed to production. Today is it actually hailed as a success story.
All these 3 guys have quit. 2 of them quit without a job. 1 found a new and better gig.
I’m still here because I need the money. There’s a tsunami of trash code waiting to fail in production, and I’m the only one left holding the fort.
Why am I surrounded by morons?
Why are these retards paid more than me?
Why are they so proud when all they produce is trash?
How on earth are they still hired?
And yeah, FML.8 -
The network starts slowing down, transactions start to fail across the 450+ stores, the website starts to spit 500 errors what is going on?
Queue a frantic running around the office working out what was going wrong... Calls from all 3 data centres, nothing is going in or out of the network.
Notice the network admin come back to his desk, his eyebrows raise and he looks left and right before unplugging his laptop ethernet from one of the server access points
The network rushes back to life, everything is fine.
That particular network mapping tool is now banned for use on production.10 -
Juniors are a fun bunch to work with.
Over confident, hero complex of that fresh graduate high, and then thrown in to the real world! Where there hopes and dreams are crushed in minutes when they see what monolithic applications really look like!!
But don't let that overwhelm you, your not going to be changing all of it any time soon, hell some of this code hasn't been touched in 5+ years and still works without fail.
Don't stress about the work load, you can only write 1 line of code at a time anyway, and hell, even seniors make mistakes.
The key about being able to manage this beast is simple, break it! Because the more you break it, the more you'll understand how a project is put together, for better or worse. Learn from the examples in front of you, and learn what not to do in the future 😎
But more importantly, plan your changes, whiteboard the high level logic of what it is you want to add, then whiteboard in the current codebase and determine where to slice this bitch up, then when it all looks well and good, take out your scalpel and slice and dice time.
Don't worry, your changes aren't going to production anytime soon, hell, you'll be lucky to get past the first pull request with this working 100% the first time, and that's a good thing, learn from tour short comings and improve your own knowledge for the next time!2 -
Product Owner: "Our definition of done is putting it on production. So you are only done if it's on production. Otherwise our sprint goal is failed."
So we put it on production.
After deploy, some content manager appears: "Why is the system doing things? I was told this should not happen today."
"Erm, we have put the feature on production as we are only done if it's on production."
"Well, yes. But it should not be live yet!"
Oh well. Communication, or the lack thereof, does never fail to amaze me ¯\_(ツ)_/¯5 -
A few days after deploying a big important Website into production, I wanted to copy the whole thing including DB back onto our test server for future testing/bug fixing if something comes up. (Last changes were done on production server before going live)
So I opened SSH, removed everything on the test sever aaaaand then I realized I was connected to production...
Took about an hour to get everything up and running again. We didn't tell the client and hoped it would not be noticed.2 -
I just love it when our clients decide to make a clone of live production server..then put it immediately online..and don't tell anyone about this.. and then start bitchin how data gets doubled all of a sudden..
Yeah, no shit sherlock.. you have two prod servers for 'hot swapping' and some services may only be running on one at a time.. You even have a manual on how to switch primary to secondary (turn off services on primary first, then turn them on on secondary and all)..or in case primary actually dies, just turn on services on secondary and you're good to go, right?
So how do ya think cloning the one with running services and putting the clone immediately online will work out?! 🤔
God, I thought it was common sense to not do that..but here I am, bitchin about how people fail to RTFM.. :/ or use brain..fuck..4 -
Debate (with rant-ish overtones):
FYI, while it is a debate, its a practiseSafeHex debate, which means there is a correct answer, i'm just interested in your responses/thoughts.
Ok lets kick off. So the remote team I work with had an opening for a new iOS developer (unrelated to anything to do with me). They interviewed and hired a guy based off his "amazing" take home challenge.
The challenge consists of 4 screens and was for a senior level position. For the challenge the interviewee created a framework (a iOS library) for each screen, included all the business logic for each screen inside, each one needs to be built separately, exposed some API/functions from each one and then created a main project to stitch it all together.
Now, my opinion is, this is highly unscalable and a ridiculous approach to take as it would add so much unnecessary overhead, for no benefit (I am correct btw).
The interviewee said he did it like this to "show off his skills and to stand out". The remote team loved it and hired him. The challenge said "show us the code standard you would be happy to release to production". I would argue that he has only demonstrated 1 extra skill, and in exchange delivered something that is unscalable, going to be a nightmare to automate and require huge on-boarding and a paradigm shift, for no reason. To me thats a fail for a senior to not realise what he's doing. This person will be required to work alone (in part), make architecture decisions, set the foundation for others etc. Having someone who is willing to just do mad shit to show off, is really not the type of person suited to this role.
Debate!11 -
Half way through a 2 hour deployment, and a fucking test fails.
Yes, it takes 2+hours to run the tests.
Rerun in test environment: pass
Rerun in prod: pass
Rerun changed test in prod: fail..
Why, why you got to hate me for?
I love it when production is the place where config gets changed.5 -
I work for a bank and every production release date it's a chaos... Like, for real, devs running to get their stories approved by the testing team and last minute scope changes that, if not made, would make the whole app fail (real shitty management as you can see).
Longstoryshort, a dev didn't finished one of his stories and create 7 major bugs with another... Today that was my breakfast, took me 4 hours and get it all done and approved... We didn't make the release tho, but I scored some major points with this.
Funny thing, tomorrow I'm telling my PM I'll leave the company for a better job, so that will be their breakfast.6 -
Doing exams at the moment. Finished phase one out of four successfully at Monday but now stuff is going bad again as usual. Seriously, with me, everything goes perfectly fine until stuff gets official, then code starts failing, self doubt comes up and fair of failure and low self esteem hit me like a bomb.
I'm using my own framework which I actually also use in production and it works fine! But then it has to start to fucking fail at the moment I need it to work the fucking most.
I've worked towards this for five years now, I don't want to fail this! I don't want to disappoint either myself or my friends or my parents.
Fuck.15 -
Java dev here. I rewrote an app and replaced a system call to ssh with a modern jaxrs post for uploading a file and (new) some additional data.
I even used a stream.
1 hour in production, first client doesn't get his file. Log says OutOfMemoryError: heap.
Me: wtf? I already use streams.
Looking at the Jersey library. Docs say nothing. An issue from 2013 says: oh if you silly don't use the Apache httpclient addon, we disable chunking and buffer the whole body, because our tests fail with the jdk included http client otherwise.
Me: meh.
No warning in the logs. Thank you soooooo much! Who could have known?4 -
In my three years experience so far I can honestly say that 100% of the developers I've worked with are narrow sighted with regards to how they develop.
As in, they lack the capacity to anticipate multiple scenarios.
They code with one unique scenario in mind and their work ends up not passing tests or generates bugs in production.
Not to say I'm the best at foreseeing every possible scenario, but I at least TRY to anticipate and test my code as much as possible to identify problems and edge cases.
I usually take much more time to complete tasks than my colleagues, but my work usually passes tests and comes back bug free. Whereas my colleagues get applauded for completing tasks quickly but end up spending lots of time fixing up after themselves when tests fail or bugs appear.
Probably more time wasted than if they had done the job correctly from the start. Yet they're considered to be effecient devs because they work "fast".
Frustrating...7 -
Okay. So my dumbass boss took this project that had a steep timeline. I told him straight up, it won't work because we won't make the timeline. If we do this, I will be the one bending over backwards to deliver. I don't like to promise and fail. I got the oh don't worry let's just try. If we don't make it that's fine. Unfortunately that's not how I work. I refuse to deliberately fail. So I say okay and we begin. I suggested open source is the fastest way to deliver bit the fucked up part is, I am the only senior dev in the team. I will be expected to reverse engineer the open source app to connect our own deployment parameters. Use tech I have never used before. Connect frontend and backend. Handle dns bullshit. I have literally been working on Vibes and coffee for the past two weeks because ofcourse I ran into so many issues. Now I have an extension for Monday and I hate to fail. So I am not sleeping or resting just working on a fucking java app I didnt build and I am expected to make it work seemlessly on our production environment. I made some progress. Deployed frontend, deployed backend. Forgot to connect production dB so I decided to go with azure database for mysql driver since we have credits on azure. Now my java app is pissing itself over ssl handshake. I generate my keystore and add it and now java socket just times out. I want to pummel somebody or a punching bag that looks like my boss.15
-
Never buy crappy, consumer-grade SSDs for use in production servers/RAIDs. This might sound obvious but at the company I used to work for, through a series of bad decisions by management and cheapness, we ended up with the cheapest consumer SSDs you can imagine powering all of our storage.
This turned into a nightmare spanning years of failed hard drives and a continues cycle of ridiculousness. Drive failed after a few days, gets taken out, sent back to manufacturer and then replaced with another equally crappy drive destined to fail within days/weeks.
Our ops people were going to the data center multiple times per week to replace failed drives. Lesson I learned: cheaping out on system-critical hardware and software can have long standing consequences and in the end usually doesn't end up actually saving money when you account for time employees have to spend dealing with issues that result from it. -
What is it with networking guys refusing to do any kind of fault finding? Pretty much everywhere I've worked they seem to be overpaid address hogs who occasionally want everyone to be proud of them for installing a new switch.
Currently seeing a production issue that's clearly due to spikes in packet loss on a certain part of the network - but oh no, it's always "our tests are fine", "we can establish a route no problem", "this is an application level issue", etc.
No you morons, when a dozen unrelated applications hosted on different cloud services fail because none of them can contact anything in your particular subnet in your data center at the same time, it's a damn networking issue. Sort it out.14 -
Does anyone else wonder why production builds fail when it gets closer to releases? That shit happens a lot!6
-
Other team lead: Hi DevOps Team, We need you to deploy this app to production. It's maintainers gave up on it in 2019, but we looked at it and it feels right.
Me: Uhm. That's not going to work. It'll fail the security scan before you can even finish the build in CI.
Other team lead: Yeah, this app is the right thing to do, and we needed it last week, but since that won't work, we'll just use this other very very infant technology that was just born yesterday. It's not stable in production, or on MySQL, or in AWS at all, but it's the other direction we can to go.
Me: What problem are you trying to solve in the first place?
Other team lead: Oh, we need access to the read from the production database.2 -
# Retrospective as Backend engineer
Once upon a time, I was rejected by a startup who tries to snag me from another company that I was working with.
They are looking for Senior / Supervisor level backend engineer and my profile looks like a fit for them.
So they contacted me, arranged a technical test, system design test, and interview with their lead backend engineer who also happens to be co-founder of the startup.
## The Interview
As usual, they asked me what are my contribution to previous workplace.
I answered them with achievements that I think are the best for each company that I worked with, and how to technologically achieve them.
One of it includes designing and implementing a `CQRS+ES` system in the backend.
With complete capability of what I `brag` as `Time Machine` through replaying event.
## The Rejection
And of course I was rejected by the startup, maybe specifically by the co-founder. As I asked around on the reason of rejection from an insider.
They insisted I am a guy who overengineer thing that are not needed, by doing `CQRS+ES`, and only suitable for RND, non-production stuffs.
Nobody needs that kind of `Time Machine`.
## Ironically
After switching jobs (to another company), becoming fullstack developer, learning about react and redux.
I can reflect back on this past experience and say this:
The same company that says `CQRS+ES` is an over engineering, also uses `React+Redux`.
Never did they realize the concept behind `React+Redux` is very similar to `CQRS+ES`.
- Separation of concern
- CQRS: `Command` is separated from `Query`
- Redux: Side effect / `Action` in `Thunk` separated from the presentation
- Managing State of Application
- ES: Through sequence of `Event` produced by `Command`
- Redux: Through action data produced / dispatched by `Action`
- Replayability
- ES: Through replaying `Event` into the `Applier`
- Redux: Through replay `Action` which trigger dispatch to `Reducer`
---
The same company that says `CQRS` is an over engineering also uses `ElasticSearch+MySQL`.
Never did they realize they are separating `WRITE` database into `MySQL` as their `Single Source Of Truth`, and `READ` database into `ElasticSearch` is also inline with `CQRS` principle.
## Value as Backend Engineer
It's a sad days as Backend Engineer these days. At least in the country I live in.
Seems like being a backend engineer is often under-appreciated.
Company (or people) seems to think of backend engineer is the guy who ONLY makes `CRUD` API endpoint to database.
- I've heard from Fullstack engineer who comes from React background complains about Backend engineers have it easy by only doing CRUD without having to worry about application.
- The same guy fails when given task in Backend to make a simple round-robin ticketing system.
- I've seen company who only hires Fullstack engineer with strong Frontend experience, fails to have basic understanding of how SQL Transaction and Connection Pool works.
- I've seen company Fullstack engineer relies on ORM to do super complex query instead of writing proper SQL, and prefer to translate SQL into ORM query language.
- I've seen company Fullstack engineer with strong React background brags about Uncle Bob clean code but fail to know on how to do basic dependency injection.
- I've heard company who made webapp criticize my way of handling `session` through http secure cookie. Saying it's a bad practice and better to use local storage. Despite my argument of `secure` in the cookie and ability to control cookie via backend.18 -
probably every time I see my tests failing.
Each time I am writing tests I'm convincing myself "it's an investment", "spend 2 hours now to save 2 days later", "unit-tests are good".
And each time I'm chasing away ideas like "perhaps they are right, perhaps writing unit tests is a waste of time..", "this code is simple, it should ever break - why test it??", "In the 2 hours I'll spend writing those UT I could build another feature"
Yes, it is terribly annoying to write tests, especially after writing the production code (code-first approach). Why test code that you know works, right?
But after a few weeks, months or years, when the time comes to change your feature: enhance it, refactor it, build an integration with/from it, etc, I feel like a child who found a forgotten favourite candy in his pocket when I see my tests failing.
It means I did a very good job writing them
It means it was not a waste of time
it means these tests will now save me hours or days of trial-and-error change→compile→deploy→test cycles.
So yeah, whenever I see my tests fail, I feel warm and fussy inside :)2 -
When your company expects you to manually change information in the production db by saying "hey, client Billy wants his stuff moved back to where it was"3
-
https://devops.com/dear-staging-wer...
What the F#*@&$# %#@$!?!?!
This person has decided to skip using staging, because it doesn't correctly reflect prod!
If that's your problem, than why don't you try to fix it? Create a DB with fake data, make one based on anonymised customer data, or even do it on non-anonymised data (with permission of course), but fix the staging env so that it reflects prod!
This is a devops site (it's literally the name!), and instead of teaching you how to make staging exactly like prod, they tell you to do what caused the creation of the staging->prod system IN THE FIRST PLACE!!!
There's all these stuff like Vagrant that are literally designed to help you as a dev mimic prod, and you just throw it all out!?
"With feature flags, I can safely test in production without fear of breaking something or negatively affecting the customer experience."
Famous last words.12 -
Oh china... just when I was so exhausted and annoyed at you... cuz i got behind and had to grind purchase/logistics/china-rel bs, to get necessities complete before your 2wks being MIA for chinese new year.
BUT THEN...
You had brought me this comedic gold in the midst of my sourcing woes!
Whoever points out my favourite 'precaution' first, wins.
...it's very rare that I say this, especially with my linguistics and algorithmic knowledge and exp making me easily peeved here...
But seriously, even Google trans with 0 knowledge of english, wouldve done better on most of these.
4 -
One of our servers had a disk fail this week. Luckily it's 1 of 3 in a RAID5 array. And, luckily, it was our mostly-dev box and didn't have any production stuff on it, except for some support things. We scheduled a disk replacement with the hosting company, took everything down, waited. Somebody at the hosting company apparently didn't know we'd scheduled the replacement, saw the machine was down, and brought it up again. Sigh. Finally they did the replacement, got it back up, but now we're seeing an ethernet port flapping, suggested they have someone go in and make sure all the jacks are fully seated, maybe one got loose when they were doing the disk switch. Bureacracy reared up again and we got the boilerplate "if there's a hardware issue suspected please boot into rescue mode and run the tests"... sigh...8
-
The meeting I will go in few minutes. Our app went in production 2 weeks ago, the managers invite us (devs) to have a drink with 'important' people I don't even know. We will celebrate the success of the project, after 6 months of pressure with everyone telling we will fail and that we are losers. Hypocrite meeting.2
-
Wordpress :(
This is some production code of a completely butchered Wordpress site I have been forced to work on (the previous developer managed to break the site so the archives section failed to work).
Words literally fail me.
1 -
Finally made my node production server stable enough that I could focus on writing tests*. I start by setting up docker, mocking cognito, preparing the database and everything. Reading up on Node test suites and following a short tut to set up my first unit test. Didn't go smoothly, but it's local and there are no deadlines so who cares. 4 days later, first assert.equal(1+1, 2) passes and I'm happy.
I start writing all sorts of tests, installing everything required into "devDependancies," and getting the joy of having some tests pass on first try with all asserts set up, feels good!
I decide to make a small update to production, so I add a test, run and see it fail, implement the feature, re-run and, it passes!
I push the feature to develop, test it, and it works as intended. Merge that to master and subsequently to one of my ec2 production servers**, and lo and behold, production server is on a bootloop claiming it "Cannot find module `graphql`". But how? I didn't change any production dependencies, and my package lock json is committed so wth?
I google the issue, but can't find anything relevant. The only thing that I could guess was that some dependencies (including graphql) were referenced*** in both, prod and dev, and were omitted when installed on a prod NODE_ENV, but googling that specific issue yielded no results, and I would have thought npm would be clever enough to see that and would always install those dependencies (spoiler: it didn't for me).
With reduced production capacity (having one server down) I decided to npm uninstall all dev dependencies anyway and see what happens. Aaaaand it works.....
So now I have a working production server, but broken local tests, and I'm not sure why npm is behaving like this...
* Yes I see the irony.
** No staging because $$$, also this is a personal project.
*** I am not directly referencing the same thing twice, it's probably a subdependency somewhere.2 -
You ever had a boss that made you feel like his bitch but he never really earned the title
You also know from a technical skill perspective you’re more competent.
And the only job he seems to do is micromanaging you. He just puts things under a microscope looking for a flaw. He always finds a flaw so in the off chance it breaks he’s always in the clear.
He’s the guy who sticks with the programs the he was taught when he was still at school and never really tried something new out of the box. He gives the reasons the he wasn’t formally trained in the other programs . I’m not talking cinema 4 here. I’m talking Matlab preference over python. Using lab-view as a production level development platform instead of going to something more approved by the industry.
He doesn’t take risk but he pushes those risks on you so if you fail he can say it wasn’t him
He’s never wrong but he’s never right either.
You’re sitting there doing the cunt work and breaking the sweat and he passes the achievements as under his management. You never really get the credit because “he guided you “. You go through hell fixing bugs and he disappears. He says he’s always a call away when what you really needed is someone taking the heavy tasks not throwing the entire project on your back.
I never call that piece of shit bcz he just throws some other bullshit that doesn’t make sense and emphasizes that might be the problem.
I once had a problem with the com port on a pc and was trying to figure out the problem. I asked him and he said that it might be bcz I’m connecting to the PC via VNC. I was like what the hell. What does that have to do with anything. I just ended up restarting the port and it bloody worked.
The saddest part is that I’m scared is that I might end up like him. In the same dead end job. Even though he guides me we work in a place where the job title doesn’t really change. Funny thing is that officially I have the same job title as him .
He’s been in the place for 5years when I came. Can someone imagine that? To work and work and then to be seized up with another brat who’s the same as you title wise.
You’re close the age of 40 and you work in a place where a 20 something year old walks in with the same Position as you.
I worry that I might end up the same if I stay long enough. That I’ll learn everything I can learn and just stop progressing and the only thing I can do is say how shit can break but wouldn’t know how to fix .
Pointing out problems because they are easier than fixing. Just plomonting into existential nihilism with no purpose.
I once told him I wanted to quit. He pretended he didn’t hear it. He then then said what do you see in this job in 5 years
I told him me not in it.
He said “seriously what do you want in this place “
I said “if I’m still her in 5 years I’ll be missing a toe because I would have shit myself in the foot”
I now realize that by convincing me to stay he might have convinced himself that staying for that long wasn’t a bad idea. He was looking for justification that he’s decision wasn’t that bad at all.
You give your life to a job and at the end it takes one away.
I don’t want to be like that and I think that’s what bugs me the most. That I’m so close to this individual that I feel sooner or later if I’m not careful I’ll end up in the same place. The same dread3 -
Production goes down because there's a memory leak due to scale.
When you say it in one sentence, it sounds too easy. Being developers we know how it all goes. It starts with an alert ping, then one server instance goes down, then the next. First you start debugging from your code, then the application servers, then the web servers and by that time, you're already on the tips of your toes. Then you realize that the application and application servers have been gradually losing memory over a period of time. If the application is one that don't get re-deployed ever so often, the complexity grows faster. No anomaly / change detection monitor can detect a gradual decrease of memory over a period of months.2 -
So, I have to begin with saying that I was 19 and my first real job. I was assigned to ongoing project for big german web company who outsourced project for german government. I am fast learner so I quickly ended up as a only developer who works on this project, because company had another important projects. And since I studied outside of my native country and I can speak Polish and English so I was also responsible for explaining everything to customer during meetings. I worked for around 4 months on the project and was heading up to the end. 3 weeks before production deadline client wanted show the results to the german government but I was still working on the functionality. My boss decided to put the web team leader to my project around 8 hours before this presentation to speed up the development. around 30 minutes before the pitch I realised that some of the latest functionalities stopped working. I was trying to figure out what I did I asked my team leader, he said that he have refactored some parts of my code. When we found the right commit it was around 3 min to presentation and after the checkout some of the .htaccess file was broken so I fixed it quickly but Germans started the meeting a bit earlier. The website was crashed almost half of the presentation. After 5 minutes my boss come to my desk and he says that he just talked with our customer and they are so freaking mad and pissed that they will not pay us. At this point I was certain I'm fired...
Suddenly the web team leader joined the conversation protecting me that it was the fault of the project menager because he should assign someone else to this project because even though I am good it is always good to have someone more experienced to work with you and review your code.
Project manager was fired about 3 months later. I was saved 😀 -
It is ok to fail and commit mistakes, that's part of the game, specially for beginner devs. Just avoid failing alone, the most you can!
I mean:
- Ask people to review your code before pushing to the source repository.
- If you are not sure how to do, ask.
- Never work in production environments without supervision. Pair with someone.
- Have a desk mate for rubberducking (https://en.wikipedia.org/wiki/...) and blame it in case you need -
So basically a friend was tasked with doing some syadmin on a propietary system running on top of GNU/Linux (they distribute the software as a distro).
Called me about an hour ago because there was some odd stuff happening so I log into the system and start figuring out what the actual fuck is up.
Just now we discovered that for a certain critical feature you just need to trust that there will be no eavesdroppers, meaning you send system credentials in cleartext over the network, and it won't work if it's not so.
Of course, some tunnels and routing later (which by the way, is "manual" configuration which is highly discouraged by the creators of this piece of crap) we kind of managed to overcome this obvious fail.
Now then, can you please explain me again how is it that these companies grab open source, make useless layers that limit it in every way possible and still profit? I mean, for fucks sake, you should at least let people manage shit with standard, well understood tools instead of "improving system administration", "easing it for...", for whom?
I'm so happy to log into our production server and be welcomed by beastie. -
Not sure how this made it to production..
Looks like Time Warner Cable updated their site to use rems with a replace all on 'px'.
https://registration.timewarnercable.com/... . -
What a fucking day.
Half a day looking for functionality on github that was lost in some branch and not merged to master. All remotes was pruned. Finally restored it from some remote on production and merged to master before Saturday release. Yeah !!!!
Month saved fuckers, pick some more hardcore shit to surprise me. To strong to fail. -
Microservices in real life:
You have 4 different project opened in IDE simultaneously editing all of them to make something to work.
All projects share same db.
Basically this is monolith but splitter to different git repos.
Communicating one another via http calls or directly altering shared db tables.
Everyone is happy until solution goes to production then :
Solution doesn't scale because db is a bottleneck.
Services can be unavailable for 1-2s so some processes are left in unfinished state forever, because everyone assume 99,9% SLA means it always available and http calls will never fail.
Http calls take time.2 -
Got a call about production was going to fail. They thought it's the application server.
I'm the end it was bogus file mods which were scrambled by the backup tool.
Why we didn't find out earlier? Because the java application was coded like this:
-------
String content;
Try {
File bla = new File
content = ... Read operation
} catch (IoException | SecurityEx | RuntimeEx ex)
// nothing we can do here
}
doWork(content);
---------
Why the fuck do we have code reviews? Why not just log or throw a Runtime Exception? Argh... I thought it would be better in enterprise applications. Perhaps I should tell them to not just use pmd, also spotbugs and sonarqube. But the department for the build tools does not have enough employees. Dang.
Anyway. Earned some money for that.
Now it's 2018 and I still get money for the same kind of bugs as 2008.3 -
Expanding a batch system on production
Set reservation to block the new nodes
Apply changes and restarted scheduler
(Reservation quietly lost, as it should persist)
Client called because jobs start to fail on new nodes…
Gahhhhhhh!!!!!! -
Spent a couple of weeks on writing a cronjob which updates a certain value in the application config, and spend the last few months on testing it in different environments to make sure it does not fail in production. Ran the deployment script, and the damn cronjob fails because of ssl certificate on production. fuck me
-
My first production support release is next week and its from midnight till 5am and be at work at 8:00am. If something goes wrong with any of the other developers from our team goes wrong I have to figure it out. And this image is exactly how I picture it goes, if any tickets fail. (🤘🏽 bring it on! )
Top Tags
Weekly Rant
View