Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
-
Worst thing you've seen another dev do? Long one, but has a happy ending.
Classic 'Dev deploys to production at 5:00PM on a Friday, and goes home.' story.
The web department was managed under the the Marketing department, so they were not required to adhere to any type of coding standards and for months we fought with them on logging. Pre-Splunk, we rolled our own logging/alerting solution and they hated being the #1 reason for phone calls/texts/emails every night.
Wanting to "get it done", 'Tony' decided to bypass the default logging and send himself an email if an exception occurred in his code.
At 5:00PM on a Friday, deploys, goes home.
Around 11:00AM on Sunday (a lot folks are still in church at this time), the VP of IS gets a call from the CEO (who does not go to church) about unable to log into his email. VP has to leave church..drive home and find out he cannot remote access the exchange server. He starts making other phone calls..forcing the entire networking department to drive in and get email back up (you can imagine not a group of happy people)
After some network-admin voodoo, by 12:00, they discover/fix the issue (know it was Tony's email that was the problem)
We find out Monday that not only did Tony deploy at 5:00 on a Friday, the deployment wasn't approved, had features no one asked for, wasn't checked into version control, and the exception during checkout cost the company over $50,000 in lost sales.
Was Tony fired? Noooo. The web is our cash cow and Tony was considered a top web developer (and he knew that), Tony decided to blame logging. While in the discovery meeting, Tony told the bosses that it wasn't his fault logging was so buggy and caused so many phone calls/texts/emails every night, if he had been trained properly, this problem could have been avoided.
Well, since I was responsible for logging, I was next in the hot seat.
For almost 30 minutes I listened to every terrible thing I had done to Tony ever since he started. I was a terrible mentor, I was mean, I was degrading, etc..etc.
Me: "Where is this coming from? I barely know Tony. We're not even in the same building. I met him once when he started, maybe saw him a couple of times in meetings."
Andrew: "Aren't you responsible for this logging fiasco?"
Me: "Good Lord no, why am I here?"
Andrew: "I'll rephrase so you'll understand, aren't you are responsible for the proper training of how developers log errors in their code? This disaster is clearly a consequence of your failure. What do you have to say for yourself?"
Me: "Nothing. Developers are responsible for their own choices. Tony made the choice to bypass our logging and send errors to himself, causing Exchange to lockup and losing sales."
Andrew: "A choice he made because he was not properly informed of the consequences? Again, that is a failure in the proper use of logging, and why you are here."
Me: "I'm done with this. Does John know I'm in here? How about you get John and you talk to him like that."
'John' was the department head at the time.
Andrew:"John, have you spoken to Tony?"
John: "Yes, and I'm very sorry and very disappointed. This won't happen again."
Me: "Um...What?"
John: "You know what. Did you even fucking talk to Tony? You just sit in your ivory tower and think your actions don't matter?"
Me: "Whoa!! What are you talking about!? My responsibility for logging stops with the work instructions. After that if Tony decides to do something else, that is on him."
John: "That is not how Tony tells it. He said he's been struggling with your logging system everyday since he's started and you've done nothing to help. This behavior ends today. We're a fucking team. Get off your damn high horse and help the little guy every once in a while."
Me: "I don't know what Tony has been telling you, but I barely know the guy. If he has been having trouble with the one line of code to log, this is the first I've heard of it."
John: "Like I said, this ends today. You are going to come up with a proper training class and learn to get out and talk to other people."
Over the next couple of weeks I become a powerpoint wizard and 'train' anyone/everyone on the proper use of logging. The one line of code to log. One line of code.
A friend 'Scott' sits close to Tony (I mean I do get out and know people) told me that Tony poured out the crocodile tears. Like cried and cried, apologizing, calling me everything but a kitchen sink,...etc. It was so bad, his manager 'Sally' was crying, her boss 'Andrew', was red in the face, when 'John' heard 'Sally' was crying, you can imagine the high levels of alpha-male 'gotta look like I'm protecting the females' hormones flowing.
Took almost another year, Tony released a change on a Friday, went home, web site crashed (losses were in the thousands of $ per minute this time), and Tony was not let back into the building on Monday (one of the best days of my life).10 -
Probably the intern who, while a decent and intelligent guy for the most part, thought it was increasingly hilarious to keep putting random cat GIFs somewhere in the product with each PR he made.
First time "ok, very funny, but you can't do that in production software, don't do it again."
The third time around the "joke" was wearing a little thin.
Eventually a script was written so that, every time he made a PR, he'd get "emailed" one of a few pre-defined messages from a random member of the team a few minutes later, telling him to remove it and stop pissing about...3 -
“In order to release on production you merge on production”
Makes sense.
“In order to release on staging you merge on release”
Wait, what?
“The CI for the staging release is called release to production”
What? No, stop!
“You also need to create a tag called pre-production-deployment, so that staging can work”
STOP FFS
“It’s very easy, you just need to read the documentation!”
How do people do not realise they are spitting out bs?3 -
One of our newly-joined junior sysadmin left a pre-production server SSH session open. Being the responsible senior (pun intended) to teach them the value of security of production (or near production, for that matter) systems, I typed in sudo rm --recursive --no-preserve-root --force / on the terminal session (I didn't hit the Enter / Return key) and left it there. The person took longer to return and the screen went to sleep. I went back to my desk and took a backup image of the machine just in case the unexpected happened.
On returning from wherever they had gone, the person hits enter / return to wake the system (they didn't even have a password-on-wake policy set up on the machine). The SSH session was stil there, the machine accepted the command and started working. This person didn't even look at the session and just navigated away elsewhere (probably to get back to work on the script they were working on).
Five minutes passes by, I get the first monitoring alert saying the server is not responding. I hoped that this person would be responsible enough to check the monitoring alerts since they had a SSH session on the machine.
Seven minutes : other dependent services on the machine start complaining that the instance is unreachable.
I assign the monitoring alert to the person of the day. They come running to me saying that they can't reach the instance but the instance is listed on the inventory list. I ask them to show me the specific terminal that ran the rm -rf command. They get the beautiful realization of the day. They freak the hell out to the point that they ask me, "Am I fired?". I reply, "You should probably ask your manager".
Lesson learnt the hard-way. I gave them a good understanding on what happened and explained the implications on what would have happened had this exact same scenario happened outside the office giving access to an outsider. I explained about why people in _our_ domain should care about security above all else.
There was a good 30+ minute downtime of the instance before I admitted that I had a backup and restored it (after the whole lecture). It wasn't critical since the environment was not user-facing and didn't have any critical data.
Since then we've been at this together - warning engineers when they leave their machines open and taking security lecture / sessions / workshops for new recruits (anyone who joins engineering).26 -
I think I want to quit my first applicantion developer job 6 months in because of just how bad the code and deployment and.. Just everything, is.
I'm a C#/.net developer. Currently I'm working on some asp.net and sql stuff for this company.
We have no code standards. Our project manager is somewhere between useless and determinental. Our clients are unreasonable (its the government, so im a bit stifled on what I can say.) and expect absurd things from us. We have 0 automated tests and before I arrived all our infrastructure wasn't correct to our documentation... And we barely had any documentation to begin with.
The code is another horror story. It's out sourced C# asp.net, js and SQL code.. And to very bad programmers in India, no offense to the good ones, I know you exist. Its all spagheti. And half of it isn't spelled correctly.
We have a single, massive constant class that probably has over 2000 constants, I don't care to count. Our SQL projects are a mess with tons of quick fix scripts to run pre and post publishing. Our folder structure makes no sense (We have root/js and root/js1 to make you cringe.) our javascript is majoritly on the asp.net pages themselves inline, so we don't even have minification most of the time.
It's... God awful. The result of a billion and one quick fixes that nobody documented. The configuration alone has to have the same value put multiple times. And now our senior developer is getting the outsourced department to work on moving every SINGLE NORMAL STRING INTO THE DATABASE. That's right. Rather then putting them into some local resource file or anything sane, our website will now be drawing every single standard string from the database. Our SENIOR DEVELOPER thinks this is a good idea. I don't need to go into detail about how slow this is. Want to do it on boot? Fine. But they do it every time the page loads. It's absurd.
Our sql database design is an absolute atrocity. You have to join several tables together just to get anything done. Half of our SP's are failing all the time because nobody really understands the design. Its gloriously awful its like.. The epitome of failed database designs.
But rather then taking a step back and dealing with all the issues, we keep adding new features and other ones get left in the dust. Hell, we don't even have complete browser support yet. There were things on the website that were still running SILVERLIGHT. In 2019. I don't even know how to feel about it.
I brought up our insane technical debt to our PM who told me that we don't have time to worry about things like technical debt. They also wouldn't spend the time to teach me anything, saying they would rather outsource everything then take the time to teach me. So i did. I learned a huge chunk of it myself.
But calling this a developer job was a sick, twisted joke. All our lives revolve around bugnet. Our work is our BN's. So every issue the client emails about becomes BN's. I haven't developed anything. All I've done is clean up others mess.
Except for the one time they did have me develop something. And I did it right and took my time. And then they told me it took too long, forced me to release before it was ready, even though I had never worked on what I was doing before. And it worked. I did it.
They then told me it likely wouldn't even be used anyway. I wasn't very happy at all.
I then discovered quickly the horrors of wanting to make changes on production. In order to make changes to it, we have to... Get this
Write a huge document explaining why. Not to our management. To the customer. The customer wants us to 'request' to fix our application.
I feel like I am literally against a wall. A huge massive wall. I can't get constent from my PM to fix the shitty code they have as a result of outsourcing. I can't make changes without the customer asking why I would work on something that doesn't add something new for them. And I can't ask for any sort of help, and half of the people I have to ask help from don't even speak english very well so it makes it double hard to understand anything.
But what can I do? If I leave my job it leaves a lasting stain on my record that I am unsure if I can shake off.
... Well, thats my tl;dr rant. Im a junior, so maybe idk what the hell im talking about.16 -
The awkward moment when the website is faster on my shitty computer than on its pre-production server...4
-
Worst thing you've seen another dev do? Here is another.
Early into our eCommerce venture, we experienced the normal growing pains.
Part of the learning process was realizing in web development, you should only access data resources on an as-needed basis.
One business object on it's creation would populate db lookups, initialize business rule engines (calling the db), etc.
Initially, this design was fine, no one noticed anything until business started to grow and started to cause problems in other systems (classic scaling problems)
VP wanted a review of the code and recommendations before throwing hardware at the problem (which they already started to do).
Over a month, I started making some aggressive changes by streamlining SQL, moving initialization, and refactoring like a mad man.
Over all page loads were not really affected, but the back-end resources were almost back to pre-eCommerce levels.
The main web developer at the time was not amused and fought my changes as much as she could.
Couple months later the CEO was speaking to everyone about his experience at a trade show when another CEO was complementing him on the changes to our web site.
The site was must faster, pages loaded without any glitches, checkout actually worked the first time, etc.
CEO wanted to thank everyone involved etc..and so on.
About a week later the VP handed out 'Thank You' certificates for the entire web team (only 4 at the time, I was on another team). I was noticeably excluded (not that I cared about a stupid piece of paper, but they also got a pizza lunch...I was much more pissed about that). My boss went to find out what was going on.
MyBoss: "Well, turned out 'Sally' did make all the web site performance improvements."
Me: "Where have you been the past 3 months? 'Sally' is the one who fought all my improvements. All my improvements are still in the production code."
MyBoss: "I'm just the messenger. What would you like me to do? I can buy you a pizza if you want. The team already reviewed the code and they are the ones who gave her the credit."
Me: "That's crap. My comments are all over that code base. I put my initials, date, what I did, why, and what was improved. I put the actual performance improvement numbers in the code!"
MyBoss: "Yea? Weird. That is what 'Tom' said why 'Sally' was put in for a promotion. For her due diligence for documenting the improvements."
Me:"What!? No. Look...lets look at the code"
Open up the file...there it was...*her* initials...the date, what changed, performance improvement numbers, etc.
WTF!
I opened version control and saw that she made one change, the day *after* the CEO thanked everyone and replaced my initials with hers.
She knew the other devs would only look at the current code to see who made the improvements (not bother to look at the code-differences)
MyBoss: "Wow...that's dirty. Best to move on and forget about it. Let them have their little party. Let us grown ups keeping doing the important things."8 -
Today was a manic-depressive kind of day. Spent the morning helping some developers with getting their code to run a stored procedure to drop old partitions, but it wasn't working on their end. It was a fairly simple proc. But working with partitions is a little like working with an array. I figured out that they were passing the wrong timestamp, and needed to add +1 to delete the right partition. Got that sorted out, and things were good. Lunch time.
After lunch I did some busy work, and then the PO comes up at about 2PM and says he's assigned some requests to me. The first was just attaching some scripts. Easy. The second, the user wants a couple of schemas exported ... at 6PM. I've been in the office since 6:45AM.
While I'm setting up some commands to run for the data export, a BA walks up and asks if I'm filling in for another DBA who is out for a few weeks. Yep. There's a change request that hasn't been assigned, and he normally does the work. I ask when it's due. Well, the pre-implementation was supposed to be done in the morning, but it wasn't, and we're in the implementation window ... half way through. I bring up the change task, and look at. Create new schema and users. That's all it says. The BA laughs. I tell I need more to go on. 10 minutes later he sends an email with the information. There's only two hours left in the window, and I can only use half of it, because the production guys have to their stuff, and we're in their window. Now I'm irritated, because I'm new to Oracle, and it's an unforgiving mistress. Fortunately, another DBA says he'll do it, so that we can get it done in time. But can't work it either, because Dev DBAs don't have access to QA, and the process required access for this task. Gets shelved until the access issue is resolved. It's now after 4:15PM. I'm going to in traffic with that 6PM deadline.
I manage to get home and to the computer by 5:45PM. Log in. Start VPN. Box pops on screen. Java needs to update. I chose skip update. Box pops up again. It won't let me log in until Java is current. Passed.
I finally get logged in, and it's 6:10PM. I'm late getting the job started. I pull up Putty and log into the first box, and paste my pre-prepared command in the command line and hit error. Command not found. I'm tired, so it's a moment to sink in. I don't have time for this.
I log into DBArtisan and pull up the first data base, use the wizard to set the job, and off it goes. Yay. Bring up the second database, and have enter the connect info. Host not found. Wut? Examine host name. Yep, it's correct. Try a different method. Host not found. Go back to Putty. Log in. Past string. Launch. Command not found. Now my brain is quitting on me. Why now? It's after 6:30PM. Fiddle with some settings, reset $Oracle home. Try again. Yay. It works. I'm done. It's after 7PM.
There is nothing like technology to snatch the euphoria of a success away from you. It's a love-hate thing, but I wouldn't trade it for anything else. I'm done. Good night.3 -
When company gave me a SSH I am a noob I have a filezilla what I do is delete all temp files and clear log files
Because the log file size goes 1gb After next day all are looking weird
Senior developer:wtf are you doing yesterday
Me: delete all the tmp files and log files 😂😂20 -
Going through the conversation for xxxxth time with my business partner, why we will not launch a new product on top of pre-made PHP script / plugin.
Just got our company into TDD, and automated QA via CI server & code checks etc, PLEASE stop trying to drag us back into the land of spaghetti code & bug legions in production. That's all thxbye. -
I'm still studying computer science/programming, I still have one year to do in order to graduate (Master). I am in a work study program so I'm working for a company half of the time, and I'm studying the other half. It is important to mention that I am the only web developer of the company
When I arrived in the company 9 months ago, I was given a Vue project which had been developed by a trainee a few weeks before my arrival and I was asked to correct a few things, it was mostly about css. Then, I was ask to add a few functionalities, nothing really hard to code, and we were supposed to test the solution in a staging environment, and if everything was ok, deploy it to prod.
However, the more I did what I was asked, the more functionalities I had to implement, until I reached a point where I had to modify the API, create new routes, etc. I'm not complaining about that, that's my job and I like it. But the solution was supposed to be ready when I arrived, it was also supposed to be tested and deployed.
The problem is, the person emitting these demands (let's call him guy X) is not from the IT service, it's a future user of the website in the admin side. The demands kept going and going and going because, according to him, the solution was not in a good enough state to be deployed, it missed too many (un)necessary features. It kept going for a few months.
The best is yet to come though : guy X was obviously a superior, and HIS superior started putting pressure on me through mails, saying the app was already supposed to be in production and he was implying that I wasn't working fast enough. Luckily, my IT supervisor was aware of what was going on and knew I obviously wasn't to blame.
In the end, the solution was eagerly deployed in production, didn't go through the staging environment and was opened to the users. Now, guy X receives complaints because none of what I did was tested (it was by me, but I wasn't going to test every single little thing because I didn't have time). Some users couldn't connect or use this or that feature and I am literally drowning in mails, all from guy X, asking me to correct things because users are blocked and it's time consuming for him to do some of the things the website was doing manually.
We are here now just because things have been done in a rush, I'm still working on it and trying to fix prod problems and it's pissing me off because we HAVE a staging environment that was supposed to prevent me from working against the clock.
On a final note, what's funny is that the code I'm modifying, the pre-existing one needs to be refactored because bits and pieces are repeated sometimes 5 times where it should have been externalized and imported from another file. But I don't know when and if I will ever be able to do that.
I could have given more context but it's 4am and I'm kinda tired, sorry if I'm not clear or anything. That's my first rant -
I HATE the idea of only releasing on pre-determined schedules despite work being completed and just waiting for that day to arrive.
I'm a co-founder of a small software company. We have partnered with another particular company that also writes software. Some of our clients have access to paid content of that company's services through our application.
Every once in a while, our clients will report issues with that company's service to us, because they access it through our application. They think it's our issue.
We then pass the report on to the partner company, telling them that their stuff is broken. Their reply goes like this:
"Ok. We'll get the bug fix scheduled, and we'll release it next Thursday."
"Next Thursday? The issue is now, they can't use the service."
"That's our scheduled release date."
O.M.G.
We voluntarily walked away from our safe, cushy jobs working for other people, taking enormous pay cuts to start this company. Now, we're 6+ years in, disrupting established fat-and-happy competitors in this space. I GUARANTEE you that if we had that same attitude, we would have been absolutely obliterated early on.
We are quick. Guided by kanban boards, our suite of unit tests and integration tests is vast and kick-ass. With continuous integration and the click of a button we know if we broke something or if the piece we're working on is ready to be pushed to production, IMMEDIATELY. Our "release schedule" is when the damn thing is complete.
It isn't all bad. Our integration with them has been beneficial for both of us. I just loathe their snail's pace which negatively affects our mutual customers. It can make us look bad, and we can do nothing about it.
Blah.3 -
Who the hell hardcodes their localhost ports in a web.config without updating the release config to the correct production URLs? And why doesn't our ops team pick up on this shit before clicking their fancy deploy button? And why in holy heaven do we even have a pre-production server if it isn't an exact mirror of production?
God help me, I need a drink. -
I was asked to integrate a piece of industrial software which was pre-alpha in a prod to align with corporate objectives. It would have caused all kinds of production problems and instability. Nobody would even notice that it wasn’t there. I procrastinated for eight months until version 1 was released.
I told my manager it was done. Then he informed me that it had been working the whole time and he got plenty of positive feedback so i should roll back to the old version until we can do a planned upgrade. I nodded in agreement and put off the downgrade. -
Today, my co-workers went nuts about the fact their version of our product doesn't work on the pre-production since everything is fine on their local dev machines ^^
After a quick sighting, I figured out what was going on.
There was a package called from non-dev code which was required as a package for development.
The build plan of the application consists of a task which purges dev-packages within the vendor folder, using composer install --no-dev ^^
So the build plan runs perfectly fine, without a single error, but runtime was yelling about a missing class.
It's a delight to be one of the only guys with dev-ops experience in the whole damn building.
xD -
After an idea has rolled into the production stage I become bored with it. I no longer want to maintain or even review git issues.
However when even the hint of a new project rolls in I start "pre-developing" in anticipation and excitement of a new project. Development and base design of a project is what keeps me going.
#firstworlddeveloperproblems3 -
Just found out our pre-production environment is the qa for another team.
Infra team has recommended to ignore the nomenclature. It's just a name they said. -
So, I rarely ask questions, I usually find the answer myself, but, I'm just tired today and maybe my fellow dev ranters can help here.
Is there a way to write a bash script that automatically runs a pre-defined mysql query on local DB and throws that up into production DB?
it's just a simple select and then insert.
Been looking at select into out file, from infile etc
I could do it manually, but I'm going to have to do this more than a few times over the next few weeks and would love to just alias it and be done with it
doesn't seem possible without actually logging into mysql cli each time...on both sides, which sucks
this isn't export/import a table, it's export/import the result of a query
thanks5 -
Released the product to customers without realising someone had pre-filled a production test accounted login at the login screen. Obviously it was used for testing purposes and the dev forgot to remove it before committing.
I loled when I found out about it, and some customers started to login and made transactions.
When I checked the commit history... It was under my name :/2 -
I think I just realized what my biggest gripe about our career paths that I hate the most.
This is something that has worsened over time, especially the last 2 to 3 years.
As developers, we have far too many options. Some of the most powerful apps are written with languages that have hard, and I mean HARD, guardrails in place. If the app is written in a language that does not meet this criteria usually a framework has been used to install those guardrails.
We just get our minds so wrapped around the possibilities and the opportunities in the software, that we just can't focus on the end result. We're like puppies that are excited about something and we just piss all over everything.
In my career I have met far too many developers that don't have the capacity and mental fortitude to take control of their actions. Because of this I think the only way for us to stop this corruption, that I feel we are nurturing, the solutions/services that we use need to push back on us and install those guardrails for us.
All this came from a change that Microsoft put in place that seems well intended, but introduces yet another choice and a multitude of opinions in how you release code.
It used to be a simple check box. If it was checked it was pre-release, if it was unchecked it was a production release. That's it. On or off. The simplest choice you ever needed to make on a release.
Now though, there are two check boxes. One for a pre-release and one for a latest release. You can also not check either for some "ephemeral" release? So now something as easy as on or off has been made into a difficult decision on how this works within my pipeline. Now every time I make a release I have to ask myself, "which one do I check?"
I shouldn't need to spend more than a second to identify a path forward on simple shit like this, but here we are with a third choice.
Can we just stop overcomplicating shit?6 -
most cutting edge?
that would be that invoice processing system that is in production despite its pre-pre-pre-alpha status.2 -
In my initial days as a web developer, i was assigned a task, to implement a cart share functionality in an e commerce company.
I made the functionality and tested on my system.
Result: working good.
Pushed it to beta testing environment.
Resilt: working good.
Pushed to pre production environment.
Result: working good.
Pushed to live site.
Result: 😀 Error in live site..
So a call comes to me from my team lead..
Asks what was the issue...
Me: i dont know either.
....
After 3-4 hrs:
I found the reason.
My system, beta test env, pre prod env are all having latest php version (5.6 i guess)
But the live server had old version of php.
Me: laughed like anything.
I didn't know that these things would matter in such a great level.
Moral of the story:
Be one with the force (server in this case)2
Top Tags
Weekly Rant
View