Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
-
Good Morning!, its time for practiseSafeHex's most incompetent co-worker!
Todays contestant is a very special one.
*sitcom audience: WHY?*
Glad you asked, you see if you were to look at his linkedin profile, you would see a job title unlike any you've seen before.
*sitcom audience oooooooohhhhhh*
were not talking software developer, engineer, tech lead, designer, CTO, CEO or anything like that, No No our new entrant "G" surpasses all of those with the title ..... "Software extraordinaire".
*sitcom audience laughs hysterically*
I KNOW!, wtf does that even mean! as a previous dev-ranter pointed out does this mean he IS quality code? I'd say he's more like a trash can ... where his code belongs
*ba dum tsssss*
Ok ok, lets get on with the show, heres some reasons why "G" is on the show:
One of G's tasks was to build an analytics gathering library for iOS, similar to google analytics where you track pages and events (we couldn't use google's). G was SO good at this job he implemented 2 features we didn't even ask for:
- If the library was unable to load its config file (for any reason) it would throw an uncatchable system integrity error, crashing the app.
- If anything was passed into any of the functions that wasn't expected (null, empty array etc.) it would crash the app as it was "more efficient" to not do any sanity checks inside the library.
This caused a lot of issues as some of the data needed to come from the clients server. The day we launched the app, within the first 3 hours we had over 40k crash logs and a VERY angry client.
Now, what makes this story important is not the bugs themselves, come on how many times have we all done something stupid? No the issue here was G defended all of this as the right thing to do!
.. and no he wasn't stoned or drunk!
G claimed if he couldn't get the right settings / params he wouldn't be able to track the event and then our CEO wouldn't have our usage data. To which I replied:
"So your solution was to not give the client an app instead? ... which also doesn't give the CEO his data".
He got very angry and asked me "what would you do then?". I offered a solution something like why not have a default tag for "error" or "unknown" where if theres an issue, we send up whatever we have, plus the file name and store it somewhere else. I was told I was being ridiculous as it wasn't built to track anything like that and that would never work ... his solution? ... pull the library out of the app and forget it.
... once again giving everyone no data.
G later moved onto another cross-platform style project. Backend team were particularly unhappy as they got no spec of what needed to be done. All they knew was it was a single endpoint dealing with very complex model. There was no Java classes, super classes, abstract classes or even interfaces, just this huge chunk of mocked data. So myself and the lead sat down with him, and asked where the interfaces for the backend where, or designs / architecture for them etc.
His response, to this day frightens me ... not makes me angry, not bewilders me ... scares the living shit out of me that people like this exist in the world and have successful careers.
G: "hhhmmm, I know how to build an interface, but i've never understood them ... Like lets say I have an interface, what now? how does that help me in any way? I can't physically use it, does it not just use up time building it for no reason?"
us: "... ... how are the backend team suppose to understand the model, its types, integrate it into the other systems?"
G: "Can I not just tell them and they can write it down?"
**
I'll just pause here for a moment, as you'll likely need to read that again out of sheer disbelief
**
I've never seen someone die inside the way the lead did. He started a syllable and his face just dropped, eyes glazed over and he instantly lost all the will to live. He replied:
" wel ............... it doesn't matter ... its not important ... I have to go, good luck with the project"
*killed the screen share and left the room*
now I know you are all dying in suspense to know what happened to that project, I can drop the shocking bombshell that it was in fact cancelled. Thankfully only ~350 man hours were spent on it
... yep, not a typo.
G's crowning achievement however will go down in history. VERY long story short, backend got deployed to the server and EVERYTHING broke. Lead investigated, found mistakes and config issues on every second line, load balancer wasn't even starting up. When asked had this been tested before it was deployed:
G: "Yeah I tested it on my machine, it worked fine"
lead: "... and on the server?"
G: "no, my machine will do the same thing"
lead: "do you have a load balancer and multiple VM's?"
G: "no, but Java is Java"
... and with that its time to end todays episode. Will G be our most incompetent? ... maybe.
Tune in later for more practiceSafeHex's most incompetent co-worker!!!31 -
Write a small js function using setInterval to fire a request every second ... then copy paste the code 450 times (literally, not an exaggeration) into a massive file to create a load test script.
This load test script also had no means to gather metrics or test response times or anything useful. It was literally a “did the server crash” test.9 -
Worst dev team failure I've experienced?
One of several.
Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.
I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.
The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.
3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.
3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.
4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).
About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.
3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.
About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.
Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.
A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.
Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."11 -
This has to be the funniest IT fail ever. https://m.reddit.com/r/...
Found it in a comment somewhere on devrant (i forgot who posted originally, but this is me giving credit)9 -
So...
I'm penetrationtesting a network and the servers on said network
The network administrator and IT security officer knows this, because they hired me..
TL;DR a scan caused the network to crash.
Today I received a very angry email going "Stop scanning NOW!" from one of the IT departments.
Apparently I crashed their login server and thus their entire network...
It happened d the first time I scanned the network from the outside and they had spend an entire day figuring out how and repairing the service they thought was the problem, but then it crashed again, when I scanned from within the network.
Now they want to send me a list of IP's that I'm not allowed to scan and want to know exactly what and when I'm scanning...
How crap can they be at their job, if they weren't able to spot a scan... The only reason they found out it was me was because the NA had whitelistet my IP, so that I could scan in peace...5 -
>give person jailed access to my server.
>person breaks out, and notifies me.
>i fix the hole
>three weeks later i decide hes trustworthy enough and the jail is causing too many problems so i give him unjailed access
>first thing he does is crash the system
albeit unintentionally, but seriously?1 -
"Almond, I thought you said the cause of the outage the other week was that our server crashed?"
"The Tomcat server crashed, yeah. Not the physical server." (And you won't give me the time or budget to spin up any kind of redundant one, but that's besides the point...)
"Ok, but I've spoken to ops and they say none of the servers have gone offline in the last month?"
"Yup, the physical server was fine, it was the Tomcat server running on it that crashed."
"...so the server didn't crash?"
"We're mixing terms here. There's two things that can be referred to as the server. One is the physical machine, and one is an application running on it. The physical machine was fine, but
the application running on it crashed."
"What?! It's a very simple question. Did the server crash, or didn't it?!"
🤦♂️13 -
> Customer logs Jira ticket claiming app is not working
< I restart the app, investigate and explain tht their server has issues
ø Client closes the ticket as Resolved
-- a couple of days pass by ---
> Customer logs Jira ticket claiming app is not working
< I restart the app, investigate and explain tht their server has issues
ø Client closes the ticket as Resolved
-- a couple of days pass by ---
> Customer logs Jira ticket claiming app is not working
< I restart the app, investigate and explain tht their server has issues
ø Client closes the ticket as Resolved
-- a couple of days pass by ---
<...>
< I log a JIRA ticket explaining what and how is wrong with the server with suggestions how to fix the problem so the app will not crash any longer (client own the server, has his own sysadmins -- I don't even had permissions to open syslog.. had to hack dmesg on their PROD server to pin-point the issue)
> no reaction from customer for weeks. I ping the ticket
× app crashes again
> no reaction from customer for weeks. I ping the ticket
> customer leaves a comment that their sysadmins are looking at it trying to figure out what might be wrong (ignoring what I wrote in ticket's description??? srsly?)
× app crashes again
< I post detail investigation details: snips from logs, screenshots, everything with crystal clear explanations.
> no reaction for weeks
......
well that's fun..6 -
We had a server that crashed every single Christmas four years a row. Last year I put a cross and garlic on it for fun. Everybody laughed at me... but it didn't crash!
Soo... should I protect it from the Christmas ghost again, this year too? 🤔👻1 -
Okay, story time.
Back during 2016, I decided to do a little experiment to test the viability of multithreading in a JavaScript server stack, and I'm not talking about the Node.js way of queuing I/O on background threads, or about WebWorkers that box and convert your arguments to JSON and back during a simple call across two JS contexts.
I'm talking about JavaScript code running concurrently on all cores. I'm talking about replacing the god-awful single-threaded event loop of ECMAScript – the biggest bottleneck in software history – with an honest-to-god, lock-free thread-pool scheduler that executes JS code in parallel, on all cores.
I'm talking about concurrent access to shared mutable state – a big, rightfully-hated mess when done badly – in JavaScript.
This rant is about the many mistakes I made at the time, specifically the biggest – but not the first – of which: publishing some preliminary results very early on.
Every time I showed my work to a JavaScript developer, I'd get negative feedback. Like, unjustified hatred and immediate denial, or outright rejection of the entire concept. Some were even adamantly trying to discourage me from this project.
So I posted a sarcastic question to the Software Engineering Stack Exchange, which was originally worded differently to reflect my frustration, but was later edited by mods to be more serious.
You can see the responses for yourself here: https://goo.gl/poHKpK
Most of the serious answers were along the lines of "multithreading is hard". The top voted response started with this statement: "1) Multithreading is extremely hard, and unfortunately the way you've presented this idea so far implies you're severely underestimating how hard it is."
While I'll admit that my presentation was initially lacking, I later made an entire page to explain the synchronisation mechanism in place, and you can read more about it here, if you're interested:
http://nexusjs.com/architecture/
But what really shocked me was that I had never understood the mindset that all the naysayers adopted until I read that response.
Because the bottom-line of that entire response is an argument: an argument against change.
The average JavaScript developer doesn't want a multithreaded server platform for JavaScript because it means a change of the status quo.
And this is exactly why I started this project. I wanted a highly performant JavaScript platform for servers that's more suitable for real-time applications like transcoding, video streaming, and machine learning.
Nexus does not and will not hold your hand. It will not repeat Node's mistakes and give you nice ways to shoot yourself in the foot later, like `process.on('uncaughtException', ...)` for a catch-all global error handling solution.
No, an uncaught exception will be dealt with like any other self-respecting language: by not ignoring the problem and pretending it doesn't exist. If you write bad code, your program will crash, and you can't rectify a bug in your code by ignoring its presence entirely and using duct tape to scrape something together.
Back on the topic of multithreading, though. Multithreading is known to be hard, that's true. But how do you deal with a difficult solution? You simplify it and break it down, not just disregard it completely; because multithreading has its great advantages, too.
Like, how about we talk performance?
How about distributed algorithms that don't waste 40% of their computing power on agent communication and pointless overhead (like the serialisation/deserialisation of messages across the execution boundary for every single call)?
How about vertical scaling without forking the entire address space (and thus multiplying your application's memory consumption by the number of cores you wish to use)?
How about utilising logical CPUs to the fullest extent, and allowing them to execute JavaScript? Something that isn't even possible with the current model implemented by Node?
Some will say that the performance gains aren't worth the risk. That the possibility of race conditions and deadlocks aren't worth it.
That's the point of cooperative multithreading. It is a way to smartly work around these issues.
If you use promises, they will execute in parallel, to the best of the scheduler's abilities, and if you chain them then they will run consecutively as planned according to their dependency graph.
If your code doesn't access global variables or shared closure variables, or your promises only deal with their provided inputs without side-effects, then no contention will *ever* occur.
If you only read and never modify globals, no contention will ever occur.
Are you seeing the same trend I'm seeing?
Good JavaScript programming practices miraculously coincide with the best practices of thread-safety.
When someone says we shouldn't use multithreading because it's hard, do you know what I like to say to that?
"To multithread, you need a pair."18 -
The website for our biggest client went down and the server went haywire. Though for this client we don’t provide any infrastructure, so we called their it partner to start figuring this out.
They started blaming us, asking is if we had upgraded the website or changed any PHP settings, which all were a firm no from us. So they told us they had competent people working on the matter.
TL;DR their people isn’t competent and I ended up fixing the issue.
Hours go by, nothing happens, client calls us and we call the it partner, nothing, they don’t understand anything. Told us they can’t find any logs etc.
So we setup a conference call with our CXO, me, another dev and a few people from the it partner.
At this point I’m just asking them if they’ve looked at this and this, no good answer, I fetch a long ethernet cable from my desk, pull it to the CXO’s office and hook up my laptop to start looking into things myself.
IT partner still can’t find anything wrong. I tail the httpd error log and see thousands upon thousands of warning messages about mysql being loaded twice, but that’s not the issue here.
Check top and see there’s 257 instances of httpd, whereas 256 is spawned by httpd, mysql is using 600% cpu and whenever I try to connect to mysql through cli it throws me a too many connections error.
I heard the IT partner talking about a ddos attack, so I asked them to pull it off the public network and only give us access through our vpn. They do that, reboot server, same problems.
Finally we get the it partner to rollback the vm to earlier last night. Everything works great, 30 min later, it crashes again. At this point I’m getting tired and frustrated, this isn’t my job, I thought they had competent people working on this.
I noticed that the db had a few corrupted tables, and ask the it partner to get a dba to look at it. No prevail.
5’o’clock is here, we decide to give the vm rollback another try, but first we go home, get some dinner and resume at 6pm. I had told them I wanted to be in on this call, and said let me try this time.
They spend ages doing the rollback, and then for some reason they have to reconfigure the network and shit. Once it booted, I told their tech to stop mysqld and httpd immediately and prevent it from start at boot.
I can now look at the logs that is leading to this issue. I noticed our debug flag was on and had generated a 30gb log file. Tail it and see it’s what I’d expect, warmings and warnings, And all other logs for mysql and apache is huge, so the drive is full. Just gotta delete it.
I quietly start apache and mysql, see the website is working fine, shut it down and just take a copy of the var/lib/mysql directory and etc directory just go have backups.
Starting to connect a few dots, but I wasn’t exactly sure if it was right. Had the full drive caused mysql to corrupt itself? Only one way to find out. Start apache and mysql back up, and just wait and see. Meanwhile I fixed that mysql being loaded twice. Some genius had put load mysql.so at the top and bottom of php ini.
While waiting on the server to crash again, I’m talking to the it support guy, who told me they haven’t updated anything on the server except security patches now and then, and they didn’t have anyone familiar with this setup. No shit, it’s running php 5.3 -.-
Website up and running 1.5 later, mission accomplished.6 -
mysql server crashes every 18 days, no oom, no crash logs, no sigkill being sent (used auditd). so I figure it's a unknown corner case bug in mysql. now I use a cron job to restart the damn thing every week at 3am, not a problem anymore8
-
Visual Studio Code.
I've tried you because of hearing a lot of good stuff about you. I'd switch back to netbeans regardless because I love netbeans and I always try to use as little as possible from companies like Google/microsoft/facebook (and others) but what you're pulling right now is un-fucking-believable.
I've disabled ANY AND EVERY form of calling home I could (find) in your settings. Crash reports, automatic updates, metrics, you name it. I've searched all the fucking settings but I can't find any other home calling thing that's enabled and yet:
I'm monitoring every goddamn DNS request (through my own DNS server) and I'm still seeing calls to a Microsoft owned domain. Closed all my browser sessions and you as well and it stopped. Started browser again but not you, nothing.
Started you again: BAM. Calls to that damned Microsoft owned domain again.
If you can't honour my decision for disabling any form of home calls, go fuck yourself.
Netbeans, I'm back, I've missed you 💜34 -
Work: there is little work for our freelancer we're getting rid of him at the end of the year
Me: are you sure, that will put a lot of work on my schedule.
Work: I'm sure you can handle it
*two weeks later*
Work: have you finished feature xyz yet
Me: nope had a server crash today so pushed it back
Work: why didn't you get "free lancers name here" to do it for you
Me: I could of if you didn't fire him, he could of also done the other features you want too
Work: ah ok1 -
Before becoming a developer, I used to work as a sales rep at this company that spent a good amount of time building what they believed to be an innovative state-of-the-art “code generator”. It was basically a scaffolding tool for generating software.
They were using it to auto generate customized iOS and Android native mobile app templates, along with a web backed.
The problem was that the generated code was shit, and the developers on the team basically spent more time fixing bugs than if they had built everything from scratch. But their passion for the product meant they just kept using it.
For some reason they never fixed issues in the original templates, so basically all the bugs that were found, kept showing up with each new app!
I have never seen apps like this that essentially had more bugs than features. Opening more than 10 app screen meant the app would freeze and crash. Sign up forms were actually dummy forms. The list goes on...
All the apps had the same shitty UI. For example, Product pages had a product image area that was like 5% of the screen view!
Last but not least, apps had a backend IP address hardcoded pointing to a server with an IP address that was temporary. So one day they had to restart the server and suddenly all customer apps stopped working and required a software update to work!
It was amazing seeing how a team of 3 developers trying to fix messy autogenerated code, couldn’t accomplish what was essentially a website on an app that I managed to build in my free time.
That’s how I knew it was time to quit my job and code full time.2 -
The story of the shittiest, FUCKING WORST day of work.
TLDR: shitty day at work, car crash to end the day.
So, let tell you about what could possibly be the worst day I had since I started working.
This morning, my alarm didn't work, woke up 30 minutes before an appointment I had with a client.
Arrived late at the client, as I start deploying. They don't have any way to transfer the deployment package to the secured server. Lost 45 minutes there.
Deployment goes pretty well. My client asks me to stay while they load some data into the app. Everything's pretty easy to work out. Just need to input 3 CSV with the correct format (which the client defined since the beginning).
I end up watching an Excel Macro called "Brigitte" (I'm not fucking kinding, could'nt have thought of that) work for 4 hours straight. Files are badly formatted and don't work.
Troubbleshooting thoses files with a fucking loader that does not tell you anything about why it failed (our fault on that one)
I leave the client at 7:30pm, going back at work, leave at 9pm.
At this point, I just want to buy some food, go home and watch series.
But NO, A FUCKING MORRON OF A BUS DRIVER had to switch lanes as I was overtaking him. Getting me crushed between the bus and the concrete blocks.
Cops were fucking dickheads, being very mean even tho I was still shaking from the adrenaline.
In conclusion, the day could have been worst. The devs at the clients are pretty cool guys and we actually had some fun troubleshooting. At work, there was still one of my colleagues who cheered me up telling me about his day.
And when I think of it, I could have got really hurt (or even worst) in the crash.
A bad day is a bad day, tomorrow morning I'm still going to get up and go to a job I love, with people I love working with.
Very big rant (sorry about that if someone's still reading)9 -
Taking a database class, prof insists on using Microsoft Sql server 2014. "Okay cool" said the Microsoft surface fan boy inside me as I installed it. "Holy shit this is using 6 fucking GBs?? Eh it's okay I trust" again said my Microsoft fanboy self. Finished installing, makes queries and it works. Cool.
Go to run Sql server again next day and get an error (nothing displayed, just a box pop up and then a crash) I use some Google skills. Change a bunch of shit and still it persists. "Just uninstall it and reinstall again" says my prof. I do so except random errors during installation saying Sql already exists even though I just uninstalled it. "Maybe it's some registry keys messing with it!" do some digging, remove unneeded registry keys and try again. Installation finished but a whack of features say failed to install.
I sit and try to work this shit out for the next four hours (not paying attention to my class) and still can't get Sql to completely uninstall itself. I try iobit uninstaller, command line uninstalling, fucking everything but still not working. Slowly my fanboy side is wishing that the windows symbol on the back of my machine was an apple.
I ended up having to backup all my files and reinstalling windows to get it working properly. Holy sweet fuck. The worst part is when this class is done ill probably need to reinstall yet again to save the 6gb it's sucking up. So if you're not sure whether you need something as heavy as Microsoft Sql server or not for your application, don't use it! It's a fucking virus that is super difficult to remove.
Tldr: life long Microsoft fanboy becomes apple convert in a day of using Microsoft Sql server.
9 -
24th, Christmas: BIND slaves decide to suddenly stop accepting zone transfers from the master. Half a day of raging and I still couldn't figure out why. dig axfr works fine, but the slaves refuse a zone update according to tcpdump logs.
25th, 2nd day: A server decides to go down and take half my network with it. Turns out that a Python script managed to crash the goddamn kernel.
Thank you very much technology for making the Christmas days just a little bit better ❤️
At least I didn't have anything to do during either days, because of the COVID-19 pandemic. And to be fair, I did manage to make a Telegram bot with fancy webhooks and whatnot in 5MB of memory and 18MB of storage. Maybe I should just write the whole thing and make another sacred temple where shitty code gets beaten the fuck out of the system. Terry must've been onto something...5 -
I am calling this a premonition rant, of more rants to come.
I have a feeling in my bones.
We have a newly acquired fat cat customer with bucks to blow who we have done some digital work for already and swag bag of marketing perkiness.
I will call the CEO of this whale "The Porcupine"
The Porcupine has a business degree and industry experience, nothing to do with websites or applications.
It claims to be a visual perfectionist yet never delivers an overall coherent review.
It likes to fixate on minor brand style differences in websites and apps we have built.
The Porcupine seems to be always busy with policy and legal and other things rather than participating in their own projects.
Procrastination on feedback or reviews until the day before release is common.
Many overtime hours worked, not a sliver of thanks. The haughty attitude indicative of somebody who thinks web development is like desktop publishing.
"It's just code" in response to a crash production server change they were warned was a risk that borked all of our responsive templates and took 3 hours to fix.
Their entire brand is shades of pea green, grey and lime. No serif fonts because they are suck. Arial and Helvetica are boss.
One of my devs missed a CSS style on privacy policy hyperlink text that went times new roman and I had various account directors and our CEO on phone telling me how embarrassing it was for us to let this happen.
Anyway. They pay on time and the cost estimates for all the upcoming work are juicy.
We have shitloads going on for an upcoming hard date conference and everything is already compressing.
Therefore I can already smell doom and feel those porcupine quill getting closer to my ass as I beg their AD today if we have any feedback on the 10 or so project reviews yet?
Nope.4 -
I love tools such as IntelliSense or Copilot, don't get me wrong!
But i still have a deep rooted fear that one day, developers will become so dependent on those luxuries, that we will become practically unable to write code on our own, without our cloud overlords blessings.
Until, you know.. the server for such a service will crash and no one will know how to fix it without its own help. *see Palpatine meme reference*
17 -
I’m fairly new to maintaining my own webservers. For the past week the servers (two of them) kept crashing constantly.
After some investigation I figured it was due to someone running a script trying to get ssh access.
I learned about fail2ban, DOS and DDOS attacks and had quite a fight configuring it all since I had 20 seconds on average between the server shutdowns and had to use those 20 second windows to configure fail2ban bit by bit.
Finally after a few hours it was up and running on both servers and recognized 380 individual IPs spamming random e-mail / password combos.
I fet relieved seeing that it all stopped right after fail2ban installation and thought I was safe now and went to sleep.
I wake up this morning to another e-mail stating that pinging my server failed once again.
I go back to the logs, worried that the attack became more sophisticated or whatever only to see that the 06:25 cronjob is causing another fucking crash. I can’t figure out why.
Fuck this shit. I’m setting another cronjob to restart this son of a bitch at 06:30.
I’m done.3 -
Amazing! My joke functionality worked and the server delivered it's content successfully to the client. The joke: malloc is wrapped to give that message and retry IMMEDIATELY. Expectation is that the computer that was already in panic mode gets even fucked up more because it's out of memory. My malloc is literally while((ptr=malloc()} == NULL) { show_warning()
;malloc();}. Imagine if it didn't give that warning, I would've never known that a malloc failed. Who checks their freaking malloc result? You should, but i do not see much people doing it.
The previous crash on screen is what happens if you're doing a get instead of a post. I just declare my server app indestructible btw. Ffs!
5 -
It's sometimes really anxiety inducing thinking that all data could be gone, if somebody decides to kill/discontinue/crash [see gitlab shitting 6 hours of data due to fucked backup strategy and shitty seperation of servers] your account/service, be it server, git-repos, backups, chrome syncs, games, music, sim card, ..
But there's simply no way of having a backup of absolutely everything (ignore DRM) - especially automated and abstracted away from you, so you don't have to do all that shit yourself13 -
1) API Server crash due to Uncaught Error and none of us realised it until 30 mins later (no email alerts, no mechanism in place to notify devs): expensive mistake
2) api server and database server on one frickin vps, with no caching mechanisms or memoisation on frontend: Worst nightmare of my life5 -
Aaaah ! So fuckin done with this Server error !
I am checking if a cookie is set in Php and if it is, I am redirecting user to some page, basically its a 'remember me' logic. But this fucking error comes in everytime my page redirects.
I have a similar logic to check if user is currently logged in the current session,and if he opens a new window and types the url(index.php) he is automatically logged in (obviously,duh !) and redirected,bt it dosent crash at that time!
Help 😥
15 -
If anyone has used the super Mario 64 online mod, it's extremely easy to crash all clients connected to any server. You have to send a chat message payload with a length greater than 256 characters. The clients do not do any bound checking and write the payload directly to super Mario 64 memory. This causes all clients connected to the game to crash. I will leave how to send a chat payload > 256 characters up to you. I've confirmed my method works!2
-
This is the last time Microsoft! I'm getting my old Arch image out and removing you from my life forever! Never again will my linux distro randomly uninstall itself without telling me in the middle of implementing new components and crash my development server. Never again will I have to deal with an update that refuses to STFU and go away until I, ME NOT YOU MICROSOFT, decides it's a good time to run the update. No more lack of customization and poor support of common dev tools. I'M DONE WITH YOU, WE NEED TO SEE OTHER PEOPLE.2
-
So I'm working our remote build/testing server, and all of a sudden my computer just turns off. No crash message, no error, just turned off. My co-worker tries to help troubleshoot the problem.
Nothing.
I take the computer downstairs to the hardware department. They plug it in and it starts no problems.
After it finishes updating, I take it back upstairs and plug it back in.
Nothing.
I suggest that it is the power cable, and my co-worker looks under his desk. Turns out he had kicked the switch on the surge protector. 😑1 -
When you hash time entries by year and you are freaking out to make sure your server doesn’t crash due to differing time zones while your family is celebrating New Year’s...2
-
I've every been a Arch Linux fag. It's my main OS from 5 years. With a small parenthesis of two months of FreeBSD recently, I've used before Arch Kubuntu, Ubuntu, Fedora KDE, OpenSolaris (randomly), CentOs, plus a lot of others distro for tests.
But I've never tested Debian!
So I've installed it on my small server.
Oh... My... God.... It's fantastic. PACMAN >>> apt, but damn it's really stable and out of the box even if minimal. A very surprise. I think it can be my favorite remote Linux for a long time....
But a question rises. Why with a father like Debian... Ubuntu after the 11.04 is such a shame? The last I've tested is the 12.04 I think, but I've hated it, and I hate it even now. (Crash, driver not found, apt problem, very heavy repos and my internet sucks, UNITY, etc...)
Ubuntu, what happened to you ...? With Kubuntu 8 you were such a good guy...4 -
Old story, happened some way back. I worked part-time for a small web development company that did between other things something called SharePoint development, basically .net webforms with shit glitter on top of it.
The most weird part of it, was the fact that we were working on vms that hosted the app, it was our dev, test and staging environment, as well as were we showed the client the polished turd.
Did I say that it was on a vm? Well it was on a remote vm, that each of use had access to it, through our domain accounts, and they couldn't configure the windows server to accept more than two or three users at once to be connected.
That was our test enviroment and dev enviroment, sooo showing the app to the client meant for the rest of us to not write any code because it might crash or get stuck.
The app was accessible and discoverable by url and through google search from outside, I dont think that should have been allowed.
The most disastrous part was that we had NO source versioning whatsoever, just plain old copy and paste in different folders.
Deploying to client meant remoting to the clients host or whatever it was, and manually copying the source files
If someone wanted to debug the application you had to shout, and you also could hear it, in the office: "I'm debugging!" or "I'm deploying!". Because we were on the same machine, there was only one process with the server and it meant that if you debug or deployed it would block it for the others.
Should I talk about code quality? Maybe not.1 -
Condemning my HP stream to server-tude
going to host a minecraft server on that one on the subwoofer - let's pray that my sick beats don't cause a crash
(i'd put it in the closet, but the charger won't reach that far, and the closet has no outlet)
3 -
What game engine would you recommend to an indie developer? The type who can't afford a fucking server to run the bloated and buggy unity editor but is actually a developer so isn't afraid of typing.
I've had enough of the improper sandboxing (will crash bc of game scripts), tempfile-based crash-unaware instance tracking (won't restart afterwards) and lack of UI scaling (seriously, that's like accessibility/retina support basics) that is the unity editor. If they had command line tools I'd use them happily.10 -
I just gave a simple API which fetches recent searches by pinging an index on elasticsearch to the UI developer.
She had just one job. And ended up calling the server every time it loads on every screen thereby reaching max limit of calls per second and giving 429.
QA are not even required to break your code. UI developers are more than enough :) -
Gah... Outlook just froze because it can't connect to the mail server.... While I was drafting a long email....
Hope it doesn't crash, hope it doesn't crash.... Please please please...
-
To me this is one of the most interesting topics. I always dream about creating the perfect programming class (not aimed at absolute beginners though, in the end there should be some usable software artifact), because I had to teach myself at least half of the skills I need everyday.
The goal of the class, which has at least to be a semester long, is to be able to create industry-ready software projects with a distributed architecture (i.e. client-server).
The important thing is to have a central theme over the whole class. Which means you should go through the software lifecycle at least once.
Let's say the class consists of 10 Units à ~3 hours (with breaks ofc) and takes place once a week, because that is the absolute minimum time to enable the students to do their homework.
1. Project setup, explanation of the whole toolchain. Init repositories, create SSH keys for github/bitbucket, git crash course (provide a cheat sheet).
Create a hello world web app with $framework. Run the web server, let the students poke around with it. Let them push their projects to their repositories.
The remainder of the lesson is for Q&A, technical problems and so on.
Homework: Read the docs of $framework. Do some commits, just alter the HTML & CSS a bit, give them your personal touch.
For the homework, provide a $chat channel/forum/mailing list or whatever for questions where not only the the teacher should help, but also the students help each other.
2. Setup of CI/Build automation. This is one of the hardest parts for the teacher/uni because the university must provide the necessary hardware for it, which costs money. But the students faces when they see that a push to master automatically triggers a build and deploys it to the right place where they can reach it from the web is priceless.
This is one recurring point over the whole course, as there will be more software artifacts beside the web app, which need to be added to the build process. I do not want to go deeper here, whether you use Jenkins, or Travis or whatev and Ansible or Puppet or whatev for automation. You probably have some docker container set up for this, because this is a very tedious task for initial setup, probably way out of proportion. But in the end there needs to be a running web service for every student which they can reach over a personal URL. Depending on the students interest on the topic it may be also better to setup this already before the first class starts and only introduce them to all the concepts in a theory block and do some more coding in the second half.
Homework: Use $framework to extend your web app. Make it a bit more user interactive with buttons, forms or the like. As we still have no backend here, you can output to alert or something.
3. Create a minimal backend with $backendFramework. Only to have something which speaks with the frontend so you can create API calls going back and forth. Also create a DB, relational or not. Discuss DB schema/model and answer student questions.
Homework: Create a form which gets transformed into JSON and sent to the backend, backend stores the user information in the DB and should also provide a query to view the entry.
4. Introduce mobile apps. As it would probably too much to introduce them both to iOS and Android, something like React Native (or whatever the most popular platform-agnostic framework is then) may come in handy. Do the same as with the minimal web app and add the build artifacts to CI. Also talk about getting software to the app/play store (a common question) and signing apps.
Homework: Use the view API call from the backend to show the data on the mobile. Play around with the mobile project to display it in a nice way.
5. Introduction to refactoring (yes, really), if we are really talking about JS here, mention things like typescript, flow, elm, reason and everything with types which compiles to JS. Types make it so much easier to refactor growing codebases and imho everybody should use it.
Flowtype would make it probably easier to get gradually introduced in the already existing codebase (and it plays nice with react native) but I want to be abstract here, so that is just a suggestion (and 100% typed languages such as ELM or Reason have so much nicer errors).
Also discuss other helpful tools like linters, formatters.
Homework: Introduce types to all your API calls and some important functions.
6. Introduction to (unit) tests. Similar as above.
Homework: Write a unit test for your form.
(TBC)4 -
Requests to a soap server were failing randomly. In order to contact the API provider, I tried to provide an curl example with the same payload and the error response. Yet when sending the payload over curl, the request worked just fine. When my application was building the request, it failed.
What. The. Fuck.
I checked and double-checked the request body and headers. They were identical.
Of course, no error response was returned by the API provider and, of course, they could not tell me how what error I caused in my request.
So I created a basic dummy server, installed wireshark and compared the payload when sending a request from my application and from curl to my dummy server.
It turns out: curl, if called in a certain way, automagically strips out newlines. The soap client kept them.
So that that shitty soap server crashed due to newlines in the message body!
Stripping out the newlines was rather easy.
Shame on you, your house, and entire family for letting it crash due to them!1 -
I would appreciate if the language server wouldn't crash constantly, it's getting rather annoying.
Why is it that most C(++) tooling I have used feels like a half assed barely working project?3 -
fucking VS2017,
It takes like 1 minute to refresh stored procedures in the server explorer, It is so annoying especially if you need to constantly add new stored procedures.
and dont get me start on DBML at least once a week that shit fucking decided to crash VS2017.
Maybe I need to download more ram 🤔🤔🤔5 -
When I wrote a templating engine (great overstatement) just for myself to learn, I had this bug in php5 and regex that would crash the whole apache server.
Literally nothing worked and I didn't know why. After rebuilding everything I tried with regex using only simple string manipulation all of a sudden everything worked fine. -
Dat Moment when the exchange Server seems to crash and you found 20x1gb Mails From boss to all employes
Dau Like a boss2 -
Back when I was a freshman in high school a friend of mine put an emulator on the shared drive, so we could play NES games while in the computer lab. Didn't know better/didn't care. One day I get pulled out of class and walked into the computer guys office. In there is also the principal of the school and the Chief of police.
The computer guy tells me there was an issue last week that caused the school server to crash and it caused damage. I asked what happened and the he said one of the emulators we were playing had a script that crashed the server and caused damage. I asked how much damage and they informed me it was over 3 thousand dollars. At this point I'm very skeptical that the damage was worth about the cost of a new workstation (the old one sitting on his desk, buried in boxes), and afterwards none of the faculty knew of any kind of an outage. I asked for him to show me what broke and what had to be done to fix/replace the damaged equipment but all I got was a simple, "I'm sorry. I can't show you that at this time."
They threatened legal action for a felony of damaging a school property. Myself and the other tech savvy kids talked about it over the next couple of days wondering what would happen. They threatened expulsion for myself and a couple of other kids, but ultimately just got a talking to about keeping personal information safe.
What I got out of it was if they think I'm good with computers I must be doing something right. Now I'm in IT. This is where it went wrong. -
Risk is part of my everyday life.
I take the risk everyday when opening IDE and changing line of code that can either break database or crash other systems that are depending on one I am developing. ( not instantly but in some time in the future )
So....
Many years ago I was updating some application server production code while being drunk.
Everything went fine except me waking up in the morning and didn’t remember how I did it.
... what I learned from my developers life except that heavy drinking and updating servers is not the best idea ?
First, don’t give a fuck, do your job and ask questions even if the person in front of you said that understood everything and you think you understood all of shit.
Second, if you think you know what to do think twice.
Third, having any backup, any tests and any documentation is always better then having nothing.
And the most important.
The most risky in every business are people around you, so always have good people around and there would be no risk at all or you won’t even think about it.
✌🏽 ❤️ -
So we started to get build failures because the build server is full. Team leader asked the co-owner for more space. Got told we should just do less builds. You know stuff is outrageous when even your team leader goes on a rant about how stupid that is and how things are going to crash and burn in the future but nobody gives a damn.3
-
Cobol / Mainframe, create on your computer then ftp to server to compile and run (and hope it doesn't crash the cobol region due to infinite running) then ftp to get the output.
-
Bullshittery continues. This time around, absolutely innocent, clamav is root cause. For once not incompetent idiot, but piece of software. IDK if that makes me happy or upset.
So our email server that I configured and took care of died. RIP. Damn, better put it back together ASAP. So Im under pressure, while still pissed at everything that I ranted before (actually my last 2 rants were throttled, and in total all of that happened past 60 minutes but devrant rate limiting) I start auditing logs. You imagine, we kindda need it NOW, and it's second time last month clamav is pulling stunts and MTA refuses (properly) to work without antivirus. So pressurized, I look at logs, what the fuck went wrong.
clamav deamonize() failed - cannot allocate memory
Hmm. Intresting, but sounds like bullshit. I know server is quite micro becouse they wanted to save on costs as much as possible, but it has well over half a gig free ram just before it crashes (like 800MB) with that message. Is it allocating almost gig in one call or what? Looked carefully at trusty htop while it was starting, and indeed, suddenly it just dies with quite a bit of ram free, almost as much as it weights already. And I remember booting it up when I was configuring it, and it had fair bit of headroom.
Google, help me friend... Okay, great, so apparently at some point clamav loads virus DB into ram (dafuq?), and than forks, which causes spike of 2x the ram usage, and than immidietely frees it up.
Great, that sounds like great design decision... At least I know, I can just slap on SWAP file, restart it and call it a day.
It worked, swap file is almost empty (used 15megs, 900 megs free ram, whatever).
That leaves me wandering, who figured out to load DB to ram? That means pretty much that clamav will eat a little bit more ram each vir db update, and that milisecond "double ram" spike will confuse innocent people who just wanted to run clamav and it worked last *long period of time* and now crashes without warning without any changes to configuration.
Maybe there is logical explanation, I want to know it.8 -
Hey guys, I have a question.
If you ever had to deal with parity (Ethereum node software) and ever ran archive node, you perfectly know how long this bastard synchronizes. For our server it took almost month. Well, today or yesterday parity decided to spontaniously blow up (or crash) corruptiong database manifest file which greatly undercut us.
Anyone knows any viable way to rebuild manifest file withoud doing full sync from a scratfch?
If anyone has any suggestions other than what 95% of internet say "well, delete your database and sync from scratch" Im happy to test if it will solve our burning burning issue.
I am sure there is some way to rebuild database, especially where it's manifest file that's corrupt (Ive checked it, for wtf reason parity decided to truncate the file when it crashed).
Database backend is written in rust, and is called rocks db.
EDIT: if helps, its archive fat database (fat db means it should be easier to recover?)4 -
Optimizing images on a client site and I hope the server doesn't crash (I'm not hosting the site).4
-
Modified stuff on production server without checking documentation, because I was cocky and tought that I remembered everything. The worst thing is, that right after that I took a lunch break and only realised what have I done after that. For an hour or so anyone who opened our app experienced an instant crash...
-
Relatively often the OpenLDAP server (slapd) behaves a bit strange.
While it is little bit slow (I didn't do a benchmark but Active Directory seemed to be a bit faster but has other quirks is Windows only) with a small amount of users it's fine. slapd is the reference implementation of the LDAP protocol and I didn't expect it to be much better.
Some years ago slapd migrated to a different configuration style - instead of a configuration file and a required restart after every change made, it now uses an additional database for "live" configuration which also allows the deployment of multiple servers with the same configuration (I guess this is nice for larger setups). Many documentations online do not reflect the new configuration and so using the new configuration style requires some knowledge of LDAP itself.
It is possible to revert to the old file based method but the possibility might be removed by any future version - and restarts may take a little bit longer. So I guess, don't do that?
To access the configuration over the network (only using the command line on the server to edit the configuration is sometimes a bit... annoying) an additional internal user has to be created in the configuration database (while working on the local machine as root you are authenticated over a unix domain socket). I mean, I had to creat an administration user during the installation of the service but apparently this only for the main database...
The password in the configuration can be hashed as usual - but strangely it does only accept hashes of some passwords (a hashed version of "123456" is accepted but not hashes of different password, I mean what the...?) so I have to use a single plaintext password... (secure password hashing works for normal user and normal admin accounts).
But even worse are the default logging options: By default (atleast on Debian) the log level is set to DEBUG. Additionally if slapd detects optimization opportunities it writes them to the logs - at least once per connection, if not per query. Together with an application that did alot of connections and queries (this was not intendet and got fixed later) THIS RESULTED IN 32 GB LOG FILES IN ≤ 24 HOURS! - enough to fill up the disk and to crash other services (lessons learned: add more monitoring, monitoring, and monitoring and /var/log should be an extra partition). I mean logging optimization hints is certainly nice - it runs faster now (again, I did not do any benchmarks) - but ther verbosity was way too high.
The worst parts are the error messages: When entering a query string with a syntax errors, slapd returns the error code 80 without any additional text - the documentation reveals SO MUCH BETTER meaning: "other error", THIS IS SO HELPFULL... In the end I was able to find the reason why the input was rejected but in my experience the most error messages are little bit more precise.2 -
I was once trying to create a video player since the format was not supported by the browsers today, so i started to download the video files from the server. Only to discover a bug in my code was trying to download a video file from the server. This was way to much for the server to handle and caused it to crash. People where running in the building to get the stream back up, i sat there silently and killing my browsers process. Since then i always test my request from local if they are okay before downloading a file from a server
-
@dfox is it a known issue to have crashes on iOS when visiting the notifications tab?
If not here’s what I can gather: When opening the notifications menu, app will crash after the devRant logo wobbles 3-4 times. After reopening the app and repeating the process about 3-5 times, the app will load into the notifications menu. However, when tapping on any individual notification, it will often crash after the same 3-4 wobbles of the devRant logo.
Now, I’m not a mobile developer (Nor a react developer, if you guys are using react) but here’s some things that I would think might cause this:
- An unhandled exception?
- If you use multiple server instances, they may be out of sync?
Not sure what it could be tbh. But I hope this helps, keep up the great work!7 -
I recoded a REST endpoint that transfers large amounts of data from our db using a streaming response so it doesn't crash the server...
Pretty easy... Mostly just needed someone that knew wtf it was or has a bit of curiosity and asks questions... rather than just keep on doing what everyone else is doing...
Who hasn't seen logs updating in near real time in TeamCity, Jenkins... for the last 5yrs+... No one else ever wondered how it's done?
So yes solving a production issue with old technology and being called a genius... I guess is pretty satisfying?
Top Tags
Weekly Rant
View