Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "disruption"
-
Buzzword dictionary to deal with annoying clients:
AI—regression
Big data—data
Blockchain—database
Algorithm—automated decision-making
Cloud—Internet
Crypto—cryptocurrency
Dark web—Onion service
Data science—statistics done by nonstatisticians
Disruption—competition
Viral—popular
IoT—malware-ready device15 -
Had this recently with a client, mysql server of one of our shared hosting servers went down:
Senior engineer 1: heads up guys, mysql of {server name} is down, working on it! *calls second engineer in*
Support people: thanks for letting know! (in case clients call about it)
*triiiingggg*
Me: good afternoon, how can I help you?
Client: this site which we manage for a shared customer says it can't connect to the database...
M: is it hosted on {server name of mysql problems}
C: yes.
M: there's a mysql disruption there right now, we're working on it!
C: *starts guilt tripping me about thy they chose us for stability reasons and now this happens*
M: sir, I can't change this situation so you can go on and on about that but it's not going to help anyone.
C: okay, so what can I tell my client?
M: you can tell that we have a mysql server disruption right now and are working to fix it as soon as possible!
C: and what am I going to tell my client if they don't accept that answer?
M: you can tell that we are fixing this disruption as soon as possible.
C: yes you said that but what if they don't accept that answer, what am I going to tell them THEN?!
M: Listen, sir. We have a disruption right now. It's not fun but whether I tell this by writing it to you in a fairy tail or shout it at you, it's not going to make a difference.
We have a disruption and we are working on i....
*click*
Well, fuck you too.7 -
The next person who calls the server disruption/emergency line for something that is NOT related to a server wide issue/outage is going to get a rusty pipe with fucking sambal up their fucking ass.
I am so fucking done with this bullshit.11 -
Me (to friend): So all your information these days is stored in the cloud.
Friend: Yeah I know that's crazy, huh!?
Me: Yeah!
Friend: I wonder if there's any disruption of the data when planes fly through.
Me: What do you mean?
Friend: Like when a cloud breaks when a plane move through it since we store our data in the precipitation layer. Nikola Tesla would be so proud.
Me: Uh... The fuck?
Me (thinking to myself): maybe he's just joking...4 -
--- GitHub 24-hour outage post mortem ---
As many of you will remember; Github fell over earlier this month and cracked its head on the counter top on the way down. For more or less a full 24 hours the repo-wrangling behemoth had inconsistent data being presented to users, slow response times and failing requests during common user actions such as reporting issues and questioning your career choice in code reviews.
It's been revealed in a post-mortem of the incident (link at the end of the article) that DB replication was the root cause of the chaos after a failing 100G network link was being replaced during routine maintenance. I don't pretend to be a rockstar-ninja-wizard DBA but after speaking with colleagues who went a shade whiter when the term "replication" was used - It's hard to predict where a design decision will bite back and leave you untanging the web of lies and misinformation reported by the databases for weeks if not months after everything's gone a tad sideways.
When the link was yanked out of the east coast DC undergoing maintenance - Github's "Orchestrator" software did exactly what it was meant to do; It hit the "ohshi" button and failed over to another DC that wasn't reporting any issues. The hitch in the master plan was that when connectivity came back up at the east coast DC, Orchestrator was unable to (un)fail-over back to the east coast DC due to each cluster containing data the other didn't have.
At this point it's reasonable to assume that pants were turning funny colours - Monitoring systems across the board started squealing, firing off messages to engineers demanding they rouse from the land of nod and snap back to reality, that was a bit more "on-fire" than usual. A quick call to Orchestrator's API returned a result set that only contained database servers from the west coast - none of the east coast servers had responded.
Come 11pm UTC (about 10 minutes after the initial pant re-colouring) engineers realised they were well and truly backed into a corner, the site was flipped into "Yellow" status and internal mechanisms for deployments were locked out. 5 minutes later an Incident Co-ordinator was dragged from their lair by the status change and almost immediately flipped the site into "Red" status, a move i can only hope was accompanied by all the lights going red and klaxons sounding.
Even more engineers were roused from their slumber to help with the recovery effort, By this point hair was turning grey in real time - The fail-over DB cluster had been processing user data for nearly 40 minutes, every second that passed made the inevitable untangling process exponentially more difficult. Not long after this Github made the call to pause webhooks and Github Pages builds in an attempt to prevent further data loss, causing disruption to those of us using Github as a way of kicking off our deployment processes (myself included, I had to SSH in and run a git pull myself like some kind of savage).
Glossing over several more "And then things were still broken" sections of the post mortem; Clever engineers with their heads screwed on the right way successfully executed what i can only imagine was a large, complex and risky plan to untangle the mess and restore functionality. Github was picked up off the kitchen floor and promptly placed in a comfy chair with a sweet tea to recover. The enormous backlog of webhooks and Pages builds was caught up with and everything was more or less back to normal.
It goes to show that even the best laid plan rarely survives first contact with the enemy, In this case a failing 100G network link somewhere inside an east coast data center.
Link to the post mortem: https://blog.github.com/2018-10-30-...6 -
Been awake since 4 am (an hour ago) for a disruption and finally called a fellow engineer because I can't fix this.
It's starting to get light.
There goes my night 😞4 -
Our website once had it’s config file (“old” .cgi app) open and available if you knew the file name. It was ‘obfuscated’ with the file name “Name of the cgi executable”.txt. So browsing, browsing.cgi, config file was browsing.txt.
After discovering the sql server admin password in plain text and reporting it to the VP, he called a meeting.
VP: “I have a report that you are storing the server admin password in plain text.”
WebMgr: “No, that is not correct.”
Me: “Um, yes it is, or we wouldn’t be here.”
WebMgr: “It’s not a network server administrator, it’s SQL Server’s SA account. Completely secure since that login has no access to the network.”
<VP looks over at me>
VP: “Oh..I was not told *that* detail.”
Me: “Um, that doesn’t matter, we shouldn’t have any login password in plain text, anywhere. Besides, the SA account has full access to the entire database. Someone could drop tables, get customer data, even access credit card data.”
WebMgr: “You are blowing all this out of proportion. There is no way anyone could do that.”
Me: “Uh, two weeks ago I discovered the catalog page was sending raw SQL from javascript. All anyone had to do was inject a semicolon and add whatever they wanted.”
WebMgr: “Who would do that? They would have to know a lot about our systems in order to do any real damage.”
VP: “Yes, it would have to be someone in our department looking to do some damage.”
<both the VP and WebMgr look at me>
Me: “Open your browser and search on SQL Injection.”
<VP searches on SQL Injection..few seconds pass>
VP: “Oh my, this is disturbing. I did not know SQL injection was such a problem. I want all SQL removed from javascript and passwords removed from the text files.”
WebMgr: “Our team is already removing the SQL, but our apps need to read the SQL server login and password from a config file. I don’t know why this is such a big deal. The file is read-only and protected by IIS. You can’t even read it from a browser.”
VP: “Well, if it’s secured, I suppose it is OK.”
Me: “Open your browser and navigate to … browse.txt”
VP: “Oh my, there it is.”
WebMgr: “You can only see it because your laptop had administrative privileges. Anyone outside our network cannot access the file.”
VP: “OK, that makes sense. As long as IIS is securing the file …”
Me: “No..no..no.. I can’t believe this. The screen shot I sent yesterday was from my home laptop showing the file is publicly available.”
WebMgr: “But you are probably an admin on the laptop.”
<couple of awkward seconds of silence…then the light comes on>
VP: “OK, I’m stopping this meeting. I want all admin users and passwords removed from the site by the end of the day.”
Took a little longer than a day, but after reviewing what the web team changed:
- They did remove the SQL Server SA account, but replaced it with another account with full admin privileges.
- Replaced the “App Name”.txt with centrally located config file at C:\Inetpub\wwwroot\config.txt (hard-coded in the app)
When I brought this up again with my manager..
Mgr: “Yea, I know, it sucks. WebMgr showed the VP the config file was not accessible by the web site and it wasn’t using the SA password. He was satisfied by that. Web site is looking to beat projections again by 15%, so WebMgr told the other VPs that another disruption from a developer could jeopardize the quarterly numbers. I’d keep my head down for a while.”8 -
Working on a (PHP based) monitoring system and currently writing rules/testing stuff.
I wrote some rules which check a few pages of a dutch site displaying if certain bigger services have disruptions and I am pulling the pages of a few ones I'd like to be notified about when they have issues.
Started the engine and received an alert about one big ISP over here from the monitoring system. Didn't believe that it would work right away so went to check that specific disruption page and...... they have a disruption right now!
IT FUCKING WORKS.
Good monitoring system 😊 *pats system*5 -
Scheduled devRant maintenance - I'm going to be upgrading some infrastructure later and there will be some downtime, probably about 15 minutes, around 9pm EDT. Apologies for the inconvenience and devRant disruption :) It will help with working towards an even more stable service in the future.
Feel free to let me know if you have any questions!11 -
Just woke up and had a shower. I can hardly keep my fucking eyes open.
Only three nights to go. Yay for being the on call/disruption engineer 😑7 -
About everyone around me right now: OH MY GOD WHATSAPP HAS A DISRUPTION!!!!!!
Me: ahh, finally some good news today 😊10 -
I'm wondering, what's you guys/gals/linux kernels/however you identify yourself 'superpower'?
I think that nearly everyone has something which can be extremely useful (maybe not healthy) and which not many other people you know have.
In my case it's that i can manage with extremely variable sleep patterns and when needed, I can sleep very short for days in a row (3-4 hours a night) and I'm all good. Nearly all friends/family that I have NEED regular sleep patterns + at least 8 hours of sleep but i can very much manage without those. Very useful when having disruption service and stuff.
Please post yours in the comments if you're comfortable with that!41 -
Long rant ahead.. so feel free to refill your cup of coffee and have a seat 🙂
It's completely useless. At least in the school I went to, the teachers were worse than useless. It's a bit of an old story that I've told quite a few times already, but I had a dispute with said teachers at some point after which I wasn't able nor willing to fully do the classes anymore.
So, just to set the stage.. le me, die-hard Linux user, and reasonably initiated in networking and security already, to the point that I really only needed half an ear to follow along with the classes, while most of the time I was just working on my own servers to pass the time instead. I noticed that the Moodle website that the school was using to do a big chunk of the course material with, wasn't TLS-secured. So whenever the class begins and everyone logs in to the Moodle website..? Yeah.. it wouldn't be hard for anyone in that class to steal everyone else's credentials, including the teacher's (as they were using the same network).
So I brought it up a few times in the first year, teacher was like "yeah yeah we'll do it at some point". Shortly before summer break I took the security teacher aside after class and mentioned it another time - please please take the opportunity to do it during summer break.
Coming back in September.. nothing happened. Maybe I needed to bring in more evidence that this is a serious issue, so I asked the security teacher: can I make a proper PoC using my machines in my home network to steal the credentials of my own Moodle account and mail a screencast to you as a private disclosure? She said "yeah sure, that's fine".
Pro tip: make the people involved sign a written contract for this!!! It'll cover your ass when they decide to be dicks.. which spoiler alert, these teachers decided they wanted to be.
So I made the PoC, mailed it to them, yada yada yada... Soon after, next class, and I noticed that my VPN server was blocked. Now I used my personal VPN server at the time mostly to access a file server at home to securely fetch documents I needed in class, without having to carry an external hard drive with me all the time. However it was also used for gateway redirection (i.e. the main purpose of commercial VPN's, le new IP for "le onenumity"). I mean for example, if some douche in that class would've decided to ARP poison the network and steal credentials, my VPN connection would've prevented that.. it was a decent workaround. But now it's for some reason causing Moodle to throw some type of 403.
Asked the teacher for routers and switches I had a class from at the time.. why is my VPN server blocked? He replied with the statement that "yeah we blocked it because you can bypass the firewall with that and watch porn in class".
Alright, fair enough. I can indeed bypass the firewall with that. But watch porn.. in class? I mean I'm a bit of an exhibitionist too, but in a fucking class!? And why right after that PoC, while I've been using that VPN connection for over a year?
Not too long after that, I prematurely left that class out of sheer frustration (I remember browsing devRant with the intent to write about it while the teacher was watching 😂), and left while looking that teacher dead in the eyes.. and never have I been that cold to someone while calling them a fucking idiot.
Shortly after I've also received an email from them in which they stated that they wanted compensation for "the disruption of good service". They actually thought that I had hacked into their servers. Security teachers, ostensibly technical people, if I may add. Never seen anyone more incompetent than those 3 motherfuckers that plotted against me to save their own asses for making such a shitty infrastructure. Regarding that mail, I not so friendly replied to them that they could settle it in court if they wanted to.. but that I already knew who would win that case. Haven't heard of them since.
So yeah. That's why I regard those expensive shitty pieces of paper as such. The only thing they prove is that someone somewhere with some unknown degree of competence confirms that you know something. I think there's far too many unknowns in there.
Nowadays I'm putting my bets on a certification from the Linux Professional Institute - a renowned and well-regarded certification body in sysadmin. Last February at FOSDEM I did half of the LPIC-1 certification exam, next year I'll do the other half. With the amount of reputation the LPI has behind it, I believe that's a far better route to go with than some random school somewhere.25 -
Pomodoro timers work only when the task at hand is boring. Yesterday, I worked non-stop on some task I greatly enjoyed(Rust embedded) and I would get distracted if there was some disruption because some overrated technique.6
-
!rant
I didn’t mean to be all “notice me senpai” but it’s nice when an Architect tells you “I wish we would have though of that years ago” when you demo your new app.1 -
The railway service in Sydney is called CityRail.
Fuck CityRail!
It's so unreliable that service disruption is to be expected when the weather is too hot, too cold, too wet, or too dry. If it rains, some services will be cancelled while the rest face delays.
Fuck CityRail!
The rail network is so dilapidated it costs the government millions of dollars each year for maintenance, despite charging a fare that is one of the most expensive.
Fuck CityRail!
Now I'm typing this because I've be stuck on this train for the last hour and there little hope for me to reach home before fucking MIDNIGHT!
FUCK CITYRAIL!
They initially offered replacement service but later retracted.
Fuck CityRail!8 -
Email from a department mgr regarding a sharepoint site we inherited (lots of custom javascript, XLS, etc, stuff we didn't write)
Dan: "The department filter isn't showing up when I select the 'Logistics and Support' department. Was this caused by the changes you guys made? Its causing a major disruption in our processes and need it fixed ASAP."
Me: "Those changes went out almost two months ago and all the filters were working fine, at least that is what you told me when you tested it."
Dan: "I thought so, but its not working. It has probably been broken ever since you made those changes so I filed a corrective action ticket against your department for not following the documented deployment and testing processes"
Me: "Really? We've been over this. Its your department that is responsible for that sharepoint site. Previous developers hacked javacript together to make it all work, but I'm sure its something simple."
Dan: "Great. I'll start putting together a root-cause analysis to determine which of your processes we need to address."
Start looking at the javascript and found the issue..
if (dept === "Logistics & Support") {
$('deptFilter').show();
}
else {
$('deptFilter').hide();
}
Me: 'Found the issue. Did you rename the logistics department?'
Dan: 'No'
Me: 'To show or hide the filter, the code was looking for "Logistics & Support", someone changed the title to "Logistics and Support"'
Dan: "Well...I guess I did that yesterday...but I didn't change the name, just that stupid character. That shouldn't make any difference."
Me: "I can fix that right now. Are you going to need more information for your root cause analysis?"
Dan: "No, I think we're good. Thanks."1 -
Companies are laying workers off these days like it's nothing.
My company dismissed (just within my department) 90 out of 129 workers — that messed me up.
However, though, my team experienced just a partial effect: 2 Senior devs are needed out of 3 — mid and junior devs are going. I literally had to re-interview for my role again, since they need to keep only 2 senior engineers.
To cut the long story short, I was the first selected candidate — grateful I still have my job. But, I'm sad to see the disruption.
I understand that at the end of the day, it's all business, but mehnnn ...
Is anyone going through the same thing here?
How y'all coping?2 -
Curse you Cyclone Debbie. Ruining my plans. I was gonna get things done, but no, just HAD TO HAVE A CYCLONE INSTEAD1
-
I formatted my entire hard drive, installed an unlicensed windows 8.1 and ran a shit ton of cracked patching on it; even went hunting for a fuckload of drivers, just so I don't have to use Windows 10.
"But it's not so bad" you say. Well, merry Christmas to you and your six hours of unskippable, non-disablebale updates per week, strategically placed to cause maximum disruption to your work, my friends.11 -
So, for the last year or so, we've been playing with a natural language A.I.
The goal was to predict port, truck and rail service disruption due to social unrest.
The trick here is that our AI would "read between the lines" of today's news articles and spit out keywords that were likely to appear in near future articles, thus giving us an early warning before some union or army start blockading roads.
It... did not work as intended. But some very weird results came out.
Apparently, we made a robotic "kid that screams that the emperor has no clothes", yielding unlikely (but somewhat expected) keywords when fed collections of articles.
We gave it marketing content about our company. It replied "high suicide rate".10 -
You know it sucks when you log in your PC first thing in the morning and have to wait for 1 hr before it gets to show your desktop whilst your energy automatically deflates.
I could kiss my productivity levels goodbye. -
That feeling when the cluster your ovh vps runs on goes down the exact moment your application is supposed to go live, and everyone is blaming you :|3
-
Well fuck Amazon. I am trying to get into my account because for some fucking reason they say my payment method is faulty while they actually write off the subscription of prime of it. But to get into my account I need to login again with 2FA as I have that turned it on. So far so good. But since it's an old phone number I can't login. Well just change the phone number wouldn't you think? Well yes but to change the phone number I need to login in with the old phone number to which I have no longer access 🤦♂️. Eventually found a phone number I could call. I get a lovely lady on the phone which guides me to resetting my password but for that, you guessed it, I need to do the 2FA again. I get send through to the next person as she can't change it for me because of privacy reasons (oh well). That guy first askes the last 4 numbers of my creditcard like 5 times because he can't remember it (write it the fuck down then asshole) then he starts mistaking the 6 for 9 (like how the fuck do you do that) and then the text messages don't come in while I am on the phone with him which he tries to blame to my service provider because they would block Amazon (like why would they do that?). But since I got a text message of them 15 min before I shot that down quickly. Then he finally admitted that they might have a disruption going on. So I think we'll fine I'll just ask my question to him how it's possible that Prime stops working as I am watching it because my payment method is faulty according to them (but manage to write off the subscription) and he starts talking just shit. Just admit that you don't know and connect me to someone who does know how that can happen. In the the end I just hung up because I knew I wasn't getting anywhere with this guy and don't you know it, as I start writing this the text messages come in. Problem solved you would say just out that number in the website and you can change your phone number. Well no because I have to tell the number to the guy who I hung up with because the texts weren't coming in 😒. Now I should call them back but I think I'll wait till tomorrow hopefully the day shift will be a bit more knowledgeable on how shit works and can actually remember 4 digits.2
-
Not having to constantly engineer software. Honestly, this long-promised disruption is like nuclear fusion. Always 10-50 years away.
-
Bwhahahah! Even after the excitement, business disruption, unpleasantness and pain, GDPR fails at its one job
https://newstatesman.com/science-te...1