Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "sysadmin hell"
-
Navy story continued.
And continuing from the arp poisoning and boredom, I started scanning the network...
So I found plenty of WinXP computers, even some Win2k servers (I shit you not, the year was 201X) I decided to play around with merasploit a bit. I mean, this had to be a secure net, right?
Like hell it was.
Among the select douchebags I arp poisoned was a senior officer that had a VERY high idea for himself, and also believed he was tech-savvy. Now that, is a combination that is the red cloth for assholes like me. But I had to be more careful, as news of the network outage leaked, and rumours of "that guy" went amok, but because the whole sysadmin thing was on the shoulders of one guy, none could track it to me in explicit way. Not that i cared, actually, when I am pissed I act with all the subtleness of an atom bomb on steroids.
So, after some scanning and arp poisoning (changing the source MAC address this time) I said...
"Let's try this common exploit, it supposedly shouldn't work, there have been notifications about it, I've read them." Oh boy, was I in for a treat. 12 meterpreter sessions. FUCKING 12. The academy's online printer had no authentication, so I took the liberty of printing a few pages of ASCII jolly rogers (cute stuff, I know, but I was still in ITSec puberty) and decided to fuck around with the other PCs. One thing I found out is that some professors' PCs had the extreme password of 1234. Serious security, that was. Had I known earlier, I could have skipped a TON of pointless memorising...
Anyway, I was running amok the entire network, the sysad never had a chance on that, and he seemed preoccupied with EVERYTHING ELSE besides monitoring the net, like fixing (replacing) the keyboard for the commander's secretary, so...
BTW, most PCs had antivirus, but SO out of date that I didn't even need to encode the payload or do any other trick. An LDAP server was open, and the hashed admin password was the name of his wife. Go figure.
I looked at a WinXP laptop with a weird name, and fired my trusty ms08_067 on it. Passowrd: "aaw". I seriously thought that Ophcrack was broken, but I confirmed it. WTF? I started looking into the files... nothing too suspicious... wait a min, this guy is supposed to work, why his browser is showing porn?
Looking at the ""Deleted"" files (hah!) I fount a TON of documents with "SECRET" in them. Curious...
Decided to download everything, like the asshole I am, and restart his PC, AND to leave him with another desktop wallpaper and a text message. Thinking that he took the hint, I told the sysadmin about the vulnerable PCs and went to class...
In the middle of the class (I think it was anti-air warfare or anti-submarine warfare) the sysad burst through the door shouting "Stop it, that's the second-in-command's PC!".
Stunned silence. Even the professor (who was an officer). God, that was awkward. So, to make things MORE awkward (like the asshole I am) I burned every document to a DVD and the next day I took the sysad and went to the second-in-command of the academy.
Surprisingly he took the whole thing in quite the easygoing fashion. I half-expected court martial or at least a good yelling, but no. Anyway, after our conversation I cornered the sysad and barraged him with some tons of security holes, needed upgrades and settings etc. I still don't know if he managed to patch everything (I left him a detailed report) because, as I've written before, budget constraints in the military are the stuff of nightmares. Still, after that, oddly, most people wouldn't even talk to me.
God, that was a nice period of my life, not having to pretend to be interested about sports and TV shows. It would be almost like a story from highschool (if our highschool had such things as a network back then - yes, I am old).
Your stories?8 -
One of our newly-joined junior sysadmin left a pre-production server SSH session open. Being the responsible senior (pun intended) to teach them the value of security of production (or near production, for that matter) systems, I typed in sudo rm --recursive --no-preserve-root --force / on the terminal session (I didn't hit the Enter / Return key) and left it there. The person took longer to return and the screen went to sleep. I went back to my desk and took a backup image of the machine just in case the unexpected happened.
On returning from wherever they had gone, the person hits enter / return to wake the system (they didn't even have a password-on-wake policy set up on the machine). The SSH session was stil there, the machine accepted the command and started working. This person didn't even look at the session and just navigated away elsewhere (probably to get back to work on the script they were working on).
Five minutes passes by, I get the first monitoring alert saying the server is not responding. I hoped that this person would be responsible enough to check the monitoring alerts since they had a SSH session on the machine.
Seven minutes : other dependent services on the machine start complaining that the instance is unreachable.
I assign the monitoring alert to the person of the day. They come running to me saying that they can't reach the instance but the instance is listed on the inventory list. I ask them to show me the specific terminal that ran the rm -rf command. They get the beautiful realization of the day. They freak the hell out to the point that they ask me, "Am I fired?". I reply, "You should probably ask your manager".
Lesson learnt the hard-way. I gave them a good understanding on what happened and explained the implications on what would have happened had this exact same scenario happened outside the office giving access to an outsider. I explained about why people in _our_ domain should care about security above all else.
There was a good 30+ minute downtime of the instance before I admitted that I had a backup and restored it (after the whole lecture). It wasn't critical since the environment was not user-facing and didn't have any critical data.
Since then we've been at this together - warning engineers when they leave their machines open and taking security lecture / sessions / workshops for new recruits (anyone who joins engineering).26 -
Back in my sysadmin days we had an IT zoo to look after. And I mean it... Linux side was allright, but unix.... Most unices were no longer supported. Some of their vendors' companies were already long gone.
There was a distant corner in our estate known to like 2 people only, both have left the company long ago. And one server in that corner went down. It took 2 days to find any info about the device. And connecting to it looked like:
1 ssh to a jumpbox #1
2 ssh to a jumpbox #2
3 ssh to a dmz jumpbox
4 ssh to an aix workload
5 fire up a vnc server
6 open up a vnc client on my workstation, connect to than vnc server [forgot to mention, all ssh connections had to forward a vnc port to my pc]
7 in vnc viewer, open up a terminal
8 ssh to hp-uxes' jumpbox
9 ssh to the problematic hp-ux
.....6 -
Wk1:, Client wants stack deployed to AWS in a day... Does it in 30 minutes...
Wk2: client complains about cost of service so requests I downsize the stack they asked for... Does so... Just to save some £££ at loss of any DR or HA capability
Wk3: Client wants stack moved to a different AWS account just cos... Advised could cause issues... Client says carry on. Migrates to account as requested on the days they requested.
Wk4: client complains that said migration caused issues and that proper change control wasn't followed..... That was never informed on..
Wk5: issue discovered to actually be network fault linked to clients wanting every £££ saved in AWS... And as the stack specialist I still have to write up a summary and findings?
Wow just get a decent AWS admin will ya....3 -
Another incident which made a Security Researcher cry
[ NOTE : Check profile to read older incidents ]
-----------------------------------------------------------
So this all started when I was at my home (bunked the office that day xD) and I got a call from a..... Let's call him Fella as I always do . So here we go . And yeah , our Fella is a SysAdmin .
-----------------------------------------------------------
Fella - Hey man sup!
Me - Good going mate , bunked the office , weather's nice , gonna spend time with my girl today . So what's goinon?
Fella - Bruh my network sharing folders ain't working no more .
Me - Did you changed or modified anything?
Fella - Nope
Me - Okay , gimme your login creds lemme check .
Fella - Check your inbox *texts me the credentials*
*I logged in and what I'm seeing is that server runs on Windows2008R2 , checked the event logs , everything's fine and all of a sudden what I found is fucking embarrassing , this wise man closed SMB service*
Me - Did you closed SMB service?
Fella - Yeah
Me - You know what it does?
Fella - Yeah it's a protocol , I turned it off to protect the server from Wannacry .
Me - Fuckerrrr!!!!! Asshole dumbass you fuckin piece of Dodo's shit!! SMB is the service responsible for files and network sharing!!!
Fella - But....I just wanted protection
Me - 😭😭😭
*A long conversation continues with a lot of specially made words to decrease the rate of frustration which I used already*
Fella - Okay I'm turning it on .
Me - Go on....... Asshole
Fella - It worked! Thanks a lot bro
Me - Just leave me and my soul away from evil and hang up .
*Now the question is , who the hell gives them the post of SysAdmin? While thinking this question , I almost thought of committing suicide but then my girl came with coffee and my rubber duck*1 -
You know the worst thing about being a freelancer? You're expected to wear every fucking hat and you don't get normal hours.
Over the past few days I have been working with a client of a client attampting to fix his server. He's running CentOS on VMWare and somehow ended up breaking the system.
Upon inspection there was no way to fix his system remotely. It wouldn't even boot in recovery mode. So we've been attempting to recover his data so that we can reinstall CentOS and not have to start completely from scratch.
So for the past 3 days straight I have been remotely logging in to a Debian Live CD and manually sending folders to a FTP server of his. He has somewhere close to 30 sites on this server, and upwards of 1 million files in total.
Yesterday either the system freaked out or he did something, but the entire fucking system stopped responding which forced me to reboot it, reinsert the live CD, reinstall evertything, and re-mount his broken systems drives.
Here we are 3 days in, we're still not done, and I'm getting slightly pissy because if you don't know Linux well enough to fix this shit yourself, you shouldn't be acting as your own sysadmin for 30+ sites.
Also, backups are a thing right? VMWare also has snapshots. I know the extra storage isn't cheap, but it's a hell of a lot cheaper than paying soemone like me $35/hr to go and fix all of your shitty mistakes.2 -
I started to hate programming.
I started with a lot of enthusiasm 11 years ago up to become in 2 years a full stack dev, a sysadmin and had also my fair share of technical assistance on every device plus hardware experience mounting hardware like cctvs, routers, extenders, industrial printers and so on. At the time you actually had the tools to solve problems and had to crack your head and pull hairs to solve stuff and people actually was developing solution and frameworks that solved stuff.
Today I can't stand anything.
Every midschooler feels entitled to release a framework that is announed as the next cure for cancer. Web dev once was thin and simplistic, now simplicity is considered a bug and not a feature.
I'm working on an angular project for the nth time and the whole environment is a clusterfuck of problems held togheter with kids glue.
Someone did a tool/framework for everything but most of it is barely well tested or mature.
Just to start this project we had to know, beside html/css/js techs like Angular, Kafka, Kubernetes, Docker, git, Lit, npm/node, mysql/sql server, webpack/grunt and the hell that it brings, C#/Asp.NET/MVC/WebAPI, and so on, the list is long.
DAMN. Making a simple page which shows a tabbed view with some grids requires you to know a whole damn stack of technologies that need to cooperate togheter.
It's 10x more complex and I actually find it much less productive than ever.
But what bugs me most, is that 90% of that stuff is bug ridden, has some niche use case or hidden pitfall and stuff because with this whole crap of "hey we put on github you open a ticket" they just release spaghetti code and wait for people to do the debug for them.
Angular puts out a version every 2 days and create destructive updates.
I am so tired that I spend most of my 8hrs binging youtube vids in despair to procrastinate work.
I liked to do this once....13 -
>Be client
>Have an issue with incredibly slow webpage load time
>Blame memcache issues
So... I look into the problem. Yes, the page either loads up fast, or times out. So, into the logs I go. Webserver is fine (except the timeout), PHP though... Error log is fine (just notices), but slow log shows the issue is the database (of course... its always the database... ugh)
So, checking the database, there is one ugly query that seems to be an issue. 5 joins and a huge where condition.
So I run EXPLAIN on the query and... Proceed to bang my head against the wall.
OF COURSE ITS SLOW YOU FU******, NONE OF YOUR TABLES HAVE ANY INDEXES.
What do they expect when the database has to always go down the whole table and do everything in memory, until it runs out and has to dump it all on disk and work with it there.
Ugh... Some clients... -
I got a median-pay front-end job through a contractor (after a contract from hell...but yeah I didn't learn...) and I'm getting zero assignments after a month and nobody seems to know what my role is.
I'm one month in, and every week I have to email my boss to remind her to sign my paycheck, which is stressful because I'm charging for my time because my assignments are like "Research this" or "look at this Wordpress theme or brand guide". The team never communicates but once a week, and I'm beginning to believe that I'm not a good fit for the team because they are impossible to get a hold of and the sysadmin won't give me access to anything even when I CC my boss. (I don't want to grief this guy...) Despite this, I've been told privately by higher-ups on a few occasions that they plan to hire me full time by November...
My SO thinks that the reason people are so dodgy toward me is because they literally do nothing and I'm breaking the flow of that by asking for things. I'm used to agency output, which can be toxic and where everything is 'due yesterday', and I'm watching this team work on assignments ten times slower than normal. ("You want to change a phone number on a website footer? You'll get it next week...maybe." I can't step on toes because I don't have access...) I'm perfectly fine with having to wear several hats at a low-stress job, but I can't even get my first assignment and I'm still being asked who I am in weekly meetings, or asked things like, "Would you even be willing to relocate here?" (I actually live DOWN THE STREET FROM THE OFFICE!! WHY DO I HAVE TO BE REMOTE? Why am I being asked this question?) It feels like my boss impulse hired me, with zero input from the team, and had no real reason to hire me in the first place...
It could also be another issue: Yeah, my experience is in PHP/JS/React, "but here have a seven year old .NET project and a company laptop with zero documentation and make this form import data to a database we know nothing about." Lead dev won't even talk to me.
I feel like a joke.2 -
I already wrote a rant about this yesterday, but since I'm a sysadmin trying to convert to dev.. I dunno, maybe it's not a bad idea to muddy the waters a bit and talk about why not to be a sysadmin.
Personally I think it's that the perceived barrier to entry is just too high, while it isn't. You don't need a huge Ceph cluster and massive servers when you're just starting out. Why overbuild an appliance like that if it's gonna start out at maybe 5 requests a minute?
Let's take an example - DNS servers! So there's been this guy on the bind-users mailing list asking how to set up a DNS server on 2 public servers, along with a website. Nothing special I guess - you can read the thread here: https://0x0.st/ZY-d. Aside from the question being quite confusing, there was advice to read RFC's, get a book, read the BIND ARM, etc etc. And the person to deny this? No one less than Stephane Bortzmeyer, one of the people who works for nic.fr (so he maintains the .fr TLD) and wrote some of those RFC's as part of the DNSOP working group in the IETF. As for valid reasons to set up a DNS server? Could just be to learn how the DNS works, or hell even for fun. As far as professional DNS servers go.. this (https://0x0.st/ZYo9) is the nugget that powers the K root server, one of the 13 root servers that power the root zone of the internet, aka the zone apex. 2 RJ45 connections, and a console connection. The reason why this is possible is the massive recursor networks that ISP's, Google DNS, Cloudflare DNS, Quad9, etc etc provide. Point is, you don't need huge infrastructure to run a server!
Or maybe your business needs email. How many thousands of emails per second are you gonna need to build your mail server against? How many millions will you need to store? If your business has 10 employees and all of those manage about 10k emails total.. well that's easy, 100k emails total. Per second? Hundreds of emails per second per employee? Haha, of course not. Maybe you'll see an email a minute at most. That is not to say that all email services are like this - it is true that ISP's who offer email to their customers, and especially providers like Microsoft and Google do need massive mail servers that can handle thousands of emails per second. But you are not Microsoft or Google. So yeah, focus on the parts of email that are actually hard.. and there is plenty.
Among sysadmins you have this distinction between "professional" sysadmins and homelabbers. I don't mind the distinction itself but I think both augment each other. If you've started out by jumping into a heap of legacy at an established company, you will have plenty of resources, immediately high complexity, and probably a clusterfuck right away. But you will have massive amounts of resources. If you start out with a homelab, you will have not many resources, small workloads, and something completely new for you to build and learn with. And when running a server like that, you'll probably find that the resources required are quite small, to provide you with your new services. My DHCP servers take 12MB memory each. My DNS servers hover around the 40MB mark. The mail server.. to be fair that one consumes around 150. But if you'd hear the people saying that you need huge servers.. omg you need at least a TB of RAM on your server and 72 cores, massive disks and Ceph!1!
No you don't. All that does is scaring people away and creating a toxic environment for everyone. Stop it.1 -
You know what's bullshit? CS Degrees as a requirement, even for the shittiest dev jobs.
Sorry fuckers, I don't feel like killing myself over fucking math bullcrap for 3 goddamn years just to work as an intern slave for some rich CEO who prefers to hire some guy who doesn't know shit about actual working with computers but has a degree.
And this horseshit happens only in dev jobs. Why. Are devs some fucking nuclear scientists or something? I work as sysadmin and they didn't ask me for any shit degree and I earn more than the average code monkey where I live.
Goddamn HR fuckers. May Allah take you to hell.4 -
Everyone in this team calls everything a team effort, but once I start offering my help, they be like "no, I can do it. I know more than you".
Hmm. yeah, but you (sysadmin) use jQuery and vanillajs mixed. For example: $('#hello') and document.getElementById('hello').
Also you put console.logs everywhere, I don't mind putting console.logs in development, but not in production.
Oh and he copies the libraries to every folder that needs it, so there are at least 12 jquery libs in this project and the version is not even the same. Lol.... Please slap me to death.
There is another networkadmin that calls himself a (python) developer. He doesn't agree with my simplicity.
His work (just an example, changed names but you get the idea)
"A notebook that is used by x-department"
Model: Notebook
endpoint: department-notebooks
Model: DepartmentConfigs
Endpoint: notebook-department-configs
You won't believe what he put in 'department'configs, it's literally hardware vendor, model, versions.
Like... really? What the hell you doing man?!
Just have these models for example: device, department, vendor, product, category
We do not only have notebooks, but also servers, routers, switches and more.
His argument of having configs in the name is that they do more complex things. Hmm, I don't see it in the code and the data is messed up:
Microsoft, microsoft, micro soft.
He fixed it by hardcoding it in a select box. Mickysoft isn't the only vendor, fuck you!
fuck this team, fuck these people
Another fucking rant, a story was assigned to me. But that stupid fake developer worked on it immediately and message me he fixed it already. I guess he won't let me touch his baby.
Everything is just piling up. This team and people aren't fun at all.3 -
So, I work as a sysadmin junior (6 months and going), and in the past few months, I learned what my boss warned me about - Devs don't understand us admins, and we don't understand the devs.
We have this huge client who is about to migrate to our company (We do mostly server managment/Housing/Renting), and I am so gald I don't have to work on the migration myself!
Just hearing what the company devs say makes me facepalm: No, it won't work. It cannot work on just 3 machines (They use like... 20 in total), no, we won't get rid of our docker swarm, that's essential (Doing the absolute minimum in their infrastructure, just a fancy buzzword to lure people on. Though they've spent like 2 years developing the app that uses it, so they my not want to give it up).
I kid you not, once, they replied to an email that contained the phrase "To be afraid of/worried about" something during the migration, that something could break, not work, be unstable. 7 times.
Might not sound as bad, but it was a rather short mail, and when they're so afraid of everything, its kinda hard to cooperate with them.
My colleague literally spent this entire week mapping out /their/ infrastructure, because they were unable to provide us with the description themselves.
And as a cherry on top, they sent us a "graph" of relationships of all the parts of their infrastructure that was this jumbled mess of rectangles and arrows. Oh, and half of all the machines were not even in the graph at all! Stating that "We also have all this, but I really don't know how to ilustracte the interactions anymore"
Why do companies like that exist? If you build an infrastructure yourself, shouldn't at least someone know exactly how it works?1 -
So I wanted to do a quick test before going to dinner and now I'm stuck on waiting for this f*cking cloud provider to start my container.
"Provisioning 20 minutes" WHAT THE HELL!? After 20 (TWENTY) minutes my container still hasn't started!?
Is it a joke? Is some sysadmin spying on me and making me wait on purpose? What the f*1