A story about how a busy programmer became responsible for training interns.

So I was put in charge of a team of interns and had to teach them to work with Linux, coding (Bash, Python and JS) and networking overall.
None of the interns had any technical experience, skills, knowledge or talent.
Furthermore the task came to me as a surprise and I didn't have any training plan nor the time.

Case 0:
Intern is asked to connect to a VM, see which interfaces there are and bring up the one that's down (eth1). He shuts eth0 down and is immediately disconnected from the machine, being unable to connect remotely.

Case 1:
Intern researches Bash scripting via a weird android app and after a hour or so creates and runs this function: test(){test|test&}
He fork-bombed the VM all other interns used.

Case 2:
All interns used the same VM despite the fact that I created one for each.
They saved the same ssh address in Putty while giving it different names.

Case 3:
After explicitly explaining and demonstrating to the interns how to connect to their own VMs they all connect to the same machine and attempt to create file systems, map them and etc. One intern keeps running "shutdown -r" in order to test the delay flag, which he never even included.

Case 4:
All of the interns still somehow connect to the same VM despite me manually configuring their Putty "favorites". Apparently they copy-paste a dns that one of them sent to the entire team via mail. He also learned about the wall command and keeps scaring his team members with fake warnings. A female intern actually asked me "how does the screen knows what I look like?!". This after she got a wall message telling her to eat less because she gained weight.

Case 5:
The most motivated intern ran "rm -rf" from his /etc directory.
P.S. All other interns got disconnected because they still keep using his VM.

Case 6:
While giving them a presentation about cryptography and explaining how SSH (that they've been using for the past two weeks) works an intern asked "So is this like Gmail?".
I gave him the benefit of the doubt and asked if he meant the authorization process. He replied with a stupid smile "No! I mean that it can send things!".

FML. I have a huge project to finish and have to babysit these art majors who decided to earn "ezy cash many" in hightech.

Adventures will be continued.

Had 2 days of vacation. Theoretically (plus weekend, plus 2 days) 6 days.

Worked today… At Saturday.

Some administrators forgot to properly check bandwidth limitations....

*rolls eyes*

We had a major version upgrade of some server software at Monday.

Guess why I got called...

Of course it MUST be the software upgrade.

It couldn't be the new hardware that was setup 2 weeks ago and on which a lot of "important" VMs were migrated.

*eyes roll inside till only white is visible*

The even more annoying thing is that it wasn't that hard to figure out.

Looking at monitoring, we had spikes on 20 Gbit/s (roughly 2.x Gigabyte/sec - Ethernet) connection of some server at roughly 1.9 plus Gigabyte/sec.

IO latency spikes that made the graph look like a heartbeat EKG with severe tachycardia...

*additionally to white eyes starts cursing in reverse latin*

Incompetent admin answer: Booboo that can only be your fault - the developers must investigate.

Me (just a tad more polite): Meep Meep mother fucker, get your shit together. If the software would eat that much, the network would be a niece chunk of charcoal. Plus the time (sending instead of links to monitoring pictures… guess the lazy fucktard who's brain is a vacuum didn't even bother to check it)...

NOTICE SOMETHING?!

Incompetent admin: It starts at the same time. Always.

After wasting roughly another hour of time discussing with him, I just hanged up the video call.

Called someone I knew from the admin department and turns out that - drumrolls please - the incompetent admin was someone who got recruited 3 months ago…

*turning into antichrist*

I then had a not so polite discussion about how the only competent people could take days off (all except incompetent admin were on vacation) and the seemingly incompetent fresh recruit - who by the way NEVER mentioned this - was the only one left of the admin department. Which would be bad alone, but no - he even got the 24/7 emergency support role for the whole weekend.

Sometimes this company and HR especially notoriously drive me insane...

Guess next week there will be some HR barbecue.

But yeah. After a lot of raging around we nailed it down to the traffic of backups and could fix it.

Roughly 4 hours of analysis, communication, raging and hatred.

Just one hour implementing shit.

*goozfraba*

Why is it that an issue is only critical-priority until the person who's raising the biggest fuss has to do something about it?

I was notified that a website hosted in AWS went down overnight and never came back up. I was then bombarded with email after email after email while I logged into our AWS account and poked around. I'm responsible for cloud infrastructure stuff, like VMs or virtual networking or security or whatever, not the actual applications running on said infrastructure. Once I confirmed their EC2 instance was reachable and I could login with SSH, I told them they'd have to fix their application.

They told me that they had no backend developer on their development team. I'm still getting a deluge of emails from multiple people on this team and their managers and managers' managers and so on.

"Perfectly understandable," I told them, though it was anything but. "You should probably look into obtaining one."

The emails stopped immediately. I assumed they were handling it and closed my ticket and moved on. But apparently I was wrong.

Six weeks later, the site is still down, they still have no backend dev, and I'm convinced that they were lying to me when they stressed the importance of this web app because now that it's no longer my problem, not a single person seems to care that it's still broken.

Got pulled out of bed at 6 am again this morning, our VMs were acting up again. Not booting, running extremely slow, high disk usage, etc.

This was the 6 time in as many weeks this happened. And always the marching orders were the same. Find the bug, smash the bug, get it working with the least effort. I've dumped hundreds of hours maintaining this broken shitheap of a system, putting off other duties to keep mission critical stations running.

The culprits? Scummy consultants, Windows 10 1709, and Citrix Studio.

Xen Server performed well enough, likely due to its open source origins and Centos architecture.

Whelp. DasSeahawks was good and pissed. Nothing like getting rousted out of bed after a few scant hours rest for patching the same broken system.

DasSeahawks lost his temper. Things went flying. Exorcists were dispatched and promptly eaten.

Enough. No consultants, no analysts, and no experts touched it. No phone calls, no manuals, not even a google search. Just a very pissed admin and his minion declaring blitzkrieg.

We made our game plan, moved the users out, smoked our cigs, chugged monster, and queued a gnu-metal playlist on spotify.

Then we took a wrecking ball to the whole setup. User docs were saved, all else was rm -r * && shred && summon -u Poseidon -beast Land_Cracken.

Started at 3pm and finished just after midnight. Rebuilt all the vms with RDP, murdered citrix studio (and their bullshit licenses), completely blocked Windows 10 updates after 1607, and load balanced the network.

So what do we get when all the experts are fired? Stabbed lightning. VMs boot in less than 10 seconds, apps open instantly, and server resources are half their previous usage state. My VMs are now the fastest stations in our complex, as they should be.

Next to do: install our mxgpu, script up snapshots and heartbeat, destroy Windows ads/telemetry, and setup PDQ. damn its good to be good!

What i learned --> never allow testing to go to production, consultants will fuck up your shit for a buck, and vendors are half as reliable over consultants. Windows works great without Microsoft, thin clients are overpriced, and getting pissed gets things done.

This my friends, is why admins are assholes.

"Suggest an AV/AM product, Avast refuses to install."

I do malware research as a hobby and have for a while, so I can generally spot when something's up before I even run a program. If i'm unsure about it (or know something's up and wanna see its effects for S&Gs) I throw it into one of a variety of VMs, each with a prepped, clean, standardized "testing" state.

I see no point to AV/AM products, especially as they annoy me more than anything since they can't be told not to reach into and protect VMs (thereby dirtying up my VM state, my research, crashing the VM hypervisor and generally being *really* annoying) and they like to erase samples from a *read-only, MOUNTED* VHDX.

However, normal people need them, so I usually suggest this list:
• MBAM is good and has a (relatively) low memory footprint, but doesn't have free realtime protection.
• Avast is very good as it picks up a lot, but it eats a FUCKTON of resources. It also *really* likes to crash VM hypervisors if it sees anything odd in them.
• AVG is garbage. Kill it with fire.
• Using Windows Defender is like trying to block the rain with an umbrella made of 1-ply toilet paper.
• herdProtect is amazing as it's basically a VirusTotal client but it's web-based and not currently available to be downloaded. (Existing copies still work!)
• Kaspersky. Yes, it spied on US gov't workers. No, they don't care about anyone BUT US gov't workers. Yes, it's pretty good.
• BitDefender: *sees steam game* "Is this ransomware?"

hope this helps

Linux software RAID and LVM are pretty powerful.

Bought a new server case for my home file server / VM host. 3U with 16 hotswap bays. Had 2x software RAID1 mirrors already with everything on them. Inserted 2 new disks with system running. Created new RAID10 array using these, with their mirrors as "missing".

Created new physical volume. Extended volume group into it, then used pvmove to transfer every logical volume across. Shrunk volume group to no longer use the old RAID1 array, disassembled that array, added its disks to the new array... Now just waiting for the mirror disks to sync up.

All this, with the system and several VMs still running.

And with a backup, of course ;)

If you've ever tried using Go plugins raise your hand.
If you've ever tried doing plugins in Go, raise your hand.
If you think that the following rant will be interesting, raise your hand.
If you raised your hand, press [Read More]:

This is a tale of pain and sorrow, the sorrow of discovering that what could be a wonderful feature is woefully incomplete, and won't be for a very long time...

Go plugins are a cool feature: dynamically load pre-compiled code, and interact with it in a useful and relatively performant way (e.g. for dynamically extending the capabilities of your program). So far it sounds great, I know right?

Now let me list off some issues (in order of me remembering them):
1. You can't unload them (due to some bs about dlopen), so you need to restart the application...
2. They bundle the stdlib like a regular Go binary, despite the fact that they're meant to be dynamic!
3. #2 wouldn't be so bad if they didn't also require identical versions of all dependencies in both binaries (meaning you'd need to vendor the dependencies, and also hope you are using the right Go version).
4. You need to use -trimpath or everything dies...

All in all, they are broken and no one is rushing to fix it (literally, the Go team said they aren't really supporting it currently...).

So what other options are there for making plugins in Go?

There's the Hashicorp method of using RPC, where you have two separate applications one the plugin, one the plugin server, and they communicate over RPC. I don't like it. Why? Because it feels like a hack, it's not really efficient and it carries a fear of a limitation that I don't like...

Then we come to a somewhat more clever approach: using Lua (or any other scripting language), it's well known, it's what everyone uses (at least in games...). But, it simply is too hard to use, all the Go Lua VMs I could find were simply too hard to set up...

Now we come to the most creative option I've seen yet: WASM. Now you ask "WASM!? But that's a web thing, how are you gonna make that work?" Indeed, my son, it is a web thing, but that doesn't mean I can't use it! Someone made a WASM VM for Go, and the pros are that you can use any WASM supporting language (i.e. any/all of them). Problem inefficient, PITA to use, and also suffers from the same issues that were preventing me from using Lua.

Enter Yaegi, a Go interpreter created by the same guys who made (and named) Traefik. Yes, you heard me right, an INTERPRETER (i.e. like python) so while it's not super performant (and possibly suffering from large inefficiency issues), it's very easy to set up, and it means that my plugins can still be written in Go (yay)! However, don't think this method doesn't have its own issues, there's still the problem of effectively abstracting different types of plugins without requiring too much boilerplate (a hard problem that I'm actively working on, commits coming soon). However, this still feels to be the best option.

As you can see, doing plugins in Go is a very hard problem. In the coming weeks (hopefully), I'm going to (attempt to at least) benchmark all the different options, as well as publish a library that should help make using Yaegi based plugins easier. All of this stuff will go (see what I did there 😉) in a nice blog post that better explains the issues and solutions. But until then I have some coding to do...

Have a good night(/day)!

TLDR: I need advice on reasonable salary expectations for sysadmin work in the rural United States.

I need some community advice. I’m the sysadmin at a small (35 employee) credit card processing company. I began as an intern and have now become their full time sysadmin/networking specialist. Since I was hired in January I have:

-migrated their 2007 Exchange server to Office 365

-Upgraded their ailing Windows server 2003 based architecture to 2012R2

-Licensed their unlicensed VMware ESXi servers (which they had already paid for license keys for!!!) and then upgraded them to 6.5 while preventing downtime on hosted VMs using tricky transfers and deployments (without vMotion!)

-Deployed a vCenter server to manage said ESXi servers easier

-Fixed a three month gap in their backups by implementing Veeam, and verifying its functionality

-Migrated a ‘no downtime’ fileserver to a new hypervisor host, implemented a ‘hot standby’ server as a backup kept up to date by the minute with DFS replication.

-Replaced failing hard drives in a RAID array underlying their one ‘business critical’ fileserver, which had no backups for 3 months at that time

-Reorganized Active Directory and Group Policy deployment from a nightmare spiderweb of OUs and duplicate policies

-Documented the entire old network and now the new one as I’ve been upgrading this

-Audited the developers AWS instances and removed redundant machines, optimized load balancing on front end Nginx servers, joined developer run Fedora workstations to the AD domain and implemented centralized syslog monitoring on them.

-Performed network scans and rewrote firewall exceptions to tighten security

There’s more, but you get the idea. I’ve now been tasked with taking point on an upcoming PCI audit which will be my first.

I’m being paid $16/hr US, with marginal health benefits. This is roughly $32,000 a year, before taxes.

I have two years previous work experience managing a third party Apple repair facility (SimplyMac) and every Apple certification for warranty repair and software troubleshooting. I have a two year degree in general sciences, with about 4 years of college credit (Two years of a physics education and two years of computer science after I switched focus) I’m actively pursuing a CCNA and MCSA server 2016 with exams paid for and scheduled.

I’m going into a salary negotiation in two months. What is a reasonable salary to request, from your perspective, for someone in my position?

Thanks in advance!

I have a LOT of Floppies but no floppy drive that works with my current Win7 machine, and my XP one is in multiple boxes for parts... so for now I can only stare at NetWare 4 to 4.2, WfW 3.11 and DOS 6-22.

Smart me made VMs on his XP machine and transferred them to his Win7 one.

I also have data tapes... now that is NEVER gonna work on my Win7 so... anyone knows how it could *possibly* work out?

Also... I got documentation on Compaq servers... those are nice.

AAAAAAND since I’m a huge MIDI nerd... I have a SC-55mkII hooked on the UM-ONE mkII and those shitty cans that I’m gonna switch (hopefully) soon for a nice pair of Cakewalks MA-15Ds.

Also, I’m looking for one of them 5150s because 80s IBM and since I also like keyboards... the one and only Model M.

Anyone can hook me up with a cheap one?

I am not a php dev and I have nearly 0 knowledge of php. All I know about php is that xampp is your friend and you have to write that $ everywhere. But that one day I had to setup phpLDAPadmin on apache2.
I have nothing against php, but I just don't want to have anything in common, since I'm just perfectly fine with my java.
So I had to make it work. Fought my way through different incompatible versions of php and phpldapadmin only to see "not found" on phpldapadmin.
I thought, like, wtf?? Especially when index.php of apache2 is displaying just fine? I mean, I can "edit" some php code, but configs and php setups are just something like out of my world. Tried setuping it on different vms - same result. I've buried way too many hours into this with no result. Finally I gave up and contacted a friend of mine which is like php god for me. He did same thing as I did in ~ 10 mins, but the result was the same. Tweaked some configs - same. Scratched his head, sat for 5 more minutes, did something and boom - it works!
I've asked him, what is that php magic and the answer killed me:
"Index.html was missing"

At that moment I just wanted to exit through the window. Sadly, there were no way to open it.

Yes, I am this stupid in php. But I still miss all these wasted hours...

Trying to setup a ltsp server for fun. Neve done server things before.

The server and the thin client are in VMs. So I start, install openssh, and them when I try to ssh... WHY CAN’T I FUCKING CONNECT, I CHECKED THE IP WITH IFCONFIG, oh shit, forgot to configure the vm network... so ssh works! Then I setup dhcp (I really don’t know what I’m doing, just following the tutorial), the ltsp configs thing, build the client image and then, I HAVE NO FUCKING INTERNET CONNECTION. Continues, boot up the thin client... WHY CAN’T YPU FUCKING FIND THE SERVER!! Then I realise the vm is not an ubuntu one, so delete it and make a new one... WHY DOES IT STILL DOESN’T WORK!!!!!! Oh wait forgot to connect to the network! Goes to put the network adaptor, and: wait! I don’t need NAT! So I replace the NAT by the correct network, and: Wait it lets me choose the weird thingy intel/pce thingy, oh I remember now! It said we needed the “...III FAST...”! Activate it and... IT WORKS ! !!!! CONNECTS TO THE SERVER!!! GOT THE DHCP!!! WAIT!!!!! What is THAT 🤬 TFTP LOADING THING!!!:

TFTP open timeout

🤬 YOU!!!!!!!!>>

I hope my boss learned his lesson: dd if=/dev/zero of=[hdd storing DB about VM cluster]
- is a very very bad idea...

Making a hard switch to ubuntu on my desktop at home. Getting just a teeny tiny, tad, bit: absolutely fucking livid....

Trying to learn ansible, vagrant, and docker more in depth for both work and my personal projects. All that I’ve been doing is just spinning my wheels trying to figure out the stupid fuck-mothering quirks with running this shit on Windows. Yes you absolutely can use all of these tools on a Windows box. There’s plenty of ports, patches, and workarounds. But I have spent all day trying to build a few vagrant boxes and use ansible to set them up. Simple LAMP stack boxes on CentOS7. Nothing major... unfortunately I spent like 90-110 minutes trying to figure out why virtualbox wouldn’t run properly. Dumbass me forgot that I installed Hyper-V ages ago.

O...K.... whelp... hyperv provider it is...

Luckily it only took about 15 minutes to determine that Hyperv’s networking can’t be setup from vagrant because vagrant doesn’t know how to interact with the hyperv - vswitch. So networking config is ignored and all VMs run on default switch (NAT) which is annoying but workable.

Ran into other issues trying to stay SSH’ed into the VM. PowerShell core (6) ssh’es into the box perfectly fine, but every time I opened vi to edit configs my terminal color scheme and fonts got fucked harder than a 2 dollar hooker on nickel night.

I’m a bright-green text on black background kinda guy. However the terminal kept changing to bright-red text on white background! It was like getting skull-fucked by a minotaur.

After a while I said fuck it, let’s try putty. Vagrant was using it’s own ssh keypair for the boxes, at work on my mac. Works like a dream. Putty failed me hard and shit the bed, kept getting all kinds of keypair errors. At this point I was finished spent too long trying to make shit work correctly on this jankbox. With enough time and patience I probably could’ve figured all of these problems out. I’m certain that at least 70% of them were caused by user error. I’m known by many as the walking ID-10t.

But alas, I have no time left in the day to fuck around with shit that doesn’t work immediately for morons like myself. My only hang up for the longest time with a complete switch to Linux was gaming. But with Proton and WINE I’m comfortable with giving it the ol’ college try. (Shhhh, don’t remind me I dropped out of college...

...Thrice.)

The gamble here is that I’ll give more than 2 halves of a fuck about trying to get my games working. A Study environment and materials for certs and general training won’t be getting anywhere near my full attention.

So, at long last, I hope this attempt at a full *nix switch finally sticks!!!

👾

This is with nothing open. How the fuck am I supposed to work on here. Defiantly won't be firing up any VMs

--
Yes, it's all background crap, no I can't close any of it. Yes, I have rebooted

Windows Remote Desktop Connection for ALL of our VMs. No VPNs whatsoever. If an employee wants to connect to work from home, we have to set up RDP on their PC. 😐🔫

So I was just thinking scenario wise how if I founded a military organization who’s aim was to depose an insane dictator from power while remaining undetected including from all his insane supporters how I would use Linux

After I took every source package offline and modified and source checked it and disabled the ability for it to use regular tcp ip for anything but tunneling

The reasoning ? I just installed one program and it downloaded 40 some extra packages that I have no idea what are doing

Linux is great but do we actually know what the software is doing ? Same as windows only you can’t compiler that from scratch

It’s either trust a bunch of random people or trust a bunch of random people part of an evil company

Not the best of choices

But oh how beautiful it would be after I had 500 people pick every last package apart with a reasonable deadline of 2 years entirely offline at frozen versions

Then we could fork all the projects
Or only implement very carefully source reviewed patches transported via offline medium To computers that are running vms to house and test all additional patches so if it blows up we just copy our raw frozen image back over it and either scrap or repair the change

As connecting to the internet in general is not an good idea for silent running

I have done some experiments on my server in the past. It's a great way to learn new things. However, I am bound to make some mistakes and over time the sever becomes messier and messier.

A week ago I installed UNRAID on my machine and I love it! I can now have my critical infrastructure live and working in docker containers and vms.

Then if I want to do an experiment I spin up a VM in a couple of minutes to do my thing and remove it when I am done. No traces left!