Frack he did it again.
In a meeting with the department mgr and going over a request feature *we already discussed ad nauseam* that wasn’t technically feasible (do-able, just not worth the effort)

DeptMgr: “I want to see the contents of web site A embedded in web site B”
Me: “I researched that and it’s not possible. I added links to the target APM dashboard instead.”
Dev: “Yes, it’s possible. Just use an IFrame.”
DeptMgr: “I thought so. Next sprint item …what’s wrong?…you look frustrated”
Me: “Um..no…well, I said it’s not possible. I tried it and it doesn’t work”
Dev: “It’s just an IFrame. They are made to display content from another site.”
Me: “Well, yes, from a standard HTML tag, but what you are seeing is rendered HTML from the content manager’s XML. It implemented its own IFrame under the hood. We already talked about it, remember?”
Dev: “Oh, that’s right.”
DeptMgr: “So it’s possible?”
Dev: “Yea, we’ll figure it out.”
Me: “No…wait…figure what out? It doesn’t work.”
Dev: “We can use a powershell script to extract the data from A and port it to B.”
DeptMgr: “Powershell, good…Next sprint item…”
Me: “Powershell what? We discussed not using powershell, remember?”
Dev: “It’s just a script. Not a big deal.”
DeptMgr: “Powershell sounds like a right solution. Can we move on? Next sprint item….are you OK? You look upset”
Me: “No, I don’t particularly care, we already discussed executing a powershell script that would have to cross two network DMZs. Bill from networking already raised his concern about opening another port and didn’t understand why we couldn’t click a link. Then Mike from infrastructure griped about another random powershell script running on his servers just for reporting. He too raised his concern about all this work to save one person one click. Am I the only one who remembers this meeting? I mean, I don’t care, I’ll do whatever you want, but we’ll have to open up the same conversations with Networking again.”
Dev: “That meeting was a long time ago, they might be OK with running powershell scripts”
Me: “A long time ago? It was only two weeks.”
Dev: “Oh yea. Anyway, lets update the board. You’ll implement the powershell script and I’ll …”
Me: “Whoa..no…I’m not implementing anything. We haven’t discussed what this mysterious powershell script is supposed to do and we have to get Mike and Bill involved. Their whole team is involved in the migration project right now, so we won’t see them come out into the daylight until next week.”
DevMgr: “What if you talk to Eric? He knows powershell. OK…next sprint item..”
Me: “Eric is the one who organized the meeting two weeks ago, remember? He didn’t want powershell scripts hitting his APM servers. Am I the only one who remembers any of this?”
Dev: “I’m pretty good with powershell, I’ll figure it out.”
DevMgr: “Good…now can we move on?”

GAAAHH! I WANT A FLAMETHROWER!!!

Ok…feel better, thanks DevRant.

Received "emergency update" code from internal enterprise security team. Wasn't given time to do code review; was assured code was reviewed and solid.

Pushed code to over 6k lower-level servers before finding this gem buried deep within:
...
cd /foo; rm -rf *; cd /
...

(This ran as root, and yes, the cwd was / from earlier in the code).

/foo, of course, did not exist on some servers.

Now, it is those servers which do not exist.

FML

Introducing avatar pets & a new way to unlock avatar items! Pets include cats, dogs, birds and tigers. We know getting all those ++'s can take some time, so while all avatar items are just as reachable for free by earning ++'s from rants and comments, we now allow items to be purchased in-app based on the difference in ++'s you have to the ++ unlock value of the item. (To purchase, you'll need the latest app version)

Paying for avatar items helps us keep the devRant servers up and running and also is extra motivation to add more fun avatar items. If you have any questions or find a bug, please let me or @dfox know. Happy ranting!

I use a lot of dev tools, but one of my favorites is the Linux screen utility. It's awesome to be able to keep shit running on servers whether my laptop stays connected tot he server or not. It's great for jobs that take a long time, can't get interrupted, etc.

If you haven't used screen definitely give it a try!

Hacking/attack experiences...
I'm, for obvious reasons, only going to talk about the attacks I went through and the *legal* ones I did 😅 😜

Let's first get some things clear/funny facts:
I've been doing offensive security since I was 14-15. Defensive since the age of 16-17. I'm getting close to 23 now, for the record.

First system ever hacked (metasploit exploit): Windows XP.
(To be clear, at home through a pentesting environment, all legal)
Easiest system ever hacked: Windows XP yet again.

Time it took me to crack/hack into today's OS's (remote + local exploits, don't remember which ones I used by the way):
Windows: XP - five seconds (damn, those metasploit exploits are powerful)
Windows Vista: Few minutes.
Windows 7: Few minutes.
Windows 10: Few minutes.
OSX (in general): 1 Hour (finding a good exploit took some time, got to root level easily aftewards. No, I do not remember how/what exactly, it's years and years ago)
Linux (Ubuntu): A month approx. Ended up using a Java applet through Firefox when that was still a thing. Literally had to click it manually xD
Linux: (RHEL based systems): Still not exploited, SELinux is powerful, motherfucker.

Keep in mind that I had a great pentesting setup back then 😊. I don't have nor do that anymore since I love defensive security more nowadays and simply don't have the time anymore.

Dealing with attacks and getting hacked.

Keep in mind that I manage around 20 servers (including vps's and dedi's) so I get the usual amount of ssh brute force attacks (thanks for keeping me safe, CSF!) which is about 40-50K every hour. Those ip's automatically get blocked after three failed attempts within 5 minutes. No root login allowed + rsa key login with freaking strong passwords/passphrases.

linu.xxx/much-security.nl - All kinds of attacks, application attacks, brute force, DDoS sometimes but that is also mostly mitigated at provider level, to name a few. So, except for my own tests and a few ddos's on both those domains, nothing really threatening. (as in, nothing seems to have fucked anything up yet)

How did I discover that two of my servers were hacked through brute forcers while no brute force protection was in place yet? installed a barebones ubuntu server onto both. They only come with system-default applications. Tried installing Nginx next day, port 80 was already in use. I always run 'pidof apache2' to make sure it isn't running and thought I'd run that for fun while I knew I didn't install it and it didn't come with the distro. It was actually running. Checked the auth logs and saw succesful root logins - fuck me - reinstalled the servers and installed Fail2Ban. It bans any ip address which had three failed ssh logins within 5 minutes:
Enabled Fail2Ban -> checked iptables (iptables -L) literally two seconds later: 100+ banned ip addresses - holy fuck, no wonder I got hacked!

One other kind/type of attack I get regularly but if it doesn't get much worse, I'll deal with that :)

Dealing with different kinds of attacks:

Web app attacks: extensively testing everything for security vulns before releasing it into the open.
Network attacks: Nginx rate limiting/CSF rate limiting against SYN DDoS attacks for example.
System attacks: Anti brute force software (Fail2Ban or CSF), anti rootkit software, AppArmor or (which I prefer) SELinux which actually catches quite some web app attacks as well and REGULARLY UPDATING THE SERVERS/SOFTWARE.

So yah, hereby :P

- Think first, write later;
- Do not rush into purchasing servers. You might not need them for another 10 months;
- It takes away a lot of family time to do side projects;
- NOT playing any games saves tremendous amounts of time. Also applies to watching TV;
- If you get stuck - get some sleep. Morning brings you better ideas;
- Write proper abstractions or you'll end up refactoring everything way more often that you'd like;
- Side projects need a loo-ooooot of your will and determination

Hey everyone - we're currently experiencing some intermittent connectivity issues with our servers. The app may be becoming unavailable for short periods of time and we are investigating.

Thanks for your patience.

Client: Can you provide some kind of guaranteed timeline that you're going to be able to move our website to our new servers with the optimizations implemented? I know you said it should take a week, but we have 3 weeks to get this moved over and we cannot afford to be double billed. I'm waiting to fire up the new server until you can confirm.

Me: As I said, it SHOULD take about a week, but that's factoring in ONLY the modifications being made for optimization and a QA call to review the website. This does not account for your hosting provider needing to spin up a new server.

We also never offered to move your website over to said new server. I sent detailed instructions for your provider to move a copy of the entire website over and have it configured and ready to point your domain over to, in order to save time and money since your provider won't give us the access necessary to perform a server-to-server transfer. If you are implying that I need to move the website over myself, you will be billed for that migration, however long it takes.

Client: So you're telling me that we paid $950 for 10 hours of work and that DOESN'T include making the changes live?

Me: Why would you think that the 10 hours that we're logged for the process of optimizing your website include additional time that has not been measured? When you build out a custom product for a customer, do you eat the shipping charges to deliver it? That is a rhetorical question of course, because I know you charge for shipping as well. My point is that we charge for delivery just as you do, because it requires our time and manpower.

All of this could have been avoided, but you are the one that enforced the strict requirement that we cannot take the website down for even 1 hour during off-peak times to incorporate the changes we made on our testbed, so we're having to go through this circus in order to deliver the work we performed.

I'm not going to give you a guarantee of any kind because there are too many factors that are not within our control, and we're not going to trap ourselves so you have a scapegoat to throw under the bus if your boss looks to you for accountability. I will reiterate that we estimate it would take about a week to implement, test and run through a full QA together, as we have other clients within our queue and our time must be appropriately blocked out each day. However, the longer you take to pull the trigger on this new server, the longer it will take on my end to get the work scheduled within the queue.

Client: If we get double billed, we're taking that out of what we have remaining to pay you.

Me: On the subject of paying us, you signed a contract acknowledging that you would pay us the remaining 50% after you approved the changes, which you did last week, in order for us to deliver the project. Thank you for the reminder that your remaining balance has not yet been paid. I'll have our CFO resend the invoice for you to remit payment before we proceed any further.

---

I love it when clients give me shit. I just give it right back.

How not to give support..

Me: Creates ticket on support site, letting them know their webservice returns "maintenance" page.
Support: "It works on our end."
Support: *closes ticket*

Euhm.. excuse me?

Me: Creates another ticket with a screenshot and the curl response information..
Support: Sends screenshot back that it works on their end. "Maybe check your firewall"
Support: *closes ticket*

I ain't playing these games..

Me: Creates new ticket with more curl responses from 4 different servers to prove it's not "firewall" related.
Support: ..

2 days later

Me: Sends *friendly* reminder.
Support: ..

6 days later

Me: Creates ticket again saying I'm still having issues.
Support: "I'm forwarding this to our technical support"
Support: *closes ticket*

10 minutes later.

Technical Support: "Here's the manual for our integration .pdf."

Excuse me, you say what now? I KNOW HOW IT WORKS, I'VE WRITTEN THE INTEGRATION ALREADY. THE SERVICE JUST SEEMS TO BE DOWN FFS.. pls..

Me: Sends mail to their project manager who manages the clients dossier with support history and such.
Him: "I'll check it out and let you know."

1 day later.

Support: "We had some issues this and that, wasn't publically availble, works now, .."

What a nice way to waste your time..

I ranted about this guy before who thought he was a security expert while hardly knowing what the word is probably. Today I met him again at a party.

Holy fucking shit, this guy.

"we use the best servers of the netherlands"
"we use a separate server for each website and finetune them"
"we always put clusters under servers, that way we have a fallback mechanism"
"companies mostly use bv ssl certificates"
"you're on call for a week? I'm full-time on call. Why I'm drinking alcohol then? Because fuck the clients hahaha"

😥🔫

string excuses[]={
"it's not a bug it's a feature",
"it worked on my machine",
"i tested it and it worked",
"its production ready",
"your browser must be caching the old content",
"that error means it was successful",
"the client fucked it up",
"the systems crashed and the code got lost" ,
"this code wont go into the final version",
"It's a compiler issue",
"it's only a minor issue",
"this will take two weeks max",
"my code is flawless must be someone else's mistake",
"it worked a minute ago",
"that was not in the original specification",
"i will fix this",
"I was told to stop working on that when something important came up",
"You must have the wrong version",
"that's way beyond my pay grade",
"that's just an unlucky coincidence",
"i saw the new guy screw around with the systems",
"our servers must've been hacked",
"i wasn't given enough time",
"its the designers fault",
"it probably won't happen again",
"your expectations were unrealistic",
"everything's great on my end",
"that's not my code",
"it's a hardware problem",
"it's a firewall issue",
"it's a character encoding issue",
"a third party API isn't responding",
"that was only supposed to be a placeholder",
"The third party documentation is wrong",
"that was just a temporary fix.",
"We outsourced that months ago.","
"that value is only wrong half of the time.",
"the person responsible for that does not work here anymore",
"That was literally a one in a million error",
"our servers couldn't handle the traffic the app was receiving",
"your machines processors must be too slow",
"your pc is too outdated",
"that is a known issue with the programming language",
"it would take too much time and resources to rebuild from scratch",
"this is historically grown",
"users will hardly notice that",
"i will fix it" };

Funny story about the first time two of my servers got hacked. The fun part is how I noticed it.
So I purchased two new vps's for proxy server goals and thought like 'I can setup fail2ban tomorrow, I'll be fine.'

Next day I wanted to install NginX so I ran the command and it said that port 80 was already in use!

I was sitting there like no that's not possible I didn't install any server software yet. So I thought 'this can't be possible' but I ran 'pidof apache2' just to confirm. It actually returned a PID! It was a barebones Debian install so I was sure it was not installed yet by ME. Checked the auth logs and noticed that an IP address had done a huge brute force attack and managed to gain root access. Simply reinstalled debian and I put fail2ban on it RIGHT AWAY.

Checked about two seconds later if anyone tried to login again (iptables -L and keep in mind that fail2ban's default config needs six failed attempts within I think five minutes to ban an ip) and I already saw that around 8-10 addresses were banned.

Was pretty shaken up but damn I learned my lesson!

You know what? I'm fucking done with people telling me that open source alternatives to popular/proprietary suck by default.

Something does NOT suck by default just because it's FUCKING open source.

Have you got any fucking clue where we'd be right now if open source software didn't exist?!

Let me just remind you that about 80 percent of the worlds' servers run Linux. Open-FUCKING-source.

How the fuck are we supposed to innovate without open sourceness? Yes indeed, that would be about fucking impossible.

Although I've got to admit that some open source software programs don't work 100 well (in comparison to alternatives), what are you expecting? People put their free time into that shit and they've got to make money as well.

"well how are programmers supposed to feed their families if they only write open source software?"

Fuck right off. Of course we all need an income to survive. Hell, I need that as fucking well. But there's more to it than just work. Some people consider open source/working on open source software a hobby/passion. That doesn't even remotely mean thaty they don't work/don't need some kind of income.

If it wasn't for open source, we'd be nowhere (technologially seeing) right now.

Fuck Microsoft.

No, not in any relation to windows this time.

Dear Microsoft, why on earth did you put us on your spam blacklist? There haven't been any spam attacks from our side, our servers have nearly the highest 'reputation' that email servers can get, we comply to all security standards and yet you're blacklisting us.

If for some reason you think something is wrong at our side anyways, we've tried to contact you and we either get ignored or get a very late response saying that we'll get delisted again within a day/week or whatsoever.

Microsoft, please go fuck yourself.

Our company got attacked last month by what i believe was a code time bomb from a ex employee. And it was brutal, website hacked, email server not responding, locked out from database servers. The IT department asked for my help and I was more than happy to do it. Long story short I got every thing back working smoothly. The IT guys ask for a favor to not include this in my monthly progress report. Fine by me. But then they went out and tokd the top management that they are teaching me about the networks and servers so thats why I was working with them last month. Fucking assholes. Not going to help them any more.

So I'm 6 seconds behind devrant servers 😏, who's has the correct time...

So I had my exams recently and I thought I'd post some of the most hacky shit I've done there over here. One thing to keep in mind, I'm a backender so I always have to hack my way around frontend!

- Had a user level authentication library which fucked up for some reason so I literally made an array with all pages and user levels allowed so I pretty much had a hardcoded user level authentication feature/function. Hey, it worked!

- CSS. Gave every page a hight of 110 percent because that made sure that you couldn't see part of the white background under the 'background' picture. Used !important about everywhere but it worked :P.

- Completey forgot (stress, time pressure etc) to make the user ID's auto incremented. 'Fixed' that by randomly generating a user id and really hoping during every registration that that user ID did not exist in the database already. Was dirty as fuck but hey it worked!

- My 'client' insisted on using Windows server.Although I wouldn't even mind using it for once, I'd never worked with it before so that would have been fucked for me. Next to that fact, you could hear swearing from about everyone who had to use Windows server in that room, even the die hard windows users rather had linux servers. So, I just told a lot of stuff about security, stability etc and actually making half of all that shit up and my client was like 'good idea, let's go for linux server then!'. Saved myself there big time.

- CHMOD'd everything 777. It just worked that way and I was in too much time pressure to spend time on that!

- Had to use VMWare instead of VirtulBox which always fucks up for me and this time it did again. Windows 10 enjoyed corrupting the virtual network adapters after every reboot of my host so I had to re-create the whole adapter about 20 times again (and removing it again) in order to get it to work. Even the administrator had no fucking clue why that was happening.

- Used project_1.0.zip etc for version control :P.

Yup, fun times!

The past 2 years where I work:

Me: hey let's use git instead of ftp!
Boss: should we?
Some time later: he is loving it...

Me: hey let's use trello instead of excell!!??!
Boss: huumm.. Dunno... Should we?
After much convincing: whole departments are using and loving it....

Me: hey let's move from rackspace to DO!
Boss: huumm... Convince me...
Year latter: everything smooth and muuuch lower prices... Managing 6 servers instead of one...

Me: UNIT TESTS!!!
Boss: nah, this but a waste of time...

For real? Get a grip man, I only encourage solutions tested ( no pun intended, or is it ) by me for a long time...

Once upon a time there was a dev.
The dev had a resume that said he could dev.
We called the dev, he sounded intelligent.
We hired the dev, who was a bit green, on a three month probationary period.
The dev did very little.
When asked, we said he contributed to discussions, but seemed unclear about what to do, and maybe they could keep him as an intern if they wanted to have him at all.
They hired him. As a full time dev.
6 months later, that dev was shocked to find we could log into the servers with a privileged account.
We (his team mates) were sad.
We asked him to fix a few prod errors.
A little while later he said "Done!"
We then had to walk him through how to actually fix them, not just add a couple pieces of info to the table.
We were sad, again.
We asked him to fix some prod errors again.
We had to walk him through the process again
We expressed concerns to our superiors about his abilities because he was all theory, no hands on ability
They promoted him
We were sad
A few of us said "Fuck you guys, I'm going home"
They said OK
Now that guy is the only one that "knows" that code base
I get calls sometimes asking me questions.
I told them to pay me a consultant fee.
They said no
I said no
They called again
I laughed at them

Listen to the people who know when you ask them questions.
Listen to the people who know when they tell you there is a problem
Don't be like that company

Half a week later I finally found out why my DNS server "wasn't working" on any of my servers.

Let's just say that MullVad's anti-dns leaking technology works extremely well.

It was tunneling all DNS requests through its own servers to prevent DNS leaks and I never thought of it a single fucking time!

😅

When the Sales team fucks up something, they point finger at each other for the blame.

When the Dev Team fucks up something, as a Senior Developer I say to my Boss that we fucked it up no matter whose fault it is and we will fix it up by x time.

When the SysAdmin team fucks. The Dev Team is to blame.

Sorry guys, I got a bit frustrated. All our servers wasn't backed up from last week and the SysAdmin guys are saying it's our fault.

What the fuck is going on? fuck you fuck fuck fuck fuck...

The man who runs my IT department. The man who is in charge of all things and people that are technical: IT management software development, infrastructure, training, help desk, system administration​, etc. A man with a staff of fifty plus. If you were to peel back the flesh on this man's head and crack open his skull you would find dung beetles feasting on the feces that power his thoughts and motor functions. Underneath this foul membrane, if you could push past the maggots; the meal worms; his undying love for hourly binges of Johnny Walker Black on any day of the week with a name that contains a vowel; his fascination with shiny objects and his endless internal monologue wondering when they would hatch rainbow ponies that fly; his desire whenever he enters a paint store to open all the cans of paint and taste the different colors; if you could push past all of the vile crap that exists where Thomas Aquinas once theorized there was a soul, you would find a colony of paramecia at the end of their short lives laughing hysterically at how much smarter they were than the host they lived in.

This man was in charge of hiring the Manager of Software Development. The manager I report to. After seven months of ignoring this chore; after interviewing the sum total of four candidates; after making a point to tell myself and a colleague that there was no one qualified to fill this position within our company (an opinion that is both untrue and, when spoken, runs afoul of internal hiring policies) this man hired a soulless cretin with no experience in software development or with running a software development group. A man who regularly confuses web servers and SQL servers. A man who asked me how my previous manager reviewed my work, was told by me that said previous manager read my code, and then replied in his capacity as the manager of software development that "looking at code is a compete waste of time for a manager." A man so without any humanity or reason for being that he will sit silently, creepily, in conference rooms with the lights off waiting for meetings to begin. Meetings he has scheduled. That have no reason for being in the first place. Just like himself.

Shortly before the man in charge offered the Dev Manager job to the simulacrum of human flesh that is my manager, he met with me and others who had been involved in the interview process. When I informed him that hiring someone with no technical knowledge for a very technical position would be a mistake that he would suffer through for years, he replied in reference to his future hire that "his managerial experience makes up for his lack of technical knowledge."

Best. Prank. Ever. Worst prank ever too. Fuck.

Timezones. So, general rules are:

1. If you don't store timezone, always use and assume UTC. Databases, backends, whatever you use, all time must be kept be in UTC.
2. If you store timezones, ensure you store them everywhere and don't drop them anywhere.
3. It's always better to ignore backend server time in favor of database's `now().` Having a single source of truth makes time consistent (if it's the same database, obviously). If you combine backend time and database time, you likely get a violation of causality.

I've just spent a couple of hours investigating "weird random one-hour time drifts on updates." Guys violated all three rules above:
- they didn't store the timezone;
- their servers had inconsistent timezones. Java was in +XX., while the server itself in UTC. On one host, they forgot to put JVM in the same timezone;
- they dropped the timezone because they thought it was the same everywhere, so there was no point in serializing it.

In this day and age, what's my fucking excuse for not using a vpn full-time on my phone as well (next to my laptop)?

I do way more personal stuff on my phone anyways and add to that that I've got access to at least three self managed VPN servers...

Yup, going full VPN from now on.

So...
I'm penetrationtesting a network and the servers on said network
The network administrator and IT security officer knows this, because they hired me..

TL;DR a scan caused the network to crash.

Today I received a very angry email going "Stop scanning NOW!" from one of the IT departments.

Apparently I crashed their login server and thus their entire network...
It happened d the first time I scanned the network from the outside and they had spend an entire day figuring out how and repairing the service they thought was the problem, but then it crashed again, when I scanned from within the network.

Now they want to send me a list of IP's that I'm not allowed to scan and want to know exactly what and when I'm scanning...

How crap can they be at their job, if they weren't able to spot a scan... The only reason they found out it was me was because the NA had whitelistet my IP, so that I could scan in peace...

A little while ago, the concierge of my apartment building came to me about some issue with the central heating system. Totally unaware about the issue, I let him check some things in the apartment. Then he told me that apparently my thermostat has been turned off all the time. So I think that my servers may very well have been the primary source of heat for this apartment for several weeks. Servers, the new type of central heating system that even does useful work in the process!! Can we get some wanketeers on this to make it a product? 🙃

I was offered to work for a startup in August last year. It required building an online platform with video calling capabilities.
I told them it would be on learn and implement basis as I didn't know a lot of the web tech. Learnt all of it and kept implementing side by side.
I was promised a share in the company at formation, but wasn't given the same at the time of formation because of some issues in documents.
Yes, I did delay at times on the delivery date of features on the product. It was my first web app, with no prior experience. I did the entire stack myself from handling servers, domains to the entire front end. All of it was done alone by me.
Later, I also did install a proxy server to expand the platform to a forum on a new server.
And yesterday after a month of no communication from their side, I was told they are scraping the old site for a new one. As I had all the credentials of the servers except the domain registration control, they transferred the domain to a new registrar and pointed it to a new server. I have a last meeting with them. I have decided to never work with them and I know they aren't going to provide me my share as promised.
I'm still in the 3rd year of my college here in India. I flunked two subjects last semester, for the first time in my life. And for 8 months of work, this is the end result of it by being scammed. I love fitness, but my love for this is more and so I did leave all fitness activities for the time. All that work day and night got me nothing of what I expected.
Though, they don't have any of my code or credentials to the server or their user base, they got the new website up very fast.
I had no contract with them. Just did work on the basis of trust. A lesson learnt for sure.
Although, I did learn to create websites completely all alone and I can do that for anyone. I'm happy that I have those skills now.
Since, they are still in the start up phase and they don't have a lot of clients, I'm planning to partner with a trusted person and release my code with a different design and branding. The same idea basically. How does that sound to you guys?

I learned that:
. No matter what happens, never ignore your health for anybody or any reason.
. Never trust in business without a solid security.
. Web is fun.
. Self-learning is the best form of learning.
. Take business as business, don't let anyone cheat you.

"Almond, I thought you said the cause of the outage the other week was that our server crashed?"

"The Tomcat server crashed, yeah. Not the physical server." (And you won't give me the time or budget to spin up any kind of redundant one, but that's besides the point...)

"Ok, but I've spoken to ops and they say none of the servers have gone offline in the last month?"

"Yup, the physical server was fine, it was the Tomcat server running on it that crashed."

"...so the server didn't crash?"

"We're mixing terms here. There's two things that can be referred to as the server. One is the physical machine, and one is an application running on it. The physical machine was fine, but
the application running on it crashed."

"What?! It's a very simple question. Did the server crash, or didn't it?!"

🤦‍♂️

I have just concluded a post-mortem on one of my servers.
Cause of death: out of memory due to a tiny memory leak in a VPN service triggered by 66 different IPs brute-forcing the creds at the same time. Mostly from China, of course.

Dear bot writers: you made me put aside my spaghetti and write iptables rules. I hate iptables. And I love spaghetti. You should be ashamed of yourself! Did momma not teach you basic OpSec? Don't crash the target and never, ever, interrupt the sysadmin during dinner!

Customer : c
Me : m

*Few weeks ago*
C: the server is slow, it sometimes takes 7 seconds before I see our data

(the project is 7+ years old and wasn't written by someone who is very good in SQL)

M: yeah I see that, our servers are busy with this one "process" (SQL query)
C: make it faster
M: well that's possible but it will take a few days (massive SQL spaghetti that I first have to untangle)
C: 😡 nvm then

*Yesterday*
C: server is down !
M: 🤔 *loads data from server and waits ~ 7 seconds*
M: Well what's the problem?
C: I need the data but it's so slow

WELL YOU MINDLESS IMBECILE... If something is slow it doesn't mean our god damn production server is down !
That just means that you have to give us a day or two so we can optimise the (ALSO BY YOUR REQUEST) rushed project... And save you YOUR money that YOU waste on the processing time on our server...

TLDR;
Wrote a slick scheduling and communication system allowing me to assign photography resources based on time and location.

I'll tell you a little secret ... I'm not actually a dev. I'm a photographer, pretending to be a dev.

Or ... perhaps it's the other way around? (I spend most of my time writing code these days, but only for me - I write the software I use to run my business).

I own a photography studio - we specialize in youth volleyball photography (mostly 12-18 year old girls with a bit of high school, college and semi-pro thrown in for good measure - it's a hugely popular sport) and travel all over the US (and sometimes Europe) photographing.

As a point of scale, this year we photographed a tournament in Denver that featured 100 volleyball courts (in one room!), playing at the same time.

I'm based in California and fly a crew of part-time staff around to these events, but my father and I drive our booth equipment wherever it needs to go. We usually setup a 30'x90' booth with local servers, download/processing/cashier computers and 45 laptops for viewing/ordering photographs. Not to mention 16' drape and banners, tons of samples, 55' TVs, etc. It's quite the production.

We photograph by paid signup only - when there are upwards of 800 teams/9,600 athletes per weekend playing, and you only have four trained photographers, you've got to manage your resources!

This of course means you have to have a system for taking sign those sign ups, assigning teams to photographers and doing so in the most efficient manner possible based on who is available when the team is playing. (You can waste an awful lot of time walking from one court to another in a large convention center - especially if you have to navigate through large crowds - not to mention exhausting yourself).

So this year I finally added a feature I've wanted for quite some time - an interactive court map. I can take an image of the court layout from the tournament and create an HTML version in our software. As I mouse over requests in one window, the corresponding court is highlighted on the map in another browser window. Each photographer has a color associated with them. When I assign requests to a photographer, the court is color coded with the color of the photographer. This allows me to group assignments to minimize photographer walk time and keep them in a specific area. It's also very easy to look at the map and see unassigned requests and look to see what photographer is nearby.

This year I also integrated with Twilio and setup a simple set of text shortcuts that photographers can use to let our booth staff know where they are, if they have memory cards that need picking up, if they need water/coffee/snack, etc. They can also move assignments on their schedule or send and SOS for help if it looks like they aren't going to be able to photograph a team.

Kind of a CLI via the phone. :)

The additions have turned out to be really useful and has made scheduling and managing the photographers much easier that it was in the past.

So I worked on getting a server ready for about 30 hours last week to be ready for a deploy on Monday Night (last night). Not only did I work on it for 30 hours, we had two other architects and a senior engineer working on it too. We got everything done Friday and it was ready to go with a simple cutover on Monday night.

The only thing left to do was deploy a link change Monday night on the existing landing page. My part was the backend servers and application that had the complicated SSO system and the other part was just a link to get to the SSO. I asked the person responsible for deploying the landing page's link if he was ready about a dozen times. He kept saying he was deploying X (the code name for the project deploy) and that is all he was doing.

Now jump to that night. They have decided that a single landing page wasn't enough and they were going to deploy a full CMS. Well no one knew what the hell was going on and they didn't realize that the landing page was hosted externally on another host. After arguing for two hours they delayed the deployment for multiple days. 24 hours later they are still trying to figure out the CMS on a host.

30 hours and four senior engineer's time wasted to get everything done for the deadline all to be canceled because of on jackass's lack of planning. WTF

The riskiest dev choice...

How about "The riskiest thing you've done as a dev"? I have a great entry for that. and I suppose it was my choice to build the feature afterall.

I was working on an instance of a small MMO at a game company I worked for. The MMO boasted multiple servers, each of them a vastly different take on the base game. We could use, extend, or outright replace anything we wanted to, leading to everything from Zelda to pokemon to an RP haven to a top-down futuristic counterstrike. The server in this particular instance was a fantasy RPG, and I was building it a new leveling and experience system with most of the trimmings. (Talents, feats/perks, etc. were in a future update.)

A bit of background, first: the game's dev setup did not have the now-standard dev/staging/prod servers; everything ran on prod, devs worked on prod, players connected and played on prod, etc. Worse yet, there was no backup system implemented -- or not really. The CTO was really the only person with sufficient access. The techy CEO did as well, but he rarely dealt with anything technical except server hardware, occasionally. And usually just to troll/punish us devs (as in "Oops ! I pulled the cat5 ! ;)"). Neither of them were the most reliable of people, either. The CTO would occasionally remote in and make backups of each server -- we assumed whenever he happened to think of it -- and would also occasionally do it when asked, but it could take him a week, sometimes even up to a month to get around to it. So the backups were only really useful for retreiving lost code and assets, not so much for player data.

The lack of reliable backups and the lack of proper testing grounds (among the plethora of other issues at the company) made for an absolutely terrible dev setup, but that's just how it was, and that's what we dealt with. We were game devs, afterall. Terrible or not, we got to make games! What more could you ask for!? It was amazing and terrible and wonderful and the worst thing ever, all at the same time. (and no, I'm not sharing the company name, but it isn't EA or Nexon, surprisingly 😅)

Anyway, back to the story! My new leveling system also needed to migrate players' existing data, so... you can see where this is going.

I did as much testing and inspection of my code as I could, copied it from a personal dev script to the server's xp system, ... and debated if I really wanted to click [Apply]. Every time I considered it, I went back to check another part or do yet more testing. I ended up taking like 40 minutes to finally click it.

And when I did... that was the scariest button press of my life. And the scariest three seconds' wait afterwards. That one click could have ruined every single player's account, permanently lost us players ...

After applying it, I immediately checked my character to see if she was broken, checked the account data for corruption or botched flags, checked for broken interactions with the other systems....

Everything ended up working out perfectly, and the players loved all of the new features. They had no idea what went into building them, and certainly had no idea of what went into applying them, or what could have gone wrong -- which is probably a good thing.

Looking back, that entire environment was so fragile, it's a wonder things didn't go horribly wrong all the time. Really, they almost never did. Apocalypses did happen, but were exceedingly rare, and were ususally fixed quickly. I guess we were all super careful simply because everything was so fragile? or the decent devs were, at least. We never trusted the lessers with access 😅 at least on the main servers where it mattered. Some of the smaller servers... well, we never really cared about those.

But I'm honestly more surprised to realize I've never had nightmares of that button click. It was certainly terrifying enough.

But yay! Complete system overhaul and migration of stored and realtime player data! on prod! With no issues! And lots of happy players! Woooooo!

Thinking back on it makes me happy 😊

Our web department was deploying a fairly large sales campaign (equivalent to a ‘Black Friday’ for us), and the day before, at 4:00PM, one of the devs emails us and asks “Hey, just a heads up, the main sales page takes almost 30 seconds to load. Any chance you could find out why? Thanks!”
We click the URL they sent, and sure enough, 30 seconds on the dot.
Our department manager almost fell out of his chair (a few ‘F’ bombs were thrown).
DBAs sit next door, so he shouts…
Mgr: ”Hey, did you know the new sales page is taking 30 seconds to open!?”
DBA: “Yea, but it’s not the database. Are you just now hearing about this? They have had performance problems for over week now. Our traces show it’s something on their end.”
Mgr: “-bleep- no!”
Mgr tries to get a hold of anyone …no one is answering the phone..so he leaves to find someone…anyone with authority.
4:15 he comes back..
Mgr: “-beep- All the web managers were in a meeting. I had to interrupt and ask if they knew about the performance problem.”
Me: “Oh crap. I assume they didn’t know or they wouldn’t be in a meeting.”
Mgr: “-bleep- no! No one knew. Apparently the only ones who knew were the 3 developers and the DBA!”
Me: “Uh…what exactly do they want us to do?”
Mgr: “The –bleep- if I know!”
Me: “Are there any load tests we could use for the staging servers? Maybe it’s only the developer servers.”
DBA: “No, just those 3 developers testing. They could reproduce the slowness on staging, so no need for the load tests.”
Mgr: “Oh my –bleep-ing God!”
4:30 ..one of the vice presidents comes into our area…
VP: “So, do we know what the problem is? John tells me you guys are fixing the problem.”
Mgr: “No, we just heard about the problem half hour ago. DBAs said the database side is fine and the traces look like the bottleneck is on web side of things.”
VP: “Hmm, no, John said the problem is the caching. Aren’t you responsible for that?”
Mgr: “Uh…um…yea, but I don’t think anyone knows what the problem is yet.”
VP: “Well, get the caching problem fixed as soon as possible. Our sales numbers this year hinge on the deployment tomorrow.”
- VP leaves -
Me: “I looked at the cache, it’s fine. Their traffic is barely a blip. How much do you want to bet they have a bug or a mistyped url in their javascript? A consistent 30 second load time is suspiciously indicative of a timeout somewhere.”
Mgr: “I was thinking the same thing. I’ll have networking run a trace.”

4:45 Networking run their trace, and sure enough, there was some relative path of ‘something’ pointing to a local resource not on development, it was waiting/timing out after 30 seconds. Fixed the path and page loaded instantaneously. Network admin walks over..
NetworkAdmin: “We had no idea they were having problems. If they told us last week, we could have identified the issue. Did anyone else think 30 second load time was a bit suspicious?”
4:50 VP walks in (“John” is the web team manager)..
VP: “John said the caching issue is fixed. Great job everyone.”
Mgr: “It wasn’t the caching, it was a mistyped resource or something in a javascript file.”
VP: “But the caching is fixed? Right? John said it was caching. Anyway, great job everyone. We’re going to have a great day tomorrow!”
VP leaves
NetworkAdmin: “Ouch…you feel that?”
Me: “Feel what?”
NetworkAdmin: “That bus John just threw us under.”
Mgr: “Yea, but I think John just saved 3 jobs. Remember that.”

About a year ago I switched my job.
At the start everything seemed like magic. I was the It director, I've finally was able to call the shots on technologies, on new software architecture.
First step was to check the current state of the company.
"qqqq" as each pc password? Ok
No firewall from outside? Lovely
Servers running on Windows Server 2008? Spectacular
People leaving pc on after work and left the machine unlocked just not to type the password? Hell yeah
The IT dude playing games instead of working? But ofcourse
Plaintext passwords publically accessible eshop? Naturally.
The list goes on and on.
After all this time, I'm working to fix every hole like that like crazy and because it doesn't show results, I'm soon to lose my job. Well better luck next time as an intern I guess :')

This happend to me around 2 weeks ago. For some reason, I decied to post this now.

I won the lottery, yey! I mean, bot really, but I am <19yo student, "less than junior dev" in my office, but sonce I am the only one who is capable of working with hardware, I was working month back as a sysadmin for a few days. Our last sysadmin was really good working but really, really toxic guy, so he got fired on a spot after argument with some manager or whatever, no big deal, we could have another guy hired in a week. But, our backup server literally was on fire, all data probably dead because bad capacitor or whatever. This was our only backup of everything at the time. Everyone in full fucking panic mode, we had literally no other working HW we could use for backup, but then comes me, intern employed on his first dev job for 3 months. That day I bought some HW for my own personal server at home (Intel NUC with some Celeron, 4GB DDR4 RAM and two 240GB SSDs for RAID 1. My manager asked everyone in the office for sollution how to survive next 4 days before new server arrives. People there had no idea what tk do and no knowedgle about HW, I just came from a break and offered my components for a week, since there was noone else who can work with HW, servers and stuff like this, manager offered me $500+HW cost if I, random intern, can make it work. I installed Debian on that little PC, created RAID1 from both SSDs, installed MySQL server and mirrored GIT server from our last standing server (we had two before one of them went lit 🔥), made simple Python script to copy all data on that RAID, with some help of our database guy copied whole DB from production to this little computer and edited some PHP so every SQL request made on our server will run on that NUC too. Everything after ±2 hours worked perfectly. Untill a fucking PSU burned in our server and took RAID controller with him in sillicon heaven next night, so we could not access any data unltill we got a new one. Thanks to every god out there, I was able to create software RAID from survived HDDs on our production server and copy all data from that NUC on the servers software RAID and make it working at 3 AM in the night before an exam 😂. Without this, we would be next ±40 hours without aerver running and we might loose soke of our data and customers. So my little skill with Linux, Python, MySQL and most importantly my NUC hardware I got that day running as a backup server saved maybe whole company 😂.
Btw, guess who is now employee of the year with $2500 bonus? 😀
Sorry for bragging and log post, but I was so lucky an so happy when everything worked out, good luck to all sysadmins out there! 👍

TL:DR: Random intern saved company and made some money 😂

One time I had this conversation with my then PM:

PM: …so in total we need like 3 extra pages; the leaders profile, event showcase and lastly a contact page.

Me: Sure, already on it.

PM: Make it simple and quick, I told the client the updates would be live in an hour.

Me: Okay.

*{5 microseconds later}*

PM: Also the page headers need to be different from the other pages.

Me: Yes, you told me that earlier.

PM: Okay, just needed to re-emphasis.

*{sad disturbing minutes later}*

PM: I don’t know if deploying on azure would be better than having the website on AWS. The pages seems slow.

Me: Yep.

PM: Or maybe we separate the asset files from the main site using a CDN.

Me: You right.

PM: The other projects on AWS seems to perform better in terms of SEO. Don’t you think?

Me: I think.

*{this dude literally just lent me a jacket and won’t allow me put it on}*

PM: So after we are done with this update we need to inform the client about the benefits of switching servers to AWS. I believe they will agree or won’t they because the event is close by?

Me: {{pointed both hands at my PC hoping they’ll get the message}}

PM: Oh you done?

Me:

Fucking China!

The only time they ever come out of their pathetic walled internet is to attack your servers.

Fuck that country.

So I'm moving to a new/bigger place with faster Internet soon so I think it's time to rebuild my current home/remote server setup.

I want to setup the following things:
- vps for server monitoring (open source pushover alternative + netdata)
- Zero tier network for connecting all my servers to the same network
- pihole/pivpn (or the Angristan vpn installer, look it up :)
- second blocking thingy next to pihole to make sure that I literally can't access google/fb etc anymore, even if I really needed/wanted to
- bunch of general servers.

Any ideas?

~March 7~
Boss: Hey cory, guess what, you will not take control of the servers anymore so you can focus on your real job, the company hire someone to do it
Me: Great, finally i can just program, thanks for the news boss

~Yesterday~
Boss: Hey cory, guess what, the person the company hired needs help to migrate some servers so you need to help him on weekend
Me: Well, it's ok i can do the job

~Today~
Director: Hey mr cory, we need you to help jonny on weekend
Me: Fine boss, i will be on weekend
Director: That's the attitude we need in the company, I do not know how much time you need but we're going to pay you 24 extra hours
Me inside ~every went better than expected~ 🤷‍♂️

Long rant ahead.. so feel free to refill your cup of coffee and have a seat 🙂

It's completely useless. At least in the school I went to, the teachers were worse than useless. It's a bit of an old story that I've told quite a few times already, but I had a dispute with said teachers at some point after which I wasn't able nor willing to fully do the classes anymore.

So, just to set the stage.. le me, die-hard Linux user, and reasonably initiated in networking and security already, to the point that I really only needed half an ear to follow along with the classes, while most of the time I was just working on my own servers to pass the time instead. I noticed that the Moodle website that the school was using to do a big chunk of the course material with, wasn't TLS-secured. So whenever the class begins and everyone logs in to the Moodle website..? Yeah.. it wouldn't be hard for anyone in that class to steal everyone else's credentials, including the teacher's (as they were using the same network).

So I brought it up a few times in the first year, teacher was like "yeah yeah we'll do it at some point". Shortly before summer break I took the security teacher aside after class and mentioned it another time - please please take the opportunity to do it during summer break.

Coming back in September.. nothing happened. Maybe I needed to bring in more evidence that this is a serious issue, so I asked the security teacher: can I make a proper PoC using my machines in my home network to steal the credentials of my own Moodle account and mail a screencast to you as a private disclosure? She said "yeah sure, that's fine".
Pro tip: make the people involved sign a written contract for this!!! It'll cover your ass when they decide to be dicks.. which spoiler alert, these teachers decided they wanted to be.

So I made the PoC, mailed it to them, yada yada yada... Soon after, next class, and I noticed that my VPN server was blocked. Now I used my personal VPN server at the time mostly to access a file server at home to securely fetch documents I needed in class, without having to carry an external hard drive with me all the time. However it was also used for gateway redirection (i.e. the main purpose of commercial VPN's, le new IP for "le onenumity"). I mean for example, if some douche in that class would've decided to ARP poison the network and steal credentials, my VPN connection would've prevented that.. it was a decent workaround. But now it's for some reason causing Moodle to throw some type of 403.

Asked the teacher for routers and switches I had a class from at the time.. why is my VPN server blocked? He replied with the statement that "yeah we blocked it because you can bypass the firewall with that and watch porn in class".
Alright, fair enough. I can indeed bypass the firewall with that. But watch porn.. in class? I mean I'm a bit of an exhibitionist too, but in a fucking class!? And why right after that PoC, while I've been using that VPN connection for over a year?

Not too long after that, I prematurely left that class out of sheer frustration (I remember browsing devRant with the intent to write about it while the teacher was watching 😂), and left while looking that teacher dead in the eyes.. and never have I been that cold to someone while calling them a fucking idiot.

Shortly after I've also received an email from them in which they stated that they wanted compensation for "the disruption of good service". They actually thought that I had hacked into their servers. Security teachers, ostensibly technical people, if I may add. Never seen anyone more incompetent than those 3 motherfuckers that plotted against me to save their own asses for making such a shitty infrastructure. Regarding that mail, I not so friendly replied to them that they could settle it in court if they wanted to.. but that I already knew who would win that case. Haven't heard of them since.

So yeah. That's why I regard those expensive shitty pieces of paper as such. The only thing they prove is that someone somewhere with some unknown degree of competence confirms that you know something. I think there's far too many unknowns in there.

Nowadays I'm putting my bets on a certification from the Linux Professional Institute - a renowned and well-regarded certification body in sysadmin. Last February at FOSDEM I did half of the LPIC-1 certification exam, next year I'll do the other half. With the amount of reputation the LPI has behind it, I believe that's a far better route to go with than some random school somewhere.

Oh F***, not again!

Bob, every single time you "patch" the servers I run into issues, that you cannot fix.

Bob: "heuuu... I don't know anything about python or npm or any of that"

Then stop touching it!!!

1. Scripting out a team. I've built a collection of bash scripts to do what one of our teams does. Except the script does it in 30min and always does it well where that team used to take 4 to 10 hours and almost always missed something in the way.

2. Automate 70-80% of our BAU tasks with a single >4k loc bash script. Integrations with servicenow, lots of internal portals, predefined huge sets of commands to run on separate servers or lists of servers, do all sorts of diagnostics, schedule hw maintenance for DC folks, chase for approvals, track CHNG/CTSK tickets in a graphical chart so we would not miss any of them and lots lots more.
Finally we were able to afford time to make some coffee/tea.

These are the bau optimizations I'm proud of the most. And they have made significant impact on how our teams operate.

Whoever recognizes both company values in the tags and know what is that company - are they still using ´S´ in unix team? :)

Worst of 2020?

Well, I've learnt never to do free / voluntary work.

Been spending many weekends helping out a former friend, mostlyby creating APIs, sites, managing servers & scripting.

Now that I have a girlfriend, my weekends are spent mostly with her.

He got really mad, said he doesn't want to waste time.

Then went to *pay* someone else to continue what I've been doing...

I love how the Keybase Linux client installs itself straight into /keybase. Unix directory structure guidelines? Oh no, those don't apply to us. And after uninstalling the application they don't even remove the directory. Leaving dirt and not even having the courtesy to clean it up. Their engineers sure are one of a kind.

Also, remember that EFAIL case? I received an email from them at the time, stating some stuff that was about as consistent as their respect for Unix directory structure guidelines. Overtyping straight from said email here:

[…] and our filesystem all do not use PGP.
> whatever that means.

The only time you'll ever use PGP encryption in Keybase is when you're sitting there thinking "Oh, I really want to use legacy PGP encryption."
> Legacy encryption.. yeah right. Just as legacy as Vim is, isn't it?

You have PGP as part of your cryptographic identity.
> OH REALLY?! NO SHIT!!! I ACTIVELY USED 3 OS'S AND FAILED ON 2 BECAUSE OF YOUR SHITTY CLIENT, JUST TO UPLOAD MY FUCKING PUBLIC KEY!!!

You'll want to remove your PGP key from your Keybase identity.
> Hmm, yeah you might want to do so. Not because EFAIL or anything, just because Keybase clearly is a total failure on all levels.

Written quickly,
the Keybase team
> Well that's fucking clear. Could've taken some time to think before hitting "Send" though.

Don't get me wrong, I love the initiatives like this with all my heart, and greatly encourage secure messaging that leverages PGP. But when the implementation sucks this much, I start to ask myself questions about whether I should really trust this thing with my private conversations. Luckily I refrained from uploading my private key to their servers, otherwise I would've been really fucked.

Did I every tell you about that time I scared a boss (not mine, he was in the room) so much, that he was to scared to enter my office for the next couple of weeks? 😅
Good times 😊

Tl;dr: He was the reason I was working at max capacity and then he started complaining that shit wasn't working.

Full story:
I was out of office, building up a new site. I was the only IT working that day, others were out on vacation.
Suddenly I start getting flooded with calls from other sites, that nothing works. It is so bad, that my boss can't reach me on the company phone, so he calls me on my private phone.
Apparently all the servers are down.
So me into a taxi, heading for the main office.
When I get there I just start booting the servers on by one, because they didn't like that they had lost power. While I'm working, my boss is standing there, ready to help.
Another boss enters the office and goes: "I can't access Navision". To which I quickly reply something like: "Well everything is down, I'm the only one who can fix it and I'm working as fast as I can".
Two weeks later, another employee tells me, that the other boss has been running all his equipment off a battery backup, since the failure, because his power cord failed. He spilled a cup of coffee on it and therefore was the reason, that all the servers lost power (bad setup, I know). And apparently I was so frightening that he didn't have the courage to ask for a new power cord 😂
Best thing was that my boss never stopped me or told me that I did something wrong.

so my parents live close and my dad said his router died. I told him I had a spare. I loaded it dd-wrt on there and next time I went over there he asked about opening a port. I said I could do it for him. wouldn't let me touch it. he believes he is the computer genius and that I would brake it.

funny part is I work on servers for huge companies like Wells Fargo, Ally Bank and tons of others. I can configure a rack mount router but he doesn't trust me to touch a little crappy home linksys router lol.

no matter how smart you are they never want to admit their kid knows more about tech than they do.

Here's the story of my first month at CERN :) But first, a little premise...
Before arriving, I expected to be scared, alone and unguided in most of my experiences: after all I was a simple 19 year old about to leave home and friends for 3 years heading out in the world with zero experience on stuff like banking, taxes.. let alone working in a huge environment! The impostor syndrome was at an all time high on that front.
Then, I had the luck and pleasure to find an extremely competent and helpful plethora of people, ranging from my team to other CERNies (yes, that how we're called :P) who took me under their wing and introduced me to all the key aspects of living the place. When the initial stress finally soothed down thanks to this, I finally started to manage focusing more and more on my work, by following day-by-day my teammates who taught me the core aspects of the system and the many projects that are in progress during Long Shutdown 2. Within a couple weeks, I already managed to grasp various concepts that got me quickly on track, and now I managed to develop and integrate new temperature monitoring scripts into a system checking on hundreds of Single Board Computer-based servers :) It's a real rollercoaster of learning and applying under all fronts and so far I'm not regretting my choice of departing.
Luckily I've also discovered I'm pretty efficient and good at my job, which surely boosts my morale :D
Keep you updated as usual!

> be me
> spend 0.02 Ether (about €5) on one of those old-school MUD-style games
> send to the same Ethereum wallet from a previous purchase
> realize that the destination wallet changes for each purchase (probably to mitigate the fact that transaction history and contents in Ethereum wallets is entirely public)

> send an email to the game dev asking to return the transaction or pass it on to my player account
> *cricket noises*

About a week later, i.e. now:
*checks that Ethereum account that I accidentally sent that transaction to*
> $0 on it, transaction has been withdrawn

Now I couldn't care less about the €5 - it's only 2 beers worth - but what I do care about is honesty. Dear Chat Wars admin, that money wasn't yours. Also, I am one of those players that plays very few games but tends to commit to those I do play. The last one I played, I spent several hundreds of euros on over the couple of years I played it. I could've probably paid for your servers, spare time development and then some. But obviously not anymore. Choosing a quick grab of €5 over a relatively steady source of income from someone that tends to financially support what he likes... Re-evaluate your life choices.

Just like that incident with the stolen flash drive that was worth only €10... I couldn't care less about the raw value of them, but I do feel very disappointed in humanity when people go for a quick grab of such worthless things.

That's actually something that happened fairly recently.. just that I didn't have the energy left at the time to write it down. That, or I got my ass too drunk to properly write anything.. not sure actually.

So on paper I'm unemployed, but I do spend some time still on pretty much voluntary work for HackingVision, along with a handful of other people.

At the time, we were just doing the usual chit-chat in the admin channel, me still sick in my bed (actually that means that I wasn't drunk but really tired for once.. amazing!) and catching up to what happened, but unable to do any useful work in this sick state. So, tablet, typing on glass, right. I didn't have any keyboard attached at the time.

One of the staff members (a wanketeer from India) apparently had an assignment in a few hours for which he needed to write a server application in Java. Now, performance issues aside, I figured.. well I've got quite a bit of experience with servers, as well as some with client-server protocols. So I got thinking.. mail servers, way too overengineered. Web servers.. well that could work, I've done some basic netcat webservers that just sent an HTTP 200 OK and the file, those worked fine.. although super basic of course. And then there's IRC, which I've actually talked to an InspIRCd server through telnet before (which by the way is pretty much the only thing that telnet is still useful for, something that was never its purpose, lol) and realized that that protocol is actually quite easy to develop around. That's why I like it so much over modern chat protocols like XMPP, MQTT and whatnot. So I recommended that he'd write a little IRC server in Java. Or even just a chatbot like I attempted to at the time, considering that that's - with a stretch of course - a sort-of server too.

His fucking response however, so goddamn fucking infuriating. "If the protocol is so easy, then please write me down how to implement it in Java."

Essentially do his fucking work for him. I don't know Java, but as a fucking HackingVision admin, YOU SHOULD FUCKING KNOW THAT HACKERS CAN'T STAND LAZY CUNTS THAT CAN'T EVEN BE ASSED TO GOOGLE SHIT!!! If I wanted to deal with cunts like that, I'd have opened the page inbox with all its Fb h4xx0ring questions, not the fucking admin chat!

And type it on a goddamn fucking piece of glass, while fucking sick?! Get your ass fucked by a bobs and vegana horny fuck from the untouchable caste, because that's where you fucking belong for expecting THAT from me, you fucking bhenchod.

But at least I didn't get my ass enraged like that to say that to him in the admin chat. Although that probably wouldn't have been a bad thing, to get his feet right back on the ground again.

I’ve started the process of setting up the new network at work. We got a 1Gbit fibre connection.

Plan was simple, move all cables from old switch to new switch. I wish it was that easy.

The imbecile of an IT Guy at work has setup everything so complex and unnecessary stupid that I’m baffled.
We got 5 older MacPros, all running MacOS Server, but they only have one service running on them.

Then we got 2x xserve raid where there’s mounted some external NAS enclosures and another mac. Both xserve raid has to be running and connected to the main macpro who’s combining all this to a few different volumes.

Everything got a static public IP (we got a /24 block), even the workstations. Only thing that doesn’t get one ip pr machine is the guest network.
The firewall is basically set to have all ports open, allowing for easy sniffing of what services we’re running.

The “dmz” is just a /29 of our ip range, no firewall rules so the servers in the dmz can access everything in our network.

Back to the xserve, it’s accessible from the outside so employees can work from home, even though no one does it. I asked our IT guy why he hadn’t setup a VPN, his explanation was first that he didn’t manage to set it up, then he said vpn is something hackers use to hide who they are.

I’m baffled by this imbecile of an IT guy, one problem is he only works there 25% of the time because of some health issues. So when one of the NAS enclosures didn’t mount after a power outage, he wasn’t at work, and took the whole day to reply to my messages about logins to the xserve.

I can’t wait till I get my order from fs.com with new patching equipment and tonnes of cables, and once I can merge all storage devices into one large SAN. It’ll be such a good work experience.

MAINTENANCE OF STACKOVERFLOW PLANNED
SHARE TO YOUR NEAREST DEV FRIENDS

Stackoverflow and its relative partners will be closed for two days due to maintenance, new design, and moving server infrastructure from United States to 1km below the Switzerland Alps for extra layers of security. This decision was made by the recent CloudFlare data leak.
Now our servers will be able to handle data leaks because even though the data was leaked, it will fill the empty places in the rocks resulting inaccessible from attackers.
Stackoverflow and its relative partners' maintenance estimated time is February 29 - 30. We will try to finish as fast as possible and bring you guys the best experience. If the maintenance delayes, we will tweet via @StackStatus or post details in our status blog.
Thank you for your support and have a happy day.
Best regards,
Stack Exchange team

While I fucking hate front end and app development, I also hate that I'm so fucking dependant on them for the development of services and such.

Right now I'm developing a suite of services with a mate and while the backend and security (+linux servers) are something I'm good at, I find it hard sometimes to continue without being able to see my API's in action through apps and good looking interfaces.

My mate is currently handling that part but he has way less time than I do and thus I sometimes have to create interfaces to even just be able to see how my shit would work irl.

I can't fucking stand this and it sometimes entirely drains my motivation but there's also no fucking way in hell that I'll dive into frontend and/or app development.

Fucking hell.

Got an email earlier this week. It went something like this:

"It looks like your team still hasn't delivered the logging and monitoring solution that we asked for. Can you get it done in time for our production deployment next Friday?"

Um, wait, excuse me, WHAT?

1. You never actually asked for the thing you claim we didn't deliver. In fact, when we brought up the fact that you should probably have some monitoring set up for your servers, you said it would be handled entirely by your own team.
2. I HAVE BEEN WORKING ON THIS PROJECT FOR SIX MONTHS WHY DIDN'T YOU TELL ME YOUR DEADLINE UNTIL NOW
3. I won't even have time to start working on this until the Monday after your prod deployment date. Sorrynotsorry.

I really shouldn't be surprised though. This project has been a clusterfuck from the very beginning so this is just par for the course.

Hello devRant, a question for you.

I'm looking to redisign/setup my server 'infrastructure'.

It'll exist out of:
7 vps's (6+gb ram/500gb+/100mbs up/down per vps)
2 dedicated servers running as virtualization servers. (16gb/4tb/1gbit up/down and another one but let's leave that one out for now because it's gonna take a shit ton of time to solve that clusterfuck)

One server will function as an entry point for all websites I run, multiple database servers and multiple backup ones.

Any advices/tips/ideas?

Just a very serious hobby thing :)

So probably about a decade ago at this point I was working for free for a friend's start-up hosting company. He had rented out a high-end server in some data center and sold out virtualized chunks to clients.

This is back when you had only a few options for running virtual servers, but the market was taking off like a bat out of hell. In our case, we used User-Mode Linux (UML).

UML is essentially a kernel hack that lets you run the kernel in user space. That alone helps keep things separate or jailed. I'm pretty sure some of you can shed more light on it, but that's as I understood it at the time and I wasn't too shabby at hacking the kernel when we'd have driver issues.

Anyway, one of the ways my friend would on-board someone was to generate a new disk image file, mount it, and then chroot to that mount path. He'd basically use a stock image to do this and then wipe it out before putting it live.

I'm not sure exactly what he was doing at the time, but I got a panicked message on New Years Day saying that he had deleted everything. By everything, he had done an rm -fr /home as root on what he had thought was the root of a drive image.

It wasn't an image. It was the host server.

In the stoke of a single command, all user data was lost. We were pretty much screwed, but I have a knack for not giving up - so I spent a ton of time investigating linux file recovery.

Fun fact about UML - since the kernel runs in user space as a regular ol' process, anything it opens is attached to that process. I had noticed that while the files were "gone", I could still see disk usage. I ended up finding the images attached to their file pointers associated with each running kernel - and thankfully all customers were running at the time.

The next part was crazy, and I still think is crazy. I don't remember the command, but I had to essentially copy the image from the referenced path into a new image file, then shutdown the kernel and power it back on from the new image. We had configs all set aside, so that was easy. When it finally worked I was floored.

Rinse and repeat, I managed to drag every last missing bit out of /proc - with the only side effect being that all MySQL databases needed to be cleaned up.

So... We have a "network admin" who manages our network and the servers (windows) and I manage the Linux servers... He is having a real hard time to understand that the servers have no password but use ssh-keys to login and keep asking me for the credential to have them somewhere in case "something happens" like I quit or die...

Ok, I’m over windows. Done with it.

I have been a long time windows, I’ve used most versions since 3.11, and have used Linux for a few years on the side (not as a daily but have needed it for work and servers) but with yesterday’s update, not only have I lost audio for the countless time again, as far as Windows is concerned there’s nothing fucking wrong with it, besides the lack of sound and all.

Drivers are reinstalled, deleted and reinstalled, redownloaded from manufacturer, different drivers installed.
Ran a system restore back to before the update and just dropped the hdd into another laptop (it’s identical model) and still no fucking audio (to exclude the audio chip as failing)

So fuck it, I’m spending my weekend finding a distro that will work, I’m fucking done!

So, a few weeks ago I asked you guys what would you do if you were to quit your current job. I attempted a start-up with an old "friend" of mine.

He was very enthusiastic and hardworking at the beginning. However he received a job offer from somewhere and told me that he would work there just until we found the company officially and then quit and work full time on the project.

Well... I put around 300 hours into the project and developed the system, did most of my part according to the plan but the guy didn't deliver anything.

Turned out he had another secret partner whom he first introduced me as his mentor. I had my suspicions and suddenly shut down API servers, project management domain and mail server. Suddenly the quiet guy called me asking if I was alright and everything was fine.

Then, nothing happened. He went radio silence until I called him last night and he picked up the phone drunk and mumbled something like "I'm aware of you being a victim" then went to "you're passive aggresive" or something like that and I said nothing, just hung up.

I think you can figure out what went down on the other side and I'd like to hear your scenarios.

PS: now I have another start-up idea: I'm gonna pick up a flamethrower and burn the world while laughing histerically. Anyone who'd like to join is welcome.

*edits file on remote server*
WanBLowS: naah you can't 😈
*le wild BSOD appears for the over 9000-th time*

... Yeah. Windows, great job. Who needs system integrity when they're working on remote servers anyway, right?!

And to top it all off, le reboot mentions that they're working on fucking "features" again. That's what you needed to BSOD for?! For a goddamn motherfucking feature?!! Fucking piece of shit.

At least when I opened vim on that server again, it's saved everything neatly in the .swp files, ready for recovery. Now that's neat, isn't it? Microsoft, the Linux community has already moved on to nvim in terms of development, but maybe, just maybe, you can learn a thing or two from our "legacy software", vim.

As for me, maybe it's time to take out my Arch laptop again. At least that won't crap out on me because the sun and the stars are in a position that the OS doesn't like, or something stupid like that. FUCK YOU MICROSHIT!!!

Guess it's time to upgrade our office servers.

TLDR: There’s truth in the motto “fake it till you make it”

Once upon a time in January 2018 I began work as a part time sysadmin intern for a small financial firm in the rural US. This company is family owned, and the family doesn’t understand or invest in the technology their business is built on. I’m hired on because of my minor background in Cisco networking and Mac repair/administration.

I was the only staff member with vendor certifications and any background in networking / systems administration / computer hardware. There is an overtaxed web developer doing sysadmin/desktop support work and hating it.

I quickly take that part of his job and become the “if it has electricity it’s his job to fix it” guy. I troubleshoot Exchange server and Active Directory problems, configure cloudhosted web servers and DNS records, change lightbulbs and reboot printers in the office.

After realizing that I’m not an intern but actually just a cheap sysadmin I began looking for work that pays appropriately and is full time. I also change my email signature to say “Company Name: Network Administrator”

A few weeks later the “HR” department (we have 30 employees, it’s more like “The accountant who checks hiring paperwork”) sends out an email saying that certain ‘key’ departments have no coverage at inappropriate times. I don’t connect the dots.

Two days later I receive a testy email from one of the owners telling me that she is unhappy with my lack of time spent in the office. That as the Network Administrator I have responsibilities, and I need to be available for her and others 8-5 when problems need troubleshooting. Her son is my “boss” who is rarely in the office and has almost no technical acumen. He neglected to inform her that I’m a part time employee.

I arrange a meeting in which I propose that I be hired on full time as the Network Administrator to alleviate their problems. They agree but wildly underpay me. I continue searching for work but now my resume says Network Administrator.

Two weeks ago I accepted a job offer for double my current salary at a local software development firm as a junior automation engineer. They said they hired me on with so little experience specifically because of my networking background, which their ops dept is weak in. I highlighted my 6 months experience as Network Administrator during my interviews.

My take away: Perception matters more than reality. If you start acting like something, people will treat you like that.

My girlfriend configuring her e-mail account in the app because her phone had to be reset to factory :
-I can't figure out how to do these setting, annoying...
-Oh yeah the imap and smtp servers can be tricky, let me put that
(I Google the settings for her mail provider and put them in)
-It still doesn't work.
-Uuuh, maybe with another security setting, try it.
-This shit still doesn't work, seriously my phone is broken.
-Have you verified the e-mail address and carefully typed the password?
-Yes of course, I've tried it several time
(I take the phone and check all the parameters... During a looooong time... Until it hits me.)
-Hmm... Can you read the e-mail you've entered?
-Yeah, it's my mail, blabla@hotmail.com.
-No can you read it again please?
-It's blabla, why?
-No, can you *spell* your e-mail?
-Yeah it's B-L-A-B-L-A-@-H-O-M-A... Ow shit...
- ¯\_(ツ)_/¯

It's finally happened. I've used my mail servers for about a year to give out different email addresses on my domain to things I sign up for online, and only used my "actual" email address that received all this email for the whole domain but the single one that I used outbound for private communications.

This worked well for a long time as I could see when spam comes in, where it came from by looking at the email address I designated it. Each company's email would be sent not only from an email address that they choose, but also to an email address that I choose. It allowed me to easily determine where there were problems. For example, on Freenode IRC my vhost happened to make my username@host there a valid email address. It eventually got blacklisted due to too much incoming spam as crawlers started detecting it. Another one was "nickname"@my.domain as I posted it a few times here. Got crawled as well. But it allowed me to easily blacklist each.

I'd never thought my actual outbound email address, my real one, to get crawled though. That would require the mail server of a company I explicitly communicated with to get hacked. But today that happened. I wonder whose it is, but I can't tell.

Time to make my outgoing email bound to a designated email address as well. I want to know which companies this happens to, even if they don't disclose it.

I just love it when our clients decide to make a clone of live production server..then put it immediately online..and don't tell anyone about this.. and then start bitchin how data gets doubled all of a sudden..

Yeah, no shit sherlock.. you have two prod servers for 'hot swapping' and some services may only be running on one at a time.. You even have a manual on how to switch primary to secondary (turn off services on primary first, then turn them on on secondary and all)..or in case primary actually dies, just turn on services on secondary and you're good to go, right?

So how do ya think cloning the one with running services and putting the clone immediately online will work out?! 🤔

God, I thought it was common sense to not do that..but here I am, bitchin about how people fail to RTFM.. :/ or use brain..fuck..

Man, most memorable has to be the lead devops engineer from the first startup I worked at. My immediate team/friends called him Mr. DW - DW being short for Done and Working.

You see, Mr. DW was a brilliant devops engineer. He came up with excellent solutions to a lot of release, deployment, and data storage problems faced at the company (small genetics firm that ships servers with our analysis software on them). I am still very impressed by some of the solutions he came up with, and wish I had more time to study and learn about them before I left that company.

BUT - despite his brilliance, Mr. DW ALWAYS shipped broken stuff. For some reason this guy thinks that only testing a single happiest of happy path scenarios for whatever he is developing constitutes "everything will work as expected!" As soon as he said it was "done", but golly for him was it "done". By fucking God was that never the truth.

So, let me provide a basic example of how things would go:
my team: "Hey DW, we have a problem with X, can you fix this?"
DW: "Oh, sure. I bet it's a problem with <insert long explanations we don't care about we just want it fixed>"
my team: "....uhh, cool! Looking forward to the fix!"
... however long later...
DW: "OK, it's done. Here you go!"
my team: "Thanks! We'll get the fix into the processing pipelines"
... another short time later...
my team: "DW, this thing is broken. Look at all these failures"
DW: "How can that be? It was done! I tested it and it worked!"
my team: "Well, the failures say otherwise. How did you test?"
DW: "I just did <insert super basic thing>"
my team: "...... you know that's, like, not how things actually work for this part of the pipeline. right?"
DW: "..... But I thought it was XYZ?"
my team: "uhhhh, no, not even close. Can you please fix and let us know when it's done and working?"
DW: "... I'll fix it..."

And rinse and repeat the "it's done.. oh wait, it's broken" a good half dozen times on average. But, anyways, the birth of Mr. Done and Working - very often stuff was done, but rarely did it ever work!

I'm still friends with my team mates, and whenever we're talking and someone says something is done, we just have to ask if it's done AND working. We always get a laugh, sadly at the excuse of Mr. DW, but he dug his own hole in this regard.

Little cherry on top: So, the above happened with one of my friends. Mr. DW created installation media for one of our servers that was deployed in China. He tested it and "it was done!" Well, my friend flies out to China for on-site installation. He plugs the install medium in and goes for the install and it crashes and burns in a fire. Thankfully my friend knew the system well enough to be able to get everything installed and configured correctly minus the broken install media, but definitely the most insane example of "it's done!" but sure as he'll "it doesn't work!" we had from Mr. DW.

My first job was actually nontechnical - I was 18 years old and sold premium office furniture for a small store in Munich.
I did code in my free time though (PHP/JS mostly, had a litte browsergame back then - those were the days), so when my boss approached me and asked me whether I liked to take over a coding project, I agreed to the idea.

Little did I know at the time: I was supposed to work with a web agency the boss had contracted to build their online shop. Only that he had no plan or anything, he basically told them "build me an online shop like abc(a major competitor of ours at the time)"

He employed another sales lady who was supposed to manage the shop (that didn't exist yet). In the end, I think 80% of her job was to keep me from killing my boss.

As you can imagine, with this huuuuge amout of planning and these exact visions of what was supposed to be, things went south fast and far. So far that I could visit my fellow flightless birds down in the Penguin's republic of Antarctica and still need to go further.

Well... When my boss started suing the web agency, I was... ahem, asked to take over. Dumb as I was, I did - I was a PHP kid and thought that Magento, being written in PHP, would be easy to master. If you know Magento, you know that was maybe the wrongest thing I ever said.

Fast forward 3 very exhausting months, the thing was online. Not all of it worked yet, but it was online and fairly secure.

I did next to everything myself, administrating the CentOS box the shop was running on, its (own) e-mail server, the web server, all the coding required for the shop (can you spell 12 hour day for 8 hour pay?)

3 further months later, my life basically was a wreck, I dragged myself to work, the only thing I looked forward being the motorcycle ride home. The system worked though.

Mind you, I was still, at the time, working with three major customers, doing deskside support and some admin (Win Server 2008R2 at the time) - because, to quote my boss, "We could not afford a full time developer and we don't need one".

I think i stopped coding in my free time, the one hobby I used to love more than anything on the world, somewhere Decemerish 2012. I dropped out of the open source projects I was in, quit working on my browser game and let everything slide.

I didn't even care to renew the domains and servers for it, I just let it die without notice.

The little free time I had, I spent playing video games and getting drunk/high.

December 2013, 1.5 years on the job, I reached my breaking point and just left, called in sick at least a week per month because I just could not see this fucking place anymore.

I looked for another job outside of ALL of what I did before. No more Magento, no more sales, no more PHP. I didn't have to look for long, despite what I thought of my skills.

In February 2014, I told my boss that I quit. It was still seven months until my new job started, but I wanted him to know early so we could migrate and find a replacement.

The search for said replacement started in June 2014. I had considerably less work in the months before, looks like he got the hint.

In August 2014, my replacement arrived and I got him started.

I found a job, which I am still in, and still happy about after almost half a decade, at a local, medium sized ISP as a software dev and IT security guy. Got a proper training with a certificate and everything now.

My replacement lasted two months, he was external and never really did his job - the site, which until I had quit, had a total of 3 days downtime for 3 YEARS (they were the hoster's fault, not mine), was down for an entire month and he could not even tell why.

HIS followup was kicked after taking two weeks to familiarize himself with the project. Well, I think that two weeks is not even barely enough to familiarize yourself with nearly three years of work, but my boss gave him two days.

In 2016, the shop was replaced with another one. Different shop system, different OS, different CI. I don't know why and I can't say I give a damn.

Almost all the people that worked at the company back with me have left for greener pastures, taking their customers (and revenue) with them.

As for my boss' comments, instructions and lines: THAT might not be safe for work. Or kids. Or humans in general. And there wouldn't be much left if you put it through a language filter...

Moral of the story: No, it's not a bad thing to leave a place if you're mistreated there. Don't mistake loyalty with stupidity!

And, to quote one of my favourite Bands: "Nothing matters when the pain is all but gone" (Tragedy + Time by Rise Against).

Thanks to @C0D4 I rediscovered Folding@Home!

I've been running this on a very powerful server at home at full-speed for a few days now (quite some cores being used to the max right now, it's like I have a vacuum cleaner running full-time in my place 😄)

Then, last night it hit me that I have quite a few servers running close to idle (rented ones).....

I'm now running a total of 4 servers at full capacity with Folding@Home.

Can recommend!

My company just migrated our mail servers over to office365. My boss has been excited and could barely contain himself when the migration was done he was having the best day ever after he got a good deal on some new toys...Then I ruined it.

Me (setting up) > WTF!? um...well I guess I don't have email on my phone anymore. These permissions are fucked.
Him > Oh why?
Me > They are ridiculous, I won't give away this much control just to read email.
Him (panicking) > and if buy you a company phone?
Me > Not a fuck it's still a personal device. I'll just sandbox the web version.
Him > Your over reacting, they obviously need them for security blah blah...
Me (sends him the pic) > The minimum system requirement is internet.
(...silence...)

I feel kinda bad for killing his vibe - he's a nice guy and he's only trying to do right by us but now he seems down like his toy isn't shiny anymore because he respects me. I wasn't beating on the stack or his choice (mines running on thunderbird). I just can't support this trend of GOD mode permissions for email / calculator and other single feature apps. I'll use the web app instead. You have to draw the line somewhere...

On the other hand I can't deny that I'm loving the irony that Microsoft just made my life easier and have a deep sense of satisfaction that for the first time ever I got fuck up his Friday :/

My first times today:
First time a droplet on Digital Ocean.
First time Nginx.
First time trying to separate mail and website servers.
First time using UFW firewall.
First time Ubuntu webserver.
First try all alone configuration of my webserver.
First time installing all the stuff I need on my own, like MySQL, PHP and so on.
First time only SSH access from the beginning.
First time deployment from bitbucket.

Do you have any advise what I should think about. Or what software I will need. Or what I should think about.

DEAR CTOs, PLEASE ASK THE DEVELOPER OF THE SOFTWARE WHICH YOU ARE PLANNING TO BUY IN WHAT LANGUAGE AND WHAT VERSION THEY ARE WRITTEN IN.

Background: I worked a LONG time for a software company which developed a BIG crm software suite for a very niche sector. The softwary company was quite successfull and got many customers, even big companies bought our software. The thing is: The software is written in Ruby 1.8.7 and Rails 2. Even some customer servers are running debian squeeze... Yes, this setup is still in production use in 2022. (Rails 7 is the current version). I really don't get it why no one asked for the specific setup, they just bought it. We always told our boss, that we need time to upgrade. But he told every time, no one pays for an tech upgrade... So there it is, many TBs of customer data are in systems which are totally old, not updated and with possibly security issues.

Finally got ceph up and running, so far it does look really great!
At the moment we have 45 TB in the cluster, but we are able to expand it extremely easy.

Two servers with 256 GB RAM and 24 cores, three more with 32 GB and 12 cores.
Dedicated gigabit WAN connection.

The project is going forward strong, I'll let you know when it is time for beta testing.

Some time ago a salesman tried to sell me a super revolutionary solution. He introduced it with "today everyone will tell you that in order to save money you must move your servers and IT infrastructure on the CLOUD (big emphasis on the word) but we offer you a different approach: 'the on premise cloud'"
😶"so, you're basically telling me to replace my local machines with other local machines?"
😎"you don't see the whole picture: It's the cloud but INSIDE your company"

Am I dumb and I didn't see the obvious technology leap he was offering me?

Long time ago, back in a day of Microsoft Office 95 and 97, I was contracted to integrate a simple API for a payment service provider.

They've sent me the spec, I read it, it was simple enough: 1. payment OK, 2. payment FAILED. Few hours later the test environment was up and happy crediting and debiting fake accounts. Then came the push to prod.

I worked with two other guys, we shut down the servers, made a backup, connected new provider. All looked perfectly fine. First customers were paying, first shops were sending their products... Until two days later it turned out the money isn't coming through even though all we are getting from the API is "1" after "1"! I shut it off. We had 7 conference calls, 2 meetings, 3 days of trying and failing. Finally, by a mere luck, I found out what's what.

You see, Microsoft, when you invent your own file format, it's really nice to make it consistent between versions... So that the punctuation made in Microsoft Word 97 that was supposed to start from "0" didn't start from "1" when you open the file in Microsoft Word 95.

Also, if you're a moron who edits documentation in Microsoft Word, at least export it to a fucking PDF before sending out. Please.

My coworker doesn't know how to use a terminal. He talked himself into his position and instead of taking the time to learn about the basic commands he keeps asking someone else (including the teammanager, who's actually a software engineer) to do things for him.
For reference; we need the terminal to tail log files, keep track of processes, cron jobs, manipulate file structures, use scp (I use sshfs) to move things between other workstations and servers etc. Being able to use a terminal is one of the basic requirements for our job.

What.
Why.
How.

Why do people do this?

When I started my current job 6 years ago I was given a desk phone with a 100mb port. Speed didn't matter at the time as everyone was given laptops for our desks. I changed positions in the company where I'm going to be provisioning servers and whatnot over the network. Started using a desktop that didn't have a Wi-Fi adapter. I requested a new phone with a Gig switch port, if possible, so doing file transfers on the network wouldn't be limited. IT had a couple of questions...

IT-Have you noticed slowness when downloading/uploading?
Me-No, but its a 100mb port...so.
IT-Well I just did a speed test and we're getting 60up/5down. Your phone is over that.
(Working from home? Our fiber was way faster than that I thought.)
Me-That's fine, but this will be for internal network transfers. Not going out to the internet. We have gigabit switches on campus correct?
It-Yes but you shouldn't notice a speed difference.
Me, now lost-If you can't change out the phone that's fine. I'll figure our something.
IT-Now now, lets troubleshoot your issue. Can you plug your phone in?
Me-Yes I have it, but I'm remote today. There is no way for it to reach the call manager.
IT-Let's give it a try.

40 min of provisioning later he gave up and said maybe it is broke. Got a "working" one the next week.

PS first post, and writing on phone. Yay insomnia!

So, my boss was angry at me today because...
1. "Why are we taking so long to finish the software?". We started coding in March, and during that time I kept asking for requirements, design and his answer was, "You build it and we'll see." . During that time, after creating the system with only three type of user modes, he was like "Oh, I want customized user permissions." Took me 1 month to come up with a design, implementation for everything. Also during these months, nearly 2 months was wasted because he kept giving me other things to do, and I was not focusing in my current project.

Today he was mad because he expect me not only to build the infrastructure, setup servers, write backend code, do QA etc, He wants me to be a product designer. A fucking product designer. My answer to him was "If you refuse to help with designing the UX, either hire someone or I will just copy/paste things for internet. If the UI works, there's where my job is done."

Fucking hell. Not only I am being under payed, but he expects do to the job of 5 other people. Fuck this shit.

My school's servers can only support 100 connections at the same time. So they are always down. It's so lame

Best code performance incr. I made?

Many, many years ago our scaling strategy was to throw hardware at performance problems. Hardware consisted of dedicated web server and backing SQL server box, so each site instance had two servers (and data replication processes in place)

Two servers turned into 4, 4 to 8, 8 to around 16 (don't remember exactly what we ended up with). With Window's server and SQL Server licenses getting into the hundreds of thousands of dollars, the 'powers-that-be' were becoming very concerned with our IT budget. With our IT-VP and other web mgrs being hardware-centric, they simply shrugged and told the company that's just the way it is.

Taking it upon myself, started looking into utilizing web services, caching data (Microsoft's Velocity at the time), and a service that returned product data, the bottleneck for most of the performance issues. Description, price, simple stuff. Testing the scaling with our dev environment, single web server and single backing sql server, the service was able to handle 10x the traffic with much better performance.

Since the majority of the IT mgmt were hardware centric, they blew off the results saying my tests were contrived and my solution wouldn't work in 'the real world'. Not 100% wrong, I had no idea what would happen when real traffic would hit the site.

With our other hardware guys concerned the web hardware budget was tearing into everything else, they helped convince the 'powers-that-be' to give my idea a shot.

Fast forward a couple of months (lots of web code changes), early one morning we started slowly turning on the new framework (3 load balanced web service servers, 3 web servers, one sql server). 5 minutes...no issues, 10 minutes...no issues,an hour...everything is looking great. Then (A is a network admin)...
A: "Umm...guys...hardly any of the other web servers are being hit. The new servers are handling almost 100% of the traffic."
VP: "That can't be right. Something must be wrong with the load balancers. Rollback!"
A:"No, everything is fine. Load balancer is working and the performance spikes are coming from the old servers, not the new ones. Wow!, this is awesome!"
<Web manager 'Stacey'>
Stacey: "We probably still need to rollback. We'll need to do a full analysis to why the performance improved and apply it the current hardware setup."
A: "Page load times are now under 100 milliseconds from almost 3 seconds. Lets not rollback and see what happens."
Stacey:"I don't know, customers aren't used to such fast load times. They'll think something is wrong and go to a competitor. Rollback."
VP: "Agreed. We don't why this so fast. We'll need to replicate what is going on to the current architecture. Good try guys."
<later that day>
VP: "We've received hundreds of emails complementing us on the web site performance this morning and upset that the site suddenly slowed down again. CEO got wind of these emails and instructed us to move forward with the new framework."

After full implementation, we were able to scale back to only a few web servers and a single sql server, saving an initial $300,000 and a potential future savings of over $500,000. Budget analysis considering other factors, over the next 7 years, this would save the company over a million dollars.

At the semi-annual company wide meeting, our VP made a speech.

VP: "I'd like to thank everyone for this hard fought journey to get our web site up to industry standards for the benefit of our customers and stakeholders. Most of all, I'd like to thank Stacey for all her effort in designing and implementation of the scaling solution. Great job Stacy!"
<hands her a blank white envelope, hmmm...wonder what was in it?>

A few devs who sat in front of me turn around, network guys to the right, all look at me with puzzled looks with one mouth-ing "WTF?"

So one of our clients is changing the IPs of his systems (3 servers and one VPN).

It was scheduled that we should support them yesterday. They have tried to change the IPs yesterday for almost 16 hours, don't know they have done in all of that time.

Today I woke up at 8AM on a Sunday (in weekends I just want to sleep!!!!) to continue and hope that we finish fast.

BUT NO! THE fucking VPN IS NOT WORKING!! We are trying to do this over Team FUCKIN VIewer.

But guess what... they cannot either connect to the new IPs from their local systems.

First rant!

The first time I got in touch with programming was when I was about 14 years old. I started a private server for a game called Maplestory (yeah you know it, I know you do) and had one of the most popular servers.
Topping all the rankings of best servers, getting lots and lots of traffic...
Anyway, I started modding the game and implement new features and quests. Right until my father saw our bandwidth. Because the server was running on my computer in my own bedroom 24/7 and blowing nice hot air in my room.
Our bandwidth limit was reached in just a couple days in to the next billing cycle and had to shut everything down from that point. And this happened a few times.
I was devastated shutting it down but learned so much from it. And it introduced me to programming.

Up till now, I'm almost graduating in computer science, already have 2 companies that are willing to hire me, and probably even going to work with my dad on a huge app soon

Imagine yourself being a CTO back then.

Brand new Acura NSX. No MacBooks, ThinkPads are hot. Your company has its own skyscraper. CASE tools are just introduced and they’re hotter than blockchain now. You do software architecture in IBM Rational Rose, typing on your Model M and thinking really hard about Java OOP which is very hot right now.

You have Erlang servers at your own data center. You laugh at people writing in COBOL. You excited about aspect-oriented programming.

What a wonderful time.

I look after servers, softwares,vendors and write code too. I am also learning datascience in spare time. Suddenly I found that I am giving no time to family and friends.

Started work: 9:30am
Finished work: 9:30pm

Finally finished my "basically done" ticket from two days ago, though I'm still not sure if it's behaving correctly. It deals with the Apple wallet and iPhone notifications, and all I have is an emulator, so. 😕 Things work as expected maybe half the time? and idfk why. I'm going to leave the actual testing to QA since they have actual iPhones and it'll run on staging on actual servers with actual connectivity and actual pushes, so maybe then things will actually make sense. Until then? So done.

Started drinking: 9:30pm
Finished drinking: TBD

<media lyrics="I don't know what the fuck just happened but I don't really care, imma get the fuck up outta here. Fuck this shit i'm out" />

I previously worked as a Linux/unix sysadmin. There was one app team owning like 4 servers accessible in a very speciffic way.
* logon to main jumpbox
* ssh to elevated-privileges jumpbox
* logon to regional jumpbox using custom-made ssh alternative [call it fkup]
* try to fkup to the app server to confirm that fkup daemon is dead
* logon to server's mgmt node [aix frame]
* ssh to server directly to find confirm sshd is dead too
* access server's console
* place root pswd request in passwords vault, chase 2 mangers via phone for approvals [to login to the vault, find my request and aprove it]
* use root pw to login to server's console, bounce sshd and fkupd
* logout from the console
* fkup into the server to get shell.

That's not the worst part... Aix'es are stable enough to run for years w/o needing any maintenance, do all this complexity could be bearable.

However, the app team used to log a change request asking to copy a new pdf file into that server every week and drop it to app directory, chown it to app user. Why can't they do that themselves you ask? Bcuz they 'only need this pdf to get there, that's all, and we're not wasting our time to raise access requests and chase for approvals just for a pdf...'

oh, and all these steps must be repeated each time a sysadmin tties to implement the change request as all the movements and decisions must be logged and justified.

Each server access takes roughly half an hour. 4 servers -> 2hrs.

So yeah.. Surely getting your accesses sorted out once is so much more time consuming and less efficient than logging a change request for sysadmins every week and wasting 2 frickin hours of my time to just copy a simple pdf for you.. Not to mention that threr's only a small team of sysadmins maintaining tens of thousands of servers and every minute we have we spend working. Lunch time takes 10-15 minutes or so.. Almost no time for coffee or restroom. And these guys are saying sparing a few hours to get their own accesses is 'a waste of their time'...

That was the time I discovered skrillex.

Just as I wait for my train, some advertisers from a utility company here approached me. Asking what my company is etc..
Me: "well I'm making my own company..."
*Looks at their pamphlet*
"Oh, utility company you mean. My apartment building has solar panels."
Them: "oh you know about electricity right... And F-16, the fighter jets that fly at 3000km/h"
(My neighbor is a former aerospace technician who mentioned that previously, should be about right)
Them: "they fly faster than electricity!"
Me: "but um.. electricity travels at the speed of light..."
Them: *avoid subject*
Them: "yeah it travels 7 times around the globe in 1 second"
Me: *recalls ping to my servers in Italy*
"Yeah to Italy my ping is about 300ms if memory serves me right... So that'd make sense"
(Turned out to be 40ms.. close enough though, right 🙃)
Them: "don't travel too much at light speed, alright!"
*They pack up and leave*

Meanwhile me, thinking: but guys.. all I wanted to do was smoke a cigarette before my train comes. Why did you waste my time with this? And uninformed time wastage at that.

Advertisers are the worst 😶

I'm actually a Dev, mostly just a shell scripter who needs to support 500 servers which run our applications. I install the new versions and check whatever is wrong if there are customer issues.
One release weekend everything went wrong, Development had to make new builds on the fly with hardly any time for testing.
It took 18 hours with no break.
It was extremely hard to concentrate, but being in the Skype group with everyone and finally getting everything fixed was quite rewarding.
Everyone just opened a beer and we stayed on the call for about 30 more minutes just to relax.

I like our Dev team way better than I like my actual colleagues, who merely mess things up and call me for the smallest thing without even thinking.

Its 1:30am (South Africa) just had my 8 drive raid 50 fail on one of our main servers after 3 sudden drive failures at the same time after lightning struck our network and blew 2 Cisco 48 port switches...

Its gonna be a longgg fucking night.

I've been staffed on a old ongoing project, first day.
0. Compatibility has to be guaranteed down till IE9... ppf.
1. Front end made in XHTML+JS(jQuery)... bah, ok.
2. XHTML+JS is actually generated by PHP5.4, not a line is actually statically served... beh, funny, ok.
3. PHP files are the output of an XSLT transform of a bunch of XMLs... meh, seriously? Oooook.
4. XMLs are the product of the serialisation of a truck of stateful JavaEE6 DTOs populated magically (undocumented) with data coming from a SQL DB... WTF mode!!!
5. Session logics lives within PHP-land at point 2, front end makes ajax calls here that propagates to another WS out of our control that triggers -somehow- (undocumented) our Java backend at point 4 to generate new XMLs and then reach front end again. Kill me now.

Boss: look... it's too slow for the client, it's too heavy on our servers: fix it. Ah, and we sold 85% test coverage by October. You're the man for the job. (I'm a Node.js fullstacker and right now there's not even a testing scaffold, ofc).
Me: prod is on Linux or Windows?
Boss: RHEL7.
Me: rm -rf / as root. Done.
Boss: I know I know...
Me: ...

I think time has come...

Why the fuck did you guys change my vps to read only file system.

WHAT IS FUCKING WRONG WITH YOU, THIS IS LIKE THE 6TH TIME.

What do you expect to tell my clients?

I should look into if I have legal rights to migrate everything to a more stable provider.

Sick and tired of having issues because other people fck up. It's a shame I have been using this provider for 5 years without any problems. And now servers left and right start having problems.

New developers. Tip: There is no silver bullet.

If you like Python, please understand GIL's behavior before making a system that handles thousands of requests.

If you like Java, know that "Write once, run anywhere" is a fallacy. Even application servers don't like the same WAR.

If you like PHP, understand the life cycle of a request before connecting to the database from all corners.

If you like C#, don't make it a small command-line application that will be used on FreeBSD.

If you like C, meet valgrind.

If you like C++, templates are cool, but don't overdo it. And take the opportunity to meet valgrind.

Never use the same tool to do everything. Elect the language and framework for the given need with rationality.

Every time I see a "Java Man", a "C++ Chad" or anything like that, it comes to mind that if he were a carpenter, he would be tightening screws with hammers.

Every lock-in is bad.

4th grade. My parents left for the night and got a babysitter for me and my younger siblings. The babysitter showed me a game she likes in IE. You could make an account, raise a digital pet, a "neopet," play flash games for points to buy items from other users who listed them in their own stores that had custom css/html, including bg music. This was the first time I had really witnessed what the web could look like. Animated tiled gif bgs really amazed me so I took to google with which I discovered sites where one could copy css and html snippets for themes. I stored each new html tag I discovered via w3cschools.com in a powerpoint where the snippets I found were pasted somewhere randomly in the ppt. From there I learned html, CSS, and a billion other things. To date I've made websites, apps with several langs in win/Linux/osx/Android (but not ios yet). I've managed servers, and databases , and DNS records. I've in even ran website with 100k requests a day.

Absolutely seamless upgrade from El Capitan to Sierra. Total time from downloading and installing Sierra, testing all my apps, servers, VMs and VPNs....under 90 minutes 💪

"Our company encourages cryptocurrency big data agile machine learning, empowerment diversity, celebrate wellness and synergy, unpack creative cloud real-time front-end bleeding edge cross-platform modular success-driven development of digital signage, powered by an unparalleled REST API backend, driven by a neural network tail recursion AI on our cloud based big data linux servers which output real time data to our Wordpress template interactive dynamic website TypeScript applet, with deep learning tensor flow capabilities.

Don't get what the fuck I just said? Udemy offers countless courses on python based buzzwords. Be the first out of 13 people to sell your soul and private information, and you'll get the first three minutes of the course free!"

Six or seven years ago, I worked for a large financial organization as part of a very large effort to convert server assets from physical to virtual. The consultants on site were in bed with the vendor of a terrible piece of software designed for that purpose. After a couple weeks on the job I'd had it, and sat down in between sessions of "validating" the conversion procedure, and started writing my own software for converting Linux servers. After a couple days it was working great, and they wound up adopting my software as the default method for Linux conversions.

Years later, I'm interviewing for my current job and one of the interviewers tells me he used my converter some time later and loved it. Pretty sure it's what swung the interview for me.

At work we have a....err....pool of usefulness awareness situation.

You see. The school does not know all that me and the lead developer does. Our head of department does. Our manager used to do(fore she left) but as far as the rest of the 2 fucking campuses are concerned, all of our shit works through magical dwarves that live inside the servers in our server rooms.

Normally, we are fine with this. But whenever one of us misses, or decides to take the day off and some shit hits the fan...THEN and only then do our users know what we do.

So we have a "pool" of items that seem to be picked every time one of us is not there. The more it gets picked, the more people are aware of us, the more they figure out what we do...the more they start to ask for customization and shit like that.

Such a funny situation. Wonder if this will get in the way of friday morning coffee and magic the Gathering at the office.

Or me and the cms admin playing videogames together.

Oh well

A few days ago Aruba Cloud terminated my VPS's without notice (shortly after my previous rant about email spam). The reason behind it is rather mundane - while slightly tipsy I wanted to send some traffic back to those Chinese smtp-shop assholes.

Around half an hour later I found that e1.nixmagic.com had lost its network link. I logged into the admin panel at Aruba and connected to the recovery console. In the kernel log there was a mention of the main network link being unresponsive. Apparently Aruba Cloud's automated systems had cut it off.

Shortly afterwards I got an email about the suspension, requested that I get back to them within 72 hours.. despite the email being from a noreply address. Big brain right there.

Now one server wasn't yet a reason to consider this a major outage. I did have 3 edge nodes, all of which had equal duties and importance in the network. However an hour later I found that Aruba had also shut down the other 2 instances, despite those doing nothing wrong. Another hour later I found my account limited, unable to login to the admin panel. Oh and did I mention that for anything in that admin panel, you have to login to the customer area first? And that the account ID used to login there is more secure than the password? Yeah their password security is that good. Normally my passwords would be 64 random characters.. not there.

So with all my servers now gone, I immediately considered it an emergency. Aruba's employees had already left the office, and wouldn't get back to me until the next day (on-call be damned I guess?). So I had to immediately pull an all-nighter and deploy new servers elsewhere and move my DNS records to those ASAP. For that I chose Hetzner.

Now at Hetzner I was actually very pleasantly surprised at just how clean the interface was, how it puts the project front and center in everything, and just tells you "this is what this is and what it does", nothing else. Despite being a sysadmin myself, I find the hosting part of it insignificant. The project - the application that is to be hosted - that's what's important. Administration of a datacenter on the other hand is background stuff. Aruba's interface is very cluttered, on Hetzner it's super clean. Night and day difference.

Oh and the specs are better for the same price, the password security is actually decent, and the servers are already up despite me not having paid for anything yet. That's incredible if you ask me.. they actually trust a new customer to pay the bills afterwards. How about you Aruba Cloud? Oh yeah.. too much to ask for right. Even the network isn't something you can trust a long-time customer of yours with.

So everything has been set up again now, and there are some things I would like to stress about hosting providers.

You don't own the hardware. While you do have root access, you don't have hardware access at all. Remember that therefore you can't store anything on it that you can't afford to lose, have stolen, or otherwise compromised. This is something I kept in mind when I made my servers. The edge nodes do nothing but reverse proxying the services from my LXC containers at home. Therefore the edge nodes could go down, while the worker nodes still kept running. All that was necessary was a new set of reverse proxies. On the other hand, if e.g. my Gitea server were to be hosted directly on those VPS's, losing that would've been devastating. All my configs, projects, mirrors and shit are hosted there.

Also remember that your hosting provider can terminate you at any time, for any reason. Server redundancy is not enough. If you can afford multiple redundant servers, get them at different hosting providers. I've looked at Aruba Cloud's Terms of Use and this is indeed something they were legally allowed to do. Any reason, any time, no notice. They covered all their bases. Make sure you do too, and hope that you'll never need it.

Oh, right - this is a rant - Aruba Cloud you are a bunch of assholes. Kindly take a 1Gbps DDoS attack up your ass in exchange for that termination without notice, will you?

Giant, month-and-a-half-long-ticket.
After learning six or so complicated areas of the system and updating them all to work with the new changes, make them all play nicely, etc. I finally got everything working. 95% spec coverage, though no ui tests because I haven't gotten selenium working. whatever, everything's done and works.

Second dev bases her ticket off of mine and continues working. Work elsewhere continues and there's an official release, so we both merge in master. I run tests, everything passes, and go back to working on other tickets.

She finishes her ticket.
We do end-to-end testing, and everything works perfectly. Time for a demo!

She merges in master again, and pushes her branch to two staging servers. (idk why two.)

Demo starts.
We connect to the staging servers, and... none of the UI changes exist; they aren't running the correct code!

So she runs it locally for a demo instead. Two features in my ticket no longer work. She throws me under the bus. She throws me under the bus again by criticising a rake task I scrapped because she wanted to do it. Then again because I didn't update my branch to master and push it before the demo, despite having no reason to. and despite the demo being of her branch.

Then she continues to show off and brag about how she's like the "legend" (senior dev) she envies. QAbuys it.

I'm having an emotion, and it's called anger.

I think I want to quit.

I know it’s a bit of an inconvenient time with there being corona around but everything was okay up till January. I’m a junior even though I shouldn’t be. Since my manager told me and my team leader senior in my review “maybe you two should switch jobs” things have been going downhill. I think the team lead had it out for me and didn’t put me on a new project, I’ve been left with doing stupid basic shit like updating text on websites in a cms and doing fuck all and then there’s also another guy that was basically harassing me trying to put me in my place any time I was doing better than him and literally both of them been like that ... and now that I’m working from home it’s even worse. I don’t have any kind of assurance that everything okay and actually I think I’m being framed as welll since I found keyloggers on my work laptop and deleted cleaned shit up the past two weeks and changed my WiFi security as there were like 5 unknown devices on our network so yeah .. I’ve been framed and they made it out like I put a powershell script on one of the servers and it crashed a Porsche website for 8 h and all kinds of bullshit - this was yday. On Tuesday they logged me out of everything like changed the password for work vpn and kicked me out of slack and Microsoft teams for over 2 hours till the end of shift and two managers weren’t answering their phone and then next day my manager called and apologised that saying that he “accidentally” did that to me along with 15 people they let go from the company....

I’m seriously thinking of quitting being removed from team group for a moment , not being on a project and people literally trying to put me down after I know I’m genuinely smarter than them and if I had over 10 years experience like those on my team (I have 1) I’d be far higher up and better

They can genuinely just go fuck themseves !!!! And here I was going to work over weekend on something! No fucking way I just wanna quit or give in my notice but because of corona I’m divided

Tldr: I just FAILED AN EXAM because of my uni's SHITTIEST SERVERS EVER. YEAR AFTER YEAR THE EXACT SAME SHIT HAPPENS, THEIR SERVERS OR NETWORK FAIL HARD AND STUDENTS FAIL EXAMS BECAUSE OF IT, AND THEY NEVER FIX IT.
FUCK

In the middle of a 3h exam at uni, using Matlab through virtualdesktop, when suddenly it stops working entirely. Trying to restart it, no go. A few other people are having the same issue as well.
After 20 minutes of waiting, one IT tech shows up saying people in other labs are having the same issue, big problems.
Ended up wasting nearly an hour waiting for shit to work again - which it did, but only when the extra time they had given us ran out.
Desperately tried to salvage what I could of my exam, but I don't think I did enough to pass.

This is not the first time it happens. Exam after exam, shit like this keeps happening over and over again, and every time some students get fucked in the ass and have to take the exam again - the next semester.

This is supposed to be one of the top universities in the country, and top 20 engineering schools in the world.
FUCK THIS SHIT

One job I picked up was for an IoT Start Up. It was quite interesting work, reporting to the technical director, who was an electronics engineer, who was designing the hardware himself, they had a couple of firmware guys already, and just needed someone to take care of the software.
So they said they needed something in Azure that they could stream their data to and provide analytics for their clients. It had to be Azure, and it had to be Azure Native, and was to be Multi-client, as they had a deal with Microsoft to showcase how well Azure works in the IoT space at an exhibition/conference in 3 months time.
So I worked flat out for 3 months, on a whole variety of technology, from C++ to get the radio packets from their IoT chip, Python to run on the hub to take the data from the C++ and stream it to the cloud, Azure IoT Hubs in every continent to receive the data and store it an a Cosmos DB, and then Power BI analytics wrapped up in an Angular front end that the clients could log into.
Got it finished 2 days before the show, and they were so pleased I got flown business class to Singapore to be on the stand and talk to customers.
The first sign of trouble was when we arrived at the show to find we just had one of those little circular tables with two stools in the middle of the floor, about two feet across and no power.
No problem, I was able to sort that, swapping laptops in and out.
Microsoft were really happy with what we had, and couldn't believe I had thrown it all together in 3 months.
We picked up a potential customer for the system, a major Asian Telecoms company.
Then when we got home, the CEO swooped in. I had never met this guy before. Imagine one of the VC guys from Silicon Valley, or the CEO from the IT Crowd. You get the picture. Could talk the hind leg of a donkey, and real street smart, but no brains. He insisted on "taking it from here" and flew alone to strike the deal with the customer. Came back with an MOU in his pocket and said to me, their guys will be in touch with you.
Then I got a call. Can you send us the source code and tell us how what servers we have to run this on?
Um, its cloud native.
No, we can't use a cloud it has to be on our servers - your CEO told us that was no problem..
He hadn't even taken the trouble to find out what it was we had built, and what he was selling.

Got drunk during work time because fuck openvpn, fuck anyconnect, and fuck badly designed vpn servers in general.

A fucking pptp connection, man, a fucking pptp.

ATTENTION @dfox!!!!

I found a bug in the devRant REST server. When you upload a rant with a vertical image (when the height is bigger than the width), the server seems to flip between the width and the height (I don't know if it happens all the time), and when I fetch the rant, the JSON data for image width and height is flipped! (in the JSON, the attached image section contains width and height information about the image, and they are flipped in their values, meaning that the width key is equal to the height, and the height key is equal to the width).

An explanation on the image below:

The small window on the left shows the real info about the image. Notice how the height is 800 and width is 600.

The window at the bottom shows the data fetched from devRant servers. Notice how the width returned from the server is 800 and the height returned from the server is 600.

ArmA 3, a great sandbox that I "wasted" a lot time scripting modding or, if you like to call it that: developing for.

A game so great, the Multiplayer-server-browser stores the amount of players on a server in an 8 bit integer.

Someone complaint a few years ago.
Response: " be happy, it were 4 bits not to long ago"

There were Servers who ran into that problem.

To clarify, that only affected the shown number, not the amount of players, at least not directly.
Who likes to be lonely in a multiplayer game.

Fucking randsomware ecrypted 3 fricken servers in our god damn office for the 2nd time this year. These plebs need to learn to use the internet.

Time to patch all the Linux servers..
Thank you Dirty Cow

How many here know surge.sh? it seems each time I mention it somewhere, nobody knows it, checks it out and thanks me. It basically allows you to host a static webpage on their servers - for free and with custom domain if you wish. They are also the creators of Cordova/Phonegap.

Never buy crappy, consumer-grade SSDs for use in production servers/RAIDs. This might sound obvious but at the company I used to work for, through a series of bad decisions by management and cheapness, we ended up with the cheapest consumer SSDs you can imagine powering all of our storage.

This turned into a nightmare spanning years of failed hard drives and a continues cycle of ridiculousness. Drive failed after a few days, gets taken out, sent back to manufacturer and then replaced with another equally crappy drive destined to fail within days/weeks.

Our ops people were going to the data center multiple times per week to replace failed drives. Lesson I learned: cheaping out on system-critical hardware and software can have long standing consequences and in the end usually doesn't end up actually saving money when you account for time employees have to spend dealing with issues that result from it.

The problem I have with atom, vscode, sublime, and notepad++ is that none are available on the command line over SSH, inside tmux. And that's where I do the vast majority of my text editing.

The first text editor I used on the command line was pico, the technological successor of which is nano. I used it because when I was in college in the late '90s, we used pine for our email, and pico was the default editor for pine.

When I got my first job out of college in 2000, I found out about vi, and very quickly fell in love with it, and its technological successor: vim.

The only reason I've never gotten into emacs is because I've never wanted for more than vi/vim. And also because as a system administrator, I'm logging into dozens, of not hundreds of servers a day. While vi or vim is guaranteed to be on all of them, emacs is not.

So, for me, the use of a desktop text editor like the ones I mentioned at the beginning of this post, just doesn't make sense to me. I almost never edit files that live on the computer where I'm sitting, and I'm not interested in doing a commit/push every single time I want to rerun a script.

remember when our whole dev team was drunk but we went back to work anyway? turns out, someone rm -rf a whole folder of js scripts!! it took us some time before we could find the back up files which were in the old server (that crashed) because recently we changed servers (so we were transferring files) and we are now using amazon.

our boss got so pissed this morning but oh well, it was his idea to drink and feast in the first place!

Lotus Notes is absolutely the worst platform known to man. Every time a ticket comes in I want to set the Domino servers on fire.

DONT do production stuff on friday afternoon. This friday evening we had an issue on production and just wanted to do a quick fix. The fix resulted in a ddos attack that we accidentally started on our servers in an IoT project. We contacted all customers' devices and asked them for response at the same time. Funny thing is that the devices are programmed to retry if a request fails until it is successful. We ended up with 4 hours downtime on production, servers were running again at 11pm.

Italian gaming community is a shit place. In my spare time I make website for minecraft servers or whatever game you want, but I found out that people prefer to have a bad website but pay less instead of pay more for a decent website.

P.S. I'm not saying that I'm the best italian web designer.

It’s really painful how non technical people think all technical stuff is the same.

The amount of times I’ve been brought in to help someone with a printer or WiFi network.

I didn’t do a computer science degree for this.

I’m also pretty sure my last two bosses think fixing the printer is the same as creating large scale systems.

My last boss wanted me to build a Vr chatbot despite spending the last few years building web servers and react/redux apps. I mean I can probably figure it out and it’ll be fun, but don’t think I’m gonna get this done in a week with all my other responsibilities too.

My new boss wants me to singlehandedly build a massive marketplace system that would probably take a team of ten, several years.

Fed up of being around non technical managers who don’t listen to me with my feedback.

Time for a new chapter in the new year.

them: welcome new project members, this is our CI/CD pipeline which is completely different from the rest of the company, there won't be any great knowledge transfer, we just expect you to be able to know and use everything. but also, we expect you to work on your tasks and don't waste any time.
me: okay, so my tasks aren't going as fast as expected, because I need to invest some learning so i can set up my project correctly.
later: some help would be nice, i'm stuck right now
coworker: *helps me to fix my problems, which were partly due to misconfigured build servers* i know it's a lot, and unfortunately, for this topic sources on the web aren't so good. i can really recommend this book, this will give a deeper understanding of the topic.
me: okay, yeah i mean, tbh, i'll read the book if the project invests some time for me so i can learn everything that's required, but this won't happen. also, some initial workshop on the topic or anything would have been nice.
coworker: well, i mean, i am a software developer. for me, it is normal that i learn all that stuff in my free time. and i think that's what the PM expects from us.
me: okay, that's fine for you, i mean, if i'm interested in a topic, i will invest my private time. but in this case, PM would just expect me to do unpaid labor, to gain knowledge and skills that i can use in this specific project. i'm not willing to do that.
coworker: ...
me: ...

it's not that i don't want to learn. the thing is that there isn't any energy left by the end of the day. i'm actually trying to find some work life balance, because i don't feel balanced right now, haven't felt since i started this job.
also, this is only one of several projects i'm working on. it's like they expect me this project has top priority in my life. if it wasn't so annoying on different levels, maybe i'd have a more positive attitude towards it.

also, at the moment i find it fucking annoying that i have to invest so much time in this dev ops bullshit and this keeps me from doing my actual work.

if they are unhappy with my skills, either they can invest in my learning or kick me out. at this point, either is fine for me..

2017 has ended and I want to throw this fucking rock of anger with it out of the Window.

I will tell you how my last 3 to 4 years as private IT Guy (IT Dumpster) for everything was.

My Gaming Community:
It was so fucking retarded that I would like to punch everyone there in the face over a thousand times.

Seriously there are over 60 People in that Community and they take every shit for granted and only cry about everything.

I'm Paying since 4 Years over 60 € for our Server every month. Some people donated some small bucks over the time and in the end it was around 150 € Donations over 4 years.

Im thankful for the Donations, but these people really think that they can demand for this Donations private Gameservers for over years for free or think that they now own the Servers. WTF?? FUCK YOU!!

Im managing 2 Linux Server with Web, MySQL, Voice and Gameserver. What did 1 guy donate? 10 € for 4 years, thats around 15% for just a month. You would get our Teamspeak for 7 Days for that price idiots!!

I did all the damn Webdev and am now writing a App in Kotlin for Android man!!

Since last everybody in the Community gets on my nerves, because they are so ignorant and dont appreciate the work of others 😡.

My Association where I support the hole IT with a friend:

We decided in the commitee that we would take on major changes in our infrastructure.

We need to finish the barebone concept in the end of January 18.
Early December 17 I wrote my friend, that we need to have a meeting and plan everything.

Well, now I mostly designed the roadmap by myself and didnt receive any reply from my friend 💀.

I really dont understand why the shit always sticks at me and I need to finish it.

I really love to code something, install or repair PC's with joy or manage Server but the people around me are in my free time like poison and they take out my fun for it.

The only fun and enjoyable IT Part for me is at my fulltime job. Thats the only good place left for me as Safe Heaven and the people there really appreciate my knowledge and work I do.

!rant

MASSIVE UPGRADE ROUND 2:

We took it by steps, the DBA did his portion and I did mine, we had waited for the entire thing to be finalized today on Sunday since our users are probably jerking off to their waifus (as they should) and today was my part. MA BOE the DBA was with me the entire time and the whole process took us about 4 hours of both of us getting multiple heart attacks here and there and praying to the elder gods of Asgard for their devine protection as we venture into the calamity of fire and juten ass mfkers that are our fucking servers for this particular process.

Man I really hope for the pandemic to be over and take my dude out for a nice beer, some wings and some relaxation time.

Best DB/Dev team I have ever been with.

I was reminded of people's posts about preferred text editors in another post, so I thought I'd do the same, but also add some super old technology that I used along the way.

The first text editor I consistently used was pico. I used it to write my first webpage at school.edu/~username. It was a natural choice, because the it was the default text editor in pine, which is what we would all use for our email after opening a serial connection to the college's Digital Unix server. Or if we were the lucky ones who had a computer in a wired dorm, telnet. My dorm was not wired until my sophomore year.

I got my first job in tech in 2001, working as a night shift tier-one support technician. By this time, most people were using web based email, or POP3, but I wanted to keep using pine (or elm, or mutt) because I was totally in love with the command line by this time, and had been playing with Linux for two or three years by now. I arranged a handshake deal with a guy in my home town who had a couple well-connected NetBSD servers, to let me have an account on one for email and web hosting (a relatively new idea at the time).

I recall telnetting into my shared hosting account from the HP-UX workstations we had in the control room. I would look at webpages on HTML conventions and standards, and I kept seeing references to this thing called vi. I looked into it more deeply, and found that it was a text editor, and was the reason I always had to CTRL-Z out of elm. I was already finding pico to be lacking, so I found a modern implementation of vi called vim that was already installed on the aforementioned NetBSD server, and read through vimtutor on it. I was hooked instantly. The modality massively appealed to me, and I found editing files to be an absolute delight, compared to pico, and its nascent open source offspring/successor, nano.

My position on that hasn't changed in the years that have passed since then.

What's your text editor origin story?

I took like 3 years to my company to get this huge-ass client to ask us to remake their website (the client is already our client for other purposes).

The old website was hosted on their local machine, behind a proxy that was there for other 30 website servers.

The old website took like 30-40 seconds to load on a browser and had a google score of 3-6/100.

We made the new website in wordpress, since it was basically a blog and managed all of the older links to redirect to the new pages so that SEO wouldn't get affected.

We then asked the previous developers to let their domain redirect to the new one (it was like example.com => ex.example.com and now it's just example.com, so we needed them to make ex.example.com redirect to example.com).

What they did was making a redirection to the 404 page of the new website, making everything go to fuck itself.

Damn this might be the first time I despise other developers, but this move was fucking awful.

I mean, I get it, we stole your big client, but it's not our fault if we made the google score go up to 90/100 in a week just by changing server and CMS.

I was already about to hit my head against the wall: was trying to install nginx all the time, but was greeted by apache default page, over and over again I re-installed the servers, tried connecting directly to the server ip, changed server hardware, picked different distros, manually build from source, did everything possible, even searched the whole system for "apache" and different regex...

It was chrome cache........ after I wiped cache I was greeted by welcome to nginx...... 10 hours wasted.......

How I knew this was for me.... I didn't.
It kind of just happened in the natural order of things.

I was once a wii young lad who had a dream, and that dream became a smashing pile of being broke, jobless and unemployable, not a great way to start off that early life but hey, it was what it was.

So I looked at my computer one day, lousy dusty Pentium 4 with a massive 80GB HDD, in the corner, and went... fuck it, this thing is going to make me money.

So from there I picked up my old high school book on VB6 and on with it I went, forcing my self to make that calculator I couldn't do in school and a few other things, from there I got into a course for webDev, not uni, and after being dropped from that course ... that's a story for another time, I basically said fuck the system and my journey into webDev took on a life of its own.

Starting with frontend (back when layouts where tables and css was font colours) and IE5 was still a thing, and progressing into JS for a fucktonne of "onClick" events, then backend... I went down the .PHP3, PHP4 hadn't been released yet, but at the time .ASP was a thing too although it was complicated as fuck.

For many years it was just 1 thing after another, picking up MySQL, screwing around with databases, setting up linux servers, gobbling up Python a couple years later and started automating different things, just building site after site, until one day I landed a professional gig - not just casual freelance stuff, and from there when you think you know a lot, what I thought I knew got blown out the window and imposter syndrome sunk in, but I kept pushing ahead.

That saying "you don't know what you don't know", it has meaning here, you don't know what you don't know... but the moment you know you don't know enough, you either crumble or you keep waterboarding yourself in knowledge to reduce the unknown.

And somewhere along the line I accepted this path.
It may have taken me a few years to get off my feet but I'm glad I took that first step.

I kinda started programming back in the day, by breaking 2 school servers using a simple batch "virus"... it effectively opened tons of porn videos, threw the mouse cursor to the top right corner of the screen and shut down the PC after 15 seconds.
I masked the file and made sure it looked like internet explorer (since my school, back in the day, didn't use chrome and didn't allow the installation of chrome)
It was meant to troll my friend but all computers shared files with each other which meant even the private PC's of my teachers got my "virus"
Eventually it landed in the startup folder and messed everything up. I got snitched out, and I had to fix it
(I literally just wrote another batch file to delete my so called virus... because of the 15 seconds time limit I had)

At a previous job, boss & owner of company would waste hours of my time to show me, at his own desk, every small detail of some random feature he had fallen in love with on some random webpage he found, while saying "I don't want to disrupt your plans or anything, this is just something to keep in the back of your minds, as this would be a really nice thing to have, even tho none of the clients have asked for this and I have asked no one else for a second opinion, and I will most likely ask you to remove this feature in the future because I will finally have realized it wasn't that good an idea anyway."

Ok dipshit, what the fuck are we supposed to do with this information? Every week from this moment on you will ask whether we have found the time to implement this feature, even though you are fully aware that our schedule has no room for random, unplanned features and that we are already not able to meet the unreasonable deadline you pulled out of your ass two weeks into a development process that would end up taking 8+ months.

We are already overworked, we already work hours upon hours of unpaid overtime, and yet you still think it reasonable to pull us away from our work every other fucking day to talk about random extra features you want added, but don't want added to the roadmap because you want no delays... Fuck you, fuck your toxic attitude, fuck your meetings where you spend half an hour complaining about features we are still in the process of developing the backend functionality for (on test servers) not having the right font colour for the text, and fuck your legacy desktop software originally written in COBOL that you now want moved to "the cloud".

I would rather be unemployed and live as a hobo on the streets with a "will code for food" sign than work for you ever again.

I've just come to realize that in all the time I ran Netdata on my servers, it was nightly releases, automatically updated too.. that somehow never really broke. Not even minor issues.

I really like that... Moving fast doesn't always have to imply breaking things. I wonder how they're doing it.

Me: "can you please send my credentials for the database?"

Hosting provider: "Yea sure... We will set it up on our servers in no time"

*5 mins later: provider sends mail to me*

Me: *looks at mail*

Also me: *NOTHING TO DO HERE*

Congrats to jweiland.net for not being able to support their customers.

FUCKING MICROSOFT IIS SHIT.
I'm a .NET dev since 13 years and EVERY FUCKING TIME STUPID IIS MOTHERFUCKER AND STUPID WINDOWS SERVER have a different problem setting up because of some permission.
You can't never get a site up in IIS without loosing time and patience having weird 400/500.x errors because every fucking machine have to set up some tweaky and hidden permissions.
I have 2 identical fucking win servers and deploying a .NET core applications and on one works (test server) and obviously, on the production server it gives troubles.

FUCK YOU MICROSOFT FUCK YOU I would take the IIS devs personally here and whip them to death until they don't resolve the fucking thing

It’s really painful how non technical people think all technical stuff is the same.

The amount of times I’ve been brought in to help someone with a printer or WiFi network.

I didn’t do a computer science degree for this.

I’m also pretty sure my last two bosses think fixing the printer is the same as creating large scale systems.

My last boss wanted me to build a Vr chatbot despite spending the last few years building web servers and react/redux apps. I mean I can probably figure it out and it’ll be fun, but don’t think I’m gonna get this done in a week with all my other responsibilities too.

My new boss wants me to singlehandedly build a massive marketplace system that would probably take a team of ten, several years.

Fed up of being around non technical managers who don’t listen to me with my feedback.

Time for a new chapter in the new year.

#whosHiring?

i hate linux like a lot , how do you guys use it

like you guys dont want an advertising ID, how the fuck will advertisers know who you are and what you like?

open source , give me a break, you mean your os devs are soo untrustworthy that you just have to see what they wrote in the code, who does that?

free come on, how poor are you linux people, i mean, quality stuff gets paid for, free stuff just means it's trash

and the linux devs , the aint like real coders they are just hobbysts, making your os in their free time

and who wants to install their own software anyway, on other platforms the company curates restricted software that you can use, and i know you'll say its oppressive but its just customer protection.

and i do want my platform to track everything i do, it only helps them build better stuff for me.

and whenever they decide to outdate my hardware and kill support for it, it only means they care and want me to get the latest tech, how considerate.

wait , i hear you say, there are no bugs in linux, my vendor makes sure my os comes with the latest antivirus software, nothing can break my system.

and just because linux runs on servers and most super computers only shows that common users like you and me are ignored, at least my vendor is not a sellout, and still makes stuff for the masses.

you say freedom i say safety i can sleep safe and sound for am protected nutured under one echosystem of software that i can not leave.

Saw this sent into a Discord chat today:

"Warning, look out for a Discord user by the name of "shaian" with the tag #2974. He is going around sending friend requests to random Discord users, and those who accept his friend requests will have their accounts DDoSed and their groups exposed with the members inside it becoming a victim as well. Spread the word and send this to as many discord servers as you can. If you see this user, DO NOT accept his friend request and immediately block him. Discord is currently working on it. SEND THIS TO ALL THE SERVERS YOU ARE IN. This is IMPORTANT: Do not accept a friend request from shaian#2974. He is a hacker.

Tell everyone on your friends list because if somebody on your list adds one of them, they'll be on your list too. They will figure out your personal computer's IP and address, so copy & paste this message where ever you can. He is going around sending friend requests to random discord users, and those who accept his requests will have their accounts and their IP Addresses revealed to him. Spread the word and send this to as many discord servers as you can. If you see this user, DO NOT accept his friend request and immediately block him. Saw this somewhere"

I was so angry I typed up an entire feature-length rant about it (just wanted to share my anger):

"1. Unless they have access to Discord data centres or third-party data centres storing Discord user information I doubt they can obtain the IP just by sending friend requests.

2. Judging by the wording, for example, 'copy & paste this message where ever you can' and 'Spread the word and send this to as many discord servers as you can. If you see this user, DO NOT accept his friend request and immediately block him.' this is most likely BS, prob just someone pissed off at that user and is trying to ruin their reputation etc.. Sentences equivalent to 'spread the word' are literally everywhere in this wall of text.

3. So what if you block the user? You don't even have their user ID, they can change their username and discrim if they want. Also, are you assuming they won't create any alts?

4. Accounts DDoSed? Does the creator of this wall of text even understand what that means? Wouldn't it be more likely that 'shaian' will be DDoSing your computer rather than your Discord account? How would the account even be DDoSed? Does that mean DDoSing Discord's servers themselves?

5. If 'shaian' really had access to Discord's information, they wouldn't need to send friend requests in order to 'DDoS accounts'. Why whould they need to friend you? It doesn't make sense. If they already had access to Discord user IP addresses, they won't even have to interact with the users themselves. Although you could argue that they are trolling and want to get to know the victim first or smth, that would just be inefficient and pointless. If they were DDoSing lots of users it would be a waste of time and resources.

6. The phrase 'Saw this somewhere' at the end just makes it worse. There is absolutely no proof/evidence of any kind provided, let along witnesses.

How do you expect me to believe this copypasta BS scam? This is like that 'Discord will be shutting down' scam a while back.

Why do people even believe this? Do you just blindly follow what others are doing and without thinking, copy and paste random walls of text?

Spreading this false information is pointless and harmful. It only provides benefits to whoever started this whole thing, trying to bring down whoever 'shaian' is.

I don't think people who copy & paste this sort of stuff are ready to use the internet yet.

Would you really believe everything people on the internet tell you?

You would probably say 'no'.

Then why copy & paste this? Do you have a reason?

Or is it 'just because of 'spread the word''?

I'm just sick of seeing people reposting this sort of stuff

People who send this are probably like the people who click 'Yes' to allow an app to make changes in the User Account Control window without reading the information about the publisher's certificate, or the people who click 'Agree' without actually reading the terms and conditions."

Fire your whole fucking web team Bethesda

* Your design is a classic ipecac. Whatever the fuck you are doing doesn't in frontend doesn't justify the 4Mb of bandwidth I wasted on a single js file. Why the fuck can I see the whole fucking node_modules directory when looking at the sources?
I know this is supposed to be a webpage for a game development studio, but I'm seriously wondering if your budget would even get me a prostitute.
I'm a greedy fuck and want a free game. apparently your servers are only good enough to register me, but login is apparently too much to ask for. Yeah sure. Oh and also thank you for choosing an "incorrect username and password" error message by default, even though your fucking gateway timed out. Please be kind enough and punch me directly into my face next time. Not like I'll ever access that shit ever again

So, I'm still not certain if it's actually a bug or merely my lack of experience, but I've been working on a 2D platformer game (using only C++ and SDL2) for roughly 2 years now (on and off; sometimes off for months) and I'm extremely embarrassed about this, but for the life of me, I cannot seem to get the player character's movement and collision physics working properly. It's driving me absolutely insane.

I've read articles and tutorials, referenced books, and posted about it in game development communities (e.g., gamedev.com, Discord servers, etc.), but even though the fundamental structure and explanations made sense, getting the code to work has been unsuccessful, albeit not completely so, but if I get one thing working, another thing breaks. It feels like I'm trying to repair a vase that fell off of a skyscraper and turned to dust on the street below.

I've always been a very tech savvy person with a fiery passion for programming, electronics and game/software/embedded/web development, but to be honest, having such a difficult time with things like this that — in theory, at least — seem like trivial bumps in the road have made me feel like I'm never going to be successful in this field. But regardless of the depressing thoughts of worthlessness, my passion doesn't let me stop trying. Who knows, maybe it'll have to remain just a hobby. 😕

So I spent over 200 hours recovering a raid array because the the business that hired the company I work for tried to do all the work in house. Now I'm to the point of trying to quote a new server setup to the company because the last one got hit by lightning. My quote $8000-$20000 in just hardware alone. If im going to do this I want to do it right. Twin servers running xenserver holding virtual machines with a high-quality Nas to hold the virtual machine. Then have the vm's being imaged to two off site backup devices in two different locations and a cloud backup. My boss comes in after me talks to the guy in charge and tells him all that is needed is one server holding all storage on it just like the last server that died and one off-site backup for $4500. How stupid can someone be who has been doing this for over 20 years? Oh btw the software running on this server will be 911 dispatch, jail records and the database for fine payments. The sheriff making the final call me and my boss are meeting with him tomorrow if my boss tries to undercut me I'm going to tell them both that if the same shit happens next time they can spend the time themselves trying to recover stuff because I done the over 200 hours in just short of two weeks.

Tinfoil hat time!

So, for this scenario we assume a world war has is in full swing, and all communications are either completely out or extremely monitored.

Additionally, no servers can be trusted and no ISPs can be trusted. Even when downloading software you cannot know if the software you got is tampered or not.

You have to send a message to a recipient you cannot physically contact, and you have to make sure him and only him get the message, otherwise nukes are deployed and we all die. The message has to arrive within 3 months or else it's too late.

Is it possible? If yes, how?

TLDR; I was editing the wrong file, let's go to bed.

We have this huge system that receives data from an API endpoint, does a whole bunch of stuff, going through three other servers, and then via some calculation based on the data received from the UI, and data received from the endpoint, it finally sends the calculated fields to the UI via websocket.

Poor me sitting for over 4 hours debugging and changing values in the logic file trying to understand why one of the fields ends up being null.

Of course every change needs a reboot to all the 4 servers involved, and a hard refresh of the UI.

I even tried to search for the word null in that file, but to no avail.

After scattering hundreds of console logs, and pulling my hair out, I found out that I am editing the wrong file.

I guess it's time for some sleep.

In my second year of A levels in secondary school (college) we had to make a quiz game using Microsoft Visual Basic. Was my first time using the software, got annoyed within a few weeks and downloaded a template off the internet. The whole IT department thought I was some form of genius until I crashed the servers (in over 200 schools) trying to hack them 6 weeks later.

Sir if you're reading this, I'm sorry.

I am a mechanical engineer first and my companies go to sysadmin second. So software developing isnt really my main field of expertise buuttt:

WHY IS SLOOPY SOFTWARE WRITING A VIABLE EXCUSE?

Story:
Yesterday i started to migrate some stuff from our old Win 2008 Server to the new 2016. Turns out there are some MS SQL Express Servers running. Quick check for what they are turns out that they are activly used. So far so good. For other reasons we have a new MSSQL 2017 Core Licence. So i thought, hey it would be nice to just move those 2012, 2008 and 2014 Express Servers to a real one that can use the entire machines capabilities.
After some try & error with exporting one of the softwares (where i had to elevate one the user rights to sysadmin for reasons) the entire system stopped working. I didnt deleted anything or changed anything! Well, i elevated user rights. After 2 hours of support call it turns out that the software stopped working cause i gave the database user sysadmin rights. I dont know enough about MSSQL to judge wether that is logical or not, but it sounds super illogical and i suspect sloopy software writing on the manufacturers part. One way or another, the excuse from the telephone support was "yeah, our software is a very fragile child"
Okay.
After i told all that my coworkers two of them were also "yeah, that is just how the [company] software is, you have to be careful with it"

Apparently it broke in the past for other minor stuff.
As an engineer i cannot build bridges that collapse when you use the left and the right lane at the same time. For an architect it isnt okay to build an house where the front door explodes when you open a window. It is not okay for a power tool to go out in a fireball when you accidently drill plastic with it. But for some weird reasons its socially acceptable for programs to be sloopy, buggy and only working under specific conditions. Since when is it okay for a car only to work when you know specific steps to make it run? Like, throwing your spare key in the gas tank, the kick the left wheel exactly three times and finally tapping the steering wheel 5 times left, 4 times right. What? That would be ridiculous? But that is exactly how that software works. You have to follow a specific step guide to make it work, EVERY TIME.

I. JUST. DONT. GET. IT

I technically joined just after this guy left(fired) but the stories are to good to tell!

The guy was clearly off but It wasn't his fault he had to of had aspergers

He would demand! To write with two pens in one hand he said it was faster and the only way he could write neatly... (Nope)

I don't think it was to weird but he would put on music and play death metal stuff full volume, because he couldn't hear anyone the team used to make paper planes and fire them at him when they wanted his attention.

Another thing he was into furry ... Stuff but was super open about it had. Wolf's and shit like that on his desk and always had a wolf shirt.

But he was fired, he wasn't great at his job.

I came in to help sort out the mess it was the government's setup for servers and nurses and doctors computers for the NHS over in england.

He effectively skull fucked the entire system.

He magically (I to do day can not understand how) did forced updates and installed to a newer version of Windows servers the problem being the programs wouldn't work on newer windows at the time.

Most were on XP at the time and they used windows servers back then.

Luckily not nation wide just in my local area but still thousands of computers affected.

The issue became this ... You see they had this program on their computers that let them get patient documents and update etc

He removed code or added code that made it update all the laptops and desktops to a new service pack which they didn't want... Then he upgraded the servers to a new windows version I don't remember the specifics

But the updates and new version of Windows made it so the laptops etc couldn't communicate with the servers.

... The next day he got fired and I was brought in for a few weeks to help sort out the mess.

But apparently he was a super interesting guy but with way to many quirks.

It costs the tax payers a fortune! Literally a few million to sort his mistake out people were working round the clock for two weeks straight.

!coding

I used to be a sysadmin, which meant I was in charge of quarterly server patching. My team managed about 2500 servers, running various flavors of linux and legacy unix. The vast majority(95% or more) ran Linux(SLES). Our maintenance window was always in the overnight-- 10pm to 6am --so the stroke of 10pm would be a massive cascade of patching commands sent to hundreds of servers.

Before I was brought into the process, it made use of the automation product we were tasked by mgmt to use: Bigfix. It's a real piece of shit. Though we had 2500 or so servers, this environment was dominated by windows. All our vcenter servers ran it, and more importantly, our bigfix nodes were all windows machines. That meant that while we're trying to patch, the bigfix servers would get patched by the windows team. This would cause lots of failed and timed out patching, because the windows admins never quite understood that taking down the automation infrastructure would cause problems.

As such, I got tired of depending on a bunch of button-pushing checkbox-clickers who didn't know shit about shit, so I started writing an ssh-wrapped patching system. By the time I left for my current job, patching had been reduced to a single command to initiate each group's patching and reboots, and an easy check to see when servers come back up. So usually, the way it worked out was that I would send patching orders to 750 machines or so, and within about 5 minutes, they would all be done patching, and within another 20 minutes all the ones that required rebooting but about 5 would be done rebooting.

The "all-nighter" which happened every time was waiting for oracle servers to run timed fscks against a dozen or so large filesystems per server, because they were all on ext3/4, which eats complete shit. Then, several hours later, as they finished, I would have to call the DBAs to tell them to validate their shitty servers.

Work story.
We have this system that's being used nation-wide and basically there's a control panel for management (it's a website)) and an app for the regular users.

I just migrated and replaced the guy before me, I'm basically the only one on the project.

The code for the website is a mess, the servers are sometimes slow, and few security problems here and there.

Project Lead comes up to me and says that few of our clients that use the website are saying it works really slowly.
I start by analyzing the networking, and found shocking things.
First of all, let's say there's a messaging option, and the management teams that are our clients can have each a lot of groups, which all have messaging.

Upon first load, ALL OF THE IMAGES, FROM ALL GROUPS, ARE PRE LOADED. It can get up to few hundred photos being preloaded upon first load, which can explain the slow loading.
After discovering that, I discovered that the Administration control panel, which only my project lead can access, with sends heavy requests to the server and loads heavy assets, is loaded every time to every single client, generating heavy stress on our server and slowing everything down.

I tell that to my project lead and say that that's what causing the slow downs, I coded a fix that currently sits and is not being merged to the master branch to be deployed, and somehow I need to find a way to fix the slowness which all comes down to the heavy requests and slow connection with servers... And they won't merge my fix that fixes the loading of the administration panel so the stress on the servers could go down, and everything will be sped up....
Ah damnit.. sometimes I don't understand it..

We had a closed beta today for a product I've worked on for the last 6 months. We've been working late nights for the last week to make sure we get all the kinks out and make sure our demo went smoothly.

After a bit of hyping via a power point presentation by my colleague, we finally start the demo and of course the login fails. Both the API and database were offline. The servers were on but we just couldn't reach the applications. My colleague stalls and I get to debugging.

We were standing in front of 50 people who were hyped-up on a product that was failing in front of them.

But after 10 minutes (felt like an hour) of hemming and hawing, the app just decides to start running again.

What in the actual fuck? Does this happen to anyone else? Coz it feels like it happens to me every time.

Time spent getting to grips with your OS is usually time spent well. While you're not operations, it really helps being able to solve general problems yourself without calling support.

Oh, and: Set up a good bashrc, and put it on the servers you're working with.

WHY DO PEOPLE ALWAYS LEAVE CLOSE TO DEADLINES???

My team exists of three developers and a product owner. By the end of the month we have to deliver two new applications. It has not been going as smoothly as we'd hoped so far. However, one of the developers has cut his time for this project from fulltime to 1-2 days a week starting this week. The other developer is also needed on another project and has to run to the rescue whenever there is a problem with the servers, so he doesn't really solve any bugs either. Also he'll be leaving straight after deadline so he has to document everything only he knows and he'll be on holiday next week. Also, the product owner leaves tomorrow and will be back after deadline.

So.. Here I am, junior developer, have been here for about three months and I have to fix everything and do the communication to our testers as well. I'm feeling too overwhelmed right now...

There's one thing I hate about the new year. Everyone assumes it's the best time to do some cleanup, some maintenance. As the year starts fresh - everyone should start fresh, right?

NOT!

No, Hetzner, you don't need to restart my servers on Dec 26th.
No, local VPS providers, you don't need to clean-up all my /tmp/ files for me on Dec 29th, leaving me in the dark and unaware I have to restart my apps.

That's just a dick move. Your intentions might be alright, but the consequences... Damn you!

>TINFOIL GUYS!!!!!
guys don't just deactivate your FB acc, request deletion, beacuse if you deactivate and use any services like any page login or any social services like insta, and you use your FB credentials to login, the gets reactivated and your news feed even shows what you have missed in the mean time , so all of the (your) data is still in their servers

Alrighty, saturday morning rant time!

I just recieved a mail from one of my not-so-much-loved colleagues.

Now Background first: I work in IT-Support. We provide services for other companies. One of those services is monitoring servers and clients for various things. I recently took over the project (was assigned to do it) and restructured everything, wrote new scripts to test more stuff, successfully tested it internally and rolled it out over the last 2 weeks.

Now one of these scripts hooks into the Windows Update API and looks at the update history. It filters for known Windows Update Agent strings (UpdateOrchestrator, AutomaticUpdates and AutomaticUpdatesWuApp in case you also want to do something like this) and then looks for installation errors over the last 24 hours and wherever there have even been any successful updates over the last one and a half months.

Back to that mail.

My colleague sent me this lovely mail about a ticket i opened about his customers servers beeing all out-of-date on updates.

"This is all wrong, everything's fine. I disabled the checks."

...

It's on bitch.

So i logged on to my work PC via TeamViewer, opened my script, connected to the customer and was ready to debug the shit out of my script, knowing i probably won't even need to.

I looked at the update history via Windows Update itself and behold: 1st April. That's almost 50 days in the past.

So the script works, go figure.

Great, so search for new Updates then.
>None found.

Hm. What could it be? Did my super special colleague forget to care about his very special totally-needs-WSUS-customer WSUS again?

Yup.

Online-Search finds a ton of new Updates.

Screenshot, write pissed mail to colleague, re-enable checks, breakfast.

As of late I'm creating private game servers on my local network in my free time for fun / relaxing

I've already running:
a World of Warcraft vanilla;
Travian version 3;

And I am currently working on a conquer online private server, but getting the right client for it is the hard part...

Fucking shit for brains authors that think the digital world is a fantasy realm where everything can happen just to aid their story. Out of boredom i watched "scorpion" today, a tv series about a group of geniusses which are a special case task force.
They got a visitor from the government saying the servers from the federal reserve bank were encrypted with ransomware. I already twitched when they said the economic system would collapse if the servers were left inoperational for a few days. Then one guy got to his desk and "hacked" the fed network to check... he then tried to remove the malware but "it changed itself when observed". But they got the magical fingerprint of the device that uploaded it. In the end some non-programmers created the malware, but it is super fast and dangerous because it runs on a quantum computer which makes it hyper fast and dangerous. They got to the quantum computer which was a glowing cube inside another cube with lasers going into it and they had to use mirrors to divert the lasers to slow down that quantum thingy. And be careful with that, otherwise it explodes. In the end the anti-malware battled the malware and won, all in a matter of minutes.

This is a multimillion hollywood production. How can a movie this abusive to computer science even air on television? Shit like this is the reason people still think the cyberworld is some instable thing that can explode any second. It's not, it's an instable thing that can break down any second. I remember "ghost in the wires" and people had surreal imaginations about the internet already. Shit like this is why people stay dumb and think everything can be done in seconds. If i ever should encounter one of these idiots i tell him i have an app that can publish his browser history by taking a picture of his phone and watch his reaction.
Time to shuw down the tv and learn vim again.