We're using a ticket system at work that a local company wrote specifically for IT-support companies. It's missing so many (to us) essential features that they flat out ignored the feature requests for. I started dissecting their front-end code to find ways to get the site to do what we want and find a lot of ugly code.
Stuff like if(!confirm("blablabla") == false) and whole JavaScript libraries just to perform one task in one page that are loaded on every page you visit, complaining in the js console that they are loaded in the wrong order. It also uses a websocket on a completely arbitrary port making it impossible to work with it if you are on a restricted wifi. They flat out lie about their customers not wanting an offline app even though their communications platform on which they got asked this question once again got swarmed with big customers disagreeing as the mobile perofrmance and design of the mobile webpage is just atrocious.

So i dig farther and farthee adding all the features we want into a userscript with a beat little 'custom namespace' i make pretty good progress until i find a site that does asynchronous loading of its subpages all of a sudden. They never do that anywhere else. Injecting code into the overcomolicated jQuery mess that they call code is impossible to me, so i track changes via a mutationObserver (awesome stuff for userscripts, never heard of it before) and get that running too.

The userscript got such a volume of functions in such a short time that my boss even used it to demonstrate to them what we want and asked them why they couldn't do it in a reasonable timeframe.

All in all I'm pretty proud if the script, but i hate that software companies that write such a mess of code in different coding styles all over the place even get a foot into the door.

And that's just the code part: They very veeeery often just break stuff in updates that then require multiple hotfixes throughout the day after we complain about it. These errors even go so far to break functionality completely or just throw 500s in our face. It really gives you the impression that they are not testing that thing at all.

And the worst: They actively encourage their trainees to write as much code as possible to get paid more than their contract says, so of course they just break stuff all the time to write as much as possible.
Where did i get that information you ask? They state it on ther fucking career page!

We also have reverse proxy in front of that page that manages the HTTPS encryption and Let's Encrypt renewal. Guess what: They internally check if the certificate on the machine is valid and the system refuses to work if it isn't. How do you upload a certificate to the system you asked? You don't! You have to mail it to them for them to SSH into the system and install it manually. When will that be possible you ask? SOON™.
At least after a while i got them to just disable the 'feature'.

While we are at 'features' (sorry for the bad structure): They have this genius 'smart redirect' feature that is supposed to throw you right back where you were once you're done editing something. Brilliant idea, how do they do it? Using a callback libk like everyone else? Noooo. A serverside database entry that only gets correctly updated half of the time. So while multitasking in multiple tabs because the performance of that thing almost forces you to makes it a whole lot worse you are not protected from it if you don't. Example: you did work on ticket A and save that. You get redirected to ticket B you worked on this morning even though its fucking 5 o' clock in the evening. So of course you get confused over wherever you selected the right ticket to begin with. So you have to check that almost everytime.

Alright, rant over.
Let's see if i beed to make another one after their big 'all feature requests on hold, UI redesign, everything will be fixed and much better'-update.

Every year my team runs an award ceremony during which people win “awards” for mistakes throughout the year. This years was quite good.

The integration partner award- one of our sysAdmins was talking with a partner from another company over Skype and was having some issues with azure. He intended to send me a small rant but instead sent “fucking azure can go fuck itself, won’t let me update to managed disks from a vhd built on unmanaged” to our jv partner.

Sysadmin wannabe award (mine)- ran “Sudo chmod -R 700 /“ on one of our dev systems then had to spend the next day trying to fix it 😓

The ain’t no sanity clause award - someone ran a massive update query on a prod database without a where clause

The dba wannabe award - one of our support guys was clearing out a prod dB server to make some disk space and accidentally deleted one of the databases devices bringing it down.

The open source community award - one of the devs had been messing about with an apache proxy on a prod web server and it ended up as part of a botnet

There were others but I can’t remember them all

This is fucking bad. I just stumbled across a database online, unencrypted plain text containing ALL details of thousands of students at my university. Full names, ID number (SSN), student numbers, address, family info, medical aid info, physical fitness reports

What do I do? I was not on any VPN or proxy when I accessed it

I am building a website inspired by devrant but have never built a server network before, and as im still a student I have no industry experience to base a design on, so was hoping for any advice on what is important/ what I have fucked up in my plan.

The attached image is my currently planned design. Blue is for the main site, and is a cluster of app servers to handle any incoming requests.

Green is a subdomain to handle images, as I figured it would help with performance to have image uploads/downloads separated from the main webpage content. It also means I can keep cache servers and app servers separated.

Pink is internal stuff for logging and backups and probably some monitoring stuff too.

Purple is databases. One is dedicated for images, that way I can easily back them up or load them to a cache server, and the other is for normal user data and posts etc.

The brown proxy in the middle is sorta an internal proxy which the servers need to authenticate with to connect to, that way I can just open the database to the internal proxy, and deny all other requests, and then I can have as many app servers as I want and as long as they authenticate with the proxy, they can access the database without me changing any firewall rules. The other 2 proxies just distribute requests between the available servers in the pool.

Any advice would be greatly appreciated! Thanks in advanced :D

So for those of you keeping track, I've become a bit of a data munger of late, something that is both interesting and somewhat frustrating.

I work with a variety of enterprise data sources. Those of you who have done enterprise work will know what I mean. Forget lovely Web APIs with proper authentication and JSON fed by well-known open source libraries. No, I've got the output from an AS/400 to deal with (For the youngsters amongst you, AS/400 is a 1980s IBM mainframe-ish operating system that oriiganlly ran on 48-bit computers). I've got EDIFACT to deal with (for the youngsters amongst you: EDIFACT is the 1980s precursor to XML. It's all cryptic codes, + delimited fields and ' delimited lines) and I've got legacy databases to massage into newer formats, all for what is laughably called my "data warehouse".

But of course, the one system that actually gives me serious problems is the most modern one. It's web-based, on internal servers. It's got all the late-naughties buzzowrds in web development, such as AJAX and JQuery. And it now has a "Web Service" interface at the request of the bosses, that I have to use.

The programmers of this system have based it on that very well-known database: Intersystems Caché. This is an Object Database, and doesn't have an SQL driver by default, so I'm basically required to use this "Web Service".

Let's put aside the poor security. I basically pass a hard-coded human readable string as password in a password field in the GET parameters. This is a step up from no security, to be fair, though not much.

It's the fact that the thing lies. All the files it spits out start with that fateful string: '<?xml version="1.0" encoding="ISO-8859-1"?>' and it lies.

It's all UTF-8, which has made some of my parsers choke, when they're expecting latin-1.

But no, the real lie is the fact that IT IS NOT WELL-FORMED XML. Let alone Valid.

THERE IS NO ROOT ELEMENT!

So now, I have to waste my time writing a proxy for this "web service" that rewrites the XML encoding string on these files, and adds a root element, just so I can spit it at an XML parser. This means added infrastructure for my data munging, and more potential bugs introduced or points of failure.

Let's just say that the developers of this system don't really cope with people wanting to integrate with them. It's amazing that they manage to integrate with third parties at all...

A serious question: what kind of stack should I choose so I can run a web backend installing no deps whatsoever? I know that Perl works on ubuntu out of the box. Anything else? Maybe Python?
Also, what can be used to replace a reverse proxy like nginx? And what kind of database is available out of the box?

I am the responsible for the atlassian Suite at work, as I maintain the systems, set them up, and stuff.

One day, our crowd (the authentication and authorization application) just went crazy. At like lunch time it could not connect to the AD anymore. No reasons. Throwing XSRF errors (cross site scripting), because http would connect to https. "won't do it, fuck you" it told me. Out of the blue. Noone changed anything. And yea, seriously. Noone did.
It just refused to connect (as connecting to AD is connecting yourself with you own api. And refusing yourself talking to yourself). It runs behind a proxy. Therefore http/https. Well, this worked for years. But out of sudden not anymore.

Yea. Fuck you.

It was reported some hours later, at like 3pm, as people could not login to the applications using crowd as authentication and authorization server.

Tried to debug the system, where nothing was did, to make it work. At best time to fail.

First workaround: if you are logged into one of the other applications of atlassian, just refresh the site, so your SSO token gets a refresh and you are signed on again.

Then I searched more and more. And more.
But nothing worked, nothing helped.

So I addressed an emergency maintenance, take down the whole Suite, restart crowd, to apply some changes to it's settings, not knowing what happening then, because all connections of SSO will then be released. Sent out the mail like 30 minutes beforehands.

While waiting for the window, I just typed my credentials... And redid, and redid, so to type and being bored.
Three minutes before the window...

It just worked again.

Well. Wtf. Serioudl
Just came back.

No Intrusion, no changes at all. Just came back, as nothing has happened.

Kind of best part of this story... A headhunter messaged me on my way home to offer me a job as an Atlassian Suite SysAdmin for a company, at kinda the double of my salary.

At first I was thinking to go there, and when someone then asked me sth about Atlassian just start to laugh and then leave still laughing...

But then I very nicely respond that I dont want to cry at work. And wished him best luck.

I am doing some bad upgrades now on our Suite. Very painful.
And I looked into the start scripts. Some Look like the untalented intern tells another one to write scripts. Seriously wtf.

Today I followed the guide to Update a confluence and change database to Postgres. Didnt work, Postgres error.
Try it again, jquery won't load. Next try, tomcat not starting anymore. Did same thing. Every fucking time.

Yea. Maintenance window to get a nice new export soon. Will only take an hour.

To switch database in confluence, you need to set it up very fresh. And then Import your export.
Export takes an hour at our system.
Importing maybe the same time. Hope it will work (hint: Nope).

Oh, can be nice also. Just tell the Bitbucket to migrate databases, there is a fucking setting for it. Enter new database, ready, go, finished.

At least they don't raise costs very much every kinda year.
Oh sorry, yes, they do.

RDS Proxy is quickly becoming my least favourite AWS offering.

I ranted about it a while back because I had to abandon it for a project because it doesn’t support clusters very well (it only proxys to the primary node).

Well I tried to use it again for a different project with only a single RW instance. Surely it will be ideal?

Nope. It doesn’t support Postgres 13. Only goes up to 12.

What the hell Amazon?

pgbouncer it is I guess.

Disclaimer: I love open source and I adore the owasp for what they do.

BUT owasp zap has to be the most overly complicated, badly documented tool in existence. As long as one stays within its most basic functions everything is fine, setting it up as a proxy and even issuing a root cert for our test devices worked wonderfully simple.

Then I made the mistake to try to actually do anything with the data we pulled and had to dive into the scripting console.

The documentation basically consists only of "This thing exists", it provides a msg object with no information what it contains or how it's structured, has no code completion and, here comes the kicker, if the script is run and has an error it gets flagged and can't be reenabled after the error is fixed. So I'm currently at forwarder48.groovy trying to simply store the request on a database for possible diagnostics.

So right now I already know that I'll spend most of my vacation next week trying to decipher the source, document it, fix that damn "flagged as error" bullshit and jump through a billion hoops trying to get a pull request through.