Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "not a dba"
-
*Now that's what I call a Hacker*
MOTHER OF ALL AUTOMATIONS
This seems a long post. but you will definitely +1 the post after reading this.
xxx: OK, so, our build engineer has left for another company. The dude was literally living inside the terminal. You know, that type of a guy who loves Vim, creates diagrams in Dot and writes wiki-posts in Markdown... If something - anything - requires more than 90 seconds of his time, he writes a script to automate that.
xxx: So we're sitting here, looking through his, uhm, "legacy"
xxx: You're gonna love this
xxx: smack-my-bitch-up.sh - sends a text message "late at work" to his wife (apparently). Automatically picks reasons from an array of strings, randomly. Runs inside a cron-job. The job fires if there are active SSH-sessions on the server after 9pm with his login.
xxx: kumar-asshole.sh - scans the inbox for emails from "Kumar" (a DBA at our clients). Looks for keywords like "help", "trouble", "sorry" etc. If keywords are found - the script SSHes into the clients server and rolls back the staging database to the latest backup. Then sends a reply "no worries mate, be careful next time".
xxx: hangover.sh - another cron-job that is set to specific dates. Sends automated emails like "not feeling well/gonna work from home" etc. Adds a random "reason" from another predefined array of strings. Fires if there are no interactive sessions on the server at 8:45am.
xxx: (and the oscar goes to) fuckingcoffee.sh - this one waits exactly 17 seconds (!), then opens an SSH session to our coffee-machine (we had no frikin idea the coffee machine is on the network, runs linux and has SSHD up and running) and sends some weird gibberish to it. Looks binary. Turns out this thing starts brewing a mid-sized half-caf latte and waits another 24 (!) seconds before pouring it into a cup. The timing is exactly how long it takes to walk to the machine from the dudes desk.
xxx: holy sh*t I'm keeping those
Credit: http://bit.ly/1jcTuTT
The bash scripts weren't bogus, you can find his scripts on the this github URL:
https://github.com/narkoz/...53 -
Yesterday: Senior dev messages out a screenshot of someone using an extension method I wrote (he didn’t know I wrote it)..
SeniorDev: “OMG…that has to be the stupidest thing I ever saw.”
Me: “Stupid? Why?”
SeniorDev: “Why are they having to check the value from the database to see if it’s DBNull and if it is, return null. The database value is already null. So stupid.”
Me: “DBNull is not null, it has a value. When you call the .ToString, it returns an empty string.”
SeniorDev: ”No it doesn’t, it returns null.”
<oh no he didn’t….the smack down begins>
Me: “Really? Are you sure?”
SeniorDev: “Yes! And if the developer bothered to write any unit tests, he would have known.”
Me: “Unit tests? Why do you assume there aren’t any unit tests? Did you look?”
<at this moment, couple other devs take off their head phones and turn around>
SeniorDev:”Well…uh…I just assumed there aren’t because this is an obvious use case. If there was a test, it would have failed.”
Me: “Well, let’s take a look..”
<open up the test project…navigate to the specific use case>
Me: “Yep, there it is. DBNull.Value.ToString does not return a Null value.”
SeniorDev: “Huh? Must be a new feature of C#. Anyway, if the developers wrote their code correctly, they wouldn’t have to use those extension methods. It’s a mess.”
<trying really hard not drop the F-Bomb or two>
Me: “Couple of years ago the DBAs changed the data access standard so any nullable values would always default to null. So no empty strings, zeros, negative values to indicate a non-value. Downside was now the developers couldn’t assume the value returned the expected data type. What they ended up writing was a lot of code to check the value if it was DBNull. Lots of variations of ‘if …’ , ternary operators, some creative lamda expressions, which led to unexpected behavior in the user interface. Developers blamed the DBAs, DBAs blamed the developers. Remember, Tom and DBA-Sam almost got into a fist fight over it.”
SeniorDev: “Oh…yea…but that’s a management problem, not a programming problem.”
Me: “Probably, but since the developers starting using the extension methods, bug tickets related to mis-matched data has nearly disappeared. When was the last time you saw DBA-Sam complain about the developers?”
SeniorDev: “I guess not for a while, but it’s still no excuse.”
Me: “Excuse? Excuse for what?”
<couple of awkward seconds of silence>
SeniorDev: “Hey, did you guys see the video of the guy punching the kangaroo? It’s hilarious…here, check this out.. ”
Pin shoulders the mat…1 2 3….I win.6 -
Yesterday, in a meeting with project stakeholders and a dev was demoing his software when an un-handled exception occurred, causing the app to crash.
Dev: “Oh..that’s weird. Doesn’t do that on my machine. Better look at the log”
- Dev looks at the log and sees the exception was a divide by zero error.
Dev: “Ohhh…yea…the average price calculation, it’s a bug in the database.”
<I burst out laughing>
Me: “That’s funny.”
<Dev manager was not laughing>
DevMgr: “What’s funny about bugs in the database?”
Me: “Divide by zero exceptions are not an indication of a data error, it’s a bug in the code.”
Dev: “Uhh…how so? The price factor is zero, which comes from a table, so that’s a bug in the database”
Me: “Jim, will you have sales with a price factor of zero?”
StakeholderJim: “Yea, for add-on items that we’re not putting on sale. Hats, gloves, things like that.”
Dev: “Steve, did anyone tell you the factor could be zero?”
DBA-Steve: “Uh...no…just that the value couldn’t be null. You guys can put whatever you want.”
DevMgr: “So, how will you fix this bug?”
DBA-Steve: “Bug? …oh…um…I guess I could default the value to 1.”
Dev: “What if the user types in a zero? Can you switch it to a 1?”
Me: “Or you check the factor value before you try to divide. That will fix the exception and Steve won’t have to do anything.”
<awkward couple of seconds of silence>
DevMgr: “Lets wrap this up. Steve, go ahead and make the necessary database changes to make sure the factor is never zero.”
StakeholderJim: “That doesn’t sound right. Add-on items should never have a factor. A value of 1 could screw up the average.”
Dev: “Don’t worry, we’ll know the difference.”
<everyone seems happy and leaves the meeting>
I completely lost any sort of brain power to say anything after Dev said that. All the little voices kept saying were ‘WTF? WTF just happened? No really…W T F just happened!?’ over and over. I still have no idea on how to articulate to anyone with any sort of sense about what happened. Thanks DevRant for letting me rant.15 -
Boss: "Could you join the new DevOps team for a week or two, for some coaching?"
Me: "I'd rather watch you masturbate furiously in a corner of the office while you cry over your ex boyfriend"
Boss: "Yeah... that's why I ask you. You are the only one brave enough to watch"
Me: *Sigh* "But I don't know shit about what DevOps does, I'm a DBA. I've told you the difference a million times. Can't we just douse it in gasoline and set it on fire?"
Boss: "What?"
Me: "Not the team, the servers..."
Boss, imitating Gimli: "And my ex!"
Me: "I get why he left you"
Boss: "It's funny, he was actually better with computers than me, maybe even better than you. He hated me for starting this company, told me I was just chasing money instead of ideals. He just isn't grown up enough to see that there is more to the world than computer games, brewing beer, maker festivals and gay bars, that you need to take responsibility... Maybe it just never works out between managers and geeks..."
Me: "Indeed. The difference in competence is too large"
Boss: "Ugh. You are like straight version of him... but will you at least take a look?"
Me: "Fine, unzip your pants..."
Boss: "No, not that... you need to teach DevOps this docking thing, with the parallel stuff, and the horizontal growth"
Me: "Damn I really hope we're talking about servers now... Do you mean Docker?"
Boss: "That's it. They want to learn how to dock on the Windows servers. They reserved two 4xlarge on AWS. Is that enough for docking?"
Me: ...
Me: ...
Me: "You know what. I'm going back to hug my DB designs, and wash my brain with some queries. Then I'll return here to burn everything to the ground. There is no hope for you left"
Boss: "That's what he said"
Me: "You're using that meme wrong"
Boss: "OK. So what if you just stay on DB management, and I'll just give you the budget to recruit a new DevOps lead and pay for training?"
Me: "That would work"
Boss: "Why are you grinning?"
Me: "Because I have your ex's phone number"18 -
The way 90% of the population wears their face masks really explains a lot about their approach to using software, apps & websites as well.
I feel like giving up.
I am not a developer for the salary, or just to solve analytical puzzles. Those are motivators, but my main drive is to make the world more comfortable and enjoyable, better optimized, build ethical services which bring happiness into people's lives. I want to improve society, even if it's just a tiny bit.
But if users invest absolutely zero percent of their limited brain capacity into understanding a product that already has a super-clean design and responds with helpful validation messages...
...why the fuck bother.
I used to think of the gap between technology and tech-incompetent people as an optimization problem.
As something which could be fixed by spending a fortune on UX research. Write tests, hire QA employees, decrease tech debt, create a bold but unified & simple design.
But the technologically incompetent just get more entitled with every small thing you simplify.
It's never fucking fool-proof enough.
Why can't I upload a 220MB PDF as profile picture? Why doesn't the app install on my 9 year old Android Froyo phone? Why can't I sign up if my phone number contains a  U+FFFC? Why does this page load so slowly from my rural concrete bunker in East Ukraine? WHY DO I HAVE PNEUMONIA, HOW DID I GET INFECTED EVEN THOUGH I WAS WEARING A MOUTH MASK ON MY FOREHEAD?
This is why I ran away from Frontend, to Backend, to DBA.
If I could remove myself further from the end user, I would.
At least I still have a full glass of tawny port and a huge database which needs to be normalized & migrated.
Fuck humans, I'm going to hug a server.25 -
In a user-interface design meeting over a regulatory compliance implementation:
User: “We’ll need to input a city.”
Dev: “Should we validate that city against the state, zip code, and country?”
User: “You are going to make me enter all that data? Ugh…then make it a drop-down. I select the city and the state, zip code auto-fill. I don’t want to make a mistake typing any of that data in.”
Me: “I don’t think a drop-down of every city in the US is feasible.”
Manage: “Why? There cannot be that many. Drop-down is fine. What about the button? We have a few icons to choose from…”
Me: “Uh..yea…there are thousands of cities in the US. Way too much data to for anyone to realistically scroll through”
Dev: “They won’t have to scroll, I’ll filter the list when they start typing.”
Me: “That’s not really the issue and if they are typing the city anyway, just let them type it in.”
User: “What if I mistype Ch1cago? We could inadvertently be out of compliance. The system should never open the company up for federal lawsuits”
Me: “If we’re hiring individuals responsible for legal compliance who can’t spell Chicago, we should be sued by the federal government. We should validate the data the best we can, but it is ultimately your department’s responsibility for data accuracy.”
Manager: “Now now…it’s all our responsibility. What is wrong with a few thousand item drop-down?”
Me: “Um, memory, network bandwidth, database storage, who maintains this list of cities? A lot of time and resources could be saved by simply paying attention.”
Manager: “Memory? Well, memory is cheap. If the workstation needs more memory, we’ll add more”
Dev: “Creating a drop-down is easy and selecting thousands of rows from the database should be fast enough. If the selection is slow, I’ll put it in a thread.”
DBA: “Table won’t be that big and won’t take up much disk space. We’ll need to setup stored procedures, and data import jobs from somewhere to maintain the data. New cities, name changes, ect. ”
Manager: “And if the network starts becoming too slow, we’ll have the Networking dept. open up the valves.”
Me: “Am I the only one seeing all the moving parts we’re introducing just to keep someone from misspelling ‘Chicago’? I’ll admit I’m wrong or maybe I’m not looking at the problem correctly. The point of redesigning the compliance system is to make it simpler, not more complex.”
Manager: “I’m missing the point to why we’re still talking about this. Decision has been made. Drop-down of all cities in the US. Moving on to the button’s icon ..”
Me: “Where is the list of cities going to come from?”
<few seconds of silence>
Dev: “Post office I guess.”
Me: “You guess?…OK…Who is going to manage this list of cities? The manager responsible for regulations?”
User: “Thousands of cities? Oh no …no one is our area has time for that. The system should do it”
Me: “OK, the system. That falls on the DBA. Are you going to be responsible for keeping the data accurate? What is going to audit the cities to make sure the names are properly named and associated with the correct state?”
DBA: “Uh..I don’t know…um…I can set up a job to run every night”
Me: “A job to do what? Validate the data against what?”
Manager: “Do you have a point? No one said it would be easy and all of those details can be answered later.”
Me: “Almost done, and this should be easy. How many cities do we currently have to maintain compliance?”
User: “Maybe 4 or 5. Not many. Regulations are mostly on a state level.”
Me: “When was the last time we created a new city compliance?”
User: “Maybe, 8 years ago. It was before I started.”
Me: “So we’re creating all this complexity for data that, realistically, probably won’t ever change?”
User: “Oh crap, you’re right. What the hell was I thinking…Scratch the drop-down idea. I doubt we’re have a new city regulation anytime soon and how hard is it to type in a city?”
Manager: “OK, are we done wasting everyone’s time on this? No drop-down of cities...next …Let’s get back to the button’s icon …”
Simplicity 1, complexity 0.16 -
FML. An overreaching supergenius "architect" and a database team:
A: "We have decided that apps should use mysql. Install a MySQL so we match cloud"
DBA: "we don't have an image or experience with MySQL. We have mssql and Oracle "
A: "ok, use mssql in data center and mysql in production cloud"
DBA: "that's... not going to work well"
A: "just do it!"
...
Me, reading this shit, sends email: "ignoring the fact that we have more than 500 queries in this application which will need to be checked and most likely rewritten, how are we supposed to test the mysql queries without production access?"
A: "just use mssql local and MySQL in cloud"
M: "... Just to make sure I understand, you want us to write queries for mssql, test them locally, and then write separate queries, with a separate SQL connection abstraction that deploys to production? Again, how are we going to test this?"
A: "no, use same queries, should be fine"
M: "they really won't, they're different dialects"
A: "do the needful, make work!"
If karma were a thing, this person would have long since exploded into a cloud of atomized blood.18 -
--- GitHub 24-hour outage post mortem ---
As many of you will remember; Github fell over earlier this month and cracked its head on the counter top on the way down. For more or less a full 24 hours the repo-wrangling behemoth had inconsistent data being presented to users, slow response times and failing requests during common user actions such as reporting issues and questioning your career choice in code reviews.
It's been revealed in a post-mortem of the incident (link at the end of the article) that DB replication was the root cause of the chaos after a failing 100G network link was being replaced during routine maintenance. I don't pretend to be a rockstar-ninja-wizard DBA but after speaking with colleagues who went a shade whiter when the term "replication" was used - It's hard to predict where a design decision will bite back and leave you untanging the web of lies and misinformation reported by the databases for weeks if not months after everything's gone a tad sideways.
When the link was yanked out of the east coast DC undergoing maintenance - Github's "Orchestrator" software did exactly what it was meant to do; It hit the "ohshi" button and failed over to another DC that wasn't reporting any issues. The hitch in the master plan was that when connectivity came back up at the east coast DC, Orchestrator was unable to (un)fail-over back to the east coast DC due to each cluster containing data the other didn't have.
At this point it's reasonable to assume that pants were turning funny colours - Monitoring systems across the board started squealing, firing off messages to engineers demanding they rouse from the land of nod and snap back to reality, that was a bit more "on-fire" than usual. A quick call to Orchestrator's API returned a result set that only contained database servers from the west coast - none of the east coast servers had responded.
Come 11pm UTC (about 10 minutes after the initial pant re-colouring) engineers realised they were well and truly backed into a corner, the site was flipped into "Yellow" status and internal mechanisms for deployments were locked out. 5 minutes later an Incident Co-ordinator was dragged from their lair by the status change and almost immediately flipped the site into "Red" status, a move i can only hope was accompanied by all the lights going red and klaxons sounding.
Even more engineers were roused from their slumber to help with the recovery effort, By this point hair was turning grey in real time - The fail-over DB cluster had been processing user data for nearly 40 minutes, every second that passed made the inevitable untangling process exponentially more difficult. Not long after this Github made the call to pause webhooks and Github Pages builds in an attempt to prevent further data loss, causing disruption to those of us using Github as a way of kicking off our deployment processes (myself included, I had to SSH in and run a git pull myself like some kind of savage).
Glossing over several more "And then things were still broken" sections of the post mortem; Clever engineers with their heads screwed on the right way successfully executed what i can only imagine was a large, complex and risky plan to untangle the mess and restore functionality. Github was picked up off the kitchen floor and promptly placed in a comfy chair with a sweet tea to recover. The enormous backlog of webhooks and Pages builds was caught up with and everything was more or less back to normal.
It goes to show that even the best laid plan rarely survives first contact with the enemy, In this case a failing 100G network link somewhere inside an east coast data center.
Link to the post mortem: https://blog.github.com/2018-10-30-...6 -
I'm convinced code addiction is a real problem and can lead to mental illness.
Dev: "Thanks for helping me with the splunk API. Already spent two weeks and was spinning my wheels."
Me: "I sent you the example over a month ago, I guess you could have used it to save time."
Dev: "I didn't understand it. I tried getting help from NetworkAdmin-Dan, SystemAdmin-Jake, they didn't understand what you sent me either."
Me: "I thought it was pretty simple. Pass it a query, get results back. That's it"
Dev: "The results were not in a standard JSON format. I was so confused."
Me: "Yea, it's sort-of JSON. Splunk streams the result as individual JSON records. You only have to deserialize each record into your object. I sent you the code sample."
Dev: "Your code didn't work. Dan and Jake were confused too. The data I have to process uses a very different result set. I guess I could have used it if you wrote the class more generically and had unit tests."
<oh frack...he's been going behind my back and telling people smack about my code again>
Me: "My code wouldn't have worked for you, because I'm serializing the objects I need and I do have unit tests, but they are only for the internal logic."
Dev:"I don't know, it confused me. Once I figured out the JSON problem and wrote unit tests, I really started to make progress. I used a tuple for this ... functional parameters for that...added a custom event for ... Took me a few weeks, but it's all covered by unit tests."
Me: "Wow. The way you explained the project was; get data from splunk and populate data in SQLServer. With the code I sent you, sounded like a 15 minute project."
Dev: "Oooh nooo...its waaay more complicated than that. I have this very complex splunk query, which I don't understand, and then I have to perform all this parsing, update a database...which I have no idea how it works. Its really...really complicated."
Me: "The splunk query returns what..4 fields...and DBA-Joe provided the upsert stored procedure..sounds like a 15 minute project."
Dev: "Maybe for you...we're all not super geniuses that crank out code. I hope to be at your level some day."
<frack you ... condescending a-hole ...you've got the same seniority here as I do>
Me: "No seriously, the code I sent would have got you 90% done. Write your deserializer for those 4 fields, execute the stored procedure, and call it a day. I don't think the effort justifies the outcome. Isn't the data for a report they'll only run every few months?"
Dev: "Yea, but Mgr-Nick wanted unit tests and I have to follow orders. I tried to explain the situation, but you know how he is."
<fracking liar..Nick doesn't know the difference between a unit test and breathalyzer test. I know exactly what you told Nick>
Dev: "Thanks again for your help. Gotta get back to it. I put a due date of April for this project and time's running out."
APRIL?!! Good Lord he's going to drag this intern-level project for another month!
After he left, I dug around and found the splunk query, the upsert stored proc, and yep, in about 15 minutes I was done.1 -
#2 Worst thing I've seen a co-worker do?
Back before we utilized stored procedures (and had an official/credentialed DBA), we used embedded/in-line SQL to fetch data from the database.
var sql = @"Select
FieldsToSelect
From
dbo.Whatever
Where
Id = @ID"
In attempts to fix database performance issues, a developer, T, started putting all the SQL on one line of code (some sql was formatted on 10+ lines to make it readable and easily copy+paste-able with SSMS)
var sql = "Select ... From...Where...etc";
His justification was putting all the SQL on one line make the code run faster.
T: "Fewer lines of code runs faster, everyone knows that."
Mgmt bought it.
This process took him a few months to complete.
When none of the effort proved to increase performance, T blamed the in-house developed ORM we were using (I wrote it, it was a simple wrapper around ADO.Net with extension methods for creating/setting parameters)
T: "Adding extra layers causes performance problems, everyone knows that."
Mgmt bought it again.
Removing the ORM, again took several months to complete.
By this time, we hired a real DBA and his focus was removing all the in-line SQL to use stored procedures, creating optimization plans, etc (stuff a real DBA does).
In the planning meetings (I was not apart of), T was selected to lead because of his coding optimization skills.
DBA: "I've been reviewing the execution plans, are all the SQL code on one line? What a mess. That has to be worst thing I ever saw."
T: "Yes, the previous developer, PaperTrail, is incompetent. If the code was written correctly the first time using stored procedures, or even formatted so people could read it, we wouldn't have all these performance problems."
DBA didn't know me (yet) and I didn't know about T's shenanigans (aka = lies) until nearly all the database perf issues were resolved and T received a recognition award for all his hard work (which also equaled a nice raise).7 -
Our web department was deploying a fairly large sales campaign (equivalent to a ‘Black Friday’ for us), and the day before, at 4:00PM, one of the devs emails us and asks “Hey, just a heads up, the main sales page takes almost 30 seconds to load. Any chance you could find out why? Thanks!”
We click the URL they sent, and sure enough, 30 seconds on the dot.
Our department manager almost fell out of his chair (a few ‘F’ bombs were thrown).
DBAs sit next door, so he shouts…
Mgr: ”Hey, did you know the new sales page is taking 30 seconds to open!?”
DBA: “Yea, but it’s not the database. Are you just now hearing about this? They have had performance problems for over week now. Our traces show it’s something on their end.”
Mgr: “-bleep- no!”
Mgr tries to get a hold of anyone …no one is answering the phone..so he leaves to find someone…anyone with authority.
4:15 he comes back..
Mgr: “-beep- All the web managers were in a meeting. I had to interrupt and ask if they knew about the performance problem.”
Me: “Oh crap. I assume they didn’t know or they wouldn’t be in a meeting.”
Mgr: “-bleep- no! No one knew. Apparently the only ones who knew were the 3 developers and the DBA!”
Me: “Uh…what exactly do they want us to do?”
Mgr: “The –bleep- if I know!”
Me: “Are there any load tests we could use for the staging servers? Maybe it’s only the developer servers.”
DBA: “No, just those 3 developers testing. They could reproduce the slowness on staging, so no need for the load tests.”
Mgr: “Oh my –bleep-ing God!”
4:30 ..one of the vice presidents comes into our area…
VP: “So, do we know what the problem is? John tells me you guys are fixing the problem.”
Mgr: “No, we just heard about the problem half hour ago. DBAs said the database side is fine and the traces look like the bottleneck is on web side of things.”
VP: “Hmm, no, John said the problem is the caching. Aren’t you responsible for that?”
Mgr: “Uh…um…yea, but I don’t think anyone knows what the problem is yet.”
VP: “Well, get the caching problem fixed as soon as possible. Our sales numbers this year hinge on the deployment tomorrow.”
- VP leaves -
Me: “I looked at the cache, it’s fine. Their traffic is barely a blip. How much do you want to bet they have a bug or a mistyped url in their javascript? A consistent 30 second load time is suspiciously indicative of a timeout somewhere.”
Mgr: “I was thinking the same thing. I’ll have networking run a trace.”
4:45 Networking run their trace, and sure enough, there was some relative path of ‘something’ pointing to a local resource not on development, it was waiting/timing out after 30 seconds. Fixed the path and page loaded instantaneously. Network admin walks over..
NetworkAdmin: “We had no idea they were having problems. If they told us last week, we could have identified the issue. Did anyone else think 30 second load time was a bit suspicious?”
4:50 VP walks in (“John” is the web team manager)..
VP: “John said the caching issue is fixed. Great job everyone.”
Mgr: “It wasn’t the caching, it was a mistyped resource or something in a javascript file.”
VP: “But the caching is fixed? Right? John said it was caching. Anyway, great job everyone. We’re going to have a great day tomorrow!”
VP leaves
NetworkAdmin: “Ouch…you feel that?”
Me: “Feel what?”
NetworkAdmin: “That bus John just threw us under.”
Mgr: “Yea, but I think John just saved 3 jobs. Remember that.”4 -
The solution for this one isn't nearly as amusing as the journey.
I was working for one of the largest retailers in NA as an architect. Said retailer had over a thousand big box stores, IT maintenance budget of $200M/year. The kind of place that just reeks of waste and mismanagement at every level.
They had installed a system to distribute training and instructional videos to every store, as well as recorded daily broadcasts to all store employees as a way of reducing management time spend with employees in the morning. This system had cost a cool 400M USD, not including labor and upgrades for round 1. Round 2 was another 100M to add a storage buffer to each store because they'd failed to account for the fact that their internet connections at the store and the outbound pipe from the DC wasn't capable of running the public facing e-commerce and streaming all the video data to every store in realtime. Typical massive enterprise clusterfuck.
Then security gets involved. Each device at stores had a different address on a private megawan. The stores didn't generally phone home, home phoned them as an access control measure; stores calling the DC was verboten. This presented an obvious problem for the video system because it needed to pull updates.
The brilliant Infosys resources had a bright idea to solve this problem:
- Treat each device IP as an access key for that device (avg 15 per store per store).
- Verify the request ip, then issue a redirect with ANOTHER ip unique to that device that the firewall would ingress only to the video subnet
- Do it all with the F5
A few months later, the networking team comes back and announces that after months of work and 10s of people years they can't implement the solution because iRules have a size limit and they would need more than 60,000 lines or 15,000 rules to implement it. Sad trombones all around.
Then, a wild DBA appears, steps up to the plate and says he can solve the problem with the power of ORACLE! Few months later he comes back with some absolutely batshit solution that stored the individual octets of an IPV4, multiple nested queries to the same table to emulate subnet masking through some temp table spanning voodoo. Time to complete: 2-4 minutes per request. He too eventually gives up the fight, sort of, in that backhanded way DBAs tend to do everything. I wish I would have paid more attention to that abortion because the rationale and its mechanics were just staggeringly rube goldberg and should have been documented for posterity.
So I catch wind of this sitting in a CAB meeting. I hear them talking about how there's "no way to solve this problem, it's too complex, we're going to need a lot more databases to handle this." I tune in and gather all it really needs to do, since the ingress firewall is handling the origin IP checks, is convert the request IP to video ingress IP, 302 and call it a day.
While they're all grandstanding and pontificating, I fire up visual studio and:
- write a method that encodes the incoming request IP into a single uint32
- write an http module that keeps an in-memory dictionary of uint32,string for the request, response, converts the request ip and 302s the call with blackhole support
- convert all the mappings in the spreadsheet attached to the meetings into a csv, dump to disk
- write a wpf application to allow for easily managing the IP database in the short term
- deploy the solution one of our stage boxes
- add a TODO to eventually move this to a database
All this took about 5 minutes. I interrupt their conversation to ask them to retarget their test to the port I exposed on the stage box. Then watch them stare in stunned silence as the crow grows cold.
According to a friend who still works there, that code is still running in production on a single node to this day. And still running on the same static file database.
#TheValueOfEngineers2 -
The website for our biggest client went down and the server went haywire. Though for this client we don’t provide any infrastructure, so we called their it partner to start figuring this out.
They started blaming us, asking is if we had upgraded the website or changed any PHP settings, which all were a firm no from us. So they told us they had competent people working on the matter.
TL;DR their people isn’t competent and I ended up fixing the issue.
Hours go by, nothing happens, client calls us and we call the it partner, nothing, they don’t understand anything. Told us they can’t find any logs etc.
So we setup a conference call with our CXO, me, another dev and a few people from the it partner.
At this point I’m just asking them if they’ve looked at this and this, no good answer, I fetch a long ethernet cable from my desk, pull it to the CXO’s office and hook up my laptop to start looking into things myself.
IT partner still can’t find anything wrong. I tail the httpd error log and see thousands upon thousands of warning messages about mysql being loaded twice, but that’s not the issue here.
Check top and see there’s 257 instances of httpd, whereas 256 is spawned by httpd, mysql is using 600% cpu and whenever I try to connect to mysql through cli it throws me a too many connections error.
I heard the IT partner talking about a ddos attack, so I asked them to pull it off the public network and only give us access through our vpn. They do that, reboot server, same problems.
Finally we get the it partner to rollback the vm to earlier last night. Everything works great, 30 min later, it crashes again. At this point I’m getting tired and frustrated, this isn’t my job, I thought they had competent people working on this.
I noticed that the db had a few corrupted tables, and ask the it partner to get a dba to look at it. No prevail.
5’o’clock is here, we decide to give the vm rollback another try, but first we go home, get some dinner and resume at 6pm. I had told them I wanted to be in on this call, and said let me try this time.
They spend ages doing the rollback, and then for some reason they have to reconfigure the network and shit. Once it booted, I told their tech to stop mysqld and httpd immediately and prevent it from start at boot.
I can now look at the logs that is leading to this issue. I noticed our debug flag was on and had generated a 30gb log file. Tail it and see it’s what I’d expect, warmings and warnings, And all other logs for mysql and apache is huge, so the drive is full. Just gotta delete it.
I quietly start apache and mysql, see the website is working fine, shut it down and just take a copy of the var/lib/mysql directory and etc directory just go have backups.
Starting to connect a few dots, but I wasn’t exactly sure if it was right. Had the full drive caused mysql to corrupt itself? Only one way to find out. Start apache and mysql back up, and just wait and see. Meanwhile I fixed that mysql being loaded twice. Some genius had put load mysql.so at the top and bottom of php ini.
While waiting on the server to crash again, I’m talking to the it support guy, who told me they haven’t updated anything on the server except security patches now and then, and they didn’t have anyone familiar with this setup. No shit, it’s running php 5.3 -.-
Website up and running 1.5 later, mission accomplished.6 -
Every day I feel like Dr. Leonard McCoy.
"I'm an automation expert, not a ..."
* DBA
* ElasticSearch administrator
* Kafka Architect
* Email marketing specialist
* SEO Analyst
* Front end developer
* Doctor
* Project manager
* Accountant8 -
TLDR - you shouldn't expect common sense from idiots who have access to databases.
I joined a startup recently. I know startups are not known for their stable architecture, but this was next level stuff.
There is one prod mongodb server.
The db has 300 collections.
200 of those 300 collections are backups/test collections.
25 collections are used to store LOGS!! They decided to store millions of logs in a nosql db because setting up a mysql server requires effort, why do that when you've already set up mongodb. Lol 😂
Each field is indexed separately in the log.
1 collection is of 2 tb and has more than 1 billion records.
Out of the 1 billion records, 1 million records are required, the rest are obsolete. Each field has an index. Apparently the asshole DBA never knew there's something called capped collection or partial indexes.
Trying to get approval to clean up the db since 3 months, but fucking bureaucracy. Extremely high server costs plus every week the db goes down since some idiot runs a query on this mammoth collection. There's one single set of credentials for everything. Everyone from applications to interns use the same creds.
And the asshole DBA left, making me in charge of handling this shit now. I am trying to fix this but am stuck to get approval from business management. Devs like these make me feel sad that they have zero respect for their work and inability to listen to people trying to improve the system.
Going to leave this place really soon. No point in working somewhere where you are expected to show up for 8 hours, irrespective of whether you even switch on your laptop.
Wish me luck folks.3 -
So here is my week 72 as a reviewer. But first, let me ask y'all. Am I weird to think that you should finish coding the thing and testing the thing before kicking it out to review? Cuz that's how I do it. And that is the process at my work place.
So anyway, I was doing this review. And it was very wrong. Like really, really wrong. We give a thorough intro to our product (perhaps too thorough) so this guy should have known every test case he had to cover. Or at least, if he was unsure, asked. It was all documented.
Anyway, he kicks out this peer review. First thing I notice, it is not following the standard. Fair enough, we didn't give him the coding standard. BUT HE DIDN'T EVEN MAINTAIN THE FORMAT OF THE ORIGINAL FILE. HE JUST DID HIS OWN THING!!! So I email him the coding standard and make a comment in the review. He denies the finding. No reason. Just turns it down. Strike 1.
Then, I'm going through and he didn't even cover all of the core cases. I found several core cases that he missed. And every edge case. Make a not of it. He fixes only the couple of examples I gave him. Strike 2.
Guy decided to redesign a major chunk of our interfaces. Our interfaces are not just used by us (hence interfaces). We designed them the way they were for a reason. It was not a fun design process. Myself, the architect, one of our customers, and the guy that did the implementation all told him to roll back his change. Especially since it wasn't in the scope of what he was doing. He wouldn't. Strike 3.
I go to the lead and bring him in. He has a talk with him. All of the sudden he is putting out multiple builds per finding. Like most times I will put out like 2 to 4 for the whole peer review. No he kicks out minimum one per finding and chokes the review queue. Strike 4.
Strike 5: he tells me, a former DBA, that I didn't know what I was talking about when I told him to move something into a new table, even after I told him that "while in database terms it doesn't make sense, this is for product robustness".
Strike 6: he was just a condescending asshole. Bragging about how he did this job and that job over his career. His longest position held was about 18 months. Bragged about working at my company and being some hotshot at the company: only worked here for 8 months and that was 5 years ago.
You know. I have never really wanted to fight someone after about undergrad. But he came close.7 -
Got in trouble today during a Data integrity meeting, everyone kept talking about data massaging. "Massage this data", "massage that data table", "insert massaged data".
Finally I just blurted out, "yeah massage it all you want but how do I get a data happy ending?"
I thought it was hilarious. The other DBA and backend devs thought it was jokes, my manager... Not so much.
Apparently, I need to keep "thoughts and comments about data happy endings to myself moving forward".
Okay. 😆😆5 -
A couple of months back we were discussing sh with a third party vendor for a very large ass fuck system that another department uses. I had been called into the meeting because the entire I.T department counts on me to at least act as an assessor to the many issues that other departments might have.
the department for which i was working with manages the databases that our institution uses, and in this particular question the DBA (my best friend mind you) was part of the meeting.
Mind you, issues that the third party vendor were having were all fixed by our DBA, and he had documented and mentioned these items to me as I provided assistance to him through the 3 weeks prior to this meetings. Once such case was that we needed a transitioning as well as intermediary system for some processes to happen from one DB to the other and a lot of other technical babble. Well, the DBA used to be an excellent (fuck you) VB developer who recently re-learned the language into .net. He had shown me many of his old programs and even by the limitations of the language they were elegant and fascinating. They really are and ya'll devrant fam know that I ain't one to hate on tech at all.
When the DBA explained how he went around some of the issues by generating programs that could assist him, he mentioned the tech stack, I had coached him into knowing that being descriptive about the tools he used would be beneficial to everyone else. While he mentioned VB.NET the vendor snickered and my boy got quiet.
Then I broke the silence, fuck you. "what was that?" and the dude said "nothing, sorry"
So I said "no no, I want to know, I am not going past this point until you, the dude getting paid over $100 an hour for something YOU couldn't fix explain to me the little hehe moment you had"
The mfker went silent. then explained how he was aware that people were moving past vb.net and shit like that, me "imagine that, someone used a tech stack that your ignorance thought obsolete to fix something you could not solve, even though we are paying you for it, were it me or in my hands, and mind you i have direct access to the VP so this foolishness might change, I would have cut you and your little sect loose months ago, I have no patience, or appreciation from leeches like you or the rest of the "professionals" that work for your company or other similar entities, much less, as you can see, my patience runs even less when you people snicker at the solutions that our staff has to take when you all slack"
The entire meeting was uncomfortable as high heaven.
Fuck you, if someone I know manages to run shit on fucking liberty basic then so fucking be it. I will slap you 10 fucking times over, and then fuck your girl, if you try to put someone else down for the tech stacks you use.
I hate neck beards, BUT I hate fake ass neckbeards ever more
*Colin Farrell in true detective mode: FUCK....YOU13 -
I think the worst work culture you can experience is nepotism and corruption in hierarchy. What do I mean? Well, this happened (and I think is still happening) in my last job. It was a huge logistic/delivery company. I was an intern, working as assistant developer of the only developer of the site. There was also a guy that was the technician, his assistant, a DBA and that's it.
Well, my partner and I were working on a system that managed almost all the operations of the company in this city.
Well, I supplied the dev two weeks when he was on vacation. I knew almost all the system. what happened? the manager from other city came with another Dev, and I'm not saying that I was an expert or something like that, but that dev from the other city was an incompetent. He couldn't even make a small GUI change without messing it all...
Guess what? The company paid him weekly round tickets to come and go from his city to ours (two hours of flight).
I was too disappointed I started searching another job. A week after getting my degree, I left my job and started in the one I am now. Before leaving, I asked my boss if there was a realistic chance to grow up. He answered no. To be honest, that didn't surprised me :/
The thing that makes me angry about this is that a lot of companies give chances to people that come from other cities, even if they don't know anything >:v
Oh, I almost forgot it: The last five months I was working there, they quit our office and send us to trailer-offices :/1 -
Major update went good through the IT side.
Somehow on a different department, some moron botched up what he had to do (he has a technical role, but he is no programmer or analyst or anything like that and even though he is an admin (with big fucking quotes) no one here would call him so.
He fucks his shit up, and the DBA and myself have to go and fix it. Fine, whatever.
Why am I ranting? Because this bitch sent an email tagging half the board and every other VP stating that it was I.T's fault this happened. Somehow, she had forgotten that she had tagged me into that email since she tagged the Web Developer section email.
I did not give two flying fucks and in front of everyone called them out on what actually happened. I was polite and used some very non-directed "bitch it was you" comments. But it pissed me off that she did that shit behind I.T's back.
Everyone in that email replied saying thank you.....except for her.
I will slap a hoe, I swear.13 -
Never mess with a motivated developer. I will make your life difficult in return.
Me: we need server logs and stats daily for analysis
DBA: to get those, you need to open a ticket
Me: can't you just give me SFTP access and permissions to query the stats from the DB?
DBA: No.
*OK.... 🤔🤔🤔*
*Writes an Excel Template file that I basically just need to copy and paste from to create a ticket*
This process should not take me more than 2mins 👍😁😋🙂😙😙😙😙😙😙😙😙
For them.... 😈😈😈😈😈😈😈😈😈😈😈9 -
In order to reduce support costs, manager instructed his team to remove all logging/reporting of errors in the company’s CRM application.
Team’s support tickets went down 80%, manager received an award for his efforts, but mysteriously, DBA/support workload increased, bad/missing data,
increased support tickets in other areas of the business (shipping, etc. that relied on correct data from the CRM) and other side-affectual behavior.
Even after pointing this out this correlation, showing before/after code, no one believed the two were related and I was accused of not being a ‘team player’.
“You and the other teams need to learn from his example!”. As ‘punishment’ was I was moved to the team managing the CRM application.1 -
Today was a manic-depressive kind of day. Spent the morning helping some developers with getting their code to run a stored procedure to drop old partitions, but it wasn't working on their end. It was a fairly simple proc. But working with partitions is a little like working with an array. I figured out that they were passing the wrong timestamp, and needed to add +1 to delete the right partition. Got that sorted out, and things were good. Lunch time.
After lunch I did some busy work, and then the PO comes up at about 2PM and says he's assigned some requests to me. The first was just attaching some scripts. Easy. The second, the user wants a couple of schemas exported ... at 6PM. I've been in the office since 6:45AM.
While I'm setting up some commands to run for the data export, a BA walks up and asks if I'm filling in for another DBA who is out for a few weeks. Yep. There's a change request that hasn't been assigned, and he normally does the work. I ask when it's due. Well, the pre-implementation was supposed to be done in the morning, but it wasn't, and we're in the implementation window ... half way through. I bring up the change task, and look at. Create new schema and users. That's all it says. The BA laughs. I tell I need more to go on. 10 minutes later he sends an email with the information. There's only two hours left in the window, and I can only use half of it, because the production guys have to their stuff, and we're in their window. Now I'm irritated, because I'm new to Oracle, and it's an unforgiving mistress. Fortunately, another DBA says he'll do it, so that we can get it done in time. But can't work it either, because Dev DBAs don't have access to QA, and the process required access for this task. Gets shelved until the access issue is resolved. It's now after 4:15PM. I'm going to in traffic with that 6PM deadline.
I manage to get home and to the computer by 5:45PM. Log in. Start VPN. Box pops on screen. Java needs to update. I chose skip update. Box pops up again. It won't let me log in until Java is current. Passed.
I finally get logged in, and it's 6:10PM. I'm late getting the job started. I pull up Putty and log into the first box, and paste my pre-prepared command in the command line and hit error. Command not found. I'm tired, so it's a moment to sink in. I don't have time for this.
I log into DBArtisan and pull up the first data base, use the wizard to set the job, and off it goes. Yay. Bring up the second database, and have enter the connect info. Host not found. Wut? Examine host name. Yep, it's correct. Try a different method. Host not found. Go back to Putty. Log in. Past string. Launch. Command not found. Now my brain is quitting on me. Why now? It's after 6:30PM. Fiddle with some settings, reset $Oracle home. Try again. Yay. It works. I'm done. It's after 7PM.
There is nothing like technology to snatch the euphoria of a success away from you. It's a love-hate thing, but I wouldn't trade it for anything else. I'm done. Good night.3 -
SQL Rule 1. Always assume there are external processes that might affect your data. (for instance, triggers).
SQL Rule 2. In Denormalised data, never execute logic on dependant table values, always copy from the parent.
SQL Rule 3. When Denormalised data schemas are created the DBA knows what they are doing.
SQL Rule 3.1. If DBA knows what they is doing then according to Rule 1 there is no problem with adding in some triggers to maintain data clones as they are created.
SQL Rule 4. If you don't like or agree with triggers, deal with it. They are a first class tool in a first class RDBMS. In a multi-app or service environment there may be many other external processes massaging your data
SQL Rule 5. If all previous rules are not broken and the system has been running efficiently for many years DO NOT complain that there are triggers in the database that are doing and have been doing the same process that you just butchered (by violating Rule 1 and 2) in your makeshift "hello world, look what I can do from my phone" angular BS when the rest of the users are still relying on the existing runtime app.
SQL Rule 6. If you turn my triggers off, you sure as hell better turn them back on!1 -
At work, my closest relation is with the DBA. Dude is a genius when it comes to proper database management as well as having a very high level of understanding concerning server administration, how he got that good at that I have no clue, he just says that he likes to fuck around with servers, Linux in particular although he also knows a lot about Windows servers.
Thing is, the dude used to work as a dev way back when VB pre VB.NET was all the rage and has been generating different small tools for his team of analysts(I used to be a part of his team) to use with only him maintaining them. He mentioned how he did not like how Microsoft just said fk u to VB6 developers, but that he was happy as long as he could use VB. He relearned how to do most of the GUI stuff he was used to do with VB6 into VB.NEt and all was good with the world. I have seen his code, proper OOP practices and architectural decisions, etc etc. Nothing to complain about his code, seems easy enough to extend, properly documented as well.
Then he got with me in order to figure out how to breach the gap between building GUI applications into web form, so that we could just host those apps in one of our servers and his users go from there, boy was he not prepared to see the amount of fuckery that we do in the web development world. Last time my dude touched web development there was still Classic ASP with JScript and VBScript(we actually had the same employer at one point in the past in which I had to deal with said technology, not bad, but definitely not something I recommend for the current state of web development) and decided that the closest thing to what he was used was either PHP(which he did not enjoy, no problem with that really, he just didn't click with the language) and WebForms using VB.NET, which he also did not like on account of them basically being on support mode since Microsoft is really pushing for people to adopt dotnet core.
After came ASP.NET with MVC, now, he did like it, but still had that lil bug in his head that told him that sticking to core was probably a better idea since he was just starting, why not start with the newest and greatest? Then in hit(both of us actually) that to this day Microsoft still not has command line templates for building web applications in .net core using VB.NET. I thought it was weird, so I decided to look into. Turns out, that without using Razor, you can actually build Web APIs with VB.NET just fine if you just convert a C# template into VB.NET, the process was...err....tricky, and not something we would want to do for other projects, with that in we decided to look into Microsoft's reasons to not have VB.NET. We discovered how Microsoft is not keeping the same language features between both languages, having crown C# as the language of choice for everything Microsoft, to this point, it seems that Microsoft was much more focused in developing features for the excellent F# way more than it ever had for VB.NET at this point and that it was not a major strategy for them to adapt most of the .net core functionality inside of VB, we found articles when the very same Microsoft team stated of how they will be slowly adding the required support for VB and that on version 5 we would definitely have proper support for VB.NET ALTHOUGH they will not be adding any new development into the language.
Past experience with Microsoft seems to point at them getting more and more ready to completely drop the language, it does not matter how many people use it, they would still kill it :P I personally would rather keep it, or open source the language's features so that people can keep adding support to it(if they can of course) because of its historical significance rather than them just completely dropping the language. I prefer using C#, and most of my .net core applications use C#, its very similar to Java on a lot of things(although very much different in others) and I am fine with it being the main language. I just think that it sucks to leave such a large developer pool in the shadows with their preferred tool of choice and force them to use something else just like that.
My boy is currently looking at how I developed a sample api with validation, user management, mediatR and a custom project structure as well as a client side application using React and typescript swappable with another one built using Angular(i wanted to test the differences to see which one I prefer, React with Typescript is beautiful, would not want to use it without it) and he is hating every minute of it on account of how complex frontend development has become :V
Just wanted to vent a little about a non bothersome situation.6 -
Still dealing with the web department and their finger pointing after several thousand errors logged.
SeniorWebDev: “Looks like there were 250 database timeout errors at 11:02AM. DBAs might want to take a look.”
I look at the actual exceptions being logged (bulk of the over 1,600 logged errors)..
“Object reference not set to an instance of an object.”
Then I looked the email timestamp…11:00AM. We received the email notification *before* the database timeout errors occurred.
I gather some facts…when the exceptions started, when they ended, and used the stack trace to find the code not checking for null (maybe 10 minutes of junior dev detective work). Send the data to the ‘powers that be’ and carried on with my daily tasks.
I attached what I found (not the actual code, it was changed to protect the innocent)
Couple of hours later another WebDev replied…
WebDev: “These errors look like a database connectivity issue between the web site and the saleitem data service. Appears the logging framework doesn’t allow us to log any information about the database connection.”
FRACK!!...that Fracking lying piece of frack! Our team is responsible for the logging framework. I was typing up my response (having to calm down) then about a minute later the head DBA replies …
DBA: “Do you have any evidence of this? Our logs show no connectivity issues. The logging framework does have the ability to log an extensive amount of data regarding the database transaction. Database name, server, login, command text, and parameter values. Everything we need to troubleshoot. This is the link to the documentation …. If you implement the one line of code to gather the data, it will go a long way in helping us debug performance and connectivity issue. Thank you.”
DBA sends me a skype message “You’re welcome :)”
Ahh..nice to see someone else fed up with their lying bull...stuff. -
This was not exactly the worst work culture because the employees, it was because the upper level of the organization chart on the IT department.
I'm not quite sure how to translate the exact positions of that chart, but lets say that there is a General Manager, a couple of Area Managers (Infrastructure, Development), some Area Supervisors (2 or 3, by each area), and the grunts (that were us). Anyway, anything on the "Manager" was the source of all the toxicity on the department.
First and foremost, there was a lack of training for almost any employee. We were expected to know everything since day-1. Yes, the new employees had a (very) brief explanation about the technologies/languages were used, but they were expected to perform as a senior employee almost since the moment they cross the door. And forget about having some KT (Knowledge Transfer) sessions, they were none existent and if they existed, were only to solve a very immediate issue (now imagine what happened when someone quit*).
The general culture that they have to always say "yes" to the client/customer to almost anything without consulting to the development teams if that what was being asked to do was doable, or even feasible. And forget about doing a proper documentation about that change/development, as "that was needed yesterday and it needs to be done to be implemented tomorrow" (you know what I mean). This contributes to the previous point, as we didn't have enough time to train someone new because we had this absurd deadlines.
And because they cannot/wanted to say "NO", there were days when they came with an amount of new requirements that needed to be done and it didn't matter that we had other things to do. And the worst was that, until a couple of years (more or less), there was almost impossible to gather the correct requirements from the client/user, as they (managers) "had already" that requirement, and as they "know better" what the user wants, it was their vision what was being described on the requirements, not the users'...
And all that caused that, in a common basis, didn't have enough time to do all this stuff (mainly because the User Support) causing that we needed to do overtime, which almost always went unpaid (because a very ambiguous clause of the contract, and that we were "non-union workers"**). And this is my favorite point of this list, because, almost any overtime went unpaid, so basically we were expected to be working for free after the end of the work day (lets say, after the 17:00). Leaving "early" was almost a sin for the managers, as they always expected that we give more time to work that the indicated on the contract, and if not, they could raise a report to HR because the ambiguous clause allowed them to do it (among other childish things that they do).
Finally, the jewel of the crown, is that they never, but never acknowledge that they made a mistake. Never. That was impossible! If something failed on the things/systems/applications that they had assigned*** it was always our fault.
- "A report for the Finance Department is giving wrong information? It's the DBA's fault**** because although he manages that report, he couldn't imagine that I have an undocumented service (that runs before the creation the report) crashed because I modified a hidden and undocumented temporal table and forgot to update that service."
But, well, at least that's on the past. And although those aren't all the things that made that workplace so toxic, for me those were the most prominent ones.
-
* Well, here we I live it's very common to don't say anything about leaving the company until the very last day. Yes, I know that there are people that leave their "2-days notice", but it's not common (IMHO, of course). And yes, there are some of us that give a 1 or 2-weeks notice, but still it's not a common practice.
** I don't know how to translate this... We have a concept called "trusted employee", which is mainly used to describe any administrative employee, and that commonly is expected to give the 110% of what the contract says (unpaid overtimes, extra stuff to do, etc) and sadly it's an accepted condition (for whatever reasons). I chose "non-union workers" because in comparison with an union worker, we have less protections (besides the legal ways) regarding what I've described before. Curiously, there are also "operative workers", that doesn't belong to an union, but they have (sometimes) better protections that the administrative ones.
*** Yes, they were in charge of several systems, because they didn't trust us to handle/maintain them. And I'm sure that they still don't trust in their developers.
**** One of the managers, and the DBA are the only ones that handle some stuff (specially the one that involves "money"). The thing that allows to use the DBA as scapegoat is that such manager have more privileges and permissions than the DBA, as he was the previous DBA2 -
Alright lads here is the thing, have not been posting anything other than replies to things cuz I have been busy being miserable at school and dealing with work stuff.
Our manager left us back in February. Because she was leaving I decided that I wanted to try a different path and went on to become a programmer analyst for my institution, if anything I knew that it was going to be pretty boring work, but it came with nice monetary compensation and a foot in the door for other data science related jobs in the future. Thing is, the department head asked me to stay in the web technologies department because we had a lack of people there and hiring is hard as shit, we do not do remote jobs since our work usually requires a level of discretion and security. Thus I have been working in the web tech department since she left albeit with a different title since I aced the interview for the analyst position and the team there were more than happy to have me. I have done very few things for them, some reports here and there and mostly working directly with the DBA in some projects. One migration project would have costed my institution a total of 58k and we managed to save the cost by building the migration software ourselves.....honestly it was a fucking cake walk, if you had any doubts about the shaddyness of enterprise level applications regarding selling overpriced shit with different levels of complexity, keep them, enterprise is shaddy af indeed. But I digress.
I wrote the specification for the manager position along the previous manager, we had decided that the next candidate needed to be strong with development knowledge as well as other things as to properly understand and manage a software team, we made the academic requirement(fuck you, yes we did ask for academic requirements) to be either in the Computer Science/software engineering area or at least on the Business Administration side. We were willing to consider BA holders in exchange for having knowledge of the development process of different products and a complete understanding of what developers go through. NOT ONE SINGLE motherfucker was able to satisfy this, some of them were idiots that I knew from before that had ABSOLUTELY no business even considering applying to the position, the courage it took for some of these assholes to apply would have hurt their mothers, their God if they had one, and their country, they were just that fucking bad in their jobs as well as being overall shit people.
Then we had 1 candidate actually fall through the cracks enough to get an interview. My dude here was lying out of his ass through the interview process. According to him he had "lots of Laravel experience and experience managing Laravel projects" and mentioned repeatedly how it would be a technology that we should consider for our products. I was to interview him alongside the vice president of our institution due to the head of my department and the rest of the managers for I.T being on vacation leave all at the same bloody time.
Backstory before the interview:
Whilst I was going over the interview questions with the vice president literally offered me the job instead. I replied with honesty, reflecting how I did not originally wanted him but feeling that our institution was ready to settle on any candidate due to the lack of potentials. He was happy to do it since apparently both him and the HOD were expecting me to step up sooner or later. I was floored.
Regardless, out of kindness he wanted to go through the interview.
So, going back to the interview. As soon as the person in question referenced the framework I started to ask him about it, just simple questions, the first was "what are your thoughts on the Eloquent ORM? I am not too fond of it and want to know what you as a full time laravel dev think of it"
his reply: "I am sorry I am not too familiar with it, I don't know what that is" <--- I appreciated his honesty in this but thought it funny that someone would say that he was a Laravel developer whilst not knowing what an ORM was since you can't really get away from using it on the initial stages of learning about Laravel, maybe if one wanted to go through the hurdle of switching to something like doctrine...but even then, it was....odd.
So I met with the hod when he came back, he was stoked at the prospect of having me become the manager and I happily accepted the position. It will be hell, but I don't even need to hit the ground running since I have been the face of the department since ages. My team were ecstatic about it since we are all close friends and they have been following my directions without complaints(but the ocational eat a dick puto) for some time, we work well together and we are happy to finally have someone to stop the constant barrage that comes from people taking advantage of a missing manager.
Its gonna get good, its gonna get fun, and i am getting to see how shit goes.7 -
I JUST HAD ONE OF THOSE DAYS THAT MAKES ONE WANT TO BANG TWO BRICKS ON HEAD SND END THE PAIN THE STORY STARTS YESTETDAY WITH ISSUES AFTER A MIGRSTION AND THEY ASK ME TO HELP TROUBLESHOOT EVEN THOUGH I'M A DEV DBA AND THE ISSUE IS IN QA/SAT AND I HELP ANYWAY AND THEY CAN'T FIND A VIEW AND SO I LOOK EVERYWHERE AND CAN'T DOING IT EITHER AND IT DIDN'T EXIST IN PROD OR DEV SO I TELL THEM IT'S NOT THERE, AND THEY ARE LIKE, CAN YOU RETRIEVE IT FOR US AND I'M LIKE FROM WHERE? I DON'T KEEP VIEWS IN MY BUTT AND YOU GUYS ARE SMOKING CRACK AND THE GIVE ME THEIR QUERY WHICH CONTAIN THE VIEE ANYWAY AND THEY SAY CAN YOU RUN IT AND IT RUNS AND WORKS AND THEY CAN'T MAKE IT WORK AND IT WORKS BECAUSE IT DOESN'T CALL THE VIEW THEY HAVE ME SO NO PROBLEM THERE SO I FINALLY ASK THEM ARE YOU POINTING TO THE CORRECT DATABASE AND THEY'RE LIKE OH MAN WE TOLD YOU THE WRONG DATABASE AND SO I LOOK AT THE RIGHT DATABASE AND FIND THAT THE GRANTS ARE MISSING AND YEAH THANK YOU FOR TAKING EIGHT HOURS OF MY LIFE BECAUSE WE WERE IN THE WRONG DB YOU GAVE ME AND I HOPE THE FLAG OF A THOUSAND CAMELS INVEST YOUR ARMPITS AND THE CHIGGERS OF A THOUSAND SOUTHERN LAWNS INGEST YOUR SOCKS AND UNDERWEAR. YAAAAAA!!!!9
-
Ever had a day that felt like you're shoveling snow from the driveway? In a blizzard? With thunderstorms & falling unicorns? Like you shovel away one m² & turn around and no footprints visible anymore? And snow built up to your neck?
Today my work day was like that.. xcept shit..shit instead of pretty & puffy snow!!
Working on things a & b, trying to not mess either one up, then comes shit x, coworker was updating production.. ofc something went wrong.. again not testing after the update..then me 'to da rescue'.. :/ hardly patch things up, so it works..in a way.. feature c still missing due to needed workarounds.. going back to a and b.. got disrupted by the same coworker who is nver listening, but always asking too much..
And when I think I finally have the b thing figured out a f-ing blocker from one of our biggest clients.. The whole system is unresponsive.. Needles to say, same guy in support for two companies (their end), so they filed the jira blocker with the wrong customer that doesn't have a SLA so no urgent emails..and then the phone calls.. and then the hell broke loose.. checking what is happening.. After frantic calls from our dba to anyone who even knows that our customer exists if they were doing sth on the db.. noup, not a single one was fucking with the prod db.. The hell! Materialised view created 10 mins ago that blocked everything..set to recreate every 10 minutes..with a query that I am guessing couldn't even select all that data in under 15.. dafaaaq?! Then we kill it..and again it is there.. We found out that customers dbas were testing something on live environment, oblivious that they mamaged to block the entire db..
FML, I'm going pokemon hunting.. :/ codename for ingress n beer..3 -
Most awkward video conference call?
Our department is in a 'virtual' book club, reading The Unicorn Project, and I asked..
Me: "So what similarities have you seen with the Phoenix project and projects we work on here?"
Dale: "Ha ha..sooo many. The biggest is the disconnect of managers with no clue of what goes on."
<Vice president of our department also in the book club>
VP: "Really? Dale, I'd like to know more about this."
<awkward silence with blank stares all around>
DBA: "Come on Dale...spill the beans. Got the VP right there."
Dale: "Um...nope...not going there...nope"
<Dale's screen goes black>
VP: "OK, so when Maxine asks ..." -
I actually never felt the need to scream at a co-worker so let's talk about that time a co-worker screamed at me instead.
tl;dr : some asshole boss screamed and threatened me because someone else's project was shit and didn't work.
Context: I was in my third year of school internship (graded) and my experience is C, C++, C#, Python all in systems programming, no web.
I was working as an intern for a shit company that was selling a shit software to hospitals (though not medically critical, thank God) the only tech guy on site was the DBA (cool guy) the product was maintained by a single dev in VB from his house, the dude never showed up to work (you'll understand why) and an other intern who couldn't dev shit.
I was working with the DBA on an software making statistical analysis from DB exports, worked nice, no problems here if we forget the lack of specs or boundaries (except must work in ieShit).
The other intern was working on something else (don't ask me what it is) I just remember it was in GWT before the community revived it. His webapp was requesting the company http server for a file instead of having one of it's java servlet to fetch it (both apps ran on sane server) which caused a lot of shit especially CORS error. That guy left (end of contract) and leaves his shit as is, boss asked me to deploy the app, I fiddle with it to see if it works and when I find out it doesn't then that asshole starts screaming at me in front of every other employee present, starts threatening to burn me in the tech world and have me thrown out of my school for no goddamn reason than the other dude's project doesn't work.
After the screaming I leave and warn my school immediately.
I guess that's why the other dev never came to work.
I had three weeks of internship left, that I did from home and worked probably less than 2 hours a day so suck it asshole.
Still had a good grade because I was reviewed by the DBA and he was happy with the work I did.
It was only later that I realized that what he did was categorizing as harassment (at least in France) and decided that never again this would happen without a response from my lawyer.1 -
GWT.
Let me explain:
Tl;dr : someone fucked up, I took shit, it was a gwt project. In a sense I don't hate GWT because of the framework itself but because how I was introduced and forced to "work" with it.
Context:
Was working as a paid intern at a small company there were 3 devs 2 interns and one senior employee that only worked from home handling the shit ton of legacy VB6 code he wrote over several year and a boss with no technical knowledge. (Other unimportant people as well)
I was working with their DBA (cool dude) because I was writing statistic and report generating software.
Story:
The other intern was tasked of doing a gwt app that was supposed to use a input file.
Rather than asking the user to upload it with a file picker (I guess they exist in gwt I didn't got to dig in the framework) he was trying to load the file with a http request directed at the same host the app was running on.
It did not work.
Then his contract was other and the app was left in an unfinished state.
The boss then tried to have the app deployed, the remaining dev dodged the bullet invoking some bullshit because he was clearly incapable of doing it.
So it fell on me, couldn't deploy the app because it was not even close to working.
Tried to fix things and make it work.
Turns out he thought it would take me 3h to deploy when I clearly explained that the other guy didn't finish the app.
Boss got mad, threatened to ruin my studies and my future career.
Couldn't because my uni had my back.
Didn't want to see me anymore.
Couldn't break my contract.
Told me to work from home for the end of my internship.
I got 3 weeks early vacation and got paid, fuck him, fuck GWT, fuck his company.
Still got well marked for the internship as my supervisor was the DBA who was happy with my work.
Morality:
Don't let your intern unsupervised, don't let your main dev work from home when you don't know shit, don't piss me off and send me work from home. -
How many of you have started a job and ended up doing something totally different, and how did deal with it?
I was hired to be an SQL Developer, writing reports, views, stored procedures etc, but knew I would be asked to do some work on some internal c# apps.
Roll on almost 2 years and I'm pretty much a DBA in all but name, and the C# app I was supposed to be doing a little work on is now mine, so as well as being the sole programmer, I'm also the product owner, 1st, 2nd, and 3rd line support.
All of this and I've not even had a change in title, let alone a pay rise that reflect the new roles and responsibilities I've taken on.9 -
I went for an interview as a MS SQL DBA. They gave me a technical knowledge test which I passed then started asking about me, All was going well until they started telling me what the company did.
They harvested people's data from websites online questionnaires and collated it into saleable data.
This was too much for me and I'm not certain that it isn't a grey area legally. They were told that they were scum of the earth and they could go fuck themselves.4 -
Boss needs certain stats pulled from database once a year for board meeting. This time I delegate it to a junior dba/sysadmin. He looks at my 3-year-old docs that I hastily jotted down and pasted and included my rambling notes with results from way back then. Mostly they were just to jog my own memory, not to be a really neat, clean instruction guide. He does the queries correctly, but in ticket for boss he pastes also all my notes from the docs. boss gets confused, "what is this other number, I don't get it?!" We have to have a meeting of the 3 of us and waste an hour or so just to figure out what went wrong, finally I realize what junior guy accidentally did. Moral of story: to avoid baffling the nontechs, always simplify, simplify, simplify. Alternate moral of story: before delegating a task that seems old hat to you, always review your notes/docs and make sure they're ready for someone else to use them.1
-
Optimization issue pops out with one of our queries.
> Team leader: You need to do this and that, it's a thing you know NOTHING about but don't worry, the DBA already performed all the preliminary analysis, it's tested and it should work. Just change these 2 lines of code and we're good to go
> ffwd 2 days, ticket gets sent back, it's not working
> Team leader: YOU WERE SUPPOSED TO TEST IT YOUR CHANGE IS NOT WORKING
> IHateForALiving: try it on our production machine and you'll see the exact same error, it's been there for years
> Team leader: BUT YOU WERE SUPPOSED TO TEST IT
Just so we're clear, when I perform a change in the code, I test the changes I made. I don't know in which universe I should be held accountable for tards breaking features 10 years ago, but you can't seriously expect me to test the whole fucking software from scratch every time I add an index to the db.1 -
Completely fucked up replication of MySQL servers.
Remote: 2 different database Servers
--> made sense.
Except the misconfiguration. Or better: No configuration at all.
So how to solve the massiv delays and make everything even more crazy?
2 remote servers - 2 readonly slaves for reading data remote (master - slave)
2 local (internal) servers.
Remote - Local Master Master.
Unfucking this cluster fuck was a real nightmare.
It had to be done at night, cause everything needed to be ripped apart.
And the servers were the backend of a warehouse with supply chain and multiple selling channels (Amazon, eBay etcetera).
So. It had to run the next day at 05.00 clock so the incoming orders could be packaged / prepared for shipping.
That was fun. Not.
And the clusterfuck died spectaculously on my first work day - the old DBA was gone (fired....)
:) -
There are a couple:
A system that updates user accounts to connect them into our wifi system by parsing thousands of processing files written in Clojure. The project was short lived and mainly experimental, It has complete test cases and the jar generated from it is still purring silently on the main application. It was used to replace an $85k vendor application that made no fucking sense. The code has not been touched in 2 years and the jar is still there. The dba mentioned the solution to the vendor, the vendor tried buying it from me, but being that it belongs to the institution nothing was touched, still, it got the VP's attention that I can make programs that would be bought for that level, it caught his attention even more when I showed him the codebase and he recognized a Lisp variant (he is old, and was back in the day a Fortran and Cobol developer)
A small Python categorical ML program that determines certain attributes of user generated data and effectively places them on the proper categories on the main DB. The program generates estimates of the users and the predictions have a 95% correctness rate. The DBA still needs to double check the generated results before doing the db updates. I don't remember how I coded it because I was mostly drunk when I experiment on the scenario. It also got the attention of the VP and director since the web tech manager was apparently doing crazy ML shit that they were not expecting me to do, it made them paranoid that I would eventually leave for a ML role somewhere, still here, but I want more moneys!!
A program that generates PDF documentation from user data, written in Go, Python and Perl (yes Perl) I even got shit from the lead developer since I used languages outside of their current scope of work. Dude had no option but to follow along with it :P since I am his boss
Many more. I am normally proud of my work code. But my biggest moment is my current ntural language processing unit that I am trying to code for my home, but I don't have enough power to build it with my computers, currently, my AI is too stupid, but sometimes it does reply back to my commands and does the things I ask it to do (simple things, opening a browser, search for a song etc) but 7 times out of ten it wont work :P -
I'm now not only a full stack developer in the charge of my own Linux servers, Devops work, programming,and the MySQL DBA, but have been asked to take on the "small" responsibilities of our only Linux administrator (retiring). No mention of title change (which is lesser than all my work), nor salary increase. A person can only do so much. Don't think I'm accepting this lightly or quietly, but to be assumed to take on more responsibility without benefit is beyond me. Mind you, this came down from my director; my boss made me privy.5
-
Before my vacation I’d been chatting with one of our dbas about an etl tool we needed for a customer we’d already signed all the contracts with saying we would provide one for a historical database of old data. They had been looking at one from SAP but in typical fashion a license was worth more than the actual contract.
Anyway long story short on the weekend before I went back to work I rattled together a little python proof of concept using a couple of azure databases and when I went back demo’d it to the pm and dba they loved it and we built on the poc to have a working loader which saved us about £30k by not buying the SAP product and just wrote our own. -
I wanted to show our DBA an example of a web api using .net core 3 in regards of how easy it is to create such things. The reason? he has been wanting to get back into programming after many years of just sticking to dba related stuff. The dude has talent and brains, he had worked years ago as a delphi dev and a vb6 dev and we had the same employer at one point, none of this man's apps have been faced out on account of how complete they are and easy to maintain for other devs was after he left. Regardless of the ancient tech stacl, the man shows ample promise and well.
Thing is, the apps I make on the Microsoft stack usually tend to C#, and my frontends are using TS, so I am more on the curlt bracket side of things and he said he was to convert my app(very basic crud example, but with auth, authorization and everything in between to plug into the frontend) to VB.NET. I thought it wouldn't be that much of a problem but apparently microsoft does not hold templates for webapi for vb.net
I thought it was shitty. VB gave Microsoft a lot of developer market back in the VB6 days, and even though I really love c# I see no reason why they would just say fuck you like that to vb.net. Shit still polls pretty high in terms of dev popularity and you can apply the same design ideas to VB without much effort.
I just think this is very shitty from Microsoft's part. Much like how Apple is forcing people to adapt to Swift when there is a huge amount of obj c out there.
I dislike when companies shift focus on tech stacks like that.2 -
Idea for future week question, maybe not a rant 'per se', but interesting endeed: 'Describe the moment when you knew you wanted to be a programmer/developer/DBA..."
-
So at the end of our sprint retrospective we (me and one other dev) apologize to the project owner for the fact that QA had to happen on dev for a few days because the database changes were not properly applied by the DBA. It was a one time thing, a little weird but not a big deal.
A few hours later we get a call from a new DBA saying that the project owner said that we wanted the old DBA off our project. So now we sound like assholes ruining a guy's self confidence and made working with him awkward as hell.
Thanks, project owner. -
!rant
Just made the leap from a company with and I.T. Department of 8 people to a company with an I.T. Department of 5000 people. At the old company, I was a SysAdmin, DBA, Helpdesk, Programmer, etc. 7/8 were programmers. We all had our hands in pretty much every aspect of the companies technology. At the new company I am just a programmer. I only write code (well, I will soon. There is a LOT I have to learn about how this place operates). I worked at the old companies for almost 3 years (2 as an Intern, 10 months as Salaried Full-Time) so it's weird having most of those aspects of my old job stripped away, yet at the same time it's somewhat freeing in the sense that I am now only responsible for 1 slice of the pie, not the whole damn thing. Anyway, hope everyone is having a great monday!1 -
Work email conversation with a COTS app support tech:
Tech: Could you provide me the following from Sql Server? (instructions followed)
Me: I am not the DBA, but I've sent them the request.
Tech: Could you send me the following from Sql Server while we wait? (Instructions followed)
Me: As I said, I don't have permissions to access what you want.
Tech: Oh, I see now, you most likely don't have proper access to be able to retrieve the information.
Me: ..yeah. Thanks. -
Having a senior DBA can save hours if not days of struggle and save your back, if you do not know well enough how to do a more complicated query yet, without fucking up something.
Good guidance and experience is worth so much.
... and no I do not have the rights to drop databases.1 -
Today, my boss comes up to me to tell me that the CR he had me set up was not created by the release manager because there wasn't a QA test ticket or DBA review ticket. Look, if you're gonna shoot shit down without any bullets loaded, at least have the safety off because you had 3 hours on Friday to tell me that before leaving it to Monday morning when my head is hotter than the roast I'm sippin' on.
-
Anyone one here played around with CouchDB before or use it for personal projects or work?
At face value, it seems like a pretty good DB, Just wanted to get some idea from people that have used it before if it is actually pretty good.
I'm not a dba and don't know the caveats of DB tuning or management. So I'm in need of a DB with simplicity and easy management in mind Hahaha.
I'm mostly working with data that is either in JSON or hashtable/dictionary format so it felt like a NoSQL DB would be easy-ish to save my data into, plus I don't think (I hope my guess is right here) that I need regular SQL type relations with the data I'm working with.
Please help me with my noob-iness!Thank you! 😄