Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "production database fail"
-
I recently joined the dark side - an agile consulting company (why and how is a long story). The first client I was assigned to was an international bank. The client wanted a web portal, that was at its core, just a massive web form for their users to perform data entry.
My company pitched and won the project even though they didn't have a single developer on their bench. The entire project team (including myself) was fast tracked through interviews and hired very rapidly so that they could staff the project (a fact I found out months later).
Although I had ~8 years of systems programming experience, my entire web development experience amounted to 12 weeks (a part time web dev course) just before I got hired.
I introduce to you, my team ...
Scrum Master. 12 years experience on paper.
Rote memorised the agile manifesto and scrum textbooks. He constantly went “We should do X instead of (practical thing) Y, because X is the agile way.” Easily pressured by the client to include ridiculous (real time chat in a form filling webpage), and sometimes near impossible features (undo at the keystroke level). He would just nag at the devs until someone mumbled ‘yes' just so that he would stfu and go away.
UX Designer. 3 years experience on paper ... as business analyst.
Zero professional experience in UX. Can’t use design tools like AI / photoshop. All he has is 10 weeks of UX bootcamp and a massive chip on his shoulder. The client wanted a web form, he designed a monstrosity that included several custom components that just HAD to be put in, because UX. When we asked for clarification the reply was a usually condescending “you guys don’t understand UX, just do <insert unhandled edge case>, this is intended."
Developer - PHD in his first job.
Invents programming puzzles to solve where there are none. The user story asked for a upload file button. He implemented a queue system that made use of custom metadata to detect file extensions, file size, and other attributes, so that he could determine which file to synchronously upload first.
Developer - Bootlicker. 5 years experience on paper.
He tried to ingratiate himself with the management from day 1. He also writes code I would fire interns and fail students for. His very first PR corrupted the database. The most recent one didn’t even compile.
Developer - Millennial fratboy with a business degree. 8 years experience on paper.
His entire knowledge of programming amounted to a single data structures class he took on Coursera. Claims that’s all he needs. His PRs was a single 4000+ line files, of which 3500+ failed the linter, had numerous bugs / console warnings / compile warnings, and implemented 60% of functionality requested in the user story. Also forget about getting his attention whenever one of the pretty secretaries walked by. He would leap out of his seat and waltz off to flirt.
Developer - Brooding loner. 6 years experience on paper.
His code works. It runs, in exponential time. Simply ignores you when you attempt to ask.
Developer - Agile fullstack developer extraordinaire. 8 years experience on paper.
Insists on doing the absolute minimum required in the user story, because more would be a waste. Does not believe in thinking ahead for edge conditions because it isn’t in the story. Every single PR is a hack around existing code. Sometimes he hacks a hack that was initially hacked by him. No one understands the components he maintains.
Developer - Team lead. 10 years of programming experience on paper.
Writes spaghetti code with if/else blocks nested 6 levels deep. When asked "how does this work ?”, the answer “I don’t know the details, but hey it works!”. Assigned as the team lead as he had the most experience on paper. Tries organise technical discussions during which he speaks absolute gibberish that either make no sense, or are complete misunderstandings of how our system actually works.
The last 2 guys are actually highly regarded by my company and are several pay grades above me. The rest were hired because my company was desperate to staff the project.
There are a 3 more guys I didn’t mention. The 4 of us literally carried the project. The codebase is ugly as hell because the others merge in each others crap. We have no unit tests, and It’s near impossible to start because of the quality of the code. But this junk works, and was deployed to production. Today is it actually hailed as a success story.
All these 3 guys have quit. 2 of them quit without a job. 1 found a new and better gig.
I’m still here because I need the money. There’s a tsunami of trash code waiting to fail in production, and I’m the only one left holding the fort.
Why am I surrounded by morons?
Why are these retards paid more than me?
Why are they so proud when all they produce is trash?
How on earth are they still hired?
And yeah, FML.8 -
Okay. So my dumbass boss took this project that had a steep timeline. I told him straight up, it won't work because we won't make the timeline. If we do this, I will be the one bending over backwards to deliver. I don't like to promise and fail. I got the oh don't worry let's just try. If we don't make it that's fine. Unfortunately that's not how I work. I refuse to deliberately fail. So I say okay and we begin. I suggested open source is the fastest way to deliver bit the fucked up part is, I am the only senior dev in the team. I will be expected to reverse engineer the open source app to connect our own deployment parameters. Use tech I have never used before. Connect frontend and backend. Handle dns bullshit. I have literally been working on Vibes and coffee for the past two weeks because ofcourse I ran into so many issues. Now I have an extension for Monday and I hate to fail. So I am not sleeping or resting just working on a fucking java app I didnt build and I am expected to make it work seemlessly on our production environment. I made some progress. Deployed frontend, deployed backend. Forgot to connect production dB so I decided to go with azure database for mysql driver since we have credits on azure. Now my java app is pissing itself over ssl handshake. I generate my keystore and add it and now java socket just times out. I want to pummel somebody or a punching bag that looks like my boss.15
-
Other team lead: Hi DevOps Team, We need you to deploy this app to production. It's maintainers gave up on it in 2019, but we looked at it and it feels right.
Me: Uhm. That's not going to work. It'll fail the security scan before you can even finish the build in CI.
Other team lead: Yeah, this app is the right thing to do, and we needed it last week, but since that won't work, we'll just use this other very very infant technology that was just born yesterday. It's not stable in production, or on MySQL, or in AWS at all, but it's the other direction we can to go.
Me: What problem are you trying to solve in the first place?
Other team lead: Oh, we need access to the read from the production database.2 -
# Retrospective as Backend engineer
Once upon a time, I was rejected by a startup who tries to snag me from another company that I was working with.
They are looking for Senior / Supervisor level backend engineer and my profile looks like a fit for them.
So they contacted me, arranged a technical test, system design test, and interview with their lead backend engineer who also happens to be co-founder of the startup.
## The Interview
As usual, they asked me what are my contribution to previous workplace.
I answered them with achievements that I think are the best for each company that I worked with, and how to technologically achieve them.
One of it includes designing and implementing a `CQRS+ES` system in the backend.
With complete capability of what I `brag` as `Time Machine` through replaying event.
## The Rejection
And of course I was rejected by the startup, maybe specifically by the co-founder. As I asked around on the reason of rejection from an insider.
They insisted I am a guy who overengineer thing that are not needed, by doing `CQRS+ES`, and only suitable for RND, non-production stuffs.
Nobody needs that kind of `Time Machine`.
## Ironically
After switching jobs (to another company), becoming fullstack developer, learning about react and redux.
I can reflect back on this past experience and say this:
The same company that says `CQRS+ES` is an over engineering, also uses `React+Redux`.
Never did they realize the concept behind `React+Redux` is very similar to `CQRS+ES`.
- Separation of concern
- CQRS: `Command` is separated from `Query`
- Redux: Side effect / `Action` in `Thunk` separated from the presentation
- Managing State of Application
- ES: Through sequence of `Event` produced by `Command`
- Redux: Through action data produced / dispatched by `Action`
- Replayability
- ES: Through replaying `Event` into the `Applier`
- Redux: Through replay `Action` which trigger dispatch to `Reducer`
---
The same company that says `CQRS` is an over engineering also uses `ElasticSearch+MySQL`.
Never did they realize they are separating `WRITE` database into `MySQL` as their `Single Source Of Truth`, and `READ` database into `ElasticSearch` is also inline with `CQRS` principle.
## Value as Backend Engineer
It's a sad days as Backend Engineer these days. At least in the country I live in.
Seems like being a backend engineer is often under-appreciated.
Company (or people) seems to think of backend engineer is the guy who ONLY makes `CRUD` API endpoint to database.
- I've heard from Fullstack engineer who comes from React background complains about Backend engineers have it easy by only doing CRUD without having to worry about application.
- The same guy fails when given task in Backend to make a simple round-robin ticketing system.
- I've seen company who only hires Fullstack engineer with strong Frontend experience, fails to have basic understanding of how SQL Transaction and Connection Pool works.
- I've seen company Fullstack engineer relies on ORM to do super complex query instead of writing proper SQL, and prefer to translate SQL into ORM query language.
- I've seen company Fullstack engineer with strong React background brags about Uncle Bob clean code but fail to know on how to do basic dependency injection.
- I've heard company who made webapp criticize my way of handling `session` through http secure cookie. Saying it's a bad practice and better to use local storage. Despite my argument of `secure` in the cookie and ability to control cookie via backend.18 -
When your company expects you to manually change information in the production db by saying "hey, client Billy wants his stuff moved back to where it was"3
-
Finally made my node production server stable enough that I could focus on writing tests*. I start by setting up docker, mocking cognito, preparing the database and everything. Reading up on Node test suites and following a short tut to set up my first unit test. Didn't go smoothly, but it's local and there are no deadlines so who cares. 4 days later, first assert.equal(1+1, 2) passes and I'm happy.
I start writing all sorts of tests, installing everything required into "devDependancies," and getting the joy of having some tests pass on first try with all asserts set up, feels good!
I decide to make a small update to production, so I add a test, run and see it fail, implement the feature, re-run and, it passes!
I push the feature to develop, test it, and it works as intended. Merge that to master and subsequently to one of my ec2 production servers**, and lo and behold, production server is on a bootloop claiming it "Cannot find module `graphql`". But how? I didn't change any production dependencies, and my package lock json is committed so wth?
I google the issue, but can't find anything relevant. The only thing that I could guess was that some dependencies (including graphql) were referenced*** in both, prod and dev, and were omitted when installed on a prod NODE_ENV, but googling that specific issue yielded no results, and I would have thought npm would be clever enough to see that and would always install those dependencies (spoiler: it didn't for me).
With reduced production capacity (having one server down) I decided to npm uninstall all dev dependencies anyway and see what happens. Aaaaand it works.....
So now I have a working production server, but broken local tests, and I'm not sure why npm is behaving like this...
* Yes I see the irony.
** No staging because $$$, also this is a personal project.
*** I am not directly referencing the same thing twice, it's probably a subdependency somewhere.2