Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "broken dev environment"
-
The riskiest dev choice...
How about "The riskiest thing you've done as a dev"? I have a great entry for that. and I suppose it was my choice to build the feature afterall.
I was working on an instance of a small MMO at a game company I worked for. The MMO boasted multiple servers, each of them a vastly different take on the base game. We could use, extend, or outright replace anything we wanted to, leading to everything from Zelda to pokemon to an RP haven to a top-down futuristic counterstrike. The server in this particular instance was a fantasy RPG, and I was building it a new leveling and experience system with most of the trimmings. (Talents, feats/perks, etc. were in a future update.)
A bit of background, first: the game's dev setup did not have the now-standard dev/staging/prod servers; everything ran on prod, devs worked on prod, players connected and played on prod, etc. Worse yet, there was no backup system implemented -- or not really. The CTO was really the only person with sufficient access. The techy CEO did as well, but he rarely dealt with anything technical except server hardware, occasionally. And usually just to troll/punish us devs (as in "Oops ! I pulled the cat5 ! ;)"). Neither of them were the most reliable of people, either. The CTO would occasionally remote in and make backups of each server -- we assumed whenever he happened to think of it -- and would also occasionally do it when asked, but it could take him a week, sometimes even up to a month to get around to it. So the backups were only really useful for retreiving lost code and assets, not so much for player data.
The lack of reliable backups and the lack of proper testing grounds (among the plethora of other issues at the company) made for an absolutely terrible dev setup, but that's just how it was, and that's what we dealt with. We were game devs, afterall. Terrible or not, we got to make games! What more could you ask for!? It was amazing and terrible and wonderful and the worst thing ever, all at the same time. (and no, I'm not sharing the company name, but it isn't EA or Nexon, surprisingly 😅)
Anyway, back to the story! My new leveling system also needed to migrate players' existing data, so... you can see where this is going.
I did as much testing and inspection of my code as I could, copied it from a personal dev script to the server's xp system, ... and debated if I really wanted to click [Apply]. Every time I considered it, I went back to check another part or do yet more testing. I ended up taking like 40 minutes to finally click it.
And when I did... that was the scariest button press of my life. And the scariest three seconds' wait afterwards. That one click could have ruined every single player's account, permanently lost us players ...
After applying it, I immediately checked my character to see if she was broken, checked the account data for corruption or botched flags, checked for broken interactions with the other systems....
Everything ended up working out perfectly, and the players loved all of the new features. They had no idea what went into building them, and certainly had no idea of what went into applying them, or what could have gone wrong -- which is probably a good thing.
Looking back, that entire environment was so fragile, it's a wonder things didn't go horribly wrong all the time. Really, they almost never did. Apocalypses did happen, but were exceedingly rare, and were ususally fixed quickly. I guess we were all super careful simply because everything was so fragile? or the decent devs were, at least. We never trusted the lessers with access 😅 at least on the main servers where it mattered. Some of the smaller servers... well, we never really cared about those.
But I'm honestly more surprised to realize I've never had nightmares of that button click. It was certainly terrifying enough.
But yay! Complete system overhaul and migration of stored and realtime player data! on prod! With no issues! And lots of happy players! Woooooo!
Thinking back on it makes me happy 😊rant deploying straight to prod prod prod prod dev server? dev on prod you chicken migration on prod wk149 git? who's a git? you're a git! scariest deploy ever game development1 -
I spent four hours just getting my dev environment working again today.
Whenever I switch branches on this project, I keep to run a script that does migrations, seed data, test db setup, static test info, etc. etc. etc. It takes 12-15 minutes to run.
Today, that script failed.
Apparently one of its steps requires running some of the project's code to produce valid objects. Makes sense. However, my ticket involves breaking a crapton of models (removing accessors) which I've already done, and then patching the behavior, which I haven't. Which means a lot of things are currently broken. Makes sense why the script fails.
However, I can't run the script on a different branch and then switch back because that simply doesn't work (for reasons), so I needed to find some workaround. I eventually did, but every attempt cost me 12 minutes.
Today was not fun, and certainly not productive.
I wonder when they're going to fire me 😅7 -
So last week I really fucked up
I had this new implementation that was supposedly to be integrating smoothly into the rest of the service. It depended on a serialized model made by a data scientist. I test it in local, in QA environment: no problem.
So, Friday, 4pm, I decide to deploy to production. I check once from the app: the service throw an error. Panic attack, my chief is at my desk, we triy to understand what went wrong. I make calls with cUrls: no problem. Everything seems fine. I recheck from the app again: no problem.
We dedice to let it in prod, as the feature work. I go get some beers with the guys, to celebrate the deploy.
Fast-forward the next morning, 11am, my phone ring: it's a colleague of my chief. "Please check Slack, a client is trying to use the feature, it's broken"
FUUUUUUUUUUUUCK!!!
Panic attack again. I go to the computer, check the errors: two types of errors. One I can fix, the other from a missing package on the machine that the data guy used.
Needless to say, I had a fairly good weekend.
Lessons learned:
- make sure Dev, QA and Prod are exactly the same (use Ansible or Container)
- never deploy on a Friday afternoon if you don't have a quick way to revert1 -
Working with a new team and I don't understand how this is normal or ok.
Me: Does anyone need help troubleshooting the broken build or can I revert this change that broke it so I can push my change out?
Dev: Stop build shaming me, I wouldn't leave the build in a failing state.
Me: Well, I wasn't sure how long to wait, before asking.... it's been broken for 4 hours.
Dev: It's the development environment, you should expect development to be going on.
Me: Yes and appears that this project architecture doesn't support any sort of isolation for development. So nobody can deploy anything except through the development branch.
Dev: That's what development is for IMO, so it doesn't bother me.11 -
Today after longer vacation I came back to work.
Edit: wrote this rant long time ago, but never finished. Was too pissed.
Some easy meetings, then wanted to start on an easy job.
Just migrating some things from bash regex voodoo to proper tools like JQ.
Finished in roughly 1 h. Lovely.
Made some tea, ate some cookies.
Set up dev environment, found no documentation what so ever, got it running after half an hour.
Annoying, but ok.
Then I tried my scripts...
They worked... Except they didn't.
Console log empty, response code 200 with state: GENERATE_NO_FILES.
Eh. Fuck you. Just fuck you.
Fixed the logging configuration, which was broken since uhm... 2 years plus?
Well... Another half another hour gone...
Kinda pissed now.
Still script return failed...
Poking and trying to sprinkle debug all over that shit cause everything seems ... An incohesive, inconsistent diarrhea.
3 hours later...
Made the ticket to rewrite it.
I did nothing wrong at all.
The API just has no workflow at all. The
*seperate* API calls have to be in an **specific** order - as otherwise the generation will fail, as the prerequisites for the generation are not fulfilled.
Yeah. Completely logical. Especially not to give out any kind of warning or an error message like requirements not met, blablabla.
I drank that evening 2 six packs of beer. I was raging mad....
Then gave that shit to another manager, as I never want to touch that nuclear waste again....
How can someone be so brain damaged -.-1 -
FML having to take on and support a python test framework that looks like it was written by a junior C embedded dev without a mentor.
- Imports everywhere in the code
- No abstraction or OOP
- sys.path.append fest (broken imports of course)
- Global variables fest
- No docstrings
- No readme
- Somehow mixed with a jUnit test framework as well
- Uses Windows environment variables profusely
- Pycharm has a stroke when I open files from this project5 -
When the CTO/CEO of your "startup" is always AFK and it takes weeks to get anything approved by them (or even secure a meeting with them) and they have almost-exclusive access to production and the admin account for all third party services.
Want to create a new messaging channel? Too bad! What about a new repository for that cool idea you had, or that new microservice you're expected to build. Expect to be blocked for at least a week.
When they also hold themselves solely responsible for security and operations, they've built their own proprietary framework that handles all the authentication, database models and microservice communications.
Speaking of which, there's more than six microservices per developer!
Oh there's a bug or limitation in the framework? Too bad. It's a black box that nobody else in the company can touch. Good luck with the two week lead time on getting anything changed there. Oh and there's no dedicated issue tracker. Have you heard of email?
When the systems and processes in place were designed for "consistency" and "scalability" in mind you can be certain that everything is consistently broken at scale. Each microservice offers:
1. Anemic & non-idempotent CRUD APIs (Can't believe it's not a Database Table™) because the consumer should do all the work.
2. Race Conditions, because transactions are "not portable" (but not to worry, all the code is written as if it were running single threaded on a single machine).
3. Fault Intolerance, just a single failure in a chain of layered microservice calls will leave the requested operation in a partially applied and corrupted state. Ger ready for manual intervention.
4. Completely Redundant Documentation, our web documentation is automatically generated and is always of the form //[FieldName] of the [ObjectName].
5. Happy Path Support, only the intended use cases and fields work, we added a bunch of others because YouAreGoingToNeedIt™ but it won't work when you do need it. The only record of this happy path is the code itself.
Consider this, you're been building a new microservice, you've carefully followed all the unwritten highly specific technical implementation standards enforced by the CTO/CEO (that your aware of). You've decided to write some unit tests, well um.. didn't you know? There's nothing scalable and consistent about running the system locally! That's not built-in to the framework. So just use curl to test your service whilst it is deployed or connected to the development environment. Then you can open a PR and once it has been approved it will be included in the next full deployment (at least a week later).
Most new 'services' feel like the are about one to five days of writing straightforward code followed by weeks to months of integration hell, testing and blocked dependencies.
When confronted/advised about these issues the response from the CTO/CEO
varies:
(A) "yes but it's an edge case, the cloud is highly available and reliable, our software doesn't crash frequently".
(B) "yes, that's why I'm thinking about adding [idempotency] to the framework to address that when I'm not so busy" two weeks go by...
(C) "yes, but we are still doing better than all of our competitors".
(D) "oh, but you can just [highly specific sequence of undocumented steps, that probably won't work when you try it].
(E) "yes, let's setup a meeting to go through this in more detail" *doesn't show up to the meeting*.
(F) "oh, but our customers are really happy with our level of [Documentation]".
Sometimes it can feel like a bit of a cult, as all of the project managers (and some of the developers) see the CTO/CEO as a sort of 'programming god' because they are never blocked on anything they work on, they're able to bypass all the limitations and obstacles they've placed in front of the 'ordinary' developers.
There's been several instances where the CTO/CEO will suddenly make widespread changes to the codebase (to enforce some 'standard') without having to go through the same review process as everybody else, these changes will usually break something like the automatic build process or something in the dev environment and its up to the developers to pick up the pieces. I think developers find it intimidating to identify issues in the CTO/CEO's code because it's implicitly defined due to their status as the "gold standard".
It's certainly frustrating but I hope this story serves as a bit of a foil to those who wish they had a more technical CTO/CEO in their organisation. Does anybody else have a similar experience or is this situation an absolute one of a kind?2 -
Weblogic Server won't start, broken build was auto-deployed. How does a broken build even get deployed? 🙄
Told "till the server is fixed work on your next-release bugs". Sure, can investigate, maybe even code fixes, but with a broken dev environment how do we test anything?
" Hmm..Oh yeah, right" 🤔 -
“OhhHhh please fill out an entire fucking excel sheet for our test environment deployment. It helps us manage everything better and gives us a reason to fucking thumb around in our holes all day and pretend like we really mean something as managers.” Like absolutely no, you can go fuck yourself with a condom filled with broken glass shards and diseases is what YOU can do. You are a parasite.
“Senior lead developer” - but they don’t have a title - says: “please just give me the list of files I need to manually change on the env in real time”. Bitch, do you even know what CI/CD is?!?!? The fact that you have been doing this for a year straight makes me pity how much of a fucking dumbass you really are. Even if u don’t use a pipeline, just look at my fucking git changes. That’s literally why we have it. You are a fucking disgrace of a developer and I hope you know that everyone who is a competent dev would rather bathe in a bath filled with lemon juice and cactus spikes, before EVER working with YOU EVER AGAIN!!!1 -
First time linux user feedback
Linux lovers are probably gonna eat me alive but I don't give a flying fuck
Maybe its a little lenghty or boring, tell me what you think
Backstory:
I work for game extension company. We work with WinAPI and such. I've been using Windows since forever and I'm happy with it. But I thought to myself "hey, if I wanna be a good dev, I should give Linux and OS X a try, too"
I downloaded Linux Mint couple of months ago to start with. I was unable to boot it from live CD no matter what I tried, even in recovery mode. Apparently, Mint 18.3 was based on Ubuntu 16.04 which doesnt support UEFI
Wait, what the fuck, all modern PCs have UEFI so what, do all Mint users have 10 y/o laptops and PCs???
Anyway, when I heard about Mint 19 being released I thought to give it another try and I did. What a surprise, it booted successfully from Live CD. I saw the Linux desktop for the first time in my life, yay! I then installed it, GRUB appeared, my Windows was still there and wasn't broken so I was happy SOMETHING was working. I configured timeshift and applied dvorak layout system-wide. Realised dvorak layout is fucked up big time and applied normal layout for just desktop environment. Everything was really nice until couple reboots later Cinnamon stopped launching (kept returning to login screen). Okay, lets use timeshift
First big what-the-fuck was when I found out system restore can only be done using GUI??? This is absolutely retarded and I couldn't believe it is true. Login screen has a reachable console but I can't login there since I can't type the password. Fuck, fuck, fucking drovak layout was there.
Recovery mode - I've spent 20 minutes trying to type "timeshift --restore" having to press all keyboard buttons just to progress with one button. I've had another what-the-fuck when I saw "error: can't restore timeshift - partition already mounted"
Okay, this is too much. Why the fuck would you bundle a recovery mode if you can't restore a snapshot from there.
I have spent 3 hours now googling and trying to remove this fucking keyboard layout. No dice. I am making another copy of the live CD now. I'm gonna reinstall the whole shit now. I have the desire to create a custom Mint version without this abomination of a keyboard layout.
It's okay. Windows has taught me to be patient.
Fuck Dvorak, I dont know who the guy is but his keyboard layout can eat my dick7 -
ops: your dev environment shuts itself down on boot up, something is broken in the boot process and it'd be more trouble to figure out why so you're better off making a new one again
fuck me
and fuck you too AWS EC2 whatever -
- Teammate discovers a standard PaaS feature isn’t working and breaks core functionality in dev environment
- Teammate creates a support ticket to the PaaS company
- PaaS company says that they’re aware of the issue but don’t have a solution yet and advises to disable the feature for now
- Teammate ships the feature and leaves it enabled on production.
- Teammate thinks that “oh we know it’s broken, nobody is going to use it anyway”
- Customer uses the feature
- Shit hits the fan
- Teammate: *shocked pikachu face* -
Upgrading my tech skills.. Once again I feel my personal my personal dev environment and told are much more up-to-date than what I use at work.... Though the book Kim reading is on TDD and was written 3 years ago.
Maybe I should read another on in cloud services and ML... but don't have any motivation for these topics.
I need TDD for work because now we're emphasizing unit test coverage...
I usually only use manual functional tests to verify the final outputs as either the testing framework is broken (JS) or I don't have time to relearn the frameworks for the particular language...
Anyway got off topic... So questions after:
1. Do you ever feel your technologically always more ahead than what you do at work and essentially you bring skills to the job but you don't learn much out of it?
2. How do you test? I actually got into a bit of a argument/discussion with my colleagues about how to implement unit tests. Apparently there are 2 ways to test? Black box vs WhiteBox. She said she tests only Public methods using mock inputs, dependencies. She read online and seems there is an opinion that should only test public functions and if you can't then your app is designed incorrectly, not separated enough.
For me I test the private functions individually (WhiteBox/Java reflection) because the public one is like generateReport and as a whole is like a Pachinko machine, too many unique paths that would need a test case for.
So thoughts? Yes sorry for turning it into a remake I guess...24