Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "db master"
-
--- GitHub 24-hour outage post mortem ---
As many of you will remember; Github fell over earlier this month and cracked its head on the counter top on the way down. For more or less a full 24 hours the repo-wrangling behemoth had inconsistent data being presented to users, slow response times and failing requests during common user actions such as reporting issues and questioning your career choice in code reviews.
It's been revealed in a post-mortem of the incident (link at the end of the article) that DB replication was the root cause of the chaos after a failing 100G network link was being replaced during routine maintenance. I don't pretend to be a rockstar-ninja-wizard DBA but after speaking with colleagues who went a shade whiter when the term "replication" was used - It's hard to predict where a design decision will bite back and leave you untanging the web of lies and misinformation reported by the databases for weeks if not months after everything's gone a tad sideways.
When the link was yanked out of the east coast DC undergoing maintenance - Github's "Orchestrator" software did exactly what it was meant to do; It hit the "ohshi" button and failed over to another DC that wasn't reporting any issues. The hitch in the master plan was that when connectivity came back up at the east coast DC, Orchestrator was unable to (un)fail-over back to the east coast DC due to each cluster containing data the other didn't have.
At this point it's reasonable to assume that pants were turning funny colours - Monitoring systems across the board started squealing, firing off messages to engineers demanding they rouse from the land of nod and snap back to reality, that was a bit more "on-fire" than usual. A quick call to Orchestrator's API returned a result set that only contained database servers from the west coast - none of the east coast servers had responded.
Come 11pm UTC (about 10 minutes after the initial pant re-colouring) engineers realised they were well and truly backed into a corner, the site was flipped into "Yellow" status and internal mechanisms for deployments were locked out. 5 minutes later an Incident Co-ordinator was dragged from their lair by the status change and almost immediately flipped the site into "Red" status, a move i can only hope was accompanied by all the lights going red and klaxons sounding.
Even more engineers were roused from their slumber to help with the recovery effort, By this point hair was turning grey in real time - The fail-over DB cluster had been processing user data for nearly 40 minutes, every second that passed made the inevitable untangling process exponentially more difficult. Not long after this Github made the call to pause webhooks and Github Pages builds in an attempt to prevent further data loss, causing disruption to those of us using Github as a way of kicking off our deployment processes (myself included, I had to SSH in and run a git pull myself like some kind of savage).
Glossing over several more "And then things were still broken" sections of the post mortem; Clever engineers with their heads screwed on the right way successfully executed what i can only imagine was a large, complex and risky plan to untangle the mess and restore functionality. Github was picked up off the kitchen floor and promptly placed in a comfy chair with a sweet tea to recover. The enormous backlog of webhooks and Pages builds was caught up with and everything was more or less back to normal.
It goes to show that even the best laid plan rarely survives first contact with the enemy, In this case a failing 100G network link somewhere inside an east coast data center.
Link to the post mortem: https://blog.github.com/2018-10-30-...6 -
Assigned to a new project team..
Using git, in a creative way. So.. "master" is "dev" branch, usually. Everyone can push their branch to dev server .. so it's "dynamic for us". Production branch is whatever, as long as the branch has the release version. Sometimes, the release comes from "master".. that mean "dev" in normal geek..
That's just Git. The source code is a saturated spagetti of Entity framework and Caliburn. It is littered with antipatterns, especially basebean. Holy Christmas and Easter that baseclass do a lot of stuff that has no place as a base class ..
Fucking frameworks, I'm gonna start to evangelize frameworks as the no1 antipattern.
MS SQL as the main DB, but is dumped to json FILES through a scheduled task to increase read performance on web.
There is a soap endpoint to expose the json files, fml..
I am assuming I was placed here to improve stuff, I have never in my life seen anything like this before.
There is a special place in hell for this repository7 -
Found this gem of a comment in a code base written 4 years back.
/*
Invoke <Service Base URL>/asset/v2/details/<SN> to get asset details
Feeling very bad to include this call, but we really need to use this !!!
This call is gonna take ~20s to respond. I've even increased the overall timeout of this module, just for this call !!!
So, if you are looking to debug any performance issue, I wish you jump directly here,
remove this call and just use master data management (MDM)
P.S: It is not that simple, as MDM and this asset DB (both asset masters) has differences in how the asset is defined :(
*/
Still trying to understand how to remove this costly time-consuming call and replace with an efficient one !!
And, of-course, the original author left 2 years back :(3 -
Full stack developer.
I know what it's supposed to mean, but I feel like it gives discredit to the devs who perfect their area (frontend, backend, db, infrastructure). It's, to me, like calling myself a chef because I can cook dinner..
The depth, analysis and customization of the domain to shape an api to a website is never appreciated. The finicle tweaks on the frontend to make those final touches. Then comes a brat who say they are full stack, and can do all those things. Bullshit. 99.9% of them have never done anything but move data through layers and present it.
Throw these wannabes an enterprise system with monoliths and microservices willy nelly, orchestrate that shit with a vertical slice nginx ssi with disaster recovery, horizontal scaling, domain modeling, version management, a busy little bus and events flowing all decimal points of 2pi. Then, if you fully master everything going on there, I believe you are full stack.
Otherwise you just scraped the surface of what complexities software development is about. Everyone who can read a tutorial can scrape together an "in-out" website. But if your db is looking the same as your api, your highest complexity is the alignment of an infobox, I will laugh loud at your full stack.
And if you told me in an interview that you are full stack, you'd better have 10+ years experience and a good list of failed and successful projects before I'd let you stay the next two minutes..1 -
No idea how I ended up here. So basically started with simple wordpress websites, transitioned to react. This move was the hardest.
Then caught up on node and mongodb. And before i knew i was doing db, backend and front end tasks. Now i know bits and pieces of everything
I don’t like the term “full stack dev”. I personally feel like I’m a Jack of all trades and master of none. There’s so much to learn if you’re a full stack developer. Endless possibilities, endless rants and endless frustration 🤯 -
So, we (I'm the backend guy and work with a UI dev) are building this product portfolio management tool for our client and they have a set of 250 users. The team has two point of contacts for the 250 users who maintain the master data, help users with data quality, tool guidance, reporting and other stuff. So one day one of these two support users come to me and say : Hey I'm not able to add new transactions coz a customer is missing.
We have the provision to create / maintain customers.
I check the production DB, application code, try creating the customer and then the transaction, everything works perfectly fine.
I ask the user for a screen sharing session, the user starts reproducing the error like this :
We have a 3 system landscape - Dev / Test and Prod
U : Logs into the test system url, creates the customer.
U : Points out the toast saying customer creation is successful.
U : opens a new tab, opens the production system, tries creating the transaction, searches for the customer and says " see !! cant find the customer here ! the master data management apps never work !! "
FML?. -
So I made an update to my React Native app. I changed UI of a couple of screen, added a few animations here and there, refactored how my graphQL resolvers work in the backend(no breaking changes), changed how data gets loaded into the database etc.
It worked in dev so I figured hey let's deploy it. Today is(was because it's now 3am but more on that later) a national holiday so no one goes to work so no one will use my app so I have an entire day to deploy.
I started at 15:00(because i woke up at 13:00 lol). I tested the update once again in dev and proceeded to deploy it to prod. I merged backend to master, built docker images, did migrations on the db, restarted docker-compose with new images. And now for the app. I run ./gradlew assembleRelease and it starts complaining that react-native-gesture-handler is not installed. Ugh, rm -rf node_modules && yarn install. It worked. But now gradlew crashes and logs don't tell me anything. Google tells me to change a bunch of gradle settings but none of them work. Fast forward 5h, it's around 20:00 and I isolated the issue to, again, react-native-gesture-handler. They updated from 2.2.4 to 2.3.0 which didn't fucking compile. 2 more hours passed (now 22:00) and I got v2.3.1 working which fixed the problem in 2.3.0 but made my app crash on startup. YOUR FUCKING LIBRARY GETS 250K WEEKLY DOWNLOADS AND YOU DONT EVEN BOTHER CHECKING IF IT COMPILES IN PROD ON ANDROID?! WHAT THE FUCK software-mansion?
After I solved that, my app didn't crash. Now it threw an error "Type errors: Network Request Failed" every time I fetch my legacy REST API(older parts use rest and newer use graphql. I'll refactor that in the next update). I'll spare you the debugging hell i went through but another 5h passed. Its 3am. My config had misspelled url to prod but good for dev... I hate myself and even more so react-native-gesture-handler.3 -
Started Master Degree in DB just to prove to myself that I can do this. Although I'm really bad at sql I passed the admission exam . Now I'm wondering : how the fuck I passed a fucking sql exam without knowing to write a single good sql line . Just how ...8
-
Attended a game workshop session by Mark Skaggs, creator of Farmville. At one point he asked all the programmers to raise their hands. I love to code even though i aint an expert in backend db related code. I raised my hand. Then he asked all the designers to raise their hand. I love to graphic design as a hobby even though i aint an artist. I raised my hand again.
He noticed and said "Interesting..."
It made me question which trade i should focus more on. I still have years ahead to master them but should i give up one for the other?
Would be cool to hear your thoughts on this!4 -
To me this is one of the most interesting topics. I always dream about creating the perfect programming class (not aimed at absolute beginners though, in the end there should be some usable software artifact), because I had to teach myself at least half of the skills I need everyday.
The goal of the class, which has at least to be a semester long, is to be able to create industry-ready software projects with a distributed architecture (i.e. client-server).
The important thing is to have a central theme over the whole class. Which means you should go through the software lifecycle at least once.
Let's say the class consists of 10 Units à ~3 hours (with breaks ofc) and takes place once a week, because that is the absolute minimum time to enable the students to do their homework.
1. Project setup, explanation of the whole toolchain. Init repositories, create SSH keys for github/bitbucket, git crash course (provide a cheat sheet).
Create a hello world web app with $framework. Run the web server, let the students poke around with it. Let them push their projects to their repositories.
The remainder of the lesson is for Q&A, technical problems and so on.
Homework: Read the docs of $framework. Do some commits, just alter the HTML & CSS a bit, give them your personal touch.
For the homework, provide a $chat channel/forum/mailing list or whatever for questions where not only the the teacher should help, but also the students help each other.
2. Setup of CI/Build automation. This is one of the hardest parts for the teacher/uni because the university must provide the necessary hardware for it, which costs money. But the students faces when they see that a push to master automatically triggers a build and deploys it to the right place where they can reach it from the web is priceless.
This is one recurring point over the whole course, as there will be more software artifacts beside the web app, which need to be added to the build process. I do not want to go deeper here, whether you use Jenkins, or Travis or whatev and Ansible or Puppet or whatev for automation. You probably have some docker container set up for this, because this is a very tedious task for initial setup, probably way out of proportion. But in the end there needs to be a running web service for every student which they can reach over a personal URL. Depending on the students interest on the topic it may be also better to setup this already before the first class starts and only introduce them to all the concepts in a theory block and do some more coding in the second half.
Homework: Use $framework to extend your web app. Make it a bit more user interactive with buttons, forms or the like. As we still have no backend here, you can output to alert or something.
3. Create a minimal backend with $backendFramework. Only to have something which speaks with the frontend so you can create API calls going back and forth. Also create a DB, relational or not. Discuss DB schema/model and answer student questions.
Homework: Create a form which gets transformed into JSON and sent to the backend, backend stores the user information in the DB and should also provide a query to view the entry.
4. Introduce mobile apps. As it would probably too much to introduce them both to iOS and Android, something like React Native (or whatever the most popular platform-agnostic framework is then) may come in handy. Do the same as with the minimal web app and add the build artifacts to CI. Also talk about getting software to the app/play store (a common question) and signing apps.
Homework: Use the view API call from the backend to show the data on the mobile. Play around with the mobile project to display it in a nice way.
5. Introduction to refactoring (yes, really), if we are really talking about JS here, mention things like typescript, flow, elm, reason and everything with types which compiles to JS. Types make it so much easier to refactor growing codebases and imho everybody should use it.
Flowtype would make it probably easier to get gradually introduced in the already existing codebase (and it plays nice with react native) but I want to be abstract here, so that is just a suggestion (and 100% typed languages such as ELM or Reason have so much nicer errors).
Also discuss other helpful tools like linters, formatters.
Homework: Introduce types to all your API calls and some important functions.
6. Introduction to (unit) tests. Similar as above.
Homework: Write a unit test for your form.
(TBC)4 -
Sometimes in our personal projects we write crazy commit messages. I'll post mine because its a weekend and I hope someone has a well deserved start. Feel free to post yours, regex out your username, time and hash and paste chronologically. ISSA THREAD MY DUDES AND DUDETTES
--
Initialization of NDM in Kotlin
Small changes, wiping drive
Small changes, wiping drive
Lottie, Backdrop contrast and logging in implementation
Added Lotties, added Link variable to Database Manifest
Fixed menu engine, added Smart adapter, indexing, Extra menus on home and Calendar
b4 work
Added branch and few changes
really before work
Merge remote-tracking branch 'origin/master'
really before work 4 sho
Refined Search response
Added Swipe to menus and nested tabs
Added custom tab library
tabs and shh
MORE TIME WASTED ON just 3 files
api and rx
New models new handlers, new static leaky objects xd, a few icons
minor changes
minor changesqwqaweqweweqwe
db db dbbb
Added Reading display and delete function
tryin to add web socket...fail
tryin to add web socket...success
New robust content handler, linked to a web socket. :) happy data-ring lol
A lot of changes, no time to explain
minor fixes ehehhe
Added args and content builder to content id
Converted some fragments into NDMListFragments
dsa
MAjor BiG ChANgEs added Listable interface added refresh and online cache added many stuff
MAjor mAjOr BiG ChANgEs added multiClick block added in-fragment Menu (and handling) added in-fragment list irem click handling
Unformatted some code, added midi handler, new menus, added manifest
Update and Insert (upsert) extension to Listable ArrayList
Test for hymnbook offline changing
Changed menuId from int to key string :) added refresh ...global... :(
Added Scale Gesture Listener
Changed Font and size of titlebar, text selection arg. NEW NEW Readings layout.
minor fix on duplicate readings
added isUserDatabase attribute to hymn database file added markwon to stanza views
Home changes :)
Modular hymn Editing
Home changes :) part 2
Home changes :) part 3
Unified Stanza view
Perfected stanza sharing
Added Summernote!!
minor changes
Another change but from source tree :)))
Added Span Saving
Added Working Quick Access
Added a caption system, well text captions only
Added Stanza view modes...quite stable though
From work changes
JUST a [ush
Touch horizontal needs fix
Return api heruko
Added bible index
Added new settings file
Added settings and new icons
Minor changes to settings
Restored ping
Toggles and Pickers in settings
Added Section Title
Added Publishing Access Panel
Added Some new color changes on restart. When am I going to be tired of adding files :)
Before the confession
Theme Adaptation to views
Before Realm DB
Theme Activity :)
Changes to theme Activity
Changes to theme Activity part 2 mini
Some laptop changes, so you wont know what changed :)
Images...
Rush ourd
Added palette from images
Added lastModified filter
Problem with cache response
works work
Some Improvements, changed calendar recycle view
Tonic Sol-fa Screen Added
Merge Pull
Yes colors
Before leasing out to testers
Working but unformated table
Added Seperators but we have a glithchchchc
Tonic sol-fa nice, dots left, and some extras :)))
Just a nice commit on a good friday.
Just a quickie
I dont know what im committing...3 -
Do you have a db specialist in teams? I often see frontend & backend ranting but no db Devs. Isn't that role so crucial as one lost index can cause massive performance drops?5
-
AMQ cluster is misbehaving.. Master is constantly dying, slave is starting under root rather than app account, connection to AMQ DB takes 20-40 seconds (while queries execute in <0.1sec and TCP probe takes <0.01sec to succeed), monitoring is down,...
I mean it's gotta be the virus - what else could it be.2 -
Ideas I've had over the years that could pan out and be useful:
SMS-DB: Stands for SMS-Data Burst. Used to allow those with low cell signal or no data plan to transfer data between a phone and some client via the standard SMS text space. Would be slow, but would act kinda like dial-up over SMS (as mobile lines are compressed on all service levels, even LTE, so traditional dial-up wouldn't work!) I have a general idea on how packets would be laid out, but that's about it so far...
everything2PNG: Allows one to transpose any file's data into a PNG with a 3 byte per pixel (full color RGB), which allows for a "compression" of sorts (about 91, 93% on preliminary tests) AND allowing further, more efficient compression of the resulting file. (Plus... it's just kinda cool to see files transposed as PNGs.) I actually have a simple transposer to go to PNG, but can't yet go back. Large files (around 600MB) use upwards of 4GB with efficient paging and other optimizations via NumPy so far, so it's not *viable* yet, but it's coming along nicely.
RPi-GPIO Interconnection Bus: A master/slave or round robin method to allow for Raspberry Pis to communicate using GPIO, which can help free up network bandwidth in RPi cloud computing clusters. At most, this'd allow for 4 bits used for pushing to the GPIO "bus", and 4 bits used for pulling from the "bus". 8 pins total are usually unused minimum, so either 3 or 4 pins for upload, 3 or 4 for download, and potentially 1 or 2 for commands, general non-data communication, etc. I made a version of this concept using Round Robin for a client, but it was horribly slow. (I also don't have distribution rights for the code, so i'm working from scratch.) Definitely doable. -
Is it a good approach to have a master SSL key for all your servers when making the authentication?
I am a Developer, but when you work in a company with two developers and you are the senior one you have to learn a lot of stuffs. I am learning more in depth things about how to secure the servers and network.
Now, I am expanding the servers. Splitting the code and database in three different servers (code, Master DB, Slave DB) and configuring Master-Slave databases.
My questions are:
1. Is it a good approach to have a master SSL key for all your servers?
2. Is is a good approach to use the same SSL key for Master database server and Slave database server?
Any other suggestions are welcome.
Thank You in advance!2 -
Boy, I sure do love trying to figure out why our master and slave MariaDBs differ in their execution plans, even after running analyze tables on the whole DB.
Or rather, I really hope the two boxes didn't somehow magically desynced, cuz that would then beg a question of why, and how to prevent that from happening again.
I hate how databases are so necessary nowadays, but are probably the most complex and black box software I deal with. There's just so much to consider...1