Scalability

!rant

So last weekend I started collecting hardware for a small scale cluster at home to test scalability of my software. Making some decent progress.

Tomorrow I will replace the switch and this weekend I will set up storage so I can start my first application

“scaling”

A Developer is desperate: his java application servers are unresponsive, thousand of dead zombie threads are sucking all cpus, memory is leaking everywhere, garbage collector has gone crazy, the cluster sessions are fucked....

The Developer goes to the closest bridge, ties a stone to his neck and gets ready to jump.

Suddenly a bearded old man with a fiery look runs toward him, yelling:
- stop stop!!!! Your application is not scaling and misconfigured, your servers are melting, cpu usage is not sustainable anymore, but don't despair

The Developer, puzzled, looks at him:
-I've never seen you...how do you know...
- Hey, man, I'm the Devil. I know everything. All your problems are solved. I'll give you magic functions. They are called Lambda.
You'll never have to worry about your servers, scalability, security, configuration and shit.

The Developer seems astonished but relieved:
- Ok, sounds great! let's try it - suddenly suspicion creeps in - hmmmm but you are the Devil....so...you want something back, don't you?
(the Devil nods lightly with a diabolic smile)
- ...and...you want my soul, I guess...
- your soul??? come on!!! - the Devil burst in a laugh - we are in 2019. I don't care about your soul. I want your ass.
- What!???!!!?
- yes, I want to fuck your ass

The Developer, evaluates quickly the situation.
Few moments of pain or slight discomfort (?) in exchange for magic lambda. It could be worth. He accepts.

After a while of rough anal fucking, the devil asks
- Hey, how old are you anyway?
- 45, why?
- Oh jeeez...45!!!??? and you still believe in the devil?

When you about to start a project by doing a small module, but as you code, you keep thinking about scalability, security dependencies and such....

....three hours later, you still haven't written code

Software engineering project discussion:
Boy: Sir, my project is a client to manage files stored on different cloud file storage systems at one place
Faculty: Boring idea. Very easy to implement, No scope of scalability, etc
Girl: My project is an app to display the weather information
Faculty: Omg! What an innovative idea! I'm surprised how no one though of this before!

Switched jobs one month ago.

Used to work overtime on complex features, every engineer was 10x, learned a bunch, worked my ass of everyday. Switch due to overtime and because I wanted more personal time.

Anyways, at the new job I’ve completed two tickets in a month, code is shit, no one cares about the quality, scalability, etc. I’m payed 2x more and I currently work max 3 hours a day. Feels weird AF. I guess I got what I wanted, but didn’t know back then that professionally I’m going to degrade. Didn’t happen yet, but I can see that in the future.

🤷🏻‍♂️

Solved a complex puzzle on a website for a local ecommerce business, mind you in 16 and not really looking for a job but an unpaid internship would look beautiful on a resume or university application.

They wanted to see some of my code and give me a tour and none of them despite them being PHP developers for Magento could wrap their heads around laravel or how the routing worked. They also didn't understand and raw PHP whatsoever. I lost all faith and walked out of their office when they asked why I was using prepared statements and how they worked. That was after finding out that they don't understand cloud scalability whatsoever or common security practices.

At a Halloween party 2 devs started talking about how scalable PHP is...

Management: List your most significant achievement during this cycle.

Me: Developed and implemented a full stack of database micro containers that allow scalability and reusability.

Started a new gig doing APIs and deployment pipelines. After a few weeks a new boss came on the scene and decided I'm now a web dev, and the API and pipelines are going to be scrapped, and replaced by something with less availability and scalability.

Anyone looking for a good back-end dev?

😵‍💫

Dear God ....SCALABILITY

I spent a month trying to understand how to split our application and setup a pdf server to manage 10'000> pdf documents generation at the same time.

Today during our monthly meeting, our dear Key Account Manager, said he's not selling pdf generation to customers since one year at least, and he's not planning to sell it in the future.

Nobody knew about that apparently. We had also some demo guys showing actually the functionality to potential customers.

Also nobody believe me when I say we should start a project to monitor used functionalities in our application.

Just wasted one month of my fucking extra-time. Now I'm late with a lot of other tasks, apparently more urgent that the potential scalability issue of a functionality we don't use.

Key Account Manager rules.
Key Asshole Manager.

I think we should hire a Key fucking-COMMUNICATION Manager

So.. how does being in a company makes you a better engineer?

For eg, this year my team shipped these 2 interesting features (among other things) that were never part of our product.
these were crazy ideas, that got liked by our users, and might have impacted the finances of company positively.

However the engineers (not me, i worked on them even less) that worked on it, did they got their knowledge growth? they just analysed the old codebase, its shitty architecture and drawbacks, and added the feature around it in a similar manner. so basically it was building debris over the debris

but is this growth? if that engg was me, would my experience in dealing with debris and building debris over it (that somehow works) be helpful for any other organisation?

And number (2) : is my organisation even a good one if its allowing to build debris over debris and not leaving a space for discussion or cleaning the mess or thinking about better architectures, data structures for scalability and robustness?

An engineer's growth should be made by giving them a chance to explore new and best solutions and not the best hacks i guess

I LOVE NODEJS
I LOVE ANGULAR
I LOVE WRITING MY OWN FRAMEWORKS
I LOVE ELECTRON
I LOVE IONIC
I DONT CARE OF APPS SCALABILITY
I HATE HYPSTERS !!!

I got notified that tomorrow I'm gonna start a porting project from a FileNet ecosystem.

Well, I don't know what is FileNet, but at least I've enough time to study its architecture. Let's start from the official IBM page:

The FileNet® P8 platform offers enterprise-level scalability and flexibility to handle the most demanding content challenges, the most complex business processes, and integration to all your existing systems. FileNet P8 is a reliable, scalable, and highly available enterprise platform that enables you to capture, store, manage, secure, and process information to increase operational efficiency and lower total cost of ownership.

Thank you IBM, now I surely know how to use FileNet. Well, I hope that wikipedia explains me what it is:

FileNet is a company acquired by IBM, developed software to help enterprises manage their content and business processes.

Oh my god. I tried searching half an hour so far and everything I found was just advertisements and not a clue about what it is.

Then they wonder why I hate IBM so much

OMG. Just had the worst convo with my supervisor. She's a dev and she does not understand why scalability is important.

We have a really bad client who has a fucking insane amount of pages and they want to change some stuff on a few and my supervisor just said manually add it in with if satements.

I replied saying this client will want more things changed let me take the extra hlaf hour to make it scalable.

She responded no just do manually.

I wanted to get in to rant about how scalability is important but it was the end of the day and I was to tired.

Yay, Dubba didnt die.
Okay, hello'ed back, lets fucking rant.

Im fucking SICK and TIRED and ANNOYED by all freaking cloud fanboys. Yeah, sure. You get scalability. You waste less resources. Sure. But it depends on use case. There is suprising ammount of services that can be run on VPS or dedicated servers that will be a lot cheaper if you just DONT run it on cloud. And dont you dare to missinterpret me. I fully recognize that there are valid uses for cloud services. But for fuck sakes, its not anwser for ALL of your needs, and it costs insane ammount of money if you use it where you should use dedicated.

WHY we, as humanity came to this place where self-claimed "proffesionals" will anwser you "cloud" before you even ask your question.

No, fuck off, take calculator and think for around 60 seconds. And than decide if you need cloud or not.

And no, this mail server will be fucking few times more expensive on cloud, and I know it becouse I have fucking calculator and I was running it past 1.5 year and actually run sometimes htop, so stop bullshitting me.

Sorry if some cloud die-hard felt offended but it is exacly how I see and feel it.
Peace.

A C++ program that spawns as many threads as it can to open TLS 1.2 connections to a given server and send random data.

Needed this to test scalability of one of our services.

I hate React. I keep reading that people have problem of grasping it, but that's not the case for me. I get it, I understand it, but I hate with passion HOW it's done knowing how nice it's done elsewhere. What really triggers me is how ugly it looks, both from architecture and code level. To me it really say a lot when even code shown in documentation looks ugly, and while reading it you ask ourself constantly "why it's done this way?". When I read React being called an "elegant" solution something explodes in me. Did you saw Svelte? Vue? Damn, even Alpine.js?
I just cannot how overengineered this API is. Even doing simplest things there produces so much junk code written only because this is what library requires. Why? I feel like working with it is a punishment.
And scalability and maintainability? I've never seen large-scale projects more messed up than those wrote with React. And yes, you can blame teams working on them for lack of skills, but it is the library which encourages or not good practices also, and I've never seen such bad situation with other libraries/frameworks.

My image of dream career through different times of my life:
- frontend specs prodigy, css enlightenment, a member of w3c or a similar committee
- indie hacker and entrepreneur, leader of a startup community
- architecture prodigy, expert in scalability
- transsexual evangelist, popular article writer and a rockstar
- hardware engineer: Linux, C, chip and dale’s Gadget-like girlfriends, xkcd, latex, assembly, buying a radio station and a telescope
- scientist like NickyBones, papers, data, more data
- art expert

Though achieving one of this would take the entire life, I had a chance to grasp all of this. WHY does they feel so incompatible? Why do I have to choose?

Why do I feel so sad? Why do I feel like I haven’t achieved anything even though I objectively achieved what I dreamed of like five years ago?

Is it true that it’s in my nature to always seek an environment to feel like a junior in? Is feeling like a junior only pleasant to me because it reminds me of old times when I wasn’t actually this mentally ill and was still happy?

Why do I feel like that arduino and C shit is the equivalent of a red corvette?

YES!

Finished a complete version upgrade with vastly improved scalability and maintenance in just 3 weeks!

NO!

I forgot to do the database migrations as I went along. Ok... no problem.. I'll just use some kind of database comparing software, I'm sure they exist...

<googles...>

...fuck... is there not any for free?

Ok fine whatever i'll manually compare the tables and columns, fuck it. Sure there's like 50 tables and like 20 columns per... whatever...
should take like a day or so, with testing...

YES!

Finished my new version upgrade after just 6 weeks of coding!

....fuck sakes...

Salesforce. Although I wasn't involved in the purchase or the implementation, I spent many 100 hour weeks dealing with the crapshoot of an implementation. A large company abused that software to the point of no return. They used that thing for everything, and then they didn't even use it right for the one thing salesforce is good at. So I guess I don't have anything against salesforce itself besides its scalability issues, custom SOQL syntax, user model, and pricing. I'm more upset about the salesforce developers/business owners that decided it was okay to use salesforce for things it was never meant for, like inventory, project management, 3rd party sales team, and so many other things that caused what should have been sub-second queries to take 30 to 60 seconds.

Project manager: "What is a micro service? I'm dating a girl and she mentioned it and I want to impress her."

Me: "Well, you have monolithic services which tend to serve many different functions whereas a micro service tends to serve a single function or a few related functions. They are usually easier to scale and can be optimized to be faster. Still, right tool for the job."

Project manager: "Oh nice! So I can ask her 'Hey, want to see my micro service? It's quick and scalable.'"

-face palm-

He's leaving this week. I'm going to miss him though.

Seriously though, in that context, would scalability mean you're bringing friends?

Question: Does using cookies for user session handling hinder the scalability of your backend because all the API's have to live on the same domain. Basically if one API starts to get a lot of request and you want to add another server to off balance the load you would have to add an entire webserver rather than just a small micro webserver with the API running on it mainly because cookies are used to authenticate user request and cookies don't survive CORS request. Am I right or don't know what the hell i'm talking about lol need some opinions I suggested we make all API's micro services and use JWT for user sessions

When the CTO/CEO of your "startup" is always AFK and it takes weeks to get anything approved by them (or even secure a meeting with them) and they have almost-exclusive access to production and the admin account for all third party services.

Want to create a new messaging channel? Too bad! What about a new repository for that cool idea you had, or that new microservice you're expected to build. Expect to be blocked for at least a week.

When they also hold themselves solely responsible for security and operations, they've built their own proprietary framework that handles all the authentication, database models and microservice communications.

Speaking of which, there's more than six microservices per developer!

Oh there's a bug or limitation in the framework? Too bad. It's a black box that nobody else in the company can touch. Good luck with the two week lead time on getting anything changed there. Oh and there's no dedicated issue tracker. Have you heard of email?

When the systems and processes in place were designed for "consistency" and "scalability" in mind you can be certain that everything is consistently broken at scale. Each microservice offers:

1. Anemic & non-idempotent CRUD APIs (Can't believe it's not a Database Table™) because the consumer should do all the work.
2. Race Conditions, because transactions are "not portable" (but not to worry, all the code is written as if it were running single threaded on a single machine).
3. Fault Intolerance, just a single failure in a chain of layered microservice calls will leave the requested operation in a partially applied and corrupted state. Ger ready for manual intervention.
4. Completely Redundant Documentation, our web documentation is automatically generated and is always of the form //[FieldName] of the [ObjectName].
5. Happy Path Support, only the intended use cases and fields work, we added a bunch of others because YouAreGoingToNeedIt™ but it won't work when you do need it. The only record of this happy path is the code itself.

Consider this, you're been building a new microservice, you've carefully followed all the unwritten highly specific technical implementation standards enforced by the CTO/CEO (that your aware of). You've decided to write some unit tests, well um.. didn't you know? There's nothing scalable and consistent about running the system locally! That's not built-in to the framework. So just use curl to test your service whilst it is deployed or connected to the development environment. Then you can open a PR and once it has been approved it will be included in the next full deployment (at least a week later).

Most new 'services' feel like the are about one to five days of writing straightforward code followed by weeks to months of integration hell, testing and blocked dependencies.

When confronted/advised about these issues the response from the CTO/CEO
varies:

(A) "yes but it's an edge case, the cloud is highly available and reliable, our software doesn't crash frequently".
(B) "yes, that's why I'm thinking about adding [idempotency] to the framework to address that when I'm not so busy" two weeks go by...
(C) "yes, but we are still doing better than all of our competitors".
(D) "oh, but you can just [highly specific sequence of undocumented steps, that probably won't work when you try it].
(E) "yes, let's setup a meeting to go through this in more detail" *doesn't show up to the meeting*.
(F) "oh, but our customers are really happy with our level of [Documentation]".

Sometimes it can feel like a bit of a cult, as all of the project managers (and some of the developers) see the CTO/CEO as a sort of 'programming god' because they are never blocked on anything they work on, they're able to bypass all the limitations and obstacles they've placed in front of the 'ordinary' developers.

There's been several instances where the CTO/CEO will suddenly make widespread changes to the codebase (to enforce some 'standard') without having to go through the same review process as everybody else, these changes will usually break something like the automatic build process or something in the dev environment and its up to the developers to pick up the pieces. I think developers find it intimidating to identify issues in the CTO/CEO's code because it's implicitly defined due to their status as the "gold standard".

It's certainly frustrating but I hope this story serves as a bit of a foil to those who wish they had a more technical CTO/CEO in their organisation. Does anybody else have a similar experience or is this situation an absolute one of a kind?

Whether it's a good idea to host my very primitive web-app on Heroku or setting up my own VPS on DigitalOcean etc.?

Heroku will save me some time, but I'm not sure about the scalability and cost.
VPS approach means I'll have to take care of all the DevOps myself, even though I know a bit Linux to kick things off, this is still quite daunting to me.

Any suggestions? Insights? Rants?

I’m working on a react codebase and company decided to add a new module.
Now im writing markup and css to ensure UX is smooth as designers thought of it.
Imagine my horror when I start to code and find out no matter what HTML tag i use, it’s been FUCKING OVERRIDDEN in the global stylesheet. AND STYLES HAVE BEEN OVERRIDDEN WITH !important

They’re also using Ant design as a component library. Guess what, default ant design classes have been overridden too. So i try to use ant design button or card, and bam, MAGICALLY SOME DESIGN FROM SOME SHITHOLE MODULE DECIDES TO FUCK WITH MY STYLES

On top of that, styles of parts of application has been written in SASS, some part of application uses bootstrap components some use third party components like tables and responsive grids to suit to their preferences. Some parts use handwritten css. Some parts use CSS IN JS and styled components. THE FUCK IS THIS GARBAGE!!!! THE FUCKING CODEBASE HAS A MIND OF ITS OWN!!!!!! YOU NAME A WAY TO ADD STYLES TO A COMPONENT, ITS THERE!!!
And the company’s management thought a “fractal” approach to maintain each individual view is “best” for SCALABILITY!!! HOW THE FUCK DID IT NOT CROSS YOUR DUMB MIND THAT FRACTAL APPROACH ALSO GUIDES TO HAVE ALL COMMON STUFF AT ONE PLACE!!!! THIS CODEBASE HAS DUPLICATE STYLES AND DUPLICATE CODE IN ALMOST EVERY MODULE!!!!

Not to mention every developer choosing to freely decide the way they should write their code without any guidelines.

HOW THE FUCK PEOPLE WRITE THEIR CODE WITHOUT THINKING ABOUT OTHER DEVS!!! SO BASICALLY I AM NOT ONLY CLEANING SOMEONE ELSE’S SHIT BUT ALSO TRY NOT TO SHIT IN THE PROCESS!! FML

I prefer it doing 2 tasks parallely during the initial phase of requirement gathering and design phase.(makes more sense if you are working extremely new system and framework)
1. Keep collecting requirements from clients and understand them.
2. Collect different designing aspects for the project and parallely, build a POC for 2 purpose: to get hands into the new Framework and also as a demo to clients. Working on POC helps in 3 ways: Improving understanding of requirement, improving framework knowledge, and playing around with code whenever bored of designing and reading tons of existing designs..
3. Once primary requirements are clear and fixed, analyse all different designs, if possible I setup meetings with senior devs, principal engineers (they help a lot when it comes to reviews on scalability and reliability of a design)
4. The above design is mostly architectural level. Once design is fixed, then I start taking each component and prepare a detailed implementation design. (Notice that whenever I am bored of designing, I spend sometime in building POC)
5. In detail design, I focus on modularity and flexibility. Anything defined should have getters and setters for example. This will help you reuse your code. Keep the interface between components in your design as generic as possible, so that in case your MySQL is change to Postgre or NoSQL, your design should be able to adapt new features..
6. Instead of building entire project, define feature targets and deliver small features.. this will help you to be in line with the requirements with minimum deviation.

Me: “I think I’ll check linkedin today”

Clicks on video: “... good perspective on growth and blockchain micro service apis that leverage ML models for understanding interfaces that welcome scalability within a agile environment...”

Me: *jumps out window*

I've actually already discussed this one on here I believe

I see this job looking for an android developer for Kotlin with UI experience with XD & Figma and experience with Firebase. I have all of these qualifications so I throw my resume into the fray within an 2 hours the recruiters contact me. they have an offer of 76,000 and I'm looking for junior so I'm like, eh whatever, I give them a copy of my resume and we hold discussion for a few days and then radio silence. I then see a job posting EXTREMELY similar but with a "different company" so I throw my resume in and again within 2 hours I get a call only THIS TIME ITS THE INTERNAL HR. She sounds interested we have a good conversation and sets me up for 96,000 and they schedule me for my first interview within the week. Interview goes great, next I meet with the CTO and we have a pretty good conversation, I'm expecting a technical exam but it doesn't happen instead they give me a case study. they send me requirements for an app API to use, architecture, and a week time span to do it. I finish the app with extra features within 6 days, in my understanding of MVVM and I was excited and happy about this app because its JUST NICE. a week goes by and I meet with the tech team. They grill me on my application, scalability, use cases, how would I advertise or place advertisement and I'm answering everything they love the UI (I included mockups I made on XD), they say everything sounds good everyone leaves with smiles they say they have to find out on what team to place me because they have multiple apps and that HR will be in contact with me in the next few days... A WEEK GOES BY and I randomly get the declination email that next Friday. When I asked for feedback they said it wasn't true MVVM. I was devastated until the next week when I was accepted for a higher paying job that didn't require me to move. After I accepted this job guess who calls? THE FIRST RECRUITER and for this long I was wondering if this was the same job due to the very similar job description so I ask "is your client XXXXXXX?" it was I just told him "I'm good" and hung up

My 2 cents on different OSes to use.

I think Linux is best for running servers and services and having long run times with little issues (when its Console and not GUI based.) But I have a lot of issues with using its GUI distributions like Ubuntu and it feels kind of unpolished in that area.

I prefer macOS for its GUI as it actually works and has far less issues than Windows GUI and is (IMO) better than Linux GUI's by far. But macOS just doesn't feel like it was designed super users and it can feel like its holding you back a bit. Also you have to use Mac hardware which are amazing machines, they are just overpriced.

I prefer Windows for its GUI and despite its problems, it is very well designed for super users and has very well designed remote desktop features and scalability (although it is a pain to maintain.) Windows works well for connected company systems.

In my opion:
Linux: Servers, databases (no GUI)

macOS: Designers, photo/video editing, IT/programmers and general use as a standalone (not part of a company system).

Windows: IT/programmers, super users, general use but better than macOS at working together in a company setup, but macOS is better at being a personal laptop or PC.

I personally use Linux for our email and web servers. Windows for our company computers (designers use Macs) and I have a Macbook as my own personal computer.

I guess you could say that my speciality is cloud at scale. I’d say it chose me more than I chose it.
Looking back on it though, I think what I like about my speciality is the unique challenges it brings.
Every speciality has its own set of challenges, like tight resource limits in embedded, or client-server synchronisation in native/mobile.

The challenge of cloud at scale is throughput. Designing systems that can support 100K users making a bazillion requests a second, or a data pipeline firing events that you need to process in near real time without dropping a single one.
The real challenge of course is doing all this within a sensible budget. We have virtually infinite compute but we dont have infinite dollars to spend on it.

Its a fun problem to solve.

advantages of bionic systems:
- energy efficiency
- flexibility
- low friction
- compact size
- water resistance

advantages of mechanical systems:
- precision
- rotation is available
- scalability
- can change parts

Half of the courses we had in our college were about electronics. Except Microprocessors and Transistors, it's not relevant.
We even had chemistry and engineering drawing. So we essentially wasted more than half of our time.

Besides languages, weren't taught anything about real world software development.

Nothing about how to work with an existing code base, version control, design patterns, system design, creating a website, debugging, functional programming, scalability, reliability.

The industry should be involved in setting the syllabus and also contributing part time teachers.

So, the main system we develop and "maintain" is a horrible monolithic huge project that has the majority of the code in one and only one god class.

On top of that, all the main logic is implemented directly in the "main" method (yes, the "public static void main"). Now every time you have to create a method to use it in the main logic, it has to be static!

God, I hate to even look at this class, let alone touch it, I feel dirty, like I'm touching a pile of shit.

Hey devRant!

University student here. I’m trying to learn how to build scalable applications. I figured what better way to learn, than to ask this community!

So, some questions:
1. What is a good stack for scalability
2. Regarding microservices, when and where should you use node, .net, and go
3. I hear much talk about docker and kubernetes... how would you use these concerning microservices

Thanks!

so i did this nice tool with data structures, dynamic ui composing, input sanitizing, modularity and scalability with tidy and efficient javascript and useful css.

showing it to my boss: "cool what you can do with html". WAT?

(he is not a programmer but knows i´m into web-programming)

Just migrated my custom help desk software for work to cockroachdb! Scalability here I come fucking come! And no more fucking MySQL Master Master configs to deal with.

TIL stack exchange is built on ASP.NET
And there I thought the platform had serious scalability issues

Have any of you fin devs out there on the interwebs had any experience with building social networks using `Stream`??

I am looking into their API for use in a golang project of mine but I am unsure about a few things...

1. Reliability
2. Scalability and the cost of it
3. General API quality, in Golang specifically but any comments are great.

It's a stretch I'm sure but if there was a place that folks would know it would be here @ devRant

I'll remove some courses - make some optional and some courses mandatory.
I'll explain- I did my B.E. degree in I.T. I'll remove some courses like the ECE subjects (Digital electronics , Communication Theory) - something I'll never use. If I will - I can learn it at that time. Some mandatory courses like DBMS, OS etc. And some optional ones you can take according to your passion like - security courses or scalability courses etc.

Guys check out IOTA, get a light node wallet and buy some as long as it is still low. It is rising at a rising rate since some hours now.

IOTA is the solution to bitcoins speed and scalability problems. IOTA does not use the blockchain but instead a network called the tangle, which enables decentralized peer-to-peer transactions. There are no miners, no fees and the transaction validation speed depends on how many people are using the currency. One transaction requires your device to validate 2 other transactions through proof of work and therefore the system can never be overloaded by too many transaction requests.

Be warned though, the IOTA foundation is only currently building up the infrastructure, and people are just starting to trade since a few weeks so trasactions still may take a few hours.

In Postgres, parallel query works by having multiple workers....

As such, depending on workers available, queries can run theoretically fast (workers available) or slow (workers not available) - right?

Question is: are there more optimizations like this... And is my assumption correct?

It seems like a major pain in the ass... Scalability and reliability wise. (?!)

Not sure if I should implement Bootstrap on my website for cross device scalability, grid system placement and basic animations, or just stick to raw CSS3... Any suggestions?

Best book/source for learning everything devops'y'/kubernetes?

(Given that I have some sort of experience in Dockers, hosting websites and know a fracture of aws archeticture, but lack in good "cloud" thinking skills, scalability, understanding costs for production applications, cluster size, etc...)

Neo4j vs Mongo vs SQL

Which one would you choose for a high density scalable database?

Please comment with your ideas about speed and scalability.

Not a rant, but seeking advice...

Should I abandon 2 years' worth of work on migrating a personal project from SQL (M$) to a Graph database, and just stick to SQL? And only consider migrating when/if I need graph capabilities?

The project is a small social media platform. Has around ~50 monthly active users.

Why I started the migration in the first place:
• When researching databases, I read that for social media, graph is more suitable. It was, at least in terms of query structure. It was more natural, there were no "joins", and queries were much simpler than their SQL counterparts.

• In case the project got big, I didn't want to have to panic-deal with database issues that come with growth. I had some indexing issues with MSSQL, and it got me worried that at 50MAU I'm having these issues, what would happen if I get more?

• It's a personal project, and the Gremlin language and graph databases looked cool and I was motivated to learn something new.

----

Why I'm considering aborting the migration:
• It's taking too damn long. I'm unable to work on other features because this migration is taking up all my free time. Sunk cost fallacy is hitting me hard with this one.

• In local testing within docker, it's extremely slow. I tried various graph engines (janusgraph, official tinkerpop, orientdb), and the fastest one takes 4-6minutes to complete my server tests. SQL finishes the same tests in under 2 minutes, same docker environment. I also tried running my tests on a remote server (AWS neptune) and it was just as slow. Maybe my queries are bad, but can I afford to spend even more time fine tuning all queries?

• I now realise that "graph = no scalability issues" was naïve of me, and 100% wishful thinking. Scalability issues don't care what database I use, but about how well tuned and configured the whole system is.

• I really want to move on. My tech stack is falling behind and becoming outdated. I'm unable to maintain dependencies.

• I'm worried about losing those 50 MAU because they're essential to gaining traction once I release the platform. I keep telling them about the migration but at some point (2 years later) they're going to get bored I feel.

I guess partially it's a rant because I feel like I shouldn't stop now having spent 2 years on this, but at the same time I feel like I'm heading towards a dead end.

If you made it this far, thank you for reading:)

first time refactoring today, it was like being brain_fistFucked

Struggling to optimize and to scale the infrastructure of our production environement dealing with people who don't bother themselves to write scalable code.

How to get good at solving problems.
My managers says that I should consider scalability, dev efforts, Operational costs etc. I am really new in this, I really need to figure out a lot of things by understanding architecture and then there are added quality parameters of the solution, where can I read all this? How to get good at this? I know one of the solution is to actually work on it, can I still get some other resources to understand things better?

You guys probably use slack at work. Or teams. But what about using discord? Would it be a good idea to use a discord server and create and manage clients' projects that way? Its free and simple to do. I have the most experience with discord even coding custom discord hooks.

For example I'd categorize each project by discord categories, and within each category I'd have channels such as general, ui, coding etc so engineers team project managers designers and clients can communicate in real time.

The biggest downside to this approach is... Discord allows maximum 50 categories per server. What if i have more than 50 projects to work on? I wouldn't be able to create a new category for that project. The growth and scalability of a company is limited thst way. That's what sucks. I'd have to create a brand new server and repeat this cycle again. And each server having max 50 projects. This doesnt sound very efficient right?

Is there a better way? Or do i have to use slack?

How do you handle scalability and load balancing in a full stack development project?

Hello fellow devs. I wanna create a survey site, which will contain 3 surveys within. All 3 surveys, will contain around 50 questions, and will have a range of input options. The site will collect the data from a form, store it and probably live visualize them with a plot-like library .

I was thinking about using React for the front-end, Firebase for the back-end + Google-Sheets and one of the many react-libraries for plotting.

Do you think that this stack is appropriate for that kind of app ? I am a bit concerned about firebase's querying system or the scalability of the app. (Since it will hold a lot of questions and answers )

( It's survey for developers, so I hope in the future you will participate in it :D ).

Any Erlang guys over here ?

I am getting into it after trying elixir, just to get solid foundations (understand it better and otp)

The goal is to get erlang elixir and pheonix undercontrol

Now my questions:

Any good reads about maintenance scalability in a prod environment ?

Thanks