Anyone else think the emphasis on scaling is misplaced?

I think we can have more innovative products that solve real problems in specific ways if people focused on solving specific problems rather than targeting wider audience in attempt to sell to as many as possible.

Hypeful vs Helpful?

I don’t see how scalability and innovation that solves specific problems exclude each other. In fact the products I’ve worked on usually focused on the innovative part before worrying about scaling issues. And when they finally started gaining momentum, they weren’t ready to handle heavier traffic. So ideally innovation and laying the ground for scalability should be tackled together

@red-knot Because scalability, in general is hard and takes a lot of extra resource. I think this is commonly accepted view of a distributed system?

It involves building out infra for things like auto scaling/messaging/data syncing etc.

Usually involves building message passing/handling.

Sure these infra are available now and are relatively easy yo use (than before)

But they make things harder to debug and introduces more points of failure.

And before you know it. A product has hundreds of topics and pipelines that are difficult to monitor and maintain.

I think these are overhead with a diminishing return.

Wouldn't it better to divert these efforts to something else? How do we know the availability of bigger and more powerful hardware isn't limiting our imagination? I don't know if this is correct but I feel like a lot of our greatest inventions were only possible because of constraints that people had to overcome

@red-knot We actually had a different problem at my last place. We were so concerned about scalability the product never saw the light of day lol

Is there a path a product can take to help make it scalable in the future? A design strategy? Or is there some kind of technological lock in that occurs?

I think having to rewrite half your code base down the line is the opposite of what the job is.

And honestly would have benefited developers alot longer to be contracted to do the job properly the first time

I once saw an application for a certain government sector being written in fucking light switch another for uni being written using SharePoint

Stupid

@AvatarOfKaine I'm not saying to do that though. I'm saying because we want to sell a product to as many people as we can. It makes us want to scale. But doing that always results in a less than ideal product that *can* work for most people. Whereas if we focus on a solving a specific need. We may not be able to sell to as many people but we can build a solid product with innovations meant to solve a specific problem and does it really well and not have to spend a lot of effort on scaling. But still follow the other practice of properly isolating your components. Single responsibility etc etc

Obviously I'm not talking about this from a making money perspective

@iceb well I also figured you meant something larger and more complicated

It depends on the app.

Btw say something else convincing if you're still here

@AvatarOfKaine That's kinda what I think. Maybe we don't need larger, complex app.

We just have to be creative and think hard to design and create apps that's simple and solves a need really well.

After all, "Senior programmers create complex solutions to complex problems. Great programmers find simple solutions to complex problems"

I mean this is just a thought I have and want to explore.

@iceb well sometimes the use of simple concepts like polymorphism to group objects for an extensible ui or data element for later scaling is simple and does make the job easier.

What seems complex is sometimes much easier as the application grows.

My issue tends to be with failed attempts to create layout containers for example on the UI side where a few simple innate behaviors and a bit of redraw code customized would do the job better.

Css has much nicer features than it used to but why not move the renderer to the api I hate dependencies on browsers.

@iceb however I am having difficulty envisioning certain complex problems

A lot of what I see in production is rather simple just likely required some fine tuning

Fat ass pedophiles getting paid too much money tend to march to fanfare they write half the time. The normal people go along with these things to safeguard their career path

@AvatarOfKaine Aaah. I'm approaching it from a different area.

I, too. Like to play around with objects and thinking about making them extensible, easy to maintain etc. When it's nice it's really nice.

When I refered to scaling here I'm talking about scaling for distributed system.

Like syncing data across microservices, data replications, trying to get 99.999% uptime.

I think the effort put towards it is a lot of effort for little gain. It might have been more useful if we focused on something else instead

I do agree that a lot of apps are simple in essence. Yet they require a large team to maintain.

And I think a lot of it is because a desire to scale. But is that really worth it?

@iceb well I .
Mean in essence that's not the most difficult thing to conceptualize

And it's really about figuring out how to make all the instances running on different containers talk to each other and load balancing the backend

The major issue I can the major issue I can think of there is essentially whenever you have to do a right to a database for multiple sources because if you have multiple instances of the same application running in multiple containers attempting to write to the same database tables there is a chance that one or more processes are going to be competing for the same access to the same record and might overwrite each other's additions in a case where essentially they need to be aware of at least one or more fields in the updated version of the record. the reason for this is is that if you have a cluster database system you can only usually write through one node so balancing that out across several would be difficult

@iceb really you have to consider the user base that you're targeting

@iceb keep on acting in some way like you're alive and we'll keep on having a conversation

Yeah nice. I feel like I'd be looked at as a weirdo when I talk to people about these ideas

It sure isn't "difficult" to create db clusters and load balance now.

With all the challenges we have with different nodes. All the usual patterns like exponential back offs. circuit breakers. bulk head strategies are non trivial to implement. And without them the app could act very unreliably.

People made generic solutions that is meant to serve lots of different use cases. (docker, kubernetes, cloud flare, Kafka etc). they often have complicated UI or lots of cli commands for people to learn.

And we'd be subjected to their downtime/bugs etc.

More importantly they must have lots of people maintaining their product to serve their users.

As a first thought. If they only sold self hosted solutions. They wouldn't need to care about scaling and we might get a fast, stable product that can achieve 99.999 availability much easier because we only have so few users @AvatarOfKaine

@iceb I don't really know what those terms mean lol but I have considered parallelism over a cloud environment before when thinking of designing a pretty big space combat game years back.

Think not anywhere near as shitty as eve online lol

@iceb bought the thought I was having related to frame syncing between individual nodes so you didn't have to segment users onto different servers like games like wow do

@iceb see I have only seen one implementation of kubernates and I'm not impressed

@iceb but it's the right idea
And maybe someone just misconfigured it

The thing is you need the ability to have a centralized router and usage monitor reassign and shutdown and allocate new nodes on the fly

I don't know if an app hosted by a platform like kubernates can do do that for sure. It's likely someone has written an addon in the host environment that allows this but it kind of defeats the point in a shared hosting platform. No system isolation

@AvatarOfKaine I haven't seen kubernetes either. Gone through some videos and I never want to touch it haha. but you can dynamically scale nodes these days. That doesn't mean there isn't a bottleneck at the data storage level. and there are a few different ways to deal with that.

And each with their own tradeoffs. And people (users) would ideally want all the data to be current and available all the time. Which is really hard to achieve. We know something will go wrong. We just invest a lot and hope things will work well long enough before something goes wrong.

They used to say “premature optimization is the root of all evil”.

Now it seems you can’t even get through a hello world without having to consider what you’ll do when your application replaces Google.

Build your thing to work and clean it up as necessary. Perfect is the enemy of sold.