what the hell is s3

@calmyourtities aws s3

@calmyourtities AWS service to store data similar to FTP but different concepts involved.

That's basically what Snowflake is doing.

@AndSoWeCode yes it is showing this.. but how to achieve this at my end :)

@varunsinghal well there's a reason why big companies do this with large teams of superstars.
Sure, you can put your data in S3 and write a program that queries it the way you want. That will work.
But if you want ease of use, scalability, data integrity, then there's a lot of work to be done.

@AndSoWeCode thanks :)
can you give me a direction to this career path, i really want to explore or at least read about this.

@varunsinghal you need knowledge about data organization. Starting with the most obvious - RDBMS, and their scalability (sharding), as well as NoSQL databases like the usual MongoDB, CouchDB, then going to graph DBs, then going with distributed computing (Spark), distributed file systems (Hadoop), a bit of devOps (Docker, Kubernetes), serverless computing (Amazon Lambda).

One needs to understand what ACID compliance is, why it's needed, where it's needed, where it can be trade off, who can benefit from losing ACID.
An important skill is the ability to develop concurrent computing processes efficiently....

Basically you need a ton of knowledge and experience, and even more time, to be able to do this well on your own.

The people that develop these systems - I talked to them. They aren't your average hipster college graduate. These are people with doctorate degrees and decades of experience, and/or people who can compute the Fibonacci sequence using regular expressions.

Don't get me wrong, the idea you proposed is great! It has value, and such a product would benefit from a lot of attention.

The problem is that implementing it, although completely feasible, is extremely hard, and will anyway cost you a lot of time and money, since you just won't be able to do it well on your own.

@AndSoWeCode i just needed a vision behind this...to understand what we are dealing with. thanks :)