As a developer, sometimes you hammer away on some useless solo side project for a few weeks. Maybe a small game, a web interface for your home-built storage server, or an app to turn your living room lights on an off.

I often see these posts and graphs here about motivation, about a desire to conceive perfection. You want to create a self-hosted Spotify clone "but better", or you set out to make the best todo app for iOS ever written.

These rants and memes often highlight how you start with this incredible drive, how your code is perfectly clean when you begin. Then it all oscillates between states of panic and surprise, sweat, tears and euphoria, an end in a disillusioned stare at the tangled mess you created, to gather dust forever in some private repository.

Writing a physics engine from scratch was harder than you expected. You needed a lot of ugly code to get your admin panel working in Safari. Some other shiny idea came along, and you decided to bite, even though you feel a burning guilt about the ever growing pile of unfinished failures.

All I want to say is:

No time was lost.

This is how senior developers are born. You strengthen your brain, the calluses on your mind provide you with perseverance to solve problems. Even if (no, *especially* if) you gave up on your project.

Eventually, giving up is good, it's a sign of wisdom an flexibility to focus on the broader domain again.

One of the things I love about failures is how varied they tend to be, how they force you to start seeing overarching patterns.

You don't notice the things you take back from your failures, they slip back sticking to you, undetected.

You get intuitions for strengths and weaknesses in patterns. Whenever you're matching two sparse ordered indexed lists, there's this corner of your brain lighting up on how to do it efficiently. You realize it's not the ORMs which suck, it's the fundamental object-relational impedance mismatch existing in all languages which causes problems, and you feel your fingers tingling whenever you encounter its effects in the future, ready to dive in ever so slightly deeper.

You notice you can suddenly solve completely abstract data problems using the pathfinding logic from your failed game. You realize you can use vector calculations from your physics engine to compare similarities in psychological behavior. You never understood trigonometry in high school, but while building a a deficient robotic Arduino abomination it suddenly started making sense.

You're building intuitions, continuously. These intuitions are grooves which become deeper each time you encounter fundamental patterns. The more variation in environments and topics you expose yourself to, the more permanent these associations become.

Failure is inconsequential, failure even deserves respect, failure builds intuition about patterns. Every single epiphany about similarity in patterns is an incredible victory.

Please, for the love of code...

Start and fail as many projects as you can.

New to this but here's my rant I suppose. It bothers me how "non-tech" people kind of devalue what tech people do. Like they have zero understanding of it, so you make something in 30 minutes or an hour that took years of building said skills and involved complex logic and understanding of relational data and because it only took you 30 minutes or an hour it must have been "easy". Or the way you are everybody's free tech advisor with family and friends... And things are said like "I'm not good with this stuff, but you're so good with it". For the record nobody is just "great" at technology or coding from birth its been a 2+ decade craft that I've experimented with and learned and put effort into. So taking into consideration all this effort I have put in to understand all this you say you'll never remember to push that button so you'll just ask me again when the problem arises. Yes because its so fun for me to constantly maintain your electronics because you can't bother to remember to push a button.

So last night i was dealing with some mysql query issues trying to grab some complex data from a relational database and ended up giving up.

This afternoon i was about half-asleep going to nap, not even thinking about the code and randomly came up with the solution.

A few hours later i made the changes and it works as intended.

Anyone else solve code subconsciously ? 😂😂😁😁

Perhaps more of a wishlist than what I think will actually happen, but:

- Everyone realises that blockchain is nothing more than a tiny niche, and therefore everyone but a tiny niche shuts up about it.
- Starting a new JS framework every 2 seconds becomes a crime. Existing JS frameworks have a big war, until only one is left standing.
- Developing for "FaaS" (serverless, if I must use that name) type computing becomes a big thing.
- Relational database engines get to the point where special handling of "big data" isn't required anymore. Joins across billions of rows doesn't present an issue.
- Everyone wakes up one day and realises that Wordpress is a steaming pile of insecure cow dung. It's never used again, and burns in a fire.

Store POJOs as json inside a database column so that we have a dynamic relational database!

For those who don't know what a POJO
*POJO: plain old Java object
Technically I was asked to store all data models as json in a single column 🙄

Today at 'Derp & Co' a fellow co-worker decided that had duplicated data on Relational DB is good!

- Dev: 'but what if we have 2 companies in diferents groups?''

- co-worker: 'Just call it company A and Company B'

- Dev: 'but... this is not what...'

- co-worker: 'Trust me Dev, is the easy way'

I want the professional way, not the easy (and damn wrong) way :(, I can't improve myself like this.

Also, dead line is here too... TT^TT
Last sprint and still with doubts about the DB structure.

I see a trend for new devs to want to focus on JavaScript while dismissing relational databases and backend languages as boring and to rigid. A week into real jobs they want help with data and .net.

I was talking to a friend about the current state of machine learning through tensorflow and commented about the use of Javascript as a language.

He discarded the idea as he views Javascript as something that should only be used as a frontend technology rather than something to build backends or deep learning models.

I am thorn. I have always liked Javascript but will admit that I have used it mostly in the area of front end with very few backend instances(i did create a full stack intranet app in Express once, major success for the application it was hosting, it was a very basic api which had its own nosql db with no need to interact with the company's relational data, it was perfect for the occasion and still help maintaining it from time to time)

My boi states that node's biggest issue has always been npm and the quality of packages. I always contradict those statements by saying that if one uses community standards and the best packages then one does not need to worry about the quality(i.e mongoose over some unmaintained mongo wrapper etc)

I sometimes catch myself finding that my way of thinking adapts better to JS than it even does Python (which is his preference for deep learning) and whilst there are some beastly packages for python in terms of quality and usefulness such as matplotlib etc that one can do great things with the equivalent JS.

I mean, tensorflow.js came from the same wizards that did tensorflow (obviously) and i find the functional approach of JS to be more on par with how we develop solutions.

I am no deep learning expert, and sadly I have no professional experience with machine learning. But I venture to say that we should not cast aside the great strides that the JS community has done to the language in terms of evolution and tooling. Today's Js is not your grandaddy's Js and thinking that the language is crippled because of early iterations of the language would be severely biased.

What do you guys(maybe someone with professional experience) think of Js as a language for machine learning?

Do you think the language poses something worth considering in terms of tooling and power for ml?

Spending hours upon hours trying to understand C# entity framework relational data mappings, going to the extreme of looking at LINQ queries, writing and rewriting over 100 lines of code only to find I needed to write the joins in dot notation on the include!

¡rant|rant

Nice to do some refactoring of the whole data access layer of our core logistics software, let me tell an story.

The project is around 80k lines of code, with a lot of integrations with an ERP system and an sql database.

The ERP system is old, shitty api for it also, only static methods through an wrapper to an c++ library

imagine an order table.

To access an order, you would first need to open the database by calling Api.Open(...file paths) (yes, it's an fucking flat file type database)

Now the database is open, now you would open the orders table with method Api.Table(int tableId) and in return you would get an integer value, the pointer.

Now for the actual order. first you need to search for it by setting the search parameter to the column ID of the order number while checking all calls for some BS error code

Api.SetInt(int pointer, int column, int query Value)
Then call the find method.

Api.Find(int pointer)

Then to top this shitcake of an api of: if it doesn't find your shit it will use the "close enough" method of search.

And now to read a singe string 😑

First you will look in the outdated and incorrect documentation given to you from the devil himself and look for the column ID to find the length of the column.

Then you create a string variable with ALL FUCKING SPACES.

Now you call the Api.GetStr(int pointer, int column, ref string emptyString, int length)

Now you have passed your poor string to the api's demon orgy by reference.

Then some more BS error code checking.

Now you have read an string value 😀

Now keep in mind to repeat these steps for all 300+ columns in the order table.

News from the creators: SQL server? yes, sql is good so everything will be better?

Now imagine the poor developers that got tasked to convert this shitcake to use a MS SQL server, that they did.

Now I can honestly say that I found the best SQL server benchmark tool. This sucker creams out just above ~105K sql statements per second on peak and ~15K per second for 1.5 second to read an order. 1.5 second to read less than 4 fucking kilobytes!

Right at that moment I released that our software would grind to an fucking halt before even thinking about starting it. And that me & myself and I would be tasked to fix it.

4 months later and two weeks until functional beta, here I am. We created our own api with the SQL server 😀

And the outcome of all this...

Fixes bugs older than a year, Forces rewriting part of code base. Forces removal of dirty fixes. allows proper unit and integration testing and even database testing with snapshot feature.

The whole ERP system could be replaced with ~10 lines of code (provided same relational structure) on the application while adding it to our own API library.

Best part is probably the performance improvements 😀. Up to 4500 times faster and 60 times less memory usage also with only managed memory.

For all the hate that Java gets, this *not rant* is to appreciate the Spring Boot/Cloud & Netty for without them I would not be half as productive as I am at my job.

Just to highlight a few of these life savers:
- Spring security: many features but I will just mention robust authorization out of the box
- Netflix Feign & Hystrix: easy circuit breaking & fallback pattern.
- Spring Data: consistent data access patterns & out of the box functionality regardless of the data source: eg relational & document dbs, redis etc with managed offerings integrations as well. The abstraction here is something to marvel at.
- Spring Boot Actuator: Out of the box health checks that check all integrations: Db, Redis, Mail,Disk, RabbitMQ etc which are crucial for Kubernetes readiness/liveness health checks.
- Spring Cloud Stream: Another abstraction for the messaging layer that decouples application logic from the binder ie could be kafka, rabbitmq etc
- SpringFox Swagger - Fantastic swagger documentation integration that allows always up to date API docs via annotations that can be converted to a swagger.yml if need be.
- Last but not least - Netty: Implementing secure non-blocking network applications is not trivial. This framework has made it easier for us to implement a protocol server on top of UDP using Java & all the support that comes with Spring.

For these & many more am grateful for Java & the big big community of devs that love & support it.

RethinkDB is such a rediculous overengineered BIGGEST BULLSHIT I HAVE EVER UNFORTUNATELY USED.

Does anyone even use this total shit????

This shit eats RAM memory for just 1 CRUD operation as if you opened 10,000 google chrome tabs. Who the fuck thought that kind of technology is a good idea?

Yes it IS very fast, a real time database. But you'd have to have a multi-million dollar supercomputer to be able to handle so much data like a relational database can....

There's very little good use cases for mongo, change my mind.

Prototyping maybe? Rails can prototype, create/update/destroy db schemas really quickly anyways.

If you're doing a web app, there's tons of libs that let you have a store in your app, even a fake mongo on the browser.

Are the reads fast? When I need that, use with redis.

Can it be an actual replacement for an app's db? No. Safety mechanisms that relational dbs have are pretty much must haves for a production level app.
Data type checks, null checks, foreign key checks, query checks.
All this robustness, this safety is something critical to maintain the data of an app sane.

Screw ups in the app layer affecting the data are a lot less visible and don't get noticed immediately (things like this can happen with relational dbs but are a lot less likely)

Let's not even get into mutating structures. Once you pick a structure with mongo, you're pretty much set.
Redoing a structure is manual, and you better have checks afterwards.

But at the same time, this is kind of a pro for mongo, since if there's variable data, as in some fields that are not always present, you don't need to create column for them, they just go into the data.

But you can have json columns in postgres too!

Is it easier to migrate than relational dbs? yes, but docker makes everything easy also.

First rant here...
Hand full of devs have to create a huge web platform that can shovel a lot of data around in about two months which is impossible...
Project lead has left major decisions in the hands of interns like database we want to use because no question can.be answered by that person. Inexperienced intern has chosen a fucking nosql database for highly relational datasets... why? Because new tech...
Development began and a bunch of problems arised... database was accessable from internet from day one. Random crashes because out of memory exceptions. Every possible feature had a description of at most 10 words... and no standards where enforced on anything.

Now that finaaaally we switch to sql after almost a year of prototypical production everybody keeps coding on new features so i have to port all the crap to the new database...

best part: a bunch of clients on different op systems have to be ported as well!

Even better part: i have to do that cause everybody else has practically no experience in any field...

And now the joke: i got hired for gui/desktop application development

Am i a wizard now?

Junior Software Developer Job( $37k-$42k USD)

-1 year experience
- J2EE, Javascript, HTML, XML, SQL
- object oriented design and implementation
- management of relational and non-relational such as Oracle, PostGreSQL and Cassandra
- Lifecycle and Agile methods
- Familiarity with the Eclipse development environment and with tools such as Hibernate, JMS, ,TomCat/Gemini/Jetty, OSGi.
• UNIX skills, including Bash or other scripting language
• Experience installing and configuring software packages
• ActiveMQ troubleshooting/knowledge
• Experience in scientific data processing and analytical science in general
• Automated testing tools and procedures, including JUnit testing, Selenium, etc.
• Experience in interfacing with scientific instrumentation, potentially over IP networks
• Familiarity with modern web development, user interface and other ever-evolving front-end
technologies, such as React, TypeScript, Material, Jest, etc.

I am betting they don't get many people applying.

Has anyone here used MongoDB for relational databases? I am trying to use MySQL with Node but I feel like the whole thing is a mess. There are some ORMs like Knex or Sequelize for MySQL but they are pretty crap. I am thinking if I should just move onto MongoDB instead of MySQL, at the same time I feel a little dirty to store relational data in a document store :(

FUCK YOU PHP, FUCK YOU SYMFONY AND DEFINITELY FUCK YOU SHOPWARE.

Don't get me wrong, PHP has evolved a lot, but the stuff people are building with it is just the biggest load of fucking shit I have ever seen: Shopware. Shopware is the most ass-sucking abomination to extend. It's nearly impossible to develop anything beyond "use the standard features and shut the fuck up" that is more sophisticated than a fucking calculator.

The architecture of this pile of crap is the worst bullshit ever. A mix of OOP, randomly making use of non OOP concepts and features together with the unnecessarily HUGE amount of useless interfaces and classes. Sometimes I feel like it's 90% fucking shitty boilerplate shit.

And don't get me started with TWIG. It's a nice thought, but WHY THE BLOODY FUCK WOULD YOU NOT USE VUE IF YOU ARE ALREADY USING IT FOR A DIFFERENT PART OF SHOPWARE. This makes no fucking sense whatsoever and makes development of new features a huge pain in the ass. I can't comprehend how people actually like using this shit.

OH AND THE DATABASE. OH MY FUCKING GOD. This one is bad. Ever tried to figure anything out in a database where random strings (yes MySQL "relational" - you might think) that are stored as text in a JSON format make up some object or relations during runtime?? Why the fuck do you have foreign and primary keys if you don't use them properly??

Seriously you can't even figure out which data belongs to what because the architecture just sucks fucking ass. FUCK YOU Shopware wankers, you suck, your product sucks, your support sucks, your architecture sucks and you keep releasing new versions that regularly break shit even in minor versions.

I used to like PHP, but not in projects like these.

To me this is one of the most interesting topics. I always dream about creating the perfect programming class (not aimed at absolute beginners though, in the end there should be some usable software artifact), because I had to teach myself at least half of the skills I need everyday.

The goal of the class, which has at least to be a semester long, is to be able to create industry-ready software projects with a distributed architecture (i.e. client-server).

The important thing is to have a central theme over the whole class. Which means you should go through the software lifecycle at least once.

Let's say the class consists of 10 Units à ~3 hours (with breaks ofc) and takes place once a week, because that is the absolute minimum time to enable the students to do their homework.

1. Project setup, explanation of the whole toolchain. Init repositories, create SSH keys for github/bitbucket, git crash course (provide a cheat sheet).
Create a hello world web app with $framework. Run the web server, let the students poke around with it. Let them push their projects to their repositories.
The remainder of the lesson is for Q&A, technical problems and so on.

Homework: Read the docs of $framework. Do some commits, just alter the HTML & CSS a bit, give them your personal touch.
For the homework, provide a $chat channel/forum/mailing list or whatever for questions where not only the the teacher should help, but also the students help each other.

2. Setup of CI/Build automation. This is one of the hardest parts for the teacher/uni because the university must provide the necessary hardware for it, which costs money. But the students faces when they see that a push to master automatically triggers a build and deploys it to the right place where they can reach it from the web is priceless.
This is one recurring point over the whole course, as there will be more software artifacts beside the web app, which need to be added to the build process. I do not want to go deeper here, whether you use Jenkins, or Travis or whatev and Ansible or Puppet or whatev for automation. You probably have some docker container set up for this, because this is a very tedious task for initial setup, probably way out of proportion. But in the end there needs to be a running web service for every student which they can reach over a personal URL. Depending on the students interest on the topic it may be also better to setup this already before the first class starts and only introduce them to all the concepts in a theory block and do some more coding in the second half.

Homework: Use $framework to extend your web app. Make it a bit more user interactive with buttons, forms or the like. As we still have no backend here, you can output to alert or something.

3. Create a minimal backend with $backendFramework. Only to have something which speaks with the frontend so you can create API calls going back and forth. Also create a DB, relational or not. Discuss DB schema/model and answer student questions.

Homework: Create a form which gets transformed into JSON and sent to the backend, backend stores the user information in the DB and should also provide a query to view the entry.

4. Introduce mobile apps. As it would probably too much to introduce them both to iOS and Android, something like React Native (or whatever the most popular platform-agnostic framework is then) may come in handy. Do the same as with the minimal web app and add the build artifacts to CI. Also talk about getting software to the app/play store (a common question) and signing apps.

Homework: Use the view API call from the backend to show the data on the mobile. Play around with the mobile project to display it in a nice way.

5. Introduction to refactoring (yes, really), if we are really talking about JS here, mention things like typescript, flow, elm, reason and everything with types which compiles to JS. Types make it so much easier to refactor growing codebases and imho everybody should use it.
Flowtype would make it probably easier to get gradually introduced in the already existing codebase (and it plays nice with react native) but I want to be abstract here, so that is just a suggestion (and 100% typed languages such as ELM or Reason have so much nicer errors).
Also discuss other helpful tools like linters, formatters.

Homework: Introduce types to all your API calls and some important functions.

6. Introduction to (unit) tests. Similar as above.
Homework: Write a unit test for your form.

(TBC)

MongoDB database with really relational data. One main collection that had refs to four other collections, all of those references necessary to populate data for a page view. Complicated aggregate to populate all the necessary data and then filter based on criteria selected by the user. And then the client decides that he wants the information to be sortable by column. Some of those columns are fields on the main model, no problem. Others are fields on the refs, which is more of a problem. Especially given that these refs aren’t one single object. They’re arrays of objects.

The revelation was that I could just write an aggregate function to flat map the main collection, returning only the fields necessary for the search, and output it to a new collection and instead use that new collection for displaying and filtering/sorting search results.

But you can’t run the aggregate all the time, you surely say. If anything changes in the main collection, it won’t be reflected in the search results!

Mongoose post(‘findOneAndUpdate’) hooks, my friends. Mongoose post(‘findOneAndUpdate’) hooks.

Never been so happy to have a thing working properly in my life.

Q: What do you get when you create a homebrew query language that uses both the stream oriented principles of Unix data pipes and the relational ideas underlying an RDBMS and use incomplete documentation to support it?

A: A frustrated borderline homicidal engineer.

After a year of using mongo in prod and personal projects I have realised some things. Its really nice early on the project, especially when there are changing requirements and for small projects or proof of concepts.
But when you make commercial software things tend to get more complex and relational. Stakeholders want reporting and even a report building which a document store isn't the best at.
With most projects projects when they get big things get relational and this becomes more and more expensive to handle in terms of compute power and developer time.
I don't doubt mongo has its place, maybe as an secondary specialised data store or if the project is inherently document oriented.
Blog over.

Our app has web scale data. So let me take all my tables from a relational database and map them one to one to a collection in a document database.

I've been dealing with a guy who knows almost nothing about tech and always insists on asinine systems and buzz words he's been sold by 3rd party vendors. Lately he really wants to replace our relational dbs with hdfs.

Our data is sub 10 gb in size.

I've been seriously considering making a contracting company and just preying on tech fools like this, I just don't know if I could ethically swallow it.

I just look at a layout like this and see a relational database. Because minus random markers, there is a defined set of relationships some of which can be inferred or taken from OTHER data like.

"Joe travels at 8 am +/- 1 hr 99% of the time, every day of the work week for the last 52 weeks, likely joe is commuting to this location"

or you could just add a schedule table and one item could be marked commute vs a log table of data that is actually happening.

With everything else I see the same things.

I also see a possibility for graph edges and the likej to get out of control really quickly when you start adding event data into it.

so what is the use of graph and whats its really offering ?

any data worthwhile is likely going to have some kind of structure, even if you add ad hoc fields that don't exist, after enough additions those fields should be standardized !

i don't like devs who put serialized data into a relational database. guys, do you even know how to relational database? 😭😭😭

Just notice that google spreadshit can do query like actual sql query.

Wtf each cell is like data as in relational databse.

Wtf wtf wtf.... soooooo cooool

My next db spreadsheet.

Storing de-normalized data in NoSql when used to relational data feels like I am vomiting on the database

I had a discussion with my colleagues about my bachelor thesis.

Together we created within the last 18 month a REST-API where we use LDAP/LMDB as database (tree structured storage). Of course our data is relational and of course we have a high redundancy there. It's a 170 call API and I highly doubt that it's actually conforming REST.
Ensuring DB integrity is done in the backend and coding style there is "If we change it at one place, let's make sure to also change it everywhere else", so you get a good impression how much of spaghetti code we have there.

Now I proposed to code a solution in my bachelor thesis where we use a relational database (we even have an administrated Oracle DB with high availability) and have a write-only layer to also store the data in LDAP but my colleagues said that "it would add too much complexity to the system".
Instead I should write the relational layer myself and fetch the data somehow from the existing LDAP tree.

What the actual fuck, spaghetti code is what makes the system really unnecessarily complex so that no one will understand that code in 2 years.

Congratulations, you just created legacy code that went into production in 2018 while not accepting the opportunity to let that legacy code get eliminated.
Now good luck with running and maintaining that system and it's inconsistencies.

A guy who thought the only way to access data was through a relational data base and the only way to access a data base from java was JPA

Best eays to improve database performance?
I have a huge database table with a lot of data, 10+columns and around 5-6million rows and growing. We are thinking about optimisation methods, the data will be frequently accessed, but not in the same ways. even though caching is helping, some initial requests take 6-20secs. What are good ways to improve a relational db performance? The data is date and item based, every item can have some data back from 2016 or older and up to 1000 related rows in each day. Every day new data would be added. Right now working with php.

I really want to switch my career from being a Full-Stack python/javascript developer to be a Data Engineer.

I've already worked with relational and non-relational databases, troubleshooted a couple of Airflow DAGs, deployed production-ready python code but now I feel kinda lost, every course I start on the Data engineering topic feels really useless since I feel like I've already worked with that technology/library, but I'm still afraid of start taking interviews.

Any good book/course or resource that I should look in?

BTW first rant in a couple of years, this brings me memories

Account service needs migrating, to AWS cause thats where everything is going.

Manager has got it in her head that a document store would be ideal for this.
My knee jerk reaction was a big No, i was told we'd discuss this at a later time.

My main argument here is that data is inheritly relational, and now i'm looking for more.

Any ideas why a documentstore is not a good fit for accounts?
Thanks!