As a developer, sometimes you hammer away on some useless solo side project for a few weeks. Maybe a small game, a web interface for your home-built storage server, or an app to turn your living room lights on an off.

I often see these posts and graphs here about motivation, about a desire to conceive perfection. You want to create a self-hosted Spotify clone "but better", or you set out to make the best todo app for iOS ever written.

These rants and memes often highlight how you start with this incredible drive, how your code is perfectly clean when you begin. Then it all oscillates between states of panic and surprise, sweat, tears and euphoria, an end in a disillusioned stare at the tangled mess you created, to gather dust forever in some private repository.

Writing a physics engine from scratch was harder than you expected. You needed a lot of ugly code to get your admin panel working in Safari. Some other shiny idea came along, and you decided to bite, even though you feel a burning guilt about the ever growing pile of unfinished failures.

All I want to say is:

No time was lost.

This is how senior developers are born. You strengthen your brain, the calluses on your mind provide you with perseverance to solve problems. Even if (no, *especially* if) you gave up on your project.

Eventually, giving up is good, it's a sign of wisdom an flexibility to focus on the broader domain again.

One of the things I love about failures is how varied they tend to be, how they force you to start seeing overarching patterns.

You don't notice the things you take back from your failures, they slip back sticking to you, undetected.

You get intuitions for strengths and weaknesses in patterns. Whenever you're matching two sparse ordered indexed lists, there's this corner of your brain lighting up on how to do it efficiently. You realize it's not the ORMs which suck, it's the fundamental object-relational impedance mismatch existing in all languages which causes problems, and you feel your fingers tingling whenever you encounter its effects in the future, ready to dive in ever so slightly deeper.

You notice you can suddenly solve completely abstract data problems using the pathfinding logic from your failed game. You realize you can use vector calculations from your physics engine to compare similarities in psychological behavior. You never understood trigonometry in high school, but while building a a deficient robotic Arduino abomination it suddenly started making sense.

You're building intuitions, continuously. These intuitions are grooves which become deeper each time you encounter fundamental patterns. The more variation in environments and topics you expose yourself to, the more permanent these associations become.

Failure is inconsequential, failure even deserves respect, failure builds intuition about patterns. Every single epiphany about similarity in patterns is an incredible victory.

Please, for the love of code...

Start and fail as many projects as you can.

Still trying to get good.

The requirements are forever shifting, and so do the applied paradigms.

I think the first layer is learning about each paradigm.

You learn 5-10 languages/technologies, get a feeling for procedural/functional/OOP programming. You mess around with some electronics engineering, write a bit of assembly. You write an ugly GTK program, an Android todo app, check how OpenGL works. You learn about relational models, about graph databases, time series storage and key value caches. You learn about networking and protocols. You void the warranty of all the devices in your house at some point. You develop preferences for languages and systems. For certain periods of time, you even become an insufferable fanboy who claims that all databases should be replaced by MongoDB, or all applications should be written in C# -- no exceptions in your mind are possible, because you found the Perfect Thing. Temporarily.

Eventually, you get to the second layer: Instead of being a champion for a single cause, you start to see patterns of applicability.

You might have grown to prefer serverless microservice architectures driven by pub/sub event busses, but realize that some MVC framework is probably more suitable for a 5-employee company. You realize that development is not just about picking the best language and best architecture -- It's about pros and cons for every situation. You start to value consistency over hard rules. You realize that even respected books about computer science can sometimes contain lies -- or represent solutions which are only applicable to "spherical cows in a vacuum".

Then you get to the third layer: Which is about orchestrating migrations between paradigms without creating a bigger mess.

Your company started with a tiny MVC webshop written in PHP. There are now 300 employees and a few million lines of code, the framework more often gets in the way than it helps, the database is terribly strained. Big rewrite? Gradual refactor? Introduce new languages within the company or stick with what people know? Educate people about paradigms which might be more suitable, but which will feel unfamiliar? What leads to a better product, someone who is experienced with PHP, or someone just learning to use Typescript?

All that theoretical knowledge about superior paradigms won't help you now -- No clean slates! You have to build a skyscraper city to replace a swamp village while keeping the economy running, together with builders who have no clue what concrete even looks like. You might think "I'll throw my superior engineering against this, no harm done if it doesn't stick", but 9 out of 10 times that will just end in a mix of concrete rubble, corpses and mud.

I think I'm somewhere between 2 and 3.

I think I have most of the important knowledge about a wide array of languages, technologies and architectures.

I think I know how to come to a conclusion about what to use in which scenario -- most of the time.

But dealing with a giant legacy mess, transforming things into something better, without creating an ugly amalgamation of old and new systems blended together into an even bigger abomination? Nah, I don't think I'm fully there yet.

New to this but here's my rant I suppose. It bothers me how "non-tech" people kind of devalue what tech people do. Like they have zero understanding of it, so you make something in 30 minutes or an hour that took years of building said skills and involved complex logic and understanding of relational data and because it only took you 30 minutes or an hour it must have been "easy". Or the way you are everybody's free tech advisor with family and friends... And things are said like "I'm not good with this stuff, but you're so good with it". For the record nobody is just "great" at technology or coding from birth its been a 2+ decade craft that I've experimented with and learned and put effort into. So taking into consideration all this effort I have put in to understand all this you say you'll never remember to push that button so you'll just ask me again when the problem arises. Yes because its so fun for me to constantly maintain your electronics because you can't bother to remember to push a button.

I've seen several rants about dumb/useless teachers, college and the CS degree studies; today is a good day to vent out some "old" memories.

Around two semesters ago I enrolled in a Database seminar with this guy, a tall geek from the 80's with a squeaky voice, so squeaky mice could had an aneurysm if they listened to him.

Either way this guy was a mess, he said he was an awesome coder, that we were still "peasants" when it came to coding, that relational databases had nothing on him since he was an awesome freelancer and did databases every day, that we had to redo the programming course with him and with his shitty, pulled out of the ass own C++ style guide with over 64 different redacted rules.

He gave us sample code of "how it should be done" in Java...it ain't my favorite language but fuck me a fucking donkey could have written better code with his ass!! He even rewrote Java's standard input function and made it highly inefficient. He still wrote in a structural paradigm in OOP languages! And he dared to make this code reviews were he would proyect someone's code and mock it in front of the class as he took off points, sometimes going to the negative realm (3,2,1,0,-1...)

But you know what's shittier? That he actually didn't even attend, 90% of the time, it was literally this:
> Good morning class
> Checks attendance. . .
> I'll be back, I'm going to check in...
> 1 hour 45 minutes later (class was 2 hrs long) - comes back
> do you have any doubts?
> O.o no...? I'm ok.
> We're done

Not only that, he scheduled from 4 to 17 homeworks throughout the week, I did the math, that was around 354 files from everyone; of course he didn't check them, other students from higher semesters did and they gained each point taken from students making students from lower semesters get the short end of the stick.

How did I pass? He didn't understood my code or database schema and he knew he couldn't fail me as he had no ground to stand on.

Thanks for listening, if you got to the end of this long ass post and had a similar experience I'd love to read it.

Mother fucking SQL, fuck mathematicians, fuck every thing!

So let's supose we'd only need the first char of a string. Every, and I mean fucking every (php, java, javascript, ruby, python, haskell) fucking language, uses something like `substring(input, 0, 1)` as it knows the input is nothing more than a fucking array of chars, otherwise known as motherfucking String. Logically the offset for the first char is 0.

Enter SQL, there you need to put `SUBSTRING(input, 1, 1)` because fuck every one! Fucking math guys who developed relational algebra on which (most) databases are based on (I love you for it, but come on you fuckers!), Decided that the first character should be at position 1...

Fuckers

Rant/story

Ok, I've always respected my PM and took everything on me, but since a while I start getting bored at work and realised many wrong things with the company and management in general.

So, brief contextual situation for you guys, I used to be very shy, unconfident and submissive. That was 2 years ago. Now am much more confident and got my own techniques in managing my constant "in the moon"-mind and relational discussions with colleagues. No more stuttering and am now answering on the spot and focussed on the discussion.

So I was having a nice day extinguishing fires on our website, this evening my PM stressly-rushed into my office (which I share with 2 other colleagues), and pressured me into giving a phone call to some developer for a situation clarification: a Json endpoints seems to truncate text after some characters.

Just came back from the loo (not sorry for the details), had my thoughts about something else, as usual, and I was just like "chill, let me get my mind together and prepare myself to be on point for this phone call". Told her I need a few seconds and she was like "now now now" knowing me I'm a bit laid-back.

Grabbed the phone, saw she was laughing (always laughs whatever I do, I must be very funny) and went talking about me to my colleague (not backstabbing but like "I don't get why he needs to get prepped for a phone call"). I managed the phone call like a boss - like usual since I got more confident -, my pm left, I finished the call, went to her to tell her my conclusions on our issue, asked me if I checked the contract with our CMS company.

Told her nope, the Json is compatible with our DB-manager's API.

She coldly answered "right, will do it myself then", I was like ok, I know you won't do it, I'll get it done.

In 15 minutes I found the contract, notified her, analyzed it, and wrote a technical email to support.

Seriously, stop taking me for some retarded person and let me breathe

Huh.

Ok now I'm gonna tell you about my "Databases 2" exam. This is gonna be long.

I'd like to know if DB designers actually have this workflow. I'm gonna "challenge" the reader, but I'm not playing smartass. The mistakes I point out here are MY mistakes.

So, in my uni there's this course, "Databases 2" ("Databases 1" is relational algebra and theoretical stuff), which consist in one exercise: design a SQL database.

We get the description of a system. Almost a two pages pdf. Of course it could be anything. Here I'm going to pretend the project is a YouTube clone (it's one of the practice exercises).

We start designing a ER diagram that describes the system. It must be fucking accurate: e.g. if we describe a "view" as a relationship between the entities User and Video, it MUST have at least another attribute, e.g. the datetime, even if the description doesn't say it. The official reason?

"The ER relationship describes a set of couples. You can not have two elements equal, thus if you don't put any attribute, it means that any user could watch a video only once. So you must put at least something else."

Do you get my point? In this phase we're not even talking about a "database", this is an analysis phase.

Then we describe the type dictionary. So far so good, we just have to specify the type of any attribute.

And now... Constraints.
Oh my god the constraints. We have to describe every fucking constraint of our system. In FIRST ORDER LOGIC. Every entity is a set, and Entity(e) means that an element e belongs to the set Entity. "A user must leave a feedback after he saw a video" becomes like

For all u,v,dv,df,f ( User(u) and Video(v) and View(u, v, dv) and feedback(u, v, f) ) ---> dv < df

provided that dv and df are the datetimes of the view and the feedback creation (it is clear in the exercise, here seems kinda cryptic)

Of course only some of the constraints are explicitly described. This one, for example, was not in the text. If you fail to mention any "hidden" constraint, you lose a lot of points. Same thing if you not describe it correctly.

Now it's time for use cases.
You start with the usual stickman diagram. So far so good.
Then you have to describe their main functions.

In first order logic. Yes.

So, if you got the point, you may think that the following is correct to get "the average amount of feedback values on a single video" (1 to 5, like the old YT).
(let's say that feedback is a relationship with attribute between User and Video

getAv(Video v): int
Let be F = { va | feedback(v, u, va) } for any User u
Let av = (sum forall f in F) / | F |
return av

But nope, there's an error here. Can you spot it (I didn't)?

F is a set. Sets do not have duplicates! So, the F set will lose some feedback values! I can not define that as a simple set!

It has to be a set of couples, like (v, u), where v is the value and u the user; this way we can have duplicate feedback values in our set.

This concludes the analysis phase. Now, the design.

Well we just refactor everything we have done until now. Is-a relations become relationships, many-to-many relationships get an "association entity" between them, nothing new.
We write down on paper every SQL statement to build any table, entity or not. We write down every possible primary key or foreign key. The constraint that are not natively satisfied by SQL and/or foreign keys become triggers, and so on.

This exam is considered the true nightmare at our department. I just love it.

Now my question is, do actually DB designers follow this workflow? Or is this just a bloody hard training in Pai Mei style?

So about two months ago in my consulting firm I was asked to replace a colleague on a project (node and Angular). The project is only a few months old but it’s already a total clusterfuck. DB is very poorly designed. It’s supposed to be a relational database but there’s not a trace of a foreign key or any key for that matter and I’ve seen joins like tableA.name = tableB.description (seriously, that’s your relation??). The code is a mess with entire blocks of code copied from another project and many parts of the code aren’t even used. He didn’t even bother renaming variables so they would make sense in the context they were shamelessly thrown into. The code is at best poorly typed if not typed at all.

During our dailies I sometimes express my frustration with my other colleagues as I very politely allude to my predecessor’s code as being hard to work with. (They are all “good friends" with him). I always get the same response from my colleagues: "yeah but you’ve gotta understand Billybob was under a lot of pressure. The user stories were not well defined. He didn’t have time to do a proper job". That type of response just makes me boil inside.

Because you think I have time to deal with this shit? You don’t think I’m working with the same client and his user stories that are barely intelligible? How long does it take to write type definitions for parameters going into a function? That’s right, 30 seconds at most? Maybe a minute if it’s a more elaborate object? How much time do you think you’ll save yourself with a properly typed function or better yet an interface? Hard to tell but certainly A LOT MORE than those 30 seconds you lost (no, the 30 seconds you INVESTED) in writing that interface!!!

FUCK people with their excuses! Never tell me you don’t have time to do a proper job! You’ve wasted HOURS of my time just because you were too fucking lazy to type your functions, too lazy to put just a little more thought into designing your tables, too lazy to rename a variable so that it’s name actually makes sense where it’s being used. It’s not because you were short on time. You’re just lazy!
FUCK!!!!!!

There's a local server with lots of processing power and plenty of GPUs and tons RAM going critically underused. Why? There's someone who's running a process using a relational database which literally performs 40 to 60 percentage disk I/O for months. It's so bad, if you run "ls" it may take a minute to run (only if you are lucky to ssh in).

¡rant|rant

Nice to do some refactoring of the whole data access layer of our core logistics software, let me tell an story.

The project is around 80k lines of code, with a lot of integrations with an ERP system and an sql database.

The ERP system is old, shitty api for it also, only static methods through an wrapper to an c++ library

imagine an order table.

To access an order, you would first need to open the database by calling Api.Open(...file paths) (yes, it's an fucking flat file type database)

Now the database is open, now you would open the orders table with method Api.Table(int tableId) and in return you would get an integer value, the pointer.

Now for the actual order. first you need to search for it by setting the search parameter to the column ID of the order number while checking all calls for some BS error code

Api.SetInt(int pointer, int column, int query Value)
Then call the find method.

Api.Find(int pointer)

Then to top this shitcake of an api of: if it doesn't find your shit it will use the "close enough" method of search.

And now to read a singe string 😑

First you will look in the outdated and incorrect documentation given to you from the devil himself and look for the column ID to find the length of the column.

Then you create a string variable with ALL FUCKING SPACES.

Now you call the Api.GetStr(int pointer, int column, ref string emptyString, int length)

Now you have passed your poor string to the api's demon orgy by reference.

Then some more BS error code checking.

Now you have read an string value 😀

Now keep in mind to repeat these steps for all 300+ columns in the order table.

News from the creators: SQL server? yes, sql is good so everything will be better?

Now imagine the poor developers that got tasked to convert this shitcake to use a MS SQL server, that they did.

Now I can honestly say that I found the best SQL server benchmark tool. This sucker creams out just above ~105K sql statements per second on peak and ~15K per second for 1.5 second to read an order. 1.5 second to read less than 4 fucking kilobytes!

Right at that moment I released that our software would grind to an fucking halt before even thinking about starting it. And that me & myself and I would be tasked to fix it.

4 months later and two weeks until functional beta, here I am. We created our own api with the SQL server 😀

And the outcome of all this...

Fixes bugs older than a year, Forces rewriting part of code base. Forces removal of dirty fixes. allows proper unit and integration testing and even database testing with snapshot feature.

The whole ERP system could be replaced with ~10 lines of code (provided same relational structure) on the application while adding it to our own API library.

Best part is probably the performance improvements 😀. Up to 4500 times faster and 60 times less memory usage also with only managed memory.

RethinkDB is such a rediculous overengineered BIGGEST BULLSHIT I HAVE EVER UNFORTUNATELY USED.

Does anyone even use this total shit????

This shit eats RAM memory for just 1 CRUD operation as if you opened 10,000 google chrome tabs. Who the fuck thought that kind of technology is a good idea?

Yes it IS very fast, a real time database. But you'd have to have a multi-million dollar supercomputer to be able to handle so much data like a relational database can....

To me this is one of the most interesting topics. I always dream about creating the perfect programming class (not aimed at absolute beginners though, in the end there should be some usable software artifact), because I had to teach myself at least half of the skills I need everyday.

The goal of the class, which has at least to be a semester long, is to be able to create industry-ready software projects with a distributed architecture (i.e. client-server).

The important thing is to have a central theme over the whole class. Which means you should go through the software lifecycle at least once.

Let's say the class consists of 10 Units à ~3 hours (with breaks ofc) and takes place once a week, because that is the absolute minimum time to enable the students to do their homework.

1. Project setup, explanation of the whole toolchain. Init repositories, create SSH keys for github/bitbucket, git crash course (provide a cheat sheet).
Create a hello world web app with $framework. Run the web server, let the students poke around with it. Let them push their projects to their repositories.
The remainder of the lesson is for Q&A, technical problems and so on.

Homework: Read the docs of $framework. Do some commits, just alter the HTML & CSS a bit, give them your personal touch.
For the homework, provide a $chat channel/forum/mailing list or whatever for questions where not only the the teacher should help, but also the students help each other.

2. Setup of CI/Build automation. This is one of the hardest parts for the teacher/uni because the university must provide the necessary hardware for it, which costs money. But the students faces when they see that a push to master automatically triggers a build and deploys it to the right place where they can reach it from the web is priceless.
This is one recurring point over the whole course, as there will be more software artifacts beside the web app, which need to be added to the build process. I do not want to go deeper here, whether you use Jenkins, or Travis or whatev and Ansible or Puppet or whatev for automation. You probably have some docker container set up for this, because this is a very tedious task for initial setup, probably way out of proportion. But in the end there needs to be a running web service for every student which they can reach over a personal URL. Depending on the students interest on the topic it may be also better to setup this already before the first class starts and only introduce them to all the concepts in a theory block and do some more coding in the second half.

Homework: Use $framework to extend your web app. Make it a bit more user interactive with buttons, forms or the like. As we still have no backend here, you can output to alert or something.

3. Create a minimal backend with $backendFramework. Only to have something which speaks with the frontend so you can create API calls going back and forth. Also create a DB, relational or not. Discuss DB schema/model and answer student questions.

Homework: Create a form which gets transformed into JSON and sent to the backend, backend stores the user information in the DB and should also provide a query to view the entry.

4. Introduce mobile apps. As it would probably too much to introduce them both to iOS and Android, something like React Native (or whatever the most popular platform-agnostic framework is then) may come in handy. Do the same as with the minimal web app and add the build artifacts to CI. Also talk about getting software to the app/play store (a common question) and signing apps.

Homework: Use the view API call from the backend to show the data on the mobile. Play around with the mobile project to display it in a nice way.

5. Introduction to refactoring (yes, really), if we are really talking about JS here, mention things like typescript, flow, elm, reason and everything with types which compiles to JS. Types make it so much easier to refactor growing codebases and imho everybody should use it.
Flowtype would make it probably easier to get gradually introduced in the already existing codebase (and it plays nice with react native) but I want to be abstract here, so that is just a suggestion (and 100% typed languages such as ELM or Reason have so much nicer errors).
Also discuss other helpful tools like linters, formatters.

Homework: Introduce types to all your API calls and some important functions.

6. Introduction to (unit) tests. Similar as above.
Homework: Write a unit test for your form.

(TBC)

MongoDB database with really relational data. One main collection that had refs to four other collections, all of those references necessary to populate data for a page view. Complicated aggregate to populate all the necessary data and then filter based on criteria selected by the user. And then the client decides that he wants the information to be sortable by column. Some of those columns are fields on the main model, no problem. Others are fields on the refs, which is more of a problem. Especially given that these refs aren’t one single object. They’re arrays of objects.

The revelation was that I could just write an aggregate function to flat map the main collection, returning only the fields necessary for the search, and output it to a new collection and instead use that new collection for displaying and filtering/sorting search results.

But you can’t run the aggregate all the time, you surely say. If anything changes in the main collection, it won’t be reflected in the search results!

Mongoose post(‘findOneAndUpdate’) hooks, my friends. Mongoose post(‘findOneAndUpdate’) hooks.

Never been so happy to have a thing working properly in my life.

I just look at a layout like this and see a relational database. Because minus random markers, there is a defined set of relationships some of which can be inferred or taken from OTHER data like.

"Joe travels at 8 am +/- 1 hr 99% of the time, every day of the work week for the last 52 weeks, likely joe is commuting to this location"

or you could just add a schedule table and one item could be marked commute vs a log table of data that is actually happening.

With everything else I see the same things.

I also see a possibility for graph edges and the likej to get out of control really quickly when you start adding event data into it.

so what is the use of graph and whats its really offering ?

any data worthwhile is likely going to have some kind of structure, even if you add ad hoc fields that don't exist, after enough additions those fields should be standardized !

My phone was stolen so that's why I wasn't getting here to rant about this, but today I had to make a back up and remembered.
adminer's export function doesn't export the tables in the correct order for import. Doesn't take foreign keys into account.
Dude, that's the whole point of a relational database, relations between tables; if you don't take that into account then what's the point?
Is this the same for the rest of database managers or is it just adminer? Please tell me this isn't normal.

I had a discussion with my colleagues about my bachelor thesis.

Together we created within the last 18 month a REST-API where we use LDAP/LMDB as database (tree structured storage). Of course our data is relational and of course we have a high redundancy there. It's a 170 call API and I highly doubt that it's actually conforming REST.
Ensuring DB integrity is done in the backend and coding style there is "If we change it at one place, let's make sure to also change it everywhere else", so you get a good impression how much of spaghetti code we have there.

Now I proposed to code a solution in my bachelor thesis where we use a relational database (we even have an administrated Oracle DB with high availability) and have a write-only layer to also store the data in LDAP but my colleagues said that "it would add too much complexity to the system".
Instead I should write the relational layer myself and fetch the data somehow from the existing LDAP tree.

What the actual fuck, spaghetti code is what makes the system really unnecessarily complex so that no one will understand that code in 2 years.

Congratulations, you just created legacy code that went into production in 2018 while not accepting the opportunity to let that legacy code get eliminated.
Now good luck with running and maintaining that system and it's inconsistencies.

So, my experience is all with relational DBs (mssql) mainly and this job is the first time I've had to deal with mongodb.

I'm using the default compass client and I'm struggling with just how shit it is.
- A default font size of 5px high which resets every time it starts.
- Total lack of keyboard shortcuts.
- Inconsistent expansion& folding behaviours
- No saving of aggregates/queries if you accidentally click on another collection.
- ittle bitty query window which is actually multi line but with no scrollbar...
The list goes on.

And mongodb, whoever thought JavaScript is an appropriate query language... It's not.
It's probably because I don't have enough experience with it but the mix of quotes and $ seems so random...

I'm taking a look at neo4j after working a lot of mongo. Takes some getting used to but I think I could use it well for my next project.

From a little bit heated discussion I want to extract this: One big pain in the ass is the human to computer interface. Maybe it's the natural vs. formal language divide, but there's a mismatch deeper than between object and relational models that no ORM can failingly fix.
The whole point of the discussion was on such a point where some wanted an interface more human friendly and I stubbornly insisted on the way it is simple for the computer system. Like not too much human messiness should invade machine. One argument sounded as if human words were like unicode code points which meaning doesn't depend on its representation.
That's raising red flags to me: Nonono, natural language is too messy, keep it out. This poor machine could have been so clean and well designed and we already stacked up so much entropy we still dare to call OS,..
Dunno, what's your stance? Still hoping that your shell one day will be able to process our poor standard English? Or do you think, like me, all those failed attempts show there's a gap you should not even touch?

Whenever I see an ORM that supports creating and transforming objects in bulk, I can't help but think about the poor misdirected users who forced it to do that. It's an Object-Relational Mapper. It maps objects. The whole concept isn't designed for bulk operations, the point is that you add logic to each and every record and convert your operations to SQL so that you never have to keep a lot of them in memory.