Each month my department compiles a 4M row 150 column data table for compliance with a federal agency. Before submitting, we check it against about 400 rules.

The existing system was simply 400 queries that ran in sequence, table-scanning 4M rows each time, taking upwards of 6 hours, which is a huge bottleneck, especially if you have to make changes and rerun. Plus the output was rather one-dimensional.

I built a proper normalized database and created a sort of rules engine, running all 400 rules in one table scan. Not only does it complete in 30 minutes, but the reports generate automatically, and the results can be filtered on several dimensions to aid with root-cause analysis.

Management was pleased.

I'm the only programmer for my employer. Today I finished a GUI and engine to let our managers easily post annoucments to our TVs.
Nothing crazy; PHP login screen, some forms, small database, and the app to run on the TVs.
When I told my manager it was up and running she asks "Since you can build that, can you start working on a tool to run voice analytics off of our agents phone calls?".
Sure.. I'll get right on that.. Follow up with me in 4 years.

A dev team has been spending the past couple of weeks working on a 'generic rule engine' to validate a marketing process. The “Buy 5, get 10% off” kind of promotions.

The UI has all the great bits, drop-downs, various data lookups, etc etc..

What the dev is storing the database is the actual string representation FieldA=“Buy 5, get 10% off” that is “built” from the UI.

Might be OK, but now they want to apply that string to an actual order. Extract ‘5’, the word ‘Buy’ to apply to the purchase quantity rule, ‘10%’ and the word ‘off’ to subtract from the total.

Dev asked me:

Dev: “How can I use reflection to parse the string and determine what are integers, decimals, and percents?”
Me: “That sounds complicated. Why would you do that?”
Dev: “It’s only a string. Parsing it was easy. First we need to know how to extract numbers and be able to compare them.”
Me: “I’ve seen the data structures, wouldn’t it be easier to serialize the objects to JSON and store the string in the database? When you deserialize, you won’t have to parse or do any kind of reflection. You should try to keep the rule behavior as simple as possible. Developing your own tokenizer that relies on reflection and hoping the UI doesn’t change isn’t going to be reliable.”
Dev: “Tokens!...yea…tokens…that’s what we want. I’ll come up with a tokenizing algorithm that can utilize recursion and reflection to extract all the comparable data structures.”
Me: “Wow…uh…no, don’t do that. The UI already has to map the data, just make it easy on yourself and serialize that object. It’s like one line of code to serialize and deserialize.”
Dev: “I don’t know…sounds like magic. Using tokens seems like the more straightforward O-O approach. Thanks anyway.”

I probably getting too old to keep up with these kids, I have no idea what the frack he was talking about. Not sure if they are too smart or I’m too stupid/lazy. Either way, I keeping my name as far away from that project as possible.

Top 10 database engines on db-engine... I never knew the top 3 were so far ahead of the game, if that site is anything to go by.

So I started a 80hour intership today at our Department of Education and this is how it went...

Boss : Design a database to show all applicants that applied for jobs at schools.

*I start thinking which tables and columns I'm going to use and start designing the database, writes out all the tables on paper*

Me : Is there a pc I should work on or should I use my laptop? And which database engine do you use?

B : No you can use your laptop. And btw we use MS Access

*Thinking wtf kind of business even governmental uses Access for their databases. But anyway, start creating the databases and relationships when my boss walks in*

B : No, what are you doing?

M : Im creating the database you asked for.

B : No, you design it on paper. Draw all the tables, draw the report and the form then you come show it to me, if I decide its good enough you can come in tomorrow and start creating it.

*Wtf kind of place is this, are you mentally retarded? You have a IT staff of 3 people, in which only the actual fulltime intern is a qualified IT professional, but when me or him tries to do some actual work, you give us shit about doing what was asked from us*

Everything about the company is a mess. The only thing that is decent is the people. And by that I mean they aren't shit.
Workflows are fucked.
Clients are fucked. You're pressuring me to get this shit production ready before new year's eve and you still don't know what the text should say and want to make changes to the UI? The fuck?!
Design is a complete shit show. There is a design team. They only make a fucking psd to show clients how an interface would look like. No mobile version (but it's still expected to work!), no markup. Resolution is fucking inconsistent and whenever a change is requested, they are nowhere to be seen so I have to actually do designing on top of having to use this worthless fucking framework I hate it so much.
Codebases are turbo-fucked because of said framework.
Databases are an inconsistent, fucked up mess. No foreign key constraints because every single fucking table is using the MyISAM engine.

And the thing that really makes me incredibly angry is all the "custom systems" look the fucking same at the database level. Like 30 fucking useless tables made for stupid HR workflows that make no fucking sense.

I just finished designing an entire asset management pipeline and christ on a fucking pogo stick, if it isn't convoluted.

Theres a lot of game engines out there, but all of them do it a little different. They all tackle a slightly different problem, without even realizing it.

1. asset management

2. asset change management

3. behavior change management

4. data management

5. combinatorial design management.

6. Combinatorial Behavior management

7. Feature completion

ASSET MANAGEMENT is exactly what it says on the tin.

ASSET CHANGE management can be thought of handling the import, export, formatting, platform specific packing, and versioning (including forking) of an asset.

BEHAVIORAL CHANGE management is a subset of asset management, because code is a subset of assets (depending on how you define 'assets'). The oldest known example of this is commenting and uncommenting code.
Or worse, printf debugging.

This can be file versioning, basic undo services, graph management of forks and mergers, toggles for features or modules, etc.

DATA management is about anything that doesn't fall into the other categories, everything from mission text to npc dialogues, quests, location names, item stats, the works. Anything you'd be tempted to put in a database, falls under this category. Haven't yet seen many engines offer this as an explicit built in tool as of yet, because the other problems are non-trivial as is, so this is a bit of low hanging fruit that gets handled by external tools, or loaded from formats as simple as json.

COMBINATORIAL DESIGN management is the idea of prefabbing, blueprints of broader object design using nested prototypes of existing game objects, to create more complex, reusable set pieces. Unity did this well. GM does this in part.

COMBINATORIAL BEHAVIOR management is entity-component systems, plus tooling to make it easy to add, remove, and configure components and their values on entity blueprints, also not uncommon. Both stencyl and unity do this. GM has a precursor to this in the form of configurable fields, but these fields are not based on component scripts attached to objects.

FEATURE COMPLETION is that set of gameplay mechanics or styles of design that an engine naturally makes easier to include or build in a game.

I don't think I'm aiming for all that, but I think at minimum a good engine has to do asset management, behavioral change management, prefabs, and entity-component systems with management tools for that. And ideally, asset change management.

Sometimes lack of confidence in one area reveals oversight cockyness in stronger areas:

Set up a simple login system from Unity engine to php to mysql db, using android device ID as the login id. Set up database column to accept 32 length varchar for MD5 hashed strings, as I knew the method I was getting the android device ID was automatically being hashed that way and more or less was what I wanted anyway.

Spend 2 days wondering why it would insert the logins with 0 issue, but could never retrieve them. Due to lack of web development and PHP skills, I assumed I was screwing up the handling of mysqli_num_rows() (to check whether I was inserting or selecting in the query) or simply screwing up my SQL queries.

Rewrite the code a few times, even went back to a method I had used in the past.

Today it dawned on me that my testing machines deviceID had been getting trimmed to the 32 character limit. Turns out I didn't account for my workstations device ID to be automatically hashed like the android device id is.

For 2 days I was obtaining and sending a 40 character string to a 32 character limit varchar and blaming my lack of PHP skills........

Back to my niche I go!

1) Learning little to nothing useful in formal post-secondary and wasting tons of time and money just to have pain and suffering.

"Let's talk about hardware disc sectors divisions in the database course, rather than most of you might find useful for industry."

"Lemme grade based on regurgitating my exact definitions of things, later I'll talk about historical failed network protocols, that have little to no relevance/importance because they fucking lost and we don't use them. Practical networking information? Nah."

"Back in the day we used to put a cup of water on top of our desktops, and if it started to shake a lot that's how you'd know your operating system was working real hard and 'thrashing' "

"Is like differentiation but is like cat looking at crystal ball"

"Not all husbands beat their wives, but statistically...." (this one was confusing and awkward to the point that the memory is mostly dropped)

Streams & lambdas in java, were a few slides in a powerpoint & not really tested. Turns out industry loves 'em.

2) Landed my first student job and get shoved on an old legacy project nobody wants to touch. Am isolated and not being taught or helped much, do poorly. Boss gets pissed at me and is unpleasant to work with and get help from. Gets to the point where I start to wonder if he starts to try and create a show of how much of a nuisance I am. He meddle with some logo I'm fixing, getting fussy about individual pixels and shades, and makes a big deal of knowing how to use GIMP and how he's sitting with me micromanaging. Monthly one on one's were uncomfortable and had him metaphorically jerking off about his lifestory career wise.

But I think I learned in code monkey industry, you gotta be capable of learning and making things happen with effectively no help at all. It's hard as fuck though.

3) Everytime I meet an asshole who knows more and accomplish than I do (that's a lot of people) with higher TC than me (also a lot of people). I despair as I realize I might sound like that without realizing it.

4) Everytime I encounter one of my glaring gaps in my knowledge and I'm ashamed of the fact I have plenty of them. Cargo cult programming.

5) I can't do leetcode hards. Sometimes I suck at white board questions I haven't seen anything like before and anything similar to them before.

6) I also suck at some of the trivia questions in interviews. (Gosh I think I'd look that up in a search engine)

7) Mentorship is nigh non-existent. Gosh I'd love to be taught stuff so I'd know how to make technical design/architecture decisions and knowing tradeoffs between tech stack. So I can go beyond being a codemonkey.

8) Gave up and took an ok job outside of America rather than continuing to grind then try to interview into a high tier American company. Doubtful I'd ever manage to break in now, and TC would be sweet but am unsure if the rest would work out.

9) Assholes and trolls on stackoverflow, it's quite hard to ask questions sometimes it feels and now get closed, marked as dupe, or downvoted without explanation.

I was looking for alternatives of MC that are atleast usable, and found a thing called Minetest. This apparently is a Voxel Engine/Scriptable game, where you create games, that consist of mods/modules and other resources.
The cool part of it is, that mods and games etc. get handled by the game itself in a package manager type fashion, so the only thing you as a user have to do is selecting them in the ui, and putting them into your world.
It's this easy because the content is managed by a content database. This engine is built with multiplayer support by default.

Now comes the interesting part: apparently a few devs sat together and made a whole MC clone in this engine, and have called it Mineclone 2. I was testing it recently on a server and have to say, that it doesn't appear to be some low effort clone, but to my surprise is an actual playable and nicely looking game. So far i'm having fun with playing and even modding it.

Since the core is written in C++ and the mods and games content is written in LUA, you can easily writte new stuff for it, and even look at other mods stuff, to find out how to make it compatible or how to do certain things. The licenses usually allows to reuse and redistribute.

If you're looking for something like that, give Minetest + Mineclone2 a spin.

The Zen Of Ripping Off Airtable:
(patterned after The Zen Of Python. For all those shamelessly copying airtables basic functionality)

*Columns can be *reordered* for visual priority and ease of use.

* Rows are purely presentational, and mostly for grouping and formatting.

* Data cells are objects in their own right, so they can control their own rendering, and formatting.

* Columns (as objects) are where linkages and other column specific data are stored.

* Rows (as objects) are where row specific data (full-row formatting) are stored.

* Rows are views or references *into* columns which hold references to the actual data cells

* Tables are meant for managing and structuring *small* amounts of data (less than 10k rows) per table.

* Just as you might do "=A1:A5" to reference a cell range in google or excel, you might do "opt(table1:columnN)" in a column header to create a 'type' for the cells in that column.

* An enumeration is a table with a single column, useful for doing the equivalent of airtables options and tags. You will never be able to decide if it should be stored on a specific column, on a specific table for ease of reuse, or separately where it and its brothers will visually clutter your list of tables. Take a shot if you are here.

* Typing or linking a column should be accomplishable first through a command-driven type language, held in column headers and cells as text.

* Take a shot if you somehow ended up creating any of the following: an FSM, a custom regex parser, a new programming language.

* A good structuring system gives us options or tags (multiple select), selections (single select), and many other datatypes and should be first, programmatically available through a simple command-driven language like how commands are done in datacells in excel or google sheets.

* Columns are a means to organize data cells, and set constraints and formatting on an entire range.

* Row height, can be overridden by the settings of a cell. If a cell overrides the row and column render/graphics settings, then it must be drawn last--drawing over the default grid.

* The header of a column is itself a datacell.

* Columns have no order among themselves. Order is purely presentational, and stored on the table itself.

* The last statement is because this allows us to pluck individual columns out of tables for specialized views.

*Very* fast scrolling on large datasets, with row and cell height variability is complicated. Thinking about it makes me want to drink. You should drink too before you embark on implementing it.

* Wherever possible, don't use a database.
If you're thinking about using a database, see the previous koan.

* If you use a database, expect to pick and choose among column-oriented stores, and json, while factoring for platform support, api support, whether you want your front-end users to be forced to install and setup a full database,
and if not, what file-based .so or .dll database engine is out there that also supports video, audio, images, and custom types.

* For each time you ignore one of these nuggets of wisdom, take a shot, question your sanity, quit halfway, and then write another koan about what you learned.

* If you do not have liquor on hand, for each time you would take a shot, spank yourself on the ass. For those who think this is a reward, for each time you would spank yourself on the ass, instead *don't* spank yourself on the ass.

* Take a sip if you *definitely* wildly misused terms from OOP, MVP, and spreadsheets.

Without a doubt it has to be the internal company search engine/file finding tool @thewamz and I wrote.

The company has a wide UNC network with files scattered all over the place and they need a way to keep track of where the files get moved to (they can and do get moved). The original tool was written in Java/Tomcat and didn't use any frameworks or utilities beyond custom written ones, no orms, and the SQL was just raw strings. The program didn't take into account that files might be moved or deleted so it never removed anything from the database, it just kept adding files and never removing them.

It however never stores files itself, just links to files elsewhere on the UNC network.

It took six months to get it into what might be a stable beta or release candidate state. The user interface is good, very simple and intuitive, the whole thing was rewritten in python/django, there were issues with utf 8 (and mysql not fully supporting utf 8 in its own utf 8 mode), we added a regex search mode (which was sorely lacking), the search used to take up to fifteen minutes however we sped it up to less than a minute (worst case when a user simply puts "^$" as the regex search). It has a multi threaded design which does some checks to ensure it doesn't spawn too many threads and get stuck in constant Gil switching. Still some bugs to fix, like moving the processing of results returned by the server in a web worker so that the content widget doesn't lock up processing millions of search results and moving the back end to use asynchronous python might gain a performance boost. But on the whole I think the system is ready to replace the older system that all the users are frustrated with and constantly complain about.

However the annoying bit is... How to actually get the new system online, while I am responsible for the development of tools and their maintenance, I am not responsible for their initial deployment and that means I have no idea when (or even if) my new tool will even ever be released :/

Sometimes in our personal projects we write crazy commit messages. I'll post mine because its a weekend and I hope someone has a well deserved start. Feel free to post yours, regex out your username, time and hash and paste chronologically. ISSA THREAD MY DUDES AND DUDETTES
--
Initialization of NDM in Kotlin
Small changes, wiping drive
Small changes, wiping drive
Lottie, Backdrop contrast and logging in implementation
Added Lotties, added Link variable to Database Manifest
Fixed menu engine, added Smart adapter, indexing, Extra menus on home and Calendar
b4 work
Added branch and few changes
really before work
Merge remote-tracking branch 'origin/master'
really before work 4 sho
Refined Search response
Added Swipe to menus and nested tabs
Added custom tab library
tabs and shh
MORE TIME WASTED ON just 3 files
api and rx
New models new handlers, new static leaky objects xd, a few icons
minor changes
minor changesqwqaweqweweqwe
db db dbbb
Added Reading display and delete function
tryin to add web socket...fail
tryin to add web socket...success
New robust content handler, linked to a web socket. :) happy data-ring lol
A lot of changes, no time to explain
minor fixes ehehhe
Added args and content builder to content id
Converted some fragments into NDMListFragments
dsa
MAjor BiG ChANgEs added Listable interface added refresh and online cache added many stuff
MAjor mAjOr BiG ChANgEs added multiClick block added in-fragment Menu (and handling) added in-fragment list irem click handling
Unformatted some code, added midi handler, new menus, added manifest
Update and Insert (upsert) extension to Listable ArrayList
Test for hymnbook offline changing
Changed menuId from int to key string :) added refresh ...global... :(
Added Scale Gesture Listener
Changed Font and size of titlebar, text selection arg. NEW NEW Readings layout.
minor fix on duplicate readings
added isUserDatabase attribute to hymn database file added markwon to stanza views
Home changes :)
Modular hymn Editing
Home changes :) part 2
Home changes :) part 3
Unified Stanza view
Perfected stanza sharing
Added Summernote!!
minor changes
Another change but from source tree :)))
Added Span Saving
Added Working Quick Access
Added a caption system, well text captions only
Added Stanza view modes...quite stable though
From work changes
JUST a [ush
Touch horizontal needs fix
Return api heruko
Added bible index
Added new settings file
Added settings and new icons
Minor changes to settings
Restored ping
Toggles and Pickers in settings
Added Section Title
Added Publishing Access Panel
Added Some new color changes on restart. When am I going to be tired of adding files :)
Before the confession
Theme Adaptation to views
Before Realm DB
Theme Activity :)
Changes to theme Activity
Changes to theme Activity part 2 mini
Some laptop changes, so you wont know what changed :)
Images...
Rush ourd
Added palette from images
Added lastModified filter
Problem with cache response
works work
Some Improvements, changed calendar recycle view
Tonic Sol-fa Screen Added
Merge Pull
Yes colors
Before leasing out to testers
Working but unformated table
Added Seperators but we have a glithchchchc
Tonic sol-fa nice, dots left, and some extras :)))
Just a nice commit on a good friday.
Just a quickie
I dont know what im committing...

Chat Journal. A chat-based journal application.

An android app I built past month using flutter and flask for the server.

I started with flutter around 2 months ago. I believe it's the way mobile development is supposed to be. It's a treat for every mobile developer out there.

I used flask to build the server, database and even made my own analytics engine. Made an awaful lot of mistakes at the beginning but I think I'm improving at this day by day.

This probably is my biggest and definitely the coolest project so far. There's some saying
"If you have completed building 90% of something, there's 90% more to be built". It's called the 180% rule (or something similar) which literally signifies the difference between building something and building something well enough to be able to publish on the playstore. And this project taught me that.

How do you feel about not creating database tables for objects that only exist in relations?

For example, I have made a wiki engine. Because nothing on wiki pages can actually change, they aren't an entity. Revisions are an entity, and they refer to the title of the page which was changed. The same application also includes two non-version-controlled directed graphs between the pages (element of category and navigation log), which are represented by tables that link two titles. Of course the indexes are all set up so that it works like a foreign key would, but there is no Page or Article table.

Fuck Oracle, fuck you oracle! The stupidest shittiest worst nightmare company with the most user-unfriendly, productivity-killing, illogical, stupid pile of software garbage products ever! And unfortunately I want to extends my worm-fucks to all Oracle employees and maintainers and to the whole fucking community of shit that made up oracle-community and to every conscious being who ever liked, enjoyed or have found the slightest genuine interest of any product tagged "oracle".

I installed the pile of shit a.k.a Oracle 18c and imported a dumb file locally, everything was working in the slightest amount of the word (fine) before it turns to nightmare. I created a C# client to call a stored procedure in that shit of a database engine. I kept getting error related to the parameter types, specifically one which is custom type of Table of numbers. It turns out that the only of doing this is through that shit they called (unmanaged driver), the "managed" doesn't support custom types. So I had to install another package of shit they call (odbc universal install) "universal my a$$ by the way", at that moment, where everything just crashed and stopped working. I spent 3 hours trying to connect to the fucking database to no avail. I shockingly found a folder in my desktop folder called (OracleInstallation) and all windows services related to oracle installation "suddenly" got somehow (re-routed) to that folder.
In conclusion, fuck oracle.

will you just fucking upgrade to a version that supports wiredtiger or just FUCKING STOP OPERATION, STUPID FUCKTARDS. FUCK MLAB

What's the general Software Engineering rule of thumb again for frontend templating code?

If I look at certain websites, I notice some code smells in PHP such as:
$.modal = <?php echo $(base)["username"] != 'me' ?' ': echo 'style="display=none"' ?>

or just in general places in the code where PHP gets used as a templating engine for gluing together pieces of HTML code based on conditionals spread out over the codebase and the database itself too. To make things worse, this carries over to JavaScript ajax functions. As a developer, this to me just seems like spaghetticode.

On the other hand, many popular frameworks properly do templating, such as EJS, containing templating in one place and not mixing it with logic too much but just having simple output like <%= %>.

I know I've seen frameworks like Angular 1 contain pieces of HTML into directives, but maybe that's something different, more 'OO'-simulating or cleaner.

So... I've been thinking, I tend to default to LVM when trying to create easy-to-manage disk partitions, or when I want to backup a database without long lockings during a dump... Though, now... I got thinking.

What do you guys think, which is better in terms of functionality: BtrFS or LVM?

I know BtrFS offers such thing like full snapshots that allow to easily transfer just the increment over the snapshot origin off to a remote server for archival, but I never fully grew to trust btrfs as a server filesystem... Its...

Younger, and not as widespread, not to mention I don't know any performance statistics to recommend its use for this or that case (Like... Would a high-load database engine stutter flushing all those changes on disk while reading / writing temp tables and such)