Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "complex query"
-
0. Plan before you code. Document everything. You won't remember either your idea or those clever implementations next week (or next month, or next year...).
1. Don't hack your way through, unless that's what you intend to do. Name your variables, functions etc. neatly: autocomplete exists!
Protip: Sometimes you want to check a quick language feature or a piece of code from one of your modules. Resist the urge to quickly hack in the test into your actual project. Maintain a separate file where you can quickly type in and check what you're looking for without hacking on your project (For example, in Python, you can open a new terminal or IDLE window for those quick tests).
2. Keep a quiet environment where you can focus. Recommend listening to something while coding (my latest fad is on asoftmurmur.com). Don't let anything distract you and throw your contextual awareness out of whack.
3. Rubber ducks work. Really. Talking out a complex piece of logic, or that regex or SQL query aids your mind greatly in grasping the concept and clearing the idea. Bounce off code and ideas with a friend or colleague to catch errors and oversights faster. Read more here: https://en.wikipedia.org/wiki/...
4. Since everyone else is saying this (and because it merits saying), USE VERSION CONTROL. Singular most important thing to software development aside from planning and documenting.
5. Remember to flout all of the above once in a while and just make a mess of a project where you have fun throwing everything around all over the place. You'll make mistakes that you never thought were possible by someone of your caliber :) That's how you learn.
Have fun, keep learning!3 -
Fuck the memes.
Fuck the framework battles.
Fuck the language battles.
Fuck the titles.
Anybody who has been in this field long enough knows that it doesn't matter if your linus fucking torvalds, there is no human who has lived or ever will live that simultaneously understands, knows, and remembers how to implement, in multiple languages, the following:
- jest mocks for complex React components (partial mocks, full mocks, no mocks at all!)
- token cancellation for asynchronous Tasks in C#
- fullstack CRUD, REST, and websocket communication (throw in gRPC for bonus points)
- database query optimization, seeding, and design
- nginx routing, https redirection
- build automation with full test coverage and environment consideration
- docker container versioning, restoration, and cleanup
- internationalization on both the front AND backends
- secret storage, security audits
- package management, maintenence, and deprecation reviews
- integrating with dozens of APIs
- fucking how to center a div
and that's a _comically_ incomplete list; barely scratches the surface of the full range of what a dev can encounter in a given day of writing software
have many of us probably done one or even all of these at different times? surely.
but does that mean we are supposed to draw that up at a moment's notice some cookie-cutter solution like a fucking robot and spit out an answer on a fax sheet?
recruiters, if you read this site (perhaps only the good ones do anyway so its wasted oxygen), just know that whoever you hire its literally the luck of the draw of how well they perform during the interview. sure, perhaps some perform better, but you can never know how good someone is until they literally start working at your org, so... have fun with that.
Oh and I almost forgot, again for you recruiters, on top of that list which you probably won't ever understand for the entirety of your lives, you can also add writing documentation, backup scripts, and orchestrating / administrating fucking JIRA or actually any somewhat technical dashboard like a CMS or website, because once again, the devs are the only truly competent ones - and i don't even mean in a technical sense, i mean in a HUMAN sense of GETTING SHIT DONE IN GENERAL.
There's literally 2 types of people in the world: those who sit around drawing flow charts and talking on the phone all day, and those WHO LITERALLY FUCKING BUILD THE WORLD
why don't i just run the whole fucking company at this point? you guys are "celebrating" that you made literally $5 dollars from a single customer and i'm just sitting here coding 12 hours a day like all is fine and well
i'm so ANGRY its always the same no matter where i go, non-technical people have just no clue, even when you implore them how long things take, they just nod and smile and say "we'll do it the MVP way". sure, fine, you can do that like 2 or 3 times, but not for 6 fucking months until you have a stack of "MVPs" that come toppling down like the garbage they are.
How do expect to keep the "momentum" of your customers and sales (I hope you can hear the hatred of each of these market words as I type them) if the entire system is glued together with ducktape because YOU wanted to expedite the feature by doing it the EASY way instead of the RIGHT way. god, just forget it, nobody is going to listen anyway, its like the 5th time a row in my life
we NEED tests!
we NEED to know our code coverage!
we NEED to design our system to handle large amounts of traffic!
we NEED detailed logging!
we NEED to start building an exception database!
BILBO BAGGINS! I'm not trying to hurt you! I'm trying to help you!
Don't really know what this rant was, I'm just raging and all over the place at the universe. I'm going to bed.20 -
I'm convinced code addiction is a real problem and can lead to mental illness.
Dev: "Thanks for helping me with the splunk API. Already spent two weeks and was spinning my wheels."
Me: "I sent you the example over a month ago, I guess you could have used it to save time."
Dev: "I didn't understand it. I tried getting help from NetworkAdmin-Dan, SystemAdmin-Jake, they didn't understand what you sent me either."
Me: "I thought it was pretty simple. Pass it a query, get results back. That's it"
Dev: "The results were not in a standard JSON format. I was so confused."
Me: "Yea, it's sort-of JSON. Splunk streams the result as individual JSON records. You only have to deserialize each record into your object. I sent you the code sample."
Dev: "Your code didn't work. Dan and Jake were confused too. The data I have to process uses a very different result set. I guess I could have used it if you wrote the class more generically and had unit tests."
<oh frack...he's been going behind my back and telling people smack about my code again>
Me: "My code wouldn't have worked for you, because I'm serializing the objects I need and I do have unit tests, but they are only for the internal logic."
Dev:"I don't know, it confused me. Once I figured out the JSON problem and wrote unit tests, I really started to make progress. I used a tuple for this ... functional parameters for that...added a custom event for ... Took me a few weeks, but it's all covered by unit tests."
Me: "Wow. The way you explained the project was; get data from splunk and populate data in SQLServer. With the code I sent you, sounded like a 15 minute project."
Dev: "Oooh nooo...its waaay more complicated than that. I have this very complex splunk query, which I don't understand, and then I have to perform all this parsing, update a database...which I have no idea how it works. Its really...really complicated."
Me: "The splunk query returns what..4 fields...and DBA-Joe provided the upsert stored procedure..sounds like a 15 minute project."
Dev: "Maybe for you...we're all not super geniuses that crank out code. I hope to be at your level some day."
<frack you ... condescending a-hole ...you've got the same seniority here as I do>
Me: "No seriously, the code I sent would have got you 90% done. Write your deserializer for those 4 fields, execute the stored procedure, and call it a day. I don't think the effort justifies the outcome. Isn't the data for a report they'll only run every few months?"
Dev: "Yea, but Mgr-Nick wanted unit tests and I have to follow orders. I tried to explain the situation, but you know how he is."
<fracking liar..Nick doesn't know the difference between a unit test and breathalyzer test. I know exactly what you told Nick>
Dev: "Thanks again for your help. Gotta get back to it. I put a due date of April for this project and time's running out."
APRIL?!! Good Lord he's going to drag this intern-level project for another month!
After he left, I dug around and found the splunk query, the upsert stored proc, and yep, in about 15 minutes I was done.1 -
So just about to head to the pub and I got the dreaded call from my boss.
The support team had developed some fixes. They "tested" and deployed without letting us know... And you guessed it there was failures all over the shop!
So it turned out their testing was running on a local base install with no integration compared to the live system with 15 years of customisation and complex integration. My they thought this was acceptable I don't know...
And the best part was the developers who made the changes didn't understand their own code (I found the tutorial they copied online) they just blindly copied it without understanding how it worked!
So 4 hours later we found the bug, nothing like having a query and s SQL connection but not executing the query....
There goes my Saturday evening. Now we're was my beer!7 -
"I'm almost done, I'll just need to add tests!"
Booom! You did it, that was a nuke going off in my head.
No, you shouldn't just need to add tests. The tests should have been written from the get go! You most likely won't cover all the cases. You won't know if adding the tests will break your feature, as you had none, as you refactor your untested mess in order to make your code testable.
When reading your mess of a test case and the painful mocking process you went through, I silently cry out into the void: "Why oh why!? All of this suffering could have been avoided!"
Since most of the time, your mocking pain boils down to not understanding what your "unit" in your "unit test" should be.
So let it be said:
- If you want to build a parser for an XML file, then just write a function / class whose *only* purpose is: parse the XML file, return a value object. That's it. Nothing more, nothing less.
- If you want to build a parser for an XML file, it MUST NOT: download a zip, extract that zip, merge all those files to one big file, parse that big file, talk to some other random APIs as a side-effect, and then return a value object.
Because then you suddenly have to mock away a http service and deal with zip files in your test cases.
The http util of your programming language will most likely work. Your unzip library will most likely work. So just assume it working. There are valid use cases where you want to make sure you acutally send a request and get a response, yet I am talking unit test here only.
In the scope of a class, keep the public methods to a reasonable minimum. As for each public method you shall at least create one test case. If you ever have the feeling "I want to test that private method" replace that statement in your head with: "I should extract that functionality to a new class where that method public. I then can create a unit test case a for that." That new service then becomes a dependency in your current service. Problem solved.
Also, mocking away dependencies should a simple process. If your mocking process fills half the screen, your test setup is overly complicated and your class is doing too much.
That's why I currently dig functional programming so much. When you build pure functions without side effects, unit tests are easy to write. Yet you can apply pure functions to OOP as well (to a degree). Embrace immutability.
Sidenote:
It's really not helpful that a lot of developers don't understand the difference between unit, functional acceptance, integration testing. Then they wonder why they can't test something easily, write overly complex test cases, until someone points out to them: No, in the scope of unit tests, we don't need to test our persistance layer. We just assume that it works. We should only test our businsess logic. You know: "Assuming that I get that response from the database, I expect that to happen." You don't need a test db, make a real query against that, in order to test that. (That still is a valid thing to do. Yet not in the scope of unit tests.)rant developer unit test test testing fp oop writing tests get your shit together unit testing unit tests8 -
So last night i was dealing with some mysql query issues trying to grab some complex data from a relational database and ended up giving up.
This afternoon i was about half-asleep going to nap, not even thinking about the code and randomly came up with the solution.
A few hours later i made the changes and it works as intended.
Anyone else solve code subconsciously ? 😂😂😁😁12 -
I wrote a node + vue web app that consumes bing api and lets you block specific hosts with a click, and I have some thoughts I need to post somewhere.
My main motivation for this it is that the search results I've been getting with the big search engines are lacking a lot of quality. The SEO situation right now is very complex but the bottom line is that there is a lot of white hat SEO abuse.
Commercial companies are fucking up the internet very hard. Search results have become way too profit oriented thus unneutral. Personal blogs are becoming very rare. Information is losing quality and sites are losing identity. The internet is consollidating.
So, I decided to write something to help me give this situation the middle finger.
I wrote this because I consider the ability to block specific sites a basic universal right. If you were ripped off by a website or you just don't like it, then you should be able to block said site from your search results. It's not rocket science.
Google used to have this feature integrated but they removed it in 2013. They also had an extension that did this client side, but they removed it in 2018 too. We're years past the time where Google forgot their "Don't be evil" motto.
AFAIK, the only search engine on earth that lets you block sites is millionshort.com, but if you block too many sites, the performance degrades. And the company that runs it is a for profit too.
There is a third party extension that blocks sites called uBlacklist. The problem is that it only works on google. I wrote my app so as to escape google's tracking clutches, ads and their annoying products showing up in between my results.
But aside uBlacklist does the same thing as my app, including the limitation that this isn't an actual search engine, it's just filtering search results after they are generated.
This is far from ideal because filter results before the results are generated would be much more preferred.
But developing a search engine is prohibitively expensive to both index and rank pages for a single person. Which is sad, but can't do much about it.
I'm also thinking of implementing the ability promote certain sites, the opposite to blocking, so these promoted sites would get more priority within the results.
I guess I would have to move the promoted sites between all pages I fetched to the first page/s, but client side.
But this is suboptimal compared to having actual access to the rank algorithm, where you could promote sites in a smarter way, but again, I can't build a search engine by myself.
I'm using mongo to cache the results, so with a click of a button I can retrieve the results of a previous query without hitting bing. So far a couple of queries don't seem to bring much performance or space issues.
On using bing: bing is basically the only realiable API option I could find that was hobby cost worthy. Most microsoft products are usually my last choice.
Bing is giving me a 7 day free trial of their search API until I register a CC. They offer a free tier, but I'm not sure if that's only for these 7 days. Otherwise, I'm gonna need to pay like 5$.
Paying or not, having to use a CC to use this software I wrote sucks balls.
So far the usage of this app has resulted in me becoming more critical of sites and finding sites of better quality. I think overall it helps me to become a better programmer, all the while having better protection of my privacy.
One not upside is that I'm the only one curating myself, whereas I could benefit from other people that I trust own block/promote lists.
I will git push it somewhere at some point, but it does require some more work:
I would want to add a docker-compose script to make it easy to start, and I didn't write any tests unfortunately (I did use eslint for both apps, though).
The performance is not excellent (the app has not experienced blocks so far, but it does make the coolers spin after a bit) because the algorithms I wrote were very POC.
But it took me some time to write it, and I need to catch some breath.
There are other more open efforts that seem to be more ethical, but they are usually hard to use or just incomplete.
commoncrawl.org is a free index of the web. one problem I found is that it doesn't seem to index everything (for example, it doesn't seem to index the blog of a friend I know that has been writing for years and is indexed by google).
it also requires knowledge on reading warc files, which will surely require some time investment to learn.
it also seems kinda slow for responses,
it is also generated only once a month, and I would still have little idea on how to implement a pagerank algorithm, let alone code it.4 -
Want to make someone's life a misery? Here's how.
Don't base your tech stack on any prior knowledge or what's relevant to the problem.
Instead design it around all the latest trends and badges you want to put on your resume because they're frequent key words on job postings.
Once your data goes in, you'll never get it out again. At best you'll be teased with little crumbs of data but never the whole.
I know, here's a genius idea, instead of putting data into a normal data base then using a cache, lets put it all into the cache and by the way it's a volatile cache.
Here's an idea. For something as simple as a single log lets make it use a queue that goes into a queue that goes into another queue that goes into another queue all of which are black boxes. No rhyme of reason, queues are all the rage.
Have you tried: Lets use a new fangled tangle, trust me it's safe, INSERT BIG NAME HERE uses it.
Finally it all gets flushed down into this subterranean cunt of a sewerage system and good luck getting it all out again. It's like hell except it's all shitty instead of all fiery.
All I want is to export one table, a simple log table with a few GB to CSV or heck whatever generic format it supports, that's it.
So I run the export table to file command and off it goes only less than a minute later for timeout commands to start piling up until it aborts. WTF. So then I set the most obvious timeout setting in the client, no change, then another timeout setting on the client, no change, then i try to put it in the client configuration file, no change, then I set the timeout on the export query, no change, then finally I bump the timeouts in the server config, no change, then I find someone has downloaded it from both tucows and apt, but they're using the tucows version so its real config is in /dev/database.xml (don't even ask). I increase that from seconds to a minute, it's still timing out after a minute.
In the end I have to make my own and this involves working out how to parse non-standard binary formatted data structures. It's the umpteenth time I have had to do this.
These aren't some no name solutions and it really terrifies me. All this is doing is taking some access logs, store them in one place then index by timestamp. These things are all meant to be blazing fast but grep is often faster. How the hell is such a trivial thing turned into a series of one nightmare after another? Things that should take a few minutes take days of screwing around. I don't have access logs any more because I can't access them anymore.
The terror of this isn't that it's so awful, it's that all the little kiddies doing all this jazz for the first time and using all these shit wipe buzzword driven approaches have no fucking clue it's not meant to be this difficult. I'm replacing entire tens of thousands to million line enterprise systems with a few hundred lines of code that's faster, more reliable and better in virtually every measurable way time and time again.
This is constant. It's not one offender, it's not one project, it's not one company, it's not one developer, it's the industry standard. It's all over open source software and all over dev shops. Everything is exponentially becoming more bloated and difficult than it needs to be. I'm seeing people pull up a hundred cloud instances for things that'll be happy at home with a few minutes to a week's optimisation efforts. Queries that are N*N and only take a few minutes to turn to LOG(N) but instead people renting out a fucking off huge ass SQL cluster instead that not only costs gobs of money but takes a ton of time maintaining and configuring which isn't going to be done right either.
I think most people are bullshitting when they say they have impostor syndrome but when the trend in technology is to make every fucking little trivial thing a thousand times more complex than it has to be I can see how they'd feel that way. There's so bloody much you need to do that you don't need to do these days that you either can't get anything done right or the smallest thing takes an age.
I have no idea why some people put up with some of these appliances. If you bought a dish washer that made washing dishes even harder than it was before you'd return it to the store.
Every time I see the terms enterprise, fast, big data, scalable, cloud or anything of the like I bang my head on the table. One of these days I'm going to lose my fucking tits.10 -
Colleague asked me to look at his eCommerce search filtering system as the customer was complaining it was slow taking 5-6 seconds to find results.
Delving into the code deeper, I discovered he was querying the results, sticking them in an array and then sub querying the results looking at all the combinations.
On top of that each sub query looked at the database fields using "DESCRIBE" to then search them each time it found a pair!
The total query count for one page search was 14,512!!!
Why oh Why? One SQL query could of done all that in one go.
I look at other code bits he's done and he's very good in other areas. I just don't get how sometimes a good developer can make such a weird decision? It's almost as if he wanted to make it as complex as possible.6 -
# Retrospective as Backend engineer
Once upon a time, I was rejected by a startup who tries to snag me from another company that I was working with.
They are looking for Senior / Supervisor level backend engineer and my profile looks like a fit for them.
So they contacted me, arranged a technical test, system design test, and interview with their lead backend engineer who also happens to be co-founder of the startup.
## The Interview
As usual, they asked me what are my contribution to previous workplace.
I answered them with achievements that I think are the best for each company that I worked with, and how to technologically achieve them.
One of it includes designing and implementing a `CQRS+ES` system in the backend.
With complete capability of what I `brag` as `Time Machine` through replaying event.
## The Rejection
And of course I was rejected by the startup, maybe specifically by the co-founder. As I asked around on the reason of rejection from an insider.
They insisted I am a guy who overengineer thing that are not needed, by doing `CQRS+ES`, and only suitable for RND, non-production stuffs.
Nobody needs that kind of `Time Machine`.
## Ironically
After switching jobs (to another company), becoming fullstack developer, learning about react and redux.
I can reflect back on this past experience and say this:
The same company that says `CQRS+ES` is an over engineering, also uses `React+Redux`.
Never did they realize the concept behind `React+Redux` is very similar to `CQRS+ES`.
- Separation of concern
- CQRS: `Command` is separated from `Query`
- Redux: Side effect / `Action` in `Thunk` separated from the presentation
- Managing State of Application
- ES: Through sequence of `Event` produced by `Command`
- Redux: Through action data produced / dispatched by `Action`
- Replayability
- ES: Through replaying `Event` into the `Applier`
- Redux: Through replay `Action` which trigger dispatch to `Reducer`
---
The same company that says `CQRS` is an over engineering also uses `ElasticSearch+MySQL`.
Never did they realize they are separating `WRITE` database into `MySQL` as their `Single Source Of Truth`, and `READ` database into `ElasticSearch` is also inline with `CQRS` principle.
## Value as Backend Engineer
It's a sad days as Backend Engineer these days. At least in the country I live in.
Seems like being a backend engineer is often under-appreciated.
Company (or people) seems to think of backend engineer is the guy who ONLY makes `CRUD` API endpoint to database.
- I've heard from Fullstack engineer who comes from React background complains about Backend engineers have it easy by only doing CRUD without having to worry about application.
- The same guy fails when given task in Backend to make a simple round-robin ticketing system.
- I've seen company who only hires Fullstack engineer with strong Frontend experience, fails to have basic understanding of how SQL Transaction and Connection Pool works.
- I've seen company Fullstack engineer relies on ORM to do super complex query instead of writing proper SQL, and prefer to translate SQL into ORM query language.
- I've seen company Fullstack engineer with strong React background brags about Uncle Bob clean code but fail to know on how to do basic dependency injection.
- I've heard company who made webapp criticize my way of handling `session` through http secure cookie. Saying it's a bad practice and better to use local storage. Despite my argument of `secure` in the cookie and ability to control cookie via backend.18 -
[vent]
I am java dev with 5 yoe at a place which has really good engineering talent.
Was assigned a feature request.
Feature request requires me and one more older dev(in age, not in exp at company) to write the code. My piece is really super complex because of the nature of the problem and involves caching, lazy loading and tonne of other optimization. Naturally it makes up 90% of the tasks in the feature request. On the other hand, the older dev simply has to write a select query (infact he only needs to call it since a function is already written).
Older dev takes up all the credit, gives the demo, knows nothing but wrongly answers in meetings with higher ups and was recently awarded employee of qtr.
It looks as if I do the easy work whereas he is the one pulling in all the hard work.
Need advice to justify my work and make others realise it's significance, nuances of area and complexity of it.
Do not expect monetary benefits, just expect credit and recognition for the worth of work I am doing.14 -
On my first day at new job, a non-technical person used CQRS word while explaining the system. When I asked what's need of such a complex pattern for simple query type service, he simply backed off. 😝
-
Tip: if you are doing a semi complex or complex query in Django and you have doubts print the SQL statement and analyze it. i.e print(queryset.query)
Just reduced a query to 1 join instead of two by just passing a list of int's instead of a list of objects. -
Interesting...
On Friday, I was playing with the ChatGPT integration in DBeaver. I was using the DBeaver sample SQLite database. This database has a couple of tables, among them Album and Artist, where Album has a foreign key into Artist.
So, I asked it:
"give me a query that lists all albums from artists who's name starts with s"
The query I got back was:
SELECT * FROM Album
Uhh, okay.
But then, I noticed that I wrote "who's" instead of 'whose', which would be proper grammatically. So, I changed that, and then I got this query:
SELECT * FROM Album WHERE ArtistId IN (SELECT ArtistId FROM Artist WHERE Name LIKE 'S%');
Hooray, that works! I'm not sure it's the best way to write the query... I might have written:
SELECT * FROM album a, artist r WHERE a.artistid=r.artistid AND r.name LIKE 'S%'
...I'd have to check to see if one performs better than the other, and consider which syntax I find clearer, but that's a separate issue, it's just nice to see a working, reasonable query generated because that's the point, after all.
But I found it interesting that such a minor error would cause it to not work, that's my main point.
Interestingly, it seems to have learned: I just tried the same thing, and I got the right query either way. So that's pretty cool.
It's a pretty neat feature and I can see some legitimate value in it. I'm pretty good writing SQL myself... I've managed to write some truly hideously complex queries over the years... but there are definitely instances I can recall where the query didn't seem obvious at the start, and having an AI that can MAYBE produce something that is AT LEAST a starting point is definitely something I can get onboard with.9 -
I wasn't happy with one of our UI views for editing a database query that consisted of about 50 fields ("editing" being the operative term here, not just viewing. It had to be two-way). Everything was hardcoded and defined manually, with each block of ~10 lines being repeated and mostly identical apart from the occasional double inline field and name of the variable. It had "just ended up that way" over time due to the variable names in the UI being different than the names of the variables that came from the API.
I decided to overhaul it all where I defined the different input components and which fields should be included, then made a function which would generate the page based on these definitions. It was about 500 lines of modularized functions and classes where the class for the actual view was about 50 lines- compared to the 1400+ lines of the previous version.
But, it didn't work. It should, but it "just didn't". There was no error. All I got was a blank, solid white page. I could make a drastic change or try something completely different and I would get the same error, same blank page. API fetch succeeded, value assignments succeeded, the object exists, but if you iterated it it was... empty.
I started getting really discouraged that I had made it too abstract. Maybe I actually made it more complex and unreadable than before. Maybe just hardcoding it all was the better solution after all. Maybe I had gone against KISS and overdesigned it.
I was up pretty late and everyone had gone home. When the last guy left there was that mood where "yeah if I can't make this work we'll just use the current version...".
Turns out I had tried iterating over a property of the set of fields to render, rather than the entire collection. In the old method the variables were a member of an object, but now they were its own object, a change I had made to isolate the set of values which were to be viewed/edited and make them easier to pass back and forth. This member existed since I hadn't cleaned it out yet, but it was empty.
I had been banging my head against this for a whole day and I was ready to admit I had made a mistake and wasted my time before discarding it all, but then I backspaced this one property and the interface went from empty to rendering perfectly and with all functionality intact. I swear god rays were coming out of my screen. -
Using grafana together with tinc+promotheus, has been a blast.
Initially I wanted to get into ELK with Kibana and all that, but that required 8G of ram, the instructions to get it running in the open source "mode" was nearly non-existent, together with all the ready docker compose stacks out there simply not working or the images being broken.
I'm sure I could've managed around most of those issues, but the fact it is as hungry as gitlab, made it a literal no-go for the usual server resources my clients host or my own scaled down server recently.
Thankfully I remembered that there's grafana and me having experimented some time ago with tinc, so I can have very lightweight beat'esque prometheus agents deployed listening on tinc local net only, with the typical nginx auth and some whitelists to all of the servers I host and all those of my clients.
The dashboard creation was especially great in grafana (tbf promotheus does actually most of it), literally what I always wanted out of those "complicated" solutions, that do it all, but have no proper query language, complex documentation, heavy collectors with no properly named data points, expensive resource runtimes, ..
with grafana I can just easily put dashboards into folders, create users to look only at certain stats or even dashboards (opened up some interesting contracts actually, because now I can also offer proper monitoring for all things delivered), easily drag and drop around stuff to fit more information (most others fix you to a small 3x2 grid, a too big grid for a TV or simply non resizable tiles, making that one counter take up an entire row) and resize to my hearts desire
tinc of course allows me to easily create private networks that are resistant to failure across any region and the routing is done for me, so I don't have to run around it all that much either
P.S: a damn tiny fly went into one of my now 4 monitors and died right in the middle, because I thought it's just some dirt and I pressed it in while trying to wipe it off, so that monitor now serves as the top most on a vesa mount5 -
Worked the whole day on a pretty complex sql query works perfect. Run it on the production server and the hole thing hangs. It's literally the same version of MySQL en a exact clone of the data but still it managed to hang the hole server.2
-
NoSQL database solutions are only useful for event streams. Nothing else. Wrestling with DynamoDB to try and filter a complex query is unnecessarily complex .1
-
So for the past two days I had to deal with a problem where I have to do a nested query with sequelize, pretty straight forward reading the documentation, or that was I think. I implemented everything according to the docs but the query stills fails, why ? I had no idea, I double check my implementation, I googled the error, no luck, after a day searching like crazy I talked with the backend lead about this and he help me to realize that the naming convention was changing because sequelize is creating a nested (SELECT * FROM) because one of the relations has a one-to-many realtion with the root model and I'm why the heck is doing that? But we both didn't know, and the problem was solved by just modifying the names, so we let it through, and sent it to QA. The next day I see the task rejected by QA and the reason was after the changes were merged another part of the app was broken, ok np, I'll fix it right away, and oh God I found the error was caused by another query that was including the first query we fix yesterday ! It was a nested query with 3 lvls! And the names became even more complex ( like `model1->model2.colum1`), goddamit, ok, I spent most of the day searching again, nothing, read the specification of the findAll function, nope, tried to put that name in the ON clause as the docs suggested, still an error, shit, then the lead helps me again and creates a literal which can hold that name and voila! Everything is happiness, at least for that moment, but I was still curious about this behavior, so I keep digging on it and I've just found an issue where a great guy posted an option to the findAll method that is not documented in any version of sequelize ! WTF ! And this option was "subQuery" which if you set it to false it won't create that additional (SELECT * FROM) from before, FUUUCK! I can't believe it, I know that all the effort works in my favor because I learn more about sequelize, but FFS I'm still angry because this shit shouldn't happen, you need to update the god damn docs, it's just adding a row and telling the people what it does. Well to end this, after putting that in the query and replacing all the workarounds with the expected syntaxis everything works like charm.1
-
Having a senior DBA can save hours if not days of struggle and save your back, if you do not know well enough how to do a more complicated query yet, without fucking up something.
Good guidance and experience is worth so much.
... and no I do not have the rights to drop databases.1