Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
-
Hey everyone,
We have a few pieces of news we're very excited to share with everyone today. Apologies for the long post, but there's a lot to cover!
First, as some of you might have already seen, we just launched the "subscribed" tab in the devRant app on iOS and Android. This feature shows you a feed of the most recent rant posts, likes, and comments from all of the people you subscribe to. This activity feed is updated in real-time (although you have to manually refresh it right now), so you can quickly see the latest activity. Additionally, the feed also shows recommended users (based on your tastes) that you might want to subscribe to. We think both of these aspects of the feed will greatly improve the devRant content discovery experience.
This new feature leads directly into this next announcement. Tim (@trogus) and I just launched a public SaaS API service that powers the features above (and can power many more use-cases across recommendations and activity feeds, with more to come). The service is called Pipeless (https://pipeless.io) and it is currently live (beta), and we encourage everyone to check it out. All feedback is greatly appreciated. It is called Pipeless because it removes the need to create complicated pipelines to power features/algorithms, by instead utilizing the flexibility of graph databases.
Pipeless was born out of the years of experience Tim and I have had working on devRant and from the desire we've seen from the community to have more insight into our technology. One of my favorite (and earliest) devRant memories is from around when we launched, and we instantly had many questions from the community about what tech stack we were using. That interest is what encouraged us to create the "about" page in the app that gives an overview of what technologies we use for devRant.
Since launch, the biggest technology powering devRant has always been our graph database. It's been fun discussing that technology with many of you. Now, we're excited to bring this technology to everyone in the form of a very simple REST API that you can use to quickly build projects that include real-time recommendations and activity feeds. Tim and I are really looking forward to hopefully seeing members of the community make really cool and unique things with the API.
Pipeless has a free plan where you get 75,000 API calls/month and 75,000 items stored. We think this is a solid amount of calls/storage to test out and even build cool projects/features with the API. Additionally, as a thanks for continued support, for devRant++ subscribers who were subscribed before this announcement was posted, we will give some bonus calls/data storage. If you'd like that special bonus, you can just let me know in the comments (as long as your devRant email is the same as Pipeless account email) or feel free to email me (david@hexicallabs.com).
Lastly, and also related, we think Pipeless is going to help us fulfill one of the biggest pieces of feedback we’ve heard from the community. Now, it is going to be our goal to open source the various components of devRant. Although there’s been a few reasons stated in the past for why we haven’t done that, one of the biggest reasons was always the highly proprietary and complicated nature of our backend storage systems. But now, with Pipeless, it will allow us to start moving data there, and then everyone has access to the same system/technology that is powering the devRant backend. The first step for this transition was building the new “subscribed” feed completely on top of Pipeless. We will be following up with more details about this open sourcing effort soon, and we’re very excited for it and we think the community will be too.
Anyway, thank you for reading this and we are really looking forward to everyone’s feedback and seeing what members of the community create with the service. If you’re looking for a very simple way to get started, we have a full sample dataset (1 click to import!) with a tutorial that Tim put together (https://docs.pipeless.io/docs/...) and a full dev portal/documentation (https://docs.pipeless.io).
Let us know if you have any questions and thanks everyone!
- David & Tim (@dfox & @trogus)
53 -
Got laid off my by old employer back in 2019 because they have their priorities completely wrong.
Got a mail today whether I could fix something for them (ofc, they wanted me to do it essentially for free).
One of the websites I built for a customer back when I still worked there had a massive bug (that I was aware off and patched in later versions of the library causing it).
They never updated it so, I told them "just update the library".
Apparently, the idiot that was in charge of maintaining said site after I left didn't know how to and completely broke everything.
The hilarious part: While I setup everything using stuff like Docker and Git to make rollbacks easy...
That idiot went back to FTP and manually upgrading the databases through PhpMyAdmin :^)
He nuked the entire site.
Database? Gone.
Codebase? Borked (installed a version with a lot of breaking changes without properly reading the migration guide).
And knowing that shit company, they don't have any backups either.
They said "I wasn't needed because we have other good devs" when they laid me off.
Uhu, I can eh... see those good devs doing their job :^)51 -
We build a backup infrastructure at work to make sure that clients can restore their files and databases themselves when something gets fucked up.
We also have step by step tutorial on how to do this.
Every fucking day we get requests to restore backups.
Mostly used reason is "I'm a technical so I won't understand it".
With all due respect, if you don't understand this and keep asking without even trying, please don't host with us.
Because, if you did as I asked and actually read through the entire article, you would.
In case you're wondering, anytime one of us asks what part they don't understand, that question is simply ignored and they pushing for us restoring it anyways continues.
Sometimes they get angry and want to talk to someone higher up or start complaining that they're paying loads of money already and that it would just take us a second anyways.
If you would read the fucking tutorial/manual instead of trying to eat out your mother's badly shaved pussy and hopefully choke on it while you're at it, you wouldn't come asking us for it.
If you genuinely don't understand this article, feel free to ask but also provide us with cocksucking feedback.
Why do you think you have the right anyways to ask us to do it for free? We maintain the backup infrastructure which definitely isn't cheap but we do it so that you, pubic sniffing weazel, can do this shit on your fucking own.
You're entitled to ask us for help but not for asking us to restore your bullshit for free every freaking time.
Tip: give your parents some condoms. Because that way they hopefully won't reproduce again, we don't need more of you in this universe.7 -
I hate Wordpress. I hate Wordpress. I hate Wordpress.
Wordpress can take a big shit on itself and crawl into a deep dark hole far away from all that is good.
Who even uses Wordpress? Bloggers? Come on, let’s be honest, they’re using more intuitive sites like weebly, wix, and square space. So WHAT is Wordpress for? I’ll tell you, it’s just to FUCKING TORTURE PEOPLE.
So, being the “techy guy” of the family, a relative contacts me asking for some help with their website because they need to install an SSL certificate but they don’t know how to. I tell them I’d gladly do it because, sure, they’re family and how long can it possibly take to install a certificate? I’ve done it before!
Well, I get to work and log into the sluggish Wordpress dashboard and try to use a plugin that would issue a LetsEncrypt certificate because they are free and just as good as any other SSL. But one plugin after the next I keep getting errors about how my hosting wouldn’t allow it.
So I contact GoDaddy (don’t get me fucking started) and ask them about the issue. The guy tells me it’s “policy” to only be able to use GoDaddy’s certificates. How much do they cost? Oh, how about $100 a year?! Fuck you.
I figured out the only way to escape this hell was to ask them to open an economy Linux hosting account with cPanel on GoDaddy (the site was formerly hosted on a “Managed Wordpress” account which is just bullshit for not wanting to give you any control over your own goddamn content). So now I have to deal with migrating the site.
GoDaddy representative tells me that it should only take 20 minutes for me to do this (I’ve already spent way too much time on this but whatever) so I go forward with the new account. I decide I should migrate the site by exporting a backup and manually placing everything on the new server. Doesn’t it end up taking an entire hour to back up a 200MB site because GoDaddy throttled the processing speed?!
So, it’s another hour later and I’ve installed all the databases and carried over all the files. At this point, I’m really at the end of my rope and can’t wait to install the certificate and be done with this fuckery.
I install the certificate and finally get ready to be on my way, but then I see it. A warning. A warning from my browser telling me the site is only partially secure. It turns out the certificate was properly installed but whoever initially made the site HARDCODED ALL THE LINKS to images, websites, and style sheets to be http instead of https.
I’m gonna explode.
I swear, I’m gonna fucking explode.
After a total of 5 hours of work, I finally get the site secure by using search and replace on every fucking file.
Wordpress can go suck a big one. Actually, Wordpress can go suck the largest fuckin one in existence and choke on it.
TL;DR I agree to install an SSL certificate but end up with much more work than I bargained.38 -
Forgive me father, for I have sinned. Alot actually, but I'm here for technical sins. Okay, a particular series of technical sins. Sit your ass back down padre, you signed up for this shit. Where was I? Right, it has been 11429 days since my last confession. May this serve as equal parts rant, confession, and record for the poor SOB who comes after me.
Ended up in a job where everything was done manually or controlled by rickety Access "apps". Many manhours were wasted on sitting and waiting for the main system to spit out a query download so it could be parsed by hand or loaded into one of the aforementioned apps that had a nasty habit of locking up the aged hardware that we were allowed. Updates to the system were done through and awful utility that tended to cut out silently, fail loudly and randomly, or post data horrifically wrong.
Fuck that noise. Floated the idea of automating downloads and uploads to bossman. This is where I learned that the main system had no SQL socket by default, but the vendor managing the system could provide one for an obscene amount of money. There was no buy in from above, not worth the price.
Automated it anyway. Main system had a free form entry field, ostensibly for handwriting SELECT queries. Using Python, AutoHotkey, and glorified copy-pasting, it worked after a fashion. Showed the time saved by not having to do downloads manually. Got us the buy in we needed, bigwigs get negotiating with the vendor, told to start developing something based on some docs from the vendor. Keep the hacky solution running as team loves not having to waste time on downloads.
Found SQLi vulnerability in the above free form query system, brought it up to bossman to bring up the chain. Vulnerability still there months later. Test using it for automated updates. Works and is magnitudes more stable than update utility. Bring it up again and show the time we can save exploiting it. Decision made to use it while it exists, saves more time. Team happier, able to actual develop solutions uninterrupted now. Using Python, AutoHotkey, glorified copy-pasting, and SQLi in the course of day to day business critical work. Ugliest hacky thing I've ever caused to exist.
Flash forward 6 years. Automation system now in heavy use acrossed two companies. Handles all automatic downloads for several departments, 1 million+ discrete updates daily with alot of room for expansion, stuff runs 24/7 on schedule, most former Access apps now gone and written sanely and managed by the automation system. Its on real hardware with real databases and security behind it.
It is still using AutoHotkey, copy-paste, and SQLi to interface with the main system. There never was and never will be a SQL socket. Keep this hellbeast I've spawned chugging along.
I've pointed out how many ways this can all go pearshaped. I've pointed out that one day the vendor will get their shit together they'll come in post system update and nothing will work anymore. I've pointed out the danger in continuing to use the system with such a glaring SQLi vulnerability.
Noone cares. Won't be my problem soon enough.
In no particular order:
Fuck management for not fighting for a good system interface
Fuck the vendor for A) not having a SQL socket and B) leaving the SQLi vulnerability there this long
Fuck me for bringing this thing into existence5 -
So probably about a decade ago at this point I was working for free for a friend's start-up hosting company. He had rented out a high-end server in some data center and sold out virtualized chunks to clients.
This is back when you had only a few options for running virtual servers, but the market was taking off like a bat out of hell. In our case, we used User-Mode Linux (UML).
UML is essentially a kernel hack that lets you run the kernel in user space. That alone helps keep things separate or jailed. I'm pretty sure some of you can shed more light on it, but that's as I understood it at the time and I wasn't too shabby at hacking the kernel when we'd have driver issues.
Anyway, one of the ways my friend would on-board someone was to generate a new disk image file, mount it, and then chroot to that mount path. He'd basically use a stock image to do this and then wipe it out before putting it live.
I'm not sure exactly what he was doing at the time, but I got a panicked message on New Years Day saying that he had deleted everything. By everything, he had done an rm -fr /home as root on what he had thought was the root of a drive image.
It wasn't an image. It was the host server.
In the stoke of a single command, all user data was lost. We were pretty much screwed, but I have a knack for not giving up - so I spent a ton of time investigating linux file recovery.
Fun fact about UML - since the kernel runs in user space as a regular ol' process, anything it opens is attached to that process. I had noticed that while the files were "gone", I could still see disk usage. I ended up finding the images attached to their file pointers associated with each running kernel - and thankfully all customers were running at the time.
The next part was crazy, and I still think is crazy. I don't remember the command, but I had to essentially copy the image from the referenced path into a new image file, then shutdown the kernel and power it back on from the new image. We had configs all set aside, so that was easy. When it finally worked I was floored.
Rinse and repeat, I managed to drag every last missing bit out of /proc - with the only side effect being that all MySQL databases needed to be cleaned up.3 -
Our project at work goes live in 3 weeks.
The code base has no automated tests, breaks very often, has never had any level of manual testing
will not be releasing with any form of enforced roles or permissions in our first release now due to no time to enforce, however there is a whole admin api where you can literally change anything in our database including roles.
We also have teams in various countries all working separately on the same solution using microservices with shared nuget packages and they aren't using them properly.
Our pull requests are so big - as much as, 75 file changes - in our fe app that I can't keep up with it and I honestly have no idea if it even works or not due to no automated tests and no time to manually test.
We have no testing team, or qa team of any sort.
Every request into the system has to hit a minimum of 3 different databases via 3 different microservices so 1 request = 4 requests with the load on the servers.
We don't use any file streams so everything is just shoved in the buffer on the server.
Most of the people working on the angular apps cba to learn angular, no one across 2 teams cba to learn git. We use git so they constantly face problems. The guy in charge has 0 experience in angular but makes me do things how he wants architecturally so half the patterns make no sense.
No one looks at the pull requests, they just click approve so they may as well push directly to master.
Unfinished work gets put in for pull request so we don't know if the app is in a release state since aall teams are working independently, but on the same code base.
I sat down and tested the app myself for an hour and found 25 fe only issues, and 5 breaking cross browser issues.
Most of our databases are not normalised. Most of our databases make no sense. 99% of our tables have no indexing since there is no expertise with free time to do it.
No one there understands css properly. Or javascript.
Our. Net core microservices all directly use ef in the controller actions so there is no shared code there.
Our customer facing fe app is not dry because no tests so it was decided it was better this way.
Management has no idea on code state, it seems team lead is lieing to them about things like having any level of tests.
Management hire devs that claim to be experts but then it turns out they have basically no knowledge of what they were hired to do, even don't know what json is or the framework or language they are hired for, but we just leave them to get on with it and again make prs too big to review.
Honestly I have no hope that this will go well now but I am morbidly curious to watch. I've never seen anything like the train wreck that we are about to get experience.5 -
Data Disinformation: the Next Big Problem
Automatic code generation LLMs like ChatGPT are capable of producing SQL snippets. Regardless of quality, those are capable of retrieving data (from prepared datasets) based on user prompts.
That data may, however, be garbage. This will lead to garbage decisions by lowly literate stakeholders.
Like with network neutrality and pii/psi ownership, we must act now to avoid yet another calamity.
Imagine a scenario where a middle-manager level illiterate barks some prompts to the corporate AI and it writes and runs an SQL query in company databases.
The AI outputs some interactive charts that show that the average worker spends 92.4 minutes on lunch daily.
The middle manager gets furious and enacts an Orwellian policy of facial recognition punch clock in the office.
Two months and millions of dollars in contractors later, and the middle manager checks the same prompt again... and the average lunch time is now 107.2 minutes!
Finally the middle manager gets a literate person to check the data... and the piece of shit SQL behind the number is sourcing from the "off-site scheduled meetings" database.
Why? because the dataset that does have the data for lunch breaks is labeled "labour board compliance 3", and the LLM thought that the metadata for the wrong dataset better matched the user's prompt.
This, given the very real world scenario of mislabeled data and LLMs' inability to understand what they are saying or accessing, and the average manager's complete data illiteracy, we might have to wrangle some actions to prepare for this type of tomfoolery.
I don't think that access restriction will save our souls here, decision-flumberers usually have the authority to overrule RACI/ACL restrictions anyway.
Making "data analysis" an AI-GMO-Free zone is laughable, that is simply not how the tech market works. Auto tools are coming to make our jobs harder and less productive, tech people!
I thought about detecting new automation-enhanced data access and visualization, and enacting awareness policies. But it would be of poor help, after a shithead middle manager gets hooked on a surreal indicator value it is nigh impossible to yank them out of it.
Gotta get this snowball rolling, we must have some idea of future AI housetraining best practices if we are to avoid a complete social-media style meltdown of data-driven processes.
Someone cares to pitch in?14 -
Hello and welcome, to a presentation in which I will tell you my thoughts on the shortcomings of modern day computers and programming practices.
Computers are based on a very fundamental and old idea, folders, and files, a file is basically a concrete amount of data, whereas a folder is a group of files, and it comes from the real life concept of files and folders, now it might be quite obvious already that using a concept invented in 1898 by a guy called Edwin G. Seibels, might not be the best way for computers to function in the year 2020, but alas, it is.
Unless of course, you step into the world of a programmer.
A programmer’s world is much different, they use this idea of a data structure, or in simpler terms, an object. An Object is just like what you would think of as an object in your head, something with different properties that you can think about in different ways, for example your mobile phone, it has a battery percentage, it has a screen size, it has free space available. Programmers use these data structures to analyse data very quickly, like finding all phones with a screen size bigger than a certain size for example.
The problem is that programmers still use files and folders to create the programs that use these objects.
Consider this example.
Let’s say you want to create a virtual version of a drink bottle, consider what properties it will have, colour, volume, height, width, depth, material, etc..
As a programmer, you can leverage programming features and change the properties of a drink bottle directly, if you wanted to change the colour, you just say, drink bottle “dot” colour, equals blue, or red.
But if the drink bottle was represented as a file, all the drink bottles data would be inside the one file, so you would have to open the whole file, find the line or section of the file that has the colour data of the drink bottle, and select it, highlight it, delete what’s there, and type in your new value.
One way to explain this better is to imagine a folder that now represents the drink bottle, imagine adding a new file into that folder that represents each property I described before, colour, volume, etc.., well now, you could just open that folder, find the file for colour, either by looking with your eyes or you could do a file search in the folder for a file called colour, open it, and edit the value inside. This way of editing objects is the one that more closely represents the way programmers and a program itself interacts with objects inside a running programming language.
But the thing is, programmers don’t use the folder/file way of creating objects and putting them into programs, because it would be too cumbersome, they just create 1 file for an object, or have lots of objects in a file, and create all the objects in 1 file, and then run the program which creates the objects, then when they stop the program, it deletes the objects. So there is no actual link between the object in a file and the object that the program creates by reading the data from that file, if you change the object in your program, it does not get saved to the file.
So programmers created databases to house these objects, but there is still a flaw in databases, they are hard to interface with, and mostly databases are just used to send data or retrieve data from, programmatically, you can’t really browse a database the way you can browse the files on your computer. You can, but database interfaces are not made to be easily navigated the way files and folders are.
As it stands, there is no way to store objects instead of files on your computer and interact with them in complex ways the way programmers can inside the programs they create.
If the idea of an object became standard the way a file and folder is standard, I think it would empower human’s a great deal to express things far more easily and fluidly than they can today.
Thanks for reading.8 -
I've been assigned lots of database work the last couple of weeks. Managing old databases are what make or break people surely.
The customers have this proprietary mess of a Visual FoxPro database manager.
I spent the entire day navigating what I should and shouldn't export for them (as they're fortunately trying to rid themselves of the service)
When I was supposed to do the export, the proprietary hardware only allowed exports of 25 rows at a time with the free version this big company is using... I ehr... I just regret not converting it into SQL to begin with.
2 -
Any recommendations for a decent free web host service?
I'm using plain html with no databases.
My current host seems to be having issues with ftp.8 -
How about a Git for Databases?
Do your changes, delete things, create things, and then when you are sure push these changes to your server.
If you did something wrong like accidentally delete fucking everything, just revert and everything is like before again.
Also you can view history and blame people for doing something wrong.
Tell me what you think about this. Not clue how you could implement this tho... Also I have enough to do already so feel free to take this idea!10 -
DBeaver is probably the best, most underrated UI for interacting with relational databases, that's free and universal.
It's just sad that I had to try out literally everything else, which is total crap in comparison, before I stumbled upon DBeaver.1 -
Me: Assigned to do some NoSQL injections test cases in December on Jira by product owner.
After asking him about it, he said it can be vague and it’s only for developers to get an idea. I also have this restriction where I can’t really keep actually data or databases in our test sample application, so I could only mock mongodb. Product owner says just mongo is fine.
I do it. Now it’s January, product owner away for a month we so director is managing it. She then schedules me to talk to database team. I show them the very simple test cases which essentially just inject payloads I found online into different parameters specified in test case. They say if that’s it. I say yes. They say what’s the point of this. I said that it’s probably to test your database clients and ensure they’re rejecting bad Malicious input? They then keep asking but I’m just the dev and tell them the product owner is away. Then the guy calls my test case essentially useless and the others agree. Then they tell me to do it for other databases which I can’t mock like couchbase even tho my PO said it’s fine for mongo only.
Am I just being silly here? I am pretty new to working in a dev environment so please feel free to be blunt.4 -
Not a rant, but seeking advice...
Should I abandon 2 years' worth of work on migrating a personal project from SQL (M$) to a Graph database, and just stick to SQL? And only consider migrating when/if I need graph capabilities?
The project is a small social media platform. Has around ~50 monthly active users.
Why I started the migration in the first place:
• When researching databases, I read that for social media, graph is more suitable. It was, at least in terms of query structure. It was more natural, there were no "joins", and queries were much simpler than their SQL counterparts.
• In case the project got big, I didn't want to have to panic-deal with database issues that come with growth. I had some indexing issues with MSSQL, and it got me worried that at 50MAU I'm having these issues, what would happen if I get more?
• It's a personal project, and the Gremlin language and graph databases looked cool and I was motivated to learn something new.
----
Why I'm considering aborting the migration:
• It's taking too damn long. I'm unable to work on other features because this migration is taking up all my free time. Sunk cost fallacy is hitting me hard with this one.
• In local testing within docker, it's extremely slow. I tried various graph engines (janusgraph, official tinkerpop, orientdb), and the fastest one takes 4-6minutes to complete my server tests. SQL finishes the same tests in under 2 minutes, same docker environment. I also tried running my tests on a remote server (AWS neptune) and it was just as slow. Maybe my queries are bad, but can I afford to spend even more time fine tuning all queries?
• I now realise that "graph = no scalability issues" was naïve of me, and 100% wishful thinking. Scalability issues don't care what database I use, but about how well tuned and configured the whole system is.
• I really want to move on. My tech stack is falling behind and becoming outdated. I'm unable to maintain dependencies.
• I'm worried about losing those 50 MAU because they're essential to gaining traction once I release the platform. I keep telling them about the migration but at some point (2 years later) they're going to get bored I feel.
I guess partially it's a rant because I feel like I shouldn't stop now having spent 2 years on this, but at the same time I feel like I'm heading towards a dead end.
If you made it this far, thank you for reading:)10 -
!rant
Hi, guys. I'm looking to get into Android development (Xamarin) and to build some simple apps first.
What I have in mind is something trivia-like, with a hosted database (can't find any free & reliable public APIs).
Do you know of any free downloadable databases on the topics of:
- trivia
- humour
- entertainment
http://www.usabledatabases.com/ seemed promising, but it's not free (or I couldn't find any free DBs, as they don't have a price filter).
Thanks in advance! -
Really need help to upskill as a data engineer. Are there any free sample databases data out there with practice questions to upskill as one? I wanna learn database design, data lake and data warehousing skills... I don't know if I'm asking the right questions but if I'm not making sense, please help me out!
Top Tags
Weekly Rant
View