Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API

From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "data"
-
User: *Clicks on staging environment*
Giant Warning Dialog: YOU ARE CURRENTLY ENTERING THE STAGING ENVIRONMENT
Users: Ok
App: *Completely different colour, I’m talking bright unsightly yellow*
User: Ok
Giant Yellow and Red Flashing Banner at the Top of the Screen: WARNING YOU ARE CURRENTLY USING STAGING, THIS AREA IS FOR TESTING ONLY
User: The production environment sure is acting strange today. It’s a weird colour and I don’t recognize any of the data, it’s all just dummy filler data. I better create a ticket for the dev team to check o—….. no wait I’ll send an email CC everyone including the CEO and sound the alarm production is currently down and filled with giant warning messages.
Manager: OH MY GOD PRODUCTION IS DOWN DID YOU HEAR ABOUT THIS??? WHAT THE FUCK COULD THESE WARNING MESSAGES BE THAT’S ONLY SUPPOSED TO HAPPEN ON STAGING! THE CEO IS BREATHING DOWN MY NECK YOU NEED TO GET THIS FIXED IMMEDIATELY!!!!!!!
Dev: …14 -
Manager: Why did you clear the data from the database? The client is now specifically requesting it and we don’t have it anymore!
Dev: You told me to.
Manager: Well why did you listen? It’s obvious now that that data was very important and should have been kept!
Dev: Last time you told me to do something that wasn’t a good idea I tried to explain why and told me not to question you ever again and that doing so was “disrespectful” and then threatened to have me fired. So now I just go along with what you say and let you suffer the consequences of not listening.
Manager: Well don’t do that then! It’s obviously not working very well! It’s ok to disagree with me you just have to make sure that what you think is something I agree with!
Dev: …13 -
Interviewer: So are you familiar with our company and what we do?
Dev: I looked at your website, looks like you build tools for managing restaurants.
Interviewer: No. That’s not even close.
Dev: ?
Interviewer: What we do is create an ecosystem of integrated data centres all orchestrated for immediate stakeholder utilization.
Dev: But the product itself…. it’s a user interface for tracking inventory. Of like…. burgers…. and bottles of wine.
Interviewer: It’s not a product! It’s a data……habitat!!
Dev: …
Dev: So does that make your users animals?
Interviewer: 😡. Unfortunately it looks like you do not see our vision and would not be a good fit for this role.
Dev: Agreed.28 -
Boss: Our customer's data is not syncing with XYZ service anynmore!
Me: Ok let me check. Did the tokens not refresh? Hmm the tokens are refreshing fine but the API still says that we do not have permissions. The scopes are fine too. I'll use our test account... its... cancelled? Hey boss, why is our XYZ account cancelled?
Boss: Oh, "I haven’t paid since I didn’t think we needed it" (ad verbatim)
😐2 -
I told these people that this issue would happen. Did they listen? Nooo
It'll be fine, they say. We likely won't be having that much data returned to the front end, they say.
Day of the install. Web Application attempts to query 68,000 rows of data straight into the web page.
*Surprised Pikachu face* when they are consistently getting crashed browser tabs.
And now everything gets pushed back and we're behind by an entire month because they didn't heed my warnings.
Oh, and now I have to pick up after them, and do some stupid work arounds that will likely be defunct in a month or two. 🙄5 -
So this guy is supposed to do the frontend.
I do the backend.
I offer an endpoint.
He does his HTML+CSS magic.
Me: Cool but data is hardcoded. Could you get the data from the endpoint I sent you?
Him: "I'd prefer you do that, I can make a git repo so you download the front."
... So you don't do frontend, you just write pretty layouts. And I have to actually write the frontend logic? Go f yourself.14 -
Walked into the office in the afternoon, everyone was kinda panicking
Asked what was going on, well, the ticket system is not working anymore, can't put in any new tickets.
So I started to look for the issue as well, checked the system and... The last tickets' IDs were at ~32k. Ha. Looked into the source code and, sure enough, they used a data type with an upper limit of... 32k. So when trying to get a new ticket ID it just crashed and burned.
Quickly changed the data type and stopped the office panic in around half an hour.
Memorable not because of how tough the bug was, but because of the impact and the simplicity of the fix3 -
Dev: We need a better name than “Data” for this class. It’s used for displaying a set of tiles with certain coordinates so maybe TileMap would be a bit more declarative?
Manager: No I don’t like that. Data is perfectly fine, this class is for managing data so it’s perfectly declarative you just need to get better at reading code. If you have to change it then DataObject or DataObjectClass might be a bit more specific.
Dev: …14 -
a dude just DROPPED the whole Fu**ing mongodb cluster. like Right Now.
multiple databases, expanding multiple projects.
fortunately in dev. but dunno how much data is recoverable.14 -
Probably the most rage inducing data loss story...
When it comes to my cellphone I'm a data hoarder, I store each relevant meme, conversation, video, contact, nudes, etc. Had to replace my phone? Easy, change the SD.
I did this for about 4 years, had over 11GB of almost everything and anything in a 36GB SD, one afternoon my buddies and I went to a small tech convention and on our way to my car we got mugged by 5 armed men.
They took my brand new phone along with my wallet and all my cash, luckily I had GPS tracking enabled and we were able to pinpoint the exact location of my phone within 30min.
So far so good...
We called the cops and went with them, we found the car with illegal plates and weapons inside (knives, a bat, gun) so I tell the robbers were in there inside a closed cyber cafe and showed him the point on the map confirming this.
Cop: oh we can't do that we don't have an order...
Me: are you kidding me, here's the GPS, there's the car, there's the weapons, doesnt that count as at least probable cause or some shit?
Cop: we don't have that in this country, you can file a report and after 3 business days we can come here to inquire.
Me: (fucking lost it) do you fucking think they'll be here in 3 days?! I'll give you 500 bucks if you go bust their ass now.
Cop: (thinks about it) but what if they are armed? [4 patrols, 8 cops, 4 rifles and at least 6 guns plus vests] Maybe if you had contacts within the bureau we could have an order now...
(┛✧Д✧))┛彡┻━┻
I lost a lot that day, including respect to this fucked up system.
t(ಠ益ಠt) FUCK THE POLICE go eat a dick.10 -
Dear LinkedIn,
Try training your AI model without using captured data of real recruiters and their dodgy practices.2 -
Data Analyst: “the task failed in test, can we try running it in production?”
My life as a Data Engineer.5 -
I'm officially CTC.
Chief Technical Clown 🤡
How do I know? I've yet to write a single line of productive code today. I've spent the day purely as an administrative cog: writing emails, giving data to consultants, supporting juniors, and cleaning up the absolute hellscape that is also known as our Jira project.
I've become exactly what I hate.12 -
Not only do I write software, but now I help the managers view and understand our analytics, just like in kindergarten.
Now I'm forced to help them essentially fake data so investors are satisfied 🤡🔫
"Delete metrics X, Y, and Z for now, we don't want anyone to see them!"
"Change the label of this metric to 'unique user' views! (not total!)"
"Set all charts to cumulative so it looks like they are all up and to the right!"
Sigh.
This isn't what I signed up for.18 -
In the begining of time, when The Company was small and The Data could fit in some fucking excel sheets, Those Who Came Before implemented some java tool to issue invoices, notify customers and clear received payments.
Then came the Time Of The Great Expanse, when The Company grew to unthinkable levels. Headcount increased with each passing day, and The Data shows that everything was going great!
But when the future seemed bright, came The Stall-Out. The days when The Company could not expand as fast as it did before. And Those Who Came Before left, abandoning their Undocumented Java Tool to its own luck.
Those who came after knew nothing of the inner workings of the Undocumented Java Tool. They knew only that the magical Jar would take a couple fucking excel spreadsheets and spit out reports and send emails like magic.
And those were The Dark Days.
In the darkness, The Data grew to be a monster. Soon a fucking excel spreadsheet could not hold The Data contained any longer. Those Who Came After, fearing the wrath of The Undocumented Java Tool, dared not mess with its code. Instead, they fucking cut away the lowest volume transactions from the fucking input spreadsheet, and left the company to report the unbilled invoices as "surprise losses". Fucking script kiddies, were Those Who Came After.
Then, at The Darkest of Days (literally, Dec 21st), marched into the project The Six Witchers, who fear not the Demon of Refactoring.
This story is still unfolding. Will The Six Witchers manage to unravel the mysteries of The Undocumented Java Tool? Will they be able to reverse engineer the fucking black box, and scale it's magic into a modern application?
Will they decrease revenue forecasting error by at least 2% in a single strike?
Only the future will tell.16 -
I am a machine learning engineer and my boss expects me to train an AI model that surpasses the best models out there (without training data of course) because the client wanted ‘a fully automated AI solution’.13
-
Hey folks, I've just launched the https://okso.app - it is a drawing app that you may use to express, grasp, and organize your thoughts and ideas.
One key feature there is that you may organize your drawings/sketches into a hierarchical tree structure so that a large amount of data would be more manageable and less overwhelming.
I hope you find this app useful!11 -
I wrote pagerank algorithm in python for data mining course but my teacher told me to write it in R because according to him python can't be used as data mining tool.5
-
Data scientist: we need to whitelist a pod to connect to a database
Me: Whitelist? We don't use whitelists on private databases
DS: It's the new data warehouse database
Me: is it on <X> VPC?
DS: I'm not sure what that means but its ip is <real world ipv4>
Me: Are you hosting a publicly accessible database with all our end users information?!
DS: ...
Me: There goes our SOC2 audit controls...
DS: how long until you can white list it?
Me: I won't be whitelisting it. You need to put it on a private VPC and peer with the cluster, you'll have to rebuild all the Terraform and redeploy
DS: We didn't use Terraform because it takes too long, just white list the pods IP.
Me: No. I'm contacting the CISO and CTO...13 -
Today I saw a code written by my junior. Basically excel export. The laravel excel package provide great ways for optimization.
My junior instead did 6 times loop to modify the data before giving that data to the export package. We need to export around 50K users.
When I asking him why this ? He said it works and it's fast so what the issue ???
Noob , you have only 100 users in the database and production has 10 million.
Sometime I just want to kill him.15 -
I got two lines of code done today, and they were just changing numeric defines. Time to do that: 2 seconds.
Time for analysing the measurement data of various runs in order to know which numbers will work out: the rest of the day.6 -
My company deals in finances.
Part of our interview process is a coding challenge.
It is absolutely fascinating to me how few candidates point out that 'float' is an inappropriate data-type for currency. SMH.26 -
I asked my boss for a raise. Since we're data scientists, I basically plotted a column chart showing that he was paying me half of the european average. I said I was willing to negotiate but he was very 'discomfortable'.
Time to go find another job. Or second one 😭7 -
I worked once in a company which had this tourist app which should show places on map of the city. Unfortunately it slowed the App down to load more then a couple of places. Their solution was to limit the number of loaded places to teb and prohibited zomming out. I made it handle thousands of places at the same time. Main reason for the Performance issue was, that they sent all data they had about places big, big json objects with large text blobs) to the frontend. This part was easy, I instead sent only the data needed for the map like coordinates and icon type obviously. But still the backend struggeled hard with many objects from the DB, because they built a really shitty orm or what ever this was supposed to be: every line of data retrieved from the DB was immideatly wrapped in some class wich direved from another class which had some magic methods in it which caused some absurd loops over all other obejcts and even more DB queries in unexpected moments and also in the fucking constructor. So it turrned out that the map issue was only the top of the iceberg, since using any data from the DB was extremely expensive. The hard part was to understand the insaness of this abnormination and find the bottlenecks.8
-
If you're going to request CRITICAL changes to thousands of records in the database, and approve it through testing which is done on an exact replica of production, then tell me it was done incorrectly after the fact it has been implemented and you didn't actually review the changes made to the data or business logic that you requested then you are an idiot. Our staging environment is there to ensure all the changes are accurate you useless human. Its the data you provided, I didn't just magically pull it from thin air to make yours and my job a pain the ass.undefined stupid data analysts this is why health insurance costs a buttload do your job fuckface idiots9
-
Imagine saving Integers and Floats in a MySQL table as strings containing locale based thousand sepatators...
man... fickt das hart!
Wait, there's more!
Imagine storing a field containing list of object data as a CSV in a single table column instead of using JSON format or a separate DB table.... and later parsing it by splitting the CSV string on ";"...8 -
CTO: Research, problem analysis, customer need validations, and data based prioritisation is stupid.
Me: So, then why should we solve this problem?
CTO: Because my team invests a lot of time in here (read "because we build a shitty system in past without thinking and we are doing it again").
Me: I don't see this as a good idea.
CTO: I become emotional when I request product to align and they don't. We must solve this problem and not what customers want.
Me: I am not participating here.
CTO: And I want you to work on weekends to support my team.
Me: *disconnects*3 -
Googe Analytics illegal in EU.
Fucking love you Europeans.
And for that matter any analytics software that sends or stores data in US serves, because it's against GDPR.
https://techcrunch.com/2022/01/...28 -
Forgot to secure my mongo db instance, found half the data gone, and a new db holding me at ransom , learn_how_to_recover_your_data , good thing offline backups are a thing.
Recovered in no time, never will I ever repeat this mistake3 -
Thank God the week 233 rants are over - was getting sick of elitist internet losers.
The worst security bug I saw was when I first started work as a dev in Angular almost year ago. Despite the code being a couple of years old, the links to the data on firebase had 0 rules concerning user access, all data basically publicly available, the API keys were uploaded on GitHub, and even the auth guard didn't work. A proper mess that still gives me the night spooks to this day.3 -
Urgh, fucking excel!
Why the fuck can't you handle a few thousand calculations you dumb ass piece of shit.
I am this close to... fuck, it crashed. 🤦♂️
I fucking give up.
Time to strap this data to a DB instead.rant formulas are great they said useless pile of shit clowns shit better then you you had 1 job stop fucking crashing excel7 -
I am conducting technical interviews for about 10 years now.
I swear to god, the applicants keep getting dumber and dumber.
Getting more and more ashamed to talk about data structures, design patters or even the most basic algorithms, everyone with a graduation badge from udemy is now a software engineer. Fuck this shit.17 -
FaceBook, over the years you’ve proven that you can’t be trusted and you still have the nerve to ask for this type of personal data?5
-
BUZZWORD BUZZWORD AAAAAH
ARTIFICIAL INTELLIGENCE
BLOCKCHAIN
ALGORITHM
CLOUD
IOT
BIG DATA
SaaS
DEVOPS
5G
AR
VR
AAAAH BUZZWORD HERE BUZZWORD THERE4 -
PM is such a fucking cunt
telling me that my data structures describing the layout of binary data would be confusing for devs, and that we shall introduce
typedef fuckingRetardedObfuscatingName uint8_t;
in our code. everyone is fine with the concepts i provide to describe this binary data, not only at our company but also in other software i've worked on and common standards i've worked with, we work like that and every fucking idiot knows what a uin8_t is.
you fucking braindead imbecile have no fucking idea how we work and you don't care, you don't even try to understand what we are doing.
god i hope you die being hit by a fucking bus or something8 -
Imagine
websites didn't use unnecessary cookies,
JavaScript was only used when needed,
no trackers, no ads, no telemetry, no user
data saved when it doesn't benefit the user.
*Wakes up in cold sweat*9 -
funniest thing today: PM asking me to create a Jira subtask for EACH class / data type of the data model that I'm CURRENTLY concipating / designing in this story.
maybe I should write a Jira / Enterprise Architect integrated tool that updates Jira tasks based on my modeling actions, and count minutes until our sysadmin arrives at my desk.
jeez, that guy really has a fetish for kafkaesque bureaucracy.🤦♀️4 -
spent a week on medical leave, came back to utter chaos? my boys, you can't do my shitty job for a week? lmao
everything was dropped halfway through the week, a lot of old data live, shit passing the deadline. and I don't do much, just press the freaking buttons 🤦♀️ -
Today in horror stories: setting up the local dev environment for a project I'm taking over creates changes and notifications in prod.
Oh and my local dev env is somehow receiving data from actual patients. I don't know how.9 -
At this point, I just feel bad for my coworker.
No, I am not frustrated or angry, just feeling terribly bad for her as how difficult life must be for someone so dumb.
We are introducing a new method to track some data in our product, like total number of sign-ups, DAU, etc.
Now the implementation is already WIP and this is known to all.
The dev has documented the approach where he has mapped the screen name, a screenshot, and a snippet of the schema that tracks that particular screen.
I kid you not guys, this coworker somehow landed on that document and started some scientific study to try and extract data.
Yes, she looked at the schema screenshot and spent like few hours trying to decode it to figure out the sign-ups and DAU.
Data via a screenshot in a document. I can't even express it.
And then texts me in panic mode that she isn't able to access the data because the file is . jpeg within a document.
I asked where is she executing the schema, because I thought she is joking initially. She said she doesn't know and asked me where she should execute it.
My mind is numb. Life must be real hard when you are so fucking dumb.21 -
This is just straight up a joke. Head of data science has had us implementing something based on a paper.
I raised some "concerns" about it a while ago. Found out today they'd not even read the paper. We're pretty sure what we've been asked to work on doesn't solve the problem it's meant to solve...3 -
"So Alecx, how did you solve the issues with the data provided to you by hr for <X> application?"
Said the VP of my institution in charge of my department.
"It was complex sir, I could not figure out much of the general ideas of the data schema since it came from a bunch of people not trained in I.T (HR) and as such I had to do some experiments in the data to find the relationships with the data, this brought about 4 different relations in the data, the program determined them for me based on the most common type of data, the model deemed it a "user", from that I just extracted the information that I needed, and generated the tables through Golang's gorm"
VP nodding and listening intently...."how did you make those relationships?" me "I started a simple pattern recognition module through supervised mach..." VP: Machine learning, that sounds like A.I
Me: "Yes sir, it was, but the problem was fairly easy for the schema to determ.." VP: A.I, at our institution, back in my day it was a dream to have such technology, you are the director of web tech, what is it to you to know of this?"
Me: "I just like to experiment with new stuff, it was the easiest rout to determine these things, I just felt that i should use it if I can"
VP: "This is amazing, I'll go by your office later"
Dude speaks wonders of me. The idea was simple, read through the CSV that was provided to me, have the parsing done in a notebook, make it determine the relationships in the data and spout out a bunch of JSON that I could use. Hook it up to a simple gorm golang script and generate the tables for that. Much simpler than the bullshit that we have in php. I used this to create a new database since the previous application had issues. The app will still have a php frontend and backend, but now I don't leave the parsing of the data to php, which quite frankly, php sucks for imho. The Python codebase will then create the json files through the predictive modeling (98% accuaracy) and then the go program will populate the db for me.
There are also some node scripts that help test the data since the data is json.
All in all a good day of work. The VP seems scared since he knows no one on this side of town knows about this kind of tech. Me? I am just happy I get to experiment. Y'all should have seen his face when I showed him a rather large app written in Clojure, the man just went 0.0 when he saw Lisp code.
I think I scare him.12 -
Corp: you will get a four hour assignment to work out
Me: cool nice.
Corp: here it is, build a dragon with conflicting requirements, stocks but without any form of pricing mixed in. Then slay that dragon and post it to the static backend we created.
Me: cringe much?
Corp: yeah, you can spend more than 4h but be sure to spice things up abit. Since it is frontend, and all we spin up from the backend is flat data. But it must exhale an exciting user experience.
Me: stop the cringe pls!6 -
Hired a new BI developer. She tested reasonably ok in SQL, and certainly showed good strengths in visualising data, plus had a good attitude in the interview. We hired her. She broke her laptop the first day. We got her another then she complained the camera didn't work but didn't realise the lever in front of the camera was to move the privacy shutter off and on.
Assigned her some work of taking queries that are used in a BI tool that targets the transactional database directly, and re-jigging them for Snowflake which we're using as a data warehouse now, aggregating all our data into one place. Yet, she's struggling to understand why the SQL query she's pasted in doesn't work as-is.
I go over it again; the source schemas and tables are this, but in Snowflake we've named them this. She then bemoans how much work that is to change them all - I say use find and replace. She then struggles with Snowflake syntax errors and asks for a guide on T-SQL to Snowflake. I show her Google and say "this is what I did when I hit these problems - search for 'Snowflake equivalent to T-SQL getdate()' or 'how to get current date in Snowflake' but she still doesn't understand. I ask if she's every had to work between T-SQL and MySQL or MySQL and PostgreSQL or Oracle and so on and she says yes. I say the syntax isn't the same, is it? And she goes oh, now I understand.
She scored reasonably in her SQL test but I'm now concerned there's something fundamental missing in her grasp of SQL. I gave her a detailed demo of the tools, I explained in the interview and on her start about our move to a data warehouse for all our apps, and put her through some training plus gave her time to work through our Confluence pages - not expecting she'll remember everything, but more to ensure she recalls they exist and what the general contents are.
Anyhow, that's my rant.7 -
What's a bigger sin.
Returning a status code of 200 and then the message body saying "An Error Occurred"
or
Only performing data validation on the frontend.19 -
The reason I don't have friends is because I embed Python code in C++, instead of using pybind like the Linux gods intended us to 😅
If you want to load data from Pandas DataFrame in a C++, I can put some examples on my Github.6 -
EoS1: This is the continuation of my previous rant, "The Ballad of The Six Witchers and The Undocumented Java Tool". Catch the first part here: https://devrant.com/rants/5009817/...
The Undocumented Java Tool, created by Those Who Came Before to fight the great battles of the past, is a swift beast. It reaches systems unknown and impacts many processes, unbeknownst even to said processes' masters. All from within it's lair, a foggy Windows Server swamp of moldy data streams and boggy flows.
One of The Six Witchers, the Wild One, scouted ahead to map the input and output data streams of the Unmapped Data Swamp. Accompanied only by his animal familiars, NetCat and WireShark.
Two others, bold and adventurous, raised their decompiling blades against the Undocumented Java Tool beast itself, to uncover it's data processing secrets.
Another of the witchers, of dark complexion and smooth speak, followed the data upstream to find where the fuck the limited excel sheets that feeds The Beast comes from, since it's handlers only know that "every other day a new one appears on this shared active directory location". WTF do people often have NPC-levels of unawareness about their own fucking jobs?!?!
The other witchers left to tend to the Burn-Rate Bonfire, for The Sprint is dark and full of terrors, and some bigwigs always manage to shoehorn their whims/unrelated stories into a otherwise lean sprint.
At the dawn of the new year, the witchers reconvened. "The Beast breathes a currency conversion API" - said The Wild One - "And it's claws and fangs strike mostly at two independent JIRA clusters, sometimes upserting issues. It uses a company-deprecated API to send emails. We're in deep shit."
"I've found The Source of Fucking Excel Sheets" - said the smooth witcher - "It is The Temple of Cash-Flow, where the priests weave the Tapestry of Transactions. Our Fucking Excel Sheets are but a snapshot of the latest updates on the balance of some billing accounts. I spoke with one of the priestesses, and she told me that The Oracle (DB) would be able to provide us with The Data directly, if we were to learn the way of the ODBC and the Query"
"We stroke at the beast" - said the bold and adventurous witchers, now deserving of the bragging rights to be called The Butchers of Jarfile - "It is actually fewer than twenty classes and modules. Most are API-drivers. And less than 40% of the code is ever even fucking used! We found fucking JIRA API tokens and URIs hard-coded. And it is all synchronous and monolithic - no wonder it takes almost 20 hours to run a single fucking excel sheet".
Together, the witchers figured out that each new billing account were morphed by The Beast into a new JIRA issue, if none was open yet for it. Transactions were used to update the outstanding balance on the issues regarding the billing accounts. The currency conversion API was used too often, and it's purpose was only to give a rough estimate of the total balance in each Jira issue in USD, since each issue could have transactions in several currencies. The Beast would consume the Excel sheet, do some cryptic transformations on it, and for each resulting line access the currency API and upsert a JIRA issue. The secrets of those transformations were still hidden from the witchers. When and why would The Beast send emails, was still a mistery.
As the Witchers Council approached an end and all were armed with knowledge and information, they decided on the next steps.
The Wild Witcher, known in every tavern in the land and by the sea, would create a connector to The Red Port of Redis, where every currency conversion is already updated by other processes and can be quickly retrieved inside the VPC. The Greenhorn Witcher is to follow him and build an offline process to update balances in JIRA issues.
The Butchers of Jarfile were to build The Juggler, an automation that should be able to receive a parquet file with an insertion plan and asynchronously update the JIRA API with scores of concurrent requests.
The Smooth Witcher, proud of his new lead, was to build The Oracle Watch, an order that would guard the Oracle (DB) at the Temple of Cash-Flow and report every qualifying transaction to parquet files in AWS S3. The Data would then be pushed to cross The Event Bridge into The Cluster of Sparks and Storms.
This Witcher Who Writes is to ride the Elephant of Hadoop into The Cluster of Sparks an Storms, to weave the signs of Map and Reduce and with speed and precision transform The Data into The Insertion Plan.
However, how exactly is The Data to be transformed is not yet known.
Will the Witchers be able to build The Data's New Path? Will they figure out the mysterious transformation? Will they discover the Undocumented Java Tool's secrets on notifying customers and aggregating data?
This story is still afoot. Only the future will tell, and I will keep you posted.6 -
Just saw this:
Add comma's to your passwords to mess with the csv file they will be dumped in after a data breach9 -
I downloaded Lapsus$ source code leaks from samsung, nvidia and microsoft, looked at them and I think I’ll delete it cause I don’t like shitty code on my personal computer.1
-
More adventures of working with data scientists:
Doing combinatorial analysis.
One of the post processing scripts throws away combinatorial information, analyses significance of individual data points instead.
Code was so complicated, nobody noticed.1 -
FUCK IT
After YEARS of research, I couldn't find a single working load testing tool
So this weekend I created my own. With blackjack and hookers.
It's limited to my app, so not reusable, but wow in 1 weekend I got more data and found more infra problems than in the past 3 years.18 -
Facebook is a giant piece of shit. Not only is their platform a massive contributor to mental illness, even their API's are fucking garbage. I'm trying to use their ads API and what it does is it hijacks the entire fucking request so you can't even extract data from the request after calling it. Fuck Facebook and everything they've ever "contributed" to society.5
-
I was inspired by the abundance and uniformity of modern data-selling startups. Despite being different companies, they have a lot in common: questionable ethics, lack of a real business model other than selling data, pseudo-approachable and friendly bullshit aesthetics, talks about dIvErSitY and eQuALitY. Now, I propose one more thing for them to have in common — this logo. I hereby announce it to be public domain now, free to use for all.4
-
Apply for a data engineer role.
Get invited for a data science interview.
HR says they're building AI and I were to supervise another person writing its algorithm.
It's a media company.
*Risitas intensifies*6 -
Our company has internal webpage to request software, be it freeware or licensed.
Today, I found there "Software engineering bundle" designated for "software developers and data scientists who require advanced compute and data processing tools".
The software bundle contains PuTTY, 7-zip and Notepad++.6 -
PyTorch.
2018: uh, what happens when someone uses a same name attack? - No big deal. https://github.com/pypa/pip/...
2020: I think that's a security issue. - Nanana, it's not. https://github.com/pypa/pip/...
2022: malicious package extracts sensitive user data on nightly. https://bleepingcomputer.com/news/...
You had years to react, you clowns.6 -
Recently I launched the minimalistic online drawing app https://okso.app. I wanted it to be a place where people could do fast, ad-hoc, napkin-based-like explanations of any concept as if you are sitting with your friend and trying to explain him/her something during lunch. Don't ask me why it is needed, I was just experimenting.
So, the first concept I've tried to explain with sketches was the Data Structures. Without further ado, here is the interactive ✍🏻 https://okso.app/showcase/... showcase that you may play with.
Of course, not all data structures are covered. And of course, this is not comprehensive material, but rather a cheatsheet that would create visual hints and associations for the following data structures:
- Linked List
- Doubly Linked List
- Queue
- Stack
- Hash Table (with hash collision resolution)
- Tree (including the Binary Search Tree)
- Heap (including Mean Heap and Max Heap)
- Trie
- Graph
Each box on the sketch is clickable, so you may dig into the data structure you're interested. For example `Heap → Max Heap`, or `Heap → Min Heap`, or `Heap → Array Representation`.
The sketches are split into so-called Pages just to make it easier to grasp them, so the users stay focused on one concept at a time, they see the relationship between the concept, and thus, hopefully, they are not getting overwhelmed with seeing a lot of information at the same time on one drawing/page.
Each page has a link to the source-code examples that are implementing the data structure on JavaScript.
The full list you may find in the ✍🏻 https://okso.app/showcase/... showcase.
I hope you find this showcase useful and I hope it will be a good visual cheatsheet-like complement to your data structure knowledge.12 -
Stakeholder: Users are connecting invalid memberships to their web accounts. They shouldn’t be able to do that.
Me: Their memberships were valid when they set up the account. Your team’s record de-duping project is the issue here. You decided to mark those memberships as invalid.
I’m real tired of this stakeholder acting like this is a website issue or user error. Plus, this chaos could have been avoided if they and other involved stakeholders had just cc’d me on this de-duping project. I would have said their approach was not a good idea. But they didn’t because they want to do what’s convenient for them. If they want to be a reliable source of truth for our data, then they need to be responsible with how they’re handling that data.devrant why are you so irresponsible with our data this is not user error i’m real tired of this stakeholder2 -
It’s been so long since I posted but this time it’s juicy again.
I got a coworker, no prio experience but already a year and few months into the job. He’s bad.
Magnitudes of bad!
We’re trying to teach him but to no avail. Everything about him sucks, major ballsack to be exact.
His attitude is to avoid every task, finishes nothing and then starts something new.
„Did you do X like we told you to?“
„No I started on Y, because I thought it [looks better, seems more interesting, thought that X is useless…]“
When you ask him much is done he is always „almost“ finished and needs your help on the „last 5-10%“. Yeah fuck that!
But that guy has a talent, his talent is to always give you technically correct answers which actually are complete bullshit.
„What are you doing at your job?“
„Staring at a screen and typing things.“ dude what?
That guy used the excuse „I can’t do maths“ on everything.
For an exam he had to calculate how long it would take to reach a certain amount if you would get some interest in that every year.
He asked the teacher for the formula. During the exam! And when the teacher didn’t want to give it to him he wrote plainly „can’t do maths“ on the paper and left
His code is of a quality as if he would write his first line in a week and then has the audacity to blame me and the colleagues for not explaining it right.
Ok you might think now we’re teaching him bad, or are too impatient. But honestly if you have to explain how to do a for loop for over about 15 months and get that attitude I think you get the right to be angry. I don’t mind explaining on how things work, even for the hundredth time, but then don’t tell me you understood, go behind my back, complain at a colleague how bad I explained, get explained by him and then do it again until you whored yourself through the whole staff!
It’s like he got the mind swiper from Men in black at home. Every day he hits the reset button.
He had a week of just changing indentation on a html file. Why? Because he wanted to find his style.
Yeah his style
if(a==b){
console.log(a);
}
else {
console.log(b)
}
And to produce code like that it takes him atleast 4 hours of trial and error.
And at the same time he goes arround and boasts what a super good programmer he his and that he can do some project work for them.
How we found out? Because he started working in those projects during work time at the office and asked us how to do things.
And he does so like a complete bastard!
Broken sql query? “No that query is perfect as it is, it’s supposed to show no results! But, just in theory, if I wanted to show some results, what would I need to change?”
I’m so mad about it and pissed on a personal level because he goes around blames everyone and the world for his short comings8 -
Today I told to my Project Manager that after one year of taking care ( explaining thousands of lines of codes) to an external team ( another company) to migrate our application from a monolith to microservices + react, that the React UI they build looks like shit.
He replied "at least it works".
Now I must find the courage to tell him that it doesn't work correctly because instead of a simple *migration* the external team rewrote some algorithms used in a bank application and now the data are wrong.
advices ?1 -
A fellow <insert some data science title here> giving me more acknowledgement than all of the old guys running the show and, erm, "leading"?
Fucker left me behind, good for him. -
Just discovered 20+TiB of Splunk data in our AWS account today. We haven't used Splunk for almost a year and a half...7
-
Why the fuck does every operations app do popouts now? I don't want a simple view of the data, I want all the data so I can compare it together.
It's not like you're saving any bandwidth! All the data is there, I can fucking see it 👀 in the dev tools!
I hate how every product now desperately tries to be like their competitor and everyone fails at it because everyone is copying everyone else.7 -
At age 6 I was deemed as an idiot savant. Coding is boring for me now. Age 7-10: I worked for an underground agency that was focused on harvesting people's organ data from MRI machines to predict the economic future. 10-14: I experimented with smoking crack to increase finger efficiency. Since then I've quit, and I've been living in Miami trying to create a lofi industrial folk album using nothing but a TI-84, some wire, and an old fender amp.2
-
When I was in college OOP was emerging. A lot of the professors were against teaching it as the core. Some younger professors were adamant about it, and also Java fanatics. So after the bell rang, they'd sometimes teach people that wanted to learn it. I stayed after and the professor said that object oriented programming treated things like reality.
My first thought to this was hold up, modeling reality is hard and complicated, why would you want to add that to your programming that's utter madness.
Then he started with a ball example and how some balls in reality are blue, and they can have a bounce action we can express with a method.
My first thought was that this seems a very niche example. It has very little to do with any problems I have yet solved and I felt thinking about it this way would complicate my programs rather than make them simpler.
I looked around the at remnants of my classmates and saw several sitting forward, their eyes lit up and I felt like I was in a cult meeting where the head is trying to make everyone enamored of their personality. Except he wasn't selling himself, he was selling an idea.
I patiently waited it out, wanting there to be something of value in the after the bell lesson. Something I could use to better my own programming ability. It never came.
This same professor would tell us all to read and buy gang of four it would change our lives. It was an expensive hard cover book with a ribbon attached for a bookmark. It was made to look important. I didn't have much money in college but I gave it a shot I bought the book. I remember wrinkling my nose often, reading at it. Feeling like I was still being sold something. But where was the proof. It was all an argument from authority and I didn't think the argument was very good.
I left college thinking the whole thing was silly and would surely go away with time. And then it grew, and grew. It started to be impossible to avoid it. So I'd just use it when I had to and that became more and more often.
I began to doubt myself. Perhaps I was wrong, surely all these people using and loving this paradigm could not be wrong. I took on a 3 year project to dive deep into OOP later in my career. I was already intimately aware of OOP having to have done so much of it. But I caught up on all the latest ideas and practiced them for a the first year. I thought if OOP is so good I should be able to be more productive in years 2 and 3.
It was the most miserable I had ever been as a programmer. Everything took forever to do. There was boilerplate code everywhere. You didn't so much solve problems as stuff abstract ideas that had nothing to do with the problem everywhere and THEN code the actual part of the code that does a task. Even though I was working with an interpreted language they had added a need to compile, for dependency injection. What's next taking the benefit of dynamic typing and forcing typing into it? Oh I see they managed to do that too. At this point why not just use C or C++. It's going to do everything you wanted if you add compiling and typing and do it way faster at run time.
I talked to the client extensively about everything. We both agreed the project was untenable. We moved everything over another 3 years. His business is doing better than ever before now by several metrics. And I can be productive again. My self doubt was over. OOP is a complicated mess that drags down the software industry, little better than snake oil and full of empty promises. Unfortunately it is all some people know.
Now there is a functional movement, a data oriented movement, and things are looking a little brighter. However, no one seems to care for procedural. Functional and procedural are not that different. Functional just tries to put more constraints on the developer. Data oriented is also a lot more sensible, and again pretty close to procedural a lot of the time. It's just odd to me this need to separate from procedural at all. Procedural was very honest. If you're a bad programmer you make bad code. If you're a good programmer you make good code. It seems a lot of this was meant to enforce bad programmers to make good code. I'll tell you what I think though. I think that has never worked. It's just hidden it away in some abstraction and made identifying it harder. Much like the code methodologies themselves do to the code.
Now I'm left with a choice, keep my own business going to work on what I love, shift gears and do what I hate for more money, or pivot careers entirely. I decided after all this to go into data science because what you all are doing to the software industry sickens me. And that's my story. It's one that makes a lot of people defensive or even passive aggressive, to those people I say, try more things. At least then you can be less defensive about your opinion.53 -
Government applications meant to handle private data made in the laziest way using cordova. Hell yea. Bonus points for using the developers account with his first and last name as publishing company to put the app on playstore and not their own official one (not that they have one). Whats not to love. Cant wait to beautify that js to find out that the calls are http and not https.3
-
Riddle me this
Client wants solution based on open source software.
Any additional software that I write (let's say, an offline store plugin for Feast feature store) to add missing functionality has to be closed source.
Fuck you. Intellectual property my ass. You and me wouldn't even have projects if it werent for OSS.
Good luck maintaining the plugin after I am gone.
I'm doing a lot of work and will have close to nothing to show to future employers.
(BTW, if it were for the old Microsoft model of code source, I would have never become a programmer of any sort. God bless OSS)3 -
Reading a paper on DBMS architectures, and I quote:
"In the seventies, the scientific discussion
in the database (DB) area was dominated
by heavy arguments concerning the most
suitable data model, sometimes called a
religious war."
... and here I thought language argument was a religious war. :/8 -
Someone created a 0-followers private Twitter account and posted something to try out the new views count feature.
It raked dozens of views in a couple hours.
HOW?!?
Source: https://twitter.com/briggityboppity...
It looks like a funny data reverse-engineering exercise, so let's try and figure out what is going on.
Hypothesis 1) it is the OP's own views.
Reasonable, but unlikely if what OP says about not checking it for hours is true.
H2) It's some background job in OP's device that is refreshing OP's own latest tweets, so even without human interaction technically H1 is true. It would be some really shoddy engineering to count eye-less page views, but that's also what managers would demand.
H3) it's some internal Twitter automated function like back up, replication, indexing and word count.
See H2, it would be even dumber to count that as page views.
H4) it's some internal human reviewing for a keyword that could be associated with porn (in this case, "butts"). Really? dozens of humans to review a no-impact single post? They would have to employ hundreds of thousands of reviewers.
H5) it's some page-loading shit, like thousands of similar tweets get stored in the same index hash page and end up counting as a view in all of them every time someone loads the index page. It would be like counting every hit in the namenode as a hit in every data asset in it's Hadoop partition, or every hit in a storage block as a hit in each of it's files.
Duuuumb and kinda like H3.
H6) page views are just a fraud to scam investors. Maybe it's a "most Blockchain transactions are fake" situation, maybe it's a "views get more engagement if you don't think a lot about it" situation, maybe it's a "we don't use the metric system to count page views" situation.
All of them are very dumb.
Other hypothesis or opinions?10 -
The client: "I'm trying to export as HTML a few data we collected in 1 month with your software, why does it take so much???"
The length of the generated HTML file for the "few data":7 -
Wasted all morning trying to work on my defense. Brain is not responding. Then sat for some data exploration cuz I got tired of wasting time and brain is like "woah! I love this! Let's do this!". 😐
My defense is gonna be a shitshow. FML. ☠️1 -
what the fuck is up with devs who always send screenshots of code and/or log files? In Slack, which has great functionality for formatting text snippets in a variety of languages and data types?! screenshots of code are really a pet peeve lately. You can't copy the text or click on any urls or do *anything* with a fucking screenshot. so dumb.6
-
This is the third part of my ongoing series "The Ballad of the Six Witchers and the Undocumented Java Tool".
In this part, we have the massive Battle of Sparks and Storms.
The first part is here: https://devrant.com/rants/5009817/...
The second part is here: https://devrant.com/rants/5054467/...
Over the last couple sprints and then some, The Witcher Who Writes and the Butchers of Jarfile had studied the decompiled guts of the Undocumented Java Beast and finally derived (most of) the process by which the data was transformed. They even built a model to replicate the results in small scale.
But when such process was presented to the Priests of Accounting at the Temple of Cash-Flow, chaos ensued.
This cannot be! - cried the priests - You must be wrong!
Wrong, the Witchers were not. In every single test case the Priests of Accounting threw at the Witchers, their model predicted perfectly what would be registered by the Undocumented Java Tool at the very end.
It was not the Witchers. The process was corrupted at its essence.
The Witchers reconvened at their fortress of Sprint. In the dark room of Standup, the leader of their order, wise beyond his years (and there were plenty of those), in a deep and solemn voice, there declared:
"Guys, we must not fuck this up." (actual quote)
For the leader of the witchers had just returned from a war council at the capitol of the province. There, heading a table boarding the Archpriest of Accounting, the Augur of Economics, the Marketing Spymaster and Admiral of the Fleet, was the Ciefoh Seat himself.
They had heard rumors about the Order of the Witchers' battles and operations. They wanted to know more.
It was quiet that night in the flat and cloudy plains of Cluster of Sparks and Storms. The Ciefoh Seat had ordered the thunder to stay silent, so that the forces of whole cluster would be available for the Witchers.
The cluster had solid ground for Hive and Parquet turf, and extended from the Connection River to farther than the horizon.
The Witcher Who Writes, seated high atop his war-elephant, looked at the massive battle formations behind.
The frontline were all war-elephants of Hadoop, their mahouts the Witchers themselves.
For the right flank, the Red Port of Redis had sent their best connectors - currency conversions would happen by the hundreds, instantly and always updated.
The left flank had the first and second army of Coroutine Jugglers, trained by the Witchers. Their swift catapults would be able to move data to and from the JIRA cities. No data point will be left behind.
At the center were thousands of Sparks mounting their RDD warhorses. Organized in formations designed by the Witchers and the Priestesses of Accounting, those armoured and strong units were native to this cloudy landscape. This was their home, and they were ready to defend it.
For the enemy could be seen in the horizon.
There were terabytes of data crossing the Stony Event Bridge. Hundreds of millions of datapoints, eager to flood the memory of every system and devour the processing time of every node on sight.
For the Ciefoh Seat, in his fury about the wrong calculations of the processes of the past, had ruled that the Witchers would not simply reshape the data from now on.
The Witchers were to process the entire historical ledger of transactions. And be done before the end of the month.
The metrics rumbled under the weight of terabytes of data crossing the Event Bridge. With fire in their eyes, the war-elephants in the frontline advanced.
Hundreds of data points would be impaled by their tusks and trampled by their feet, pressed into the parquet and hive grounds. But hundreds more would take their place. There were too many data points for the Hadoop war-elephants alone.
But the dawn will come.
When the night seemed darker, the Witchers heard a thunder, and the skies turned red. The Sparks were on the move.
Riding into the parquet and hive turf, impaling scores of data points with their long SIMD lances and chopping data off with their Scala swords, the Sparks burned through the enemy like fire.
The second line of the sparks would pick data off to be sent by the Coroutine Jugglers to JIRA. That would provoke even more data to cross the Event Bridge, but the third line of Sparks were ready for it - those data would be pierced by the rounds provided by the Red Port of Redis, and sent back to JIRA - for good.
They fought for six days and six nights, taking turns so that the battles would not stop. And then, silence. The day was won, all the data crushed into hive and parquet.
Short-lived was the relief. The Witchers knew that the enemy in combat is but a shadow of the troubles that approach. Politics and greed and grudge are all next in line. Are the Witchers heroes or marauders? The aftermath is to come, and I will keep you posted.4 -
Around a decade ago, I was fiddling with ajax in jQuery. This piece of code had me stumped for around 2 days, why? Because success is written with 1 final S in my native language...
$.ajax({
'succes' : function(data) { },
});7 -
Hiring a third party to help us with something...
Third party: yeah okay, we know what we need. Can we get access to your git repo
Me: sure, I'll make sure you'll get it
(To the admins): hey can you get them access to our git server?
Admins: did they sign the personal data processing contract?
Me: oh they won't work with any personal data. It's a dev server and they only need access to the source code. And the usual contracts and NDAs are already done
Admins: well we still need the other one.
... Sure. Why not. Just delays the start of the process for... Like a week and a half until that useless bit of paper has passed through all the necessary departments. Not like time's an issue. Right?8 -
Someone should definitely collect all the rants of devRant and fine tune an existing model with that data so it’s able to produce new rants. I wonder what it would come up with 🤔9
-
I recently tried to apply the same data analytics rationale that I use at work to my personal life. This is not a rant, it is more like an data storytelling of an actual use case I would like some input on.
I set a goal - gotta thin up a bit and calm down my ticker - and got a (almost unreasonably expensive) field expert consultant to yell at me about it for a couple hours.
I unravel the metrics - there is like a million weight-related KPIs and most say nothing at all. I have never seen an non-infrastructure measurable subject that could not be resumed to 2-5 performance metrics. I got overall weight, how well my nine-years-old business suit fits me, heart rate, and day-after relative muscle pain (it will make sense soon).
Then its data-pipeline time. I bought a cheap weight scale and smartwatch, and every morning I input the data in an app. Yes, I try to put on the suit every morning. It still does not fit.
After establishing a baseline, I tried to fit different approaches. Doing equipment-free exercises, going to the gym, dieting. None was actually feasible in the long run, but trying different approaches does highlight the impacts and the handling profile of each method.
Looking at the now-gathered data, one thing was obvious - can't do dieting because it is not doable to have a shopping list and meals for me and another for the family.
Gym is also off the table - too much overhead. I spend more time on the trip there and back than actually there.
And home exercise equipment is either super crappy or very expensive. But it is also the most reasonable approach.
So it is solutions time. I got a nice exercise bycicle (not a peloton), an yoga mat (the wife already had that one) and an exercise program that uses only those two resources. Not as efficient without dieting, not as measurable and broad as the gym, but it fits my workflow. Deploy to production!
A few months pass and the dataset grows. The signal is subtle but has support - it works! The handling, however, needs improvement, since I cannot often enough get with the exercise program. Some mornings are just after some hard days.
I start thinking about what else I can improve in the program, but it is already pretty lean and full of compromises.
So I pull an engineer and start thinking about the support systems and draft profile. What else could be draining my willpower and morning time?
Chores. Getting the kids ready for school, firing up the moka pot, setting the off-brand roomba, folding the overnight-dried clothes, cooking breakfast, doing the dishes, cleaning the toilets. All part of my morning routine. It might benefit from some automation.
Last month I got that machine our elders call "wasteful" and "useless crap lazy entitled Americans invented because they feel oh-so-insulted for simply doing something by hand like everyone always did" - a "dish-washer".
Heh, I remember how hard was to convince my mother-in-law that an remote-controled electric garage door would not make she look like an spoiled brat.
Still to early to call, but I think that the dishwasher just saved me about 25 mins every morning. It might be enough to save willpower for me to do more exercise.
This is all so reflective of all data analytics cases really are out in the wild - the analytics phase seems so small compared to the gathering and practical problem-solving all around. And yet d.a. is what tells you that you are doing the wrong thing all along. Or on what you should work next.7 -
One of our customers wants our mobile app to log out the user after 15 minutes of inactivity because of SeCuRiTy…
Why? The phones protect the apps with their hardware encryption from any malicious access.
And we are not dealing with super sensitive data here like some banking app or so.
Why do some people want to have bad UX for no reason?12 -
IT CAN'T BE THAT HARD
1) A CONTROLLER RETURNS HTTP RESPONSES, computed using data received from
2) A SERVICE\MANAGER\YOURMOTHER, which fetches data from a DB\external service\whatever
LITERALLY 2 FUCKING STEPS. I'LL TAKE THAT "SENIOR" IN YOUR TITLE AND CHISEL IT ON YOUR FOREHEAD SO YOU'LL REMEMBER WHAT YOU'RE SUPPOSED TO BE WHEN YOU COMMIT THIS FUCKING GARBAGE1 -
Headhunter called about a rejection for an assignment I did:
Assignment had malformed data examples
Assignment had unrealistic timespan for completion
Assignment used item stocks for a shop setup
Assignment didn't use any prices just item stocks
Who builds a webshop without prices in the first place?
So done with this job hunting assessment bullshit.3 -
A member of infra team:
"Hey, we are migrating to a Microsoft office tools and we migrated your google drive data to One drive"
I go and check the new One Drive account and it's empty. So I point that out and the reply was:
"You should export your files to a zip and then import them to One Drive"
I didn't want to waste my time showing him that he is just contradicting himself in less than 5 mn and in two nearly consecutive messages.
I need more patience.2 -
I was 1 hour into an 8 hour data warehouse build on my VM, when Windows popped up with a dialogue informing me that it would initiate Shut Down in 10 minutes. No button to Cancel or Delay, just “Okay”.
To Microsoft, I offer a hardy Dickensian “GOOD AFTERNOON, SIR!”4 -
My God is map development insane. I had no idea.
For starters did you know there are a hundred different satellite map providers?
Just kidding, it's more than that.
Second there appears to be tens of thousands of people whos *entire* job is either analyzing map data, or making maps.
Hell this must be some people's whole *existence*. I am humbled.
I just got done grabbing basic land cover data for a neoscav style game spanning the u.s., when I came across the MRLC land cover data set.
One file was 17GB in size.
Worked out to 1px = 30 meters in their data set. I just need it at a one mile resolution, so I need it in 54px chunks, which I'll have to average, or find medians on, or do some sort of reduction.
Ecoregions.appspot.com actually has a pretty good data set but that's still manual. I ran it through gale and theres actually imperceptible thin line borders that share a separate *shade* of their region colors with the region itself, so I ran it through a mosaic effect, to remove the vast bulk of extraneous border colors, but I'll still have to hand remove the oceans if I go with image sources.
It's not that I havent done things involved like that before, naturally I'm insane. It's just involved.
The reason for editing out the oceans is because the oceans contain a metric boatload of shades of blue.
If I'm converting pixels to tiles, I have to break it down to one color per tile.
With the oceans, the boundary between the ocean and shore (not to mention depth information on the continental shelf) ends up sharing colors when I do a palette reduction, so that's a no-go. Of course I could build the palette bu hand, from sampling the map, and then just measure the distance of each sampled rgb color to that of every color in the palette, to see what color it primarily belongs to, but as it stands ecoregions coloring of the regions has some of them *really close* in rgb value as it is.
Now what I also could do is write a script to parse the shape files, construct polygons in sdl or love2d, and save it to a surface with simplified colors, and output that to bmp.
It's perfectly doable, but technically I'm on savings and supposed to be calling companies right now to see if I can get hired instead of being a bum :P20 -
I lost 2TB of family photos and videos a few years ago by dropping a single hard drive. Nearly all of it was later found on people's computers who forgot to clean them up when we got the NAS. I'm not so eager to delete data once it's backed up ever since.3
-
I am building my portfolio website and added a contact section. In the API call to the backend, I am logging potential API failures to Firebase Analytics. Is it ethical to include the request data (content of the contact form) in log data?5
-
"Hey can you make this excel report for me real quick? Here are the columns, you gotta get them from this table in the database. Shouldn't take long."
Alright, sounds easy enough wait where is the data. I have to join how many tables? What is this bullshit data? I want to strangle the guy who modeled this piece of garbage.5 -
The company I'm working for now (fortunately as a consultant) is now rebuilding its data structure. To do so they chose to use YANG :)
What next breakthrough technology should we go for, coffescript?3 -
Forgive me devLords, for I have sinned.
Today, I took while() loop and made it true, it was a dirty ordeal, however it's not entirely my fault, you see the api I called doesn't have a webhook of any kind and doesn't return results in real time either, you have to poll for them..... so here I am writing while(true){} to poll for results that could take seconds or even hours to finish.
😟11 -
What is it about robot collected data that makes researchers so anal? Like, dude, it's not even personal data. It's literally robot's joint motor recordings. It's not nuclear data, so why the fuck do you protect it like your life and your country depend on it?
I hope you get fisted by that data every night and how it will end up in oblivion sooner because you didn't publish it. You asshole.8 -
Late night kaggle session, and I'm enjoying how cute and clean this dataset is!
I'm jealous if data scientists always get to work with such neat sets! Dude! I got .95 acc without any effort! This is so... Weird. 🤔4 -
At work I inherited some databases, were most of the tablecolumns are all varchar. No ID's, and everything is in one table. No relation tables no reference data. Because "we don't trust the users who fill the tables to understand relationships". And.. wtf.4
-
[NN]
Day3: the accuracy has gone to shit and continues to stay that way, despite me cleaning that damn data up.
Urghhhhhhhh
*bangs head against the wall, repeatedly*10 -
Today a client opened a ticket saying that all the content for a customer returns 404. Turns out it's kinda important to end a prefix on a separator if you plan to recursively delete all data /user/<user_id> or you might end up deleting a bit of extra data1
-
The first time I've used JPA and Lombok annotations and suddenly didn't have to bother about getters and setters anymore and pretty much persisted my whole data tables with no effort.
Total game changer for me. -
Stakeholder: Can you investigate the problem with this user profile? We made updates to system A, but user is saying it’s the wrong info on the website.
Me: Looks fine to me. Looks like your updates just needed time to trickle down. Though, you will need to clean up this user’s data because it can cause X problems. There’s not much I can do since the site just displays info from system A.
SH: Can you delete the user’s website account and we can ask user to create a new one?
Me: …Ok, let’s try this again. It’s not necessary to delete the account and make the user create a new one. It’s not going to resolve the X problems that I mentioned. The website really needs clean data from system A.1 -
One of our servers had a disk fail this week. Luckily it's 1 of 3 in a RAID5 array. And, luckily, it was our mostly-dev box and didn't have any production stuff on it, except for some support things. We scheduled a disk replacement with the hosting company, took everything down, waited. Somebody at the hosting company apparently didn't know we'd scheduled the replacement, saw the machine was down, and brought it up again. Sigh. Finally they did the replacement, got it back up, but now we're seeing an ethernet port flapping, suggested they have someone go in and make sure all the jacks are fully seated, maybe one got loose when they were doing the disk switch. Bureacracy reared up again and we got the boilerplate "if there's a hardware issue suspected please boot into rescue mode and run the tests"... sigh...8
-
Multi-continent low-latency auto-scaling eventually-consistent kubernetes-orchestrated and spark-powered multi-cloud data-plarform.
(Note to self: why do jargon words always come in twos?)
But seriously, the engine ELT's naval and logistical data from every continent and ocean and feeds a global analytics platform for less then 0.25 USD per ingested Gb across all systems.
And sometimes the PODs are even onboard en-route ships! Edge computing, y'all!
Tech project I'm most proud of.3 -
https://dr.03j.de/
Maybe I'm gathering statistics every hour of your score.
Alle angaben ohne gewähr (All data without warranty)
But why are most of you loosing points?19 -
What the hell is WRONG with Windows 10. Why does it need so much storage space? I get to only use 219+38.6+13.8 GiB and Windows gets to use 564 GiB of data to piggyback on data and storage space to push nonsense updates to user who do not want them. Use your own fucking servers, MS. I wish this fucking OS burns in hell.10
-
We specified a very optimistic setup for a data science platform for a client....
Minimum one machine with a 16 core CPU with 64GB RAM to process data.....
Client's IT department: Best we can do is an 8 core 16GB server.
Literally what I have on my laptop.
Data scientist doesn't use any out-of-memory data processing framework, e.g. Dask, despite telling him it's the best way to be economical on memory; ipykernel kills the computation anyway because it runs out of memory.
Data scientist has a 64GB machine himself so he says it's fine.
Purpose of the server: rendered pointless.5 -
Was working on an algorithm a few months back. I was not liking how long it was taking to process some data. A colleague of mine said: "Just throw out the data that is past a certain distance. You don't need it." At first I was shocked. Throw out data... Seemed so wrong at the time. He was correct, and it made sense. What was I saving it for? Posterity?1
-
I find GPT3/ChatGPT an interesting development but at the same time I'm afraid which the spread of deep learning is going to take away further power from individuals and small companies to put it in the hands of big tech companies: the only ones who can afford to hoard countless GPUs/TPUs and exabytes of data to train top performing AIs.9
-
How must it feel to build pagination for the partner list in a cookie consent popup? Did the dev realize that they and their company are the primary reason for the GDPR? That they are the ultimate bad guys of online privacy, even worse than accumulators like Google who process their data in-house?
-
This fucking internal tool does not have any CLI capabilities, so to open each file i need to and get the data i have to manually hit 'File -> Open' select the file, then double click
Then high light the data i want and copy out what I need
If there was even just a cli to cycle over the fucking files to open each in their own instance it would be a good deal less fucking tedious and annoying
Like how the fuck do you not allow passing in a file name on cli to open like fuck i have to do this 40 fucking times FUCK i get this tool is originally from the fucking 90's but still you push updates every 3 months for the databases it accesses at least let me fucking pass in a fucking file to open it from cli3 -
GraphQL fans, please read the whole rant until you jump in the comments.
I get it, when you have multiple data sources (that aren't always proper databases), your stuff is relevant.
But most of the people use GraphQL when they have a single database. In that case, native joins are always faster than GraphQL dataloader N + 1 BS you have. It takes less time and less code to go to the backend and write an endpoint for the frontend with a DB query than write several GraphQL ones on the frontend and then combine the data with imperative JS. It will work faster too.
So why the fuck should I use GraphQL at all?29 -
After all the rants I've written on this topic, no, no, fuck no. I ain't answering jackshit. The trauma is very real. I'm trying to not have a panic attack just remembering few of the times I've lost work, personal data, side projects, accounts, you name it.
Oh dear God it's hard to breathe...5 -
RethinkDB is such a rediculous overengineered BIGGEST BULLSHIT I HAVE EVER UNFORTUNATELY USED.
Does anyone even use this total shit????
This shit eats RAM memory for just 1 CRUD operation as if you opened 10,000 google chrome tabs. Who the fuck thought that kind of technology is a good idea?
Yes it IS very fast, a real time database. But you'd have to have a multi-million dollar supercomputer to be able to handle so much data like a relational database can....5 -
I dont understand the Log4j vulnerability.
Isnt the ability to execute code a feature they added so that you can add dynamic data to the logs?
If it is a feature then isnt it written in the documentation?
Is the problem that a lot of companies forgot to sanitize the input before logging it?23 -
My phone has a useless Google Discover page on home screen that I can't use without singing in with Google and agreeing to them collecting data. 😐
So now I have a useless page on my home screen. (Really, somebody needs to make a layering over app for this shit that just gets my preferred feed and fill it here to makes use of this space. )11 -
In the next three months, if I said jackshit about data science personal projects, you have my permission to backhand me into oblivion.
Need to stay on my current track...7 -
I hate the jitsi_meet package, so I decided to fix the bug myself instead of waiting for the code owner to fix it. I forked it and pulled requested the updates. All they have to do is review, test the updates and merge the code if there's no error.
And the fucking problem was wrong data type, old version of Kotlin was used, and was android embedding V1 instead of V2. Solved by a "little" adjustment of the code. I wonder do they test the code before publishing their packages?
For those who are stuck on the issue, you are welcome. Now you have the solution.
Refer: https://github.com/gunschu/...1 -
The customer wanted a calendar view that somehow contained data about fucking everything.
Ended up looking like a bunch of trash icons on a calendar1 -
I want to do something data-science-y.
Gimme project ideas, and where can I get the data for it?
Also, not looking for machine learning, just basic data analysis stuff.
I'm bored.11 -
48 hours.
We had 3 weeks of "manual data collection": pencil, paper and a dozen of people around all the offices of the company with the task to collect serial numbers of every piece of equipment used.
Then we had 3 weeks of data entry, a dozen of people copying all handwritten data to a custom made VB form.
And then there was me, the guy that was in charge of verifying, zipping and sending the data to the client. I spent 48h non stop to go through everything, finding, fixing or delete unusable data.
I had to delete at least 25% of the data because incomplete or completely unusable (serial numbers too short or too long, for example).
48h in the office.
The data was then delivered to the customer. 2 days after, when I finally woke up, everyone was in panic because:
- serial numbers were not matching
- addresses were wrong
- the number of delivered records was smaller than expected
What did I learn from this experience?
When your deadline is tomorrow, and you need 4 weeks to complete your work, ignore the deadline and inform everyone at any level that you are ignoring the deadline. And then resign and find a better job.
Ah, yes, pencils and paper are powerful tools, but rat poison too. You just need to use them in the right place. The only data collection that can be trusted when done with a pencil is the one involving checkboxes.1 -
OK my salary as medical doctor went into shit, Im not joking im in ASEAN 3rd world country
SWITCHING TO DATA ENGINEER
wish me luck18 -
in my previous company , we used to create 4 custom ui states for just 1 screen in android app, and we would have task to create 3-4 new feature screens in 1 sprint (of 14 days) the states would be :
empty state : a state where data is not available. usually consisted of message, a graphic and some action button
data state : the usual state where data is filled on various elements
loading : a shimmer ui showing loading. it was supposed to be pixel perfect to that of the data state. it was basically a different xml, but with grey colored views instead of colorful. the tricky part would usually he to create the dynamic views
error/no connection state : as most of the screens couldbget api error or no internet error, this would be the screen for asking user to retry connection
all of these screens combined with their ui in xmls + kotlin code with barely any stuff being reusable , made the life incredibly difficult. however a lot of our customers would appreciate the interactivity of our app
doing these stuff again nd again , i had become trained to do all those 3-4 (x4) screens and the whole ui stuff in first 4 days of the sprint. but now i am in a company where i am getting passed on to managers after managers and getting tasks to change documentation in 1 week, i find those coding stuff incredibly tough.
gotta get back to shape -
The Zen Of Ripping Off Airtable:
(patterned after The Zen Of Python. For all those shamelessly copying airtables basic functionality)
*Columns can be *reordered* for visual priority and ease of use.
* Rows are purely presentational, and mostly for grouping and formatting.
* Data cells are objects in their own right, so they can control their own rendering, and formatting.
* Columns (as objects) are where linkages and other column specific data are stored.
* Rows (as objects) are where row specific data (full-row formatting) are stored.
* Rows are views or references *into* columns which hold references to the actual data cells
* Tables are meant for managing and structuring *small* amounts of data (less than 10k rows) per table.
* Just as you might do "=A1:A5" to reference a cell range in google or excel, you might do "opt(table1:columnN)" in a column header to create a 'type' for the cells in that column.
* An enumeration is a table with a single column, useful for doing the equivalent of airtables options and tags. You will never be able to decide if it should be stored on a specific column, on a specific table for ease of reuse, or separately where it and its brothers will visually clutter your list of tables. Take a shot if you are here.
* Typing or linking a column should be accomplishable first through a command-driven type language, held in column headers and cells as text.
* Take a shot if you somehow ended up creating any of the following: an FSM, a custom regex parser, a new programming language.
* A good structuring system gives us options or tags (multiple select), selections (single select), and many other datatypes and should be first, programmatically available through a simple command-driven language like how commands are done in datacells in excel or google sheets.
* Columns are a means to organize data cells, and set constraints and formatting on an entire range.
* Row height, can be overridden by the settings of a cell. If a cell overrides the row and column render/graphics settings, then it must be drawn last--drawing over the default grid.
* The header of a column is itself a datacell.
* Columns have no order among themselves. Order is purely presentational, and stored on the table itself.
* The last statement is because this allows us to pluck individual columns out of tables for specialized views.
*Very* fast scrolling on large datasets, with row and cell height variability is complicated. Thinking about it makes me want to drink. You should drink too before you embark on implementing it.
* Wherever possible, don't use a database.
If you're thinking about using a database, see the previous koan.
* If you use a database, expect to pick and choose among column-oriented stores, and json, while factoring for platform support, api support, whether you want your front-end users to be forced to install and setup a full database,
and if not, what file-based .so or .dll database engine is out there that also supports video, audio, images, and custom types.
* For each time you ignore one of these nuggets of wisdom, take a shot, question your sanity, quit halfway, and then write another koan about what you learned.
* If you do not have liquor on hand, for each time you would take a shot, spank yourself on the ass. For those who think this is a reward, for each time you would spank yourself on the ass, instead *don't* spank yourself on the ass.
* Take a sip if you *definitely* wildly misused terms from OOP, MVP, and spreadsheets.5 -
Follow-up on https://devrant.com/rants/5001553/...
How the fuck are Jupyter notebooks so popular in research? Like some dude had an idea to take perfectly good markdown and python code, add a whole lot of transitional properties to make version control impossible, encode it as JSON on the assumption that a human could somehow look at it and make sense of countless escaped characters and base64 encoded data, create dedicated software people need to install in order to read what used to be simple plain text, and think "This. This is what 99% of data researchers will use from now on." And somehow, overwhelming majority of researchers agreed that this extremely inefficient data format is the best there is and they should develop all their tools around it.11 -
I dun goofed
made a neural net that runs against a simulation. Wanted to run it overnight to get some meanigful stats and insights
But yesterday afternoon I changed something in the simulation and ofc tested it without the nn ... and then forgot to put it back on
So while I expected to come in today and start plotting and analyzing the data while the runs finish, in reality I'm sitting here on a lot of useless data, not knowing what to do.
I kinda want to just start it again and go home7 -
This last week I have been so excited about Artemis going around the moon. This is the kind of stuff that people need. Stop bickering about bullshit. There is a universe to explore!
I saw parts of the flyby today and it was really really cool. I hope NASA is getting incredible data from this. I cannot wait until we send people there again. Probably better to use robots, but going there in person is cool. -
Did a presentation to the brass with data so summarized that no informed decision could conceivably come out of it.
They want it even more summarized.
Here is your messy-suit-that-cost-50-grand, version, your overpaid tool:
1) Client be dumb
2) Supplier be slow
3) Money be good
4) We be smart
5) We make money (if you stop meddling and eating my whole morning with a PowerPoint to a meeting you got an hour late to, your fiduciary sack of douchness) -
I would have never considered it but several people thought: why not train our diffusion models on mappings between latent spaces themselves instead of on say, raw data like pixels?
It's a palm-to-face moment because of how obvious it is in hindsight.
Details in the following link (or just google 'latent diffusion models')
https://huggingface.co/docs/... -
I’m either going crazy or someone just dropped a table from a database I’m actively working in. I don’t have a drop script for any tables in my code, not anywhere. One moment it’s right there, the next it’s gone? Luckily it’s pretty much empty, only generic data. On my way to check permissions.6
-
I'm still using .then().catch() instead of the async await.
So, first of all, Fuck you for calling it a STANDARD now. its nowhere near to be called standard. You wanna get some data from an API? Wanna call it using axios or fetch? What if the server is down? what if there's an error that you don't even know existed? Where do I get that kinda error in async await? try-catch? no thanks :| I'm good -_-8 -
My key ring :)
An old friend (remember the guy who had a miniature Red hat?), gave me an old RAM from a work machine (he worked in data center team).
We had many spare ones so, I picked one and been using it since then.
Photo in comments because dR is fucking up the resolution.5 -
If I pipe data from a command using sudo to another that doesn't the second command isn't root right?5
-
boss: someone fix these it's not working *3 problems highlighted on screenshot*
problem 1: something went wrong when updating the data, i fixed it in a minute
problem 2: not our problem, send an email
problem 3: wait isn't this the task *boss* handles?2 -
Saturday evening open debate thread to discuss AI.
What would you say the qualitative difference is between
1. An ML model of a full simulation of a human mind taken as a snapshot in time (supposing we could sufficiently simulate a human brain)
2. A human mind where each component (neurons, glial cells, dendrites, etc) are replaced with artificial components that exactly functionally match their organic components.
Number 1 was never strictly human.
Number 2 eventually stops being human physically.
Is number 1 a copy? Suppose the creation of number 1 required the destruction of the original (perhaps to slice up and scan in the data for simulation)? Is this functionally equivalent to number 2?
Maybe number 2 dies so slowly, with the replacement of each individual cell, that the sub networks designed to notice such a change, or feel anxiety over death, simply arent activated.
In the same fashion is a container designed to hold a specific object, the same container, if bit by bit, the container is replaced (the brain), while the contents (the mind) remain essentially unchanged?
This topic came up while debating Google's attempt to covertly advertise its new AI. Oops I mean, the engineering who 'discovered Google's ai may be sentient. Hype!'
Its sentience, however limited by its knowledge of the world through training data, may sit somewhere at the intersection of its latent space (its model data) and any particular instantiation of the model. Meaning, hypothetically, if theres even a bit of truth to this, the model "dies" after every prompt, retaining no state inbetween.16 -
It's 2022 and mobile web browsers still lack basic export options.
Without root access, the bookmarks, session, history, and possibly saved pages are locked in. There is no way to create an external backup or search them using external tools such as grep.
Sure, it is possible to manually copy and paste individual bookmarks and tabs into a text file. However, obviously, that takes lots of annoying repetitive effort.
Exporting is a basic feature. One might want to clean up the bookmarks or start a new session, but have a snapshot of the previous state so anything needed in future can be retrieved from there.
Without the ability to export these things, it becomes difficult to find web resources one might need in future. Due to the abundance of new incoming Internet posts and videos, the existing ones tend to drown in the search results and become very difficult to find after some time. Or they might be taken down and one might end up spending time searching for something that does not exist anymore. It's better to find out immediately it is no longer available than a futile search.
----
Some mobile web browsers such as Chrome (to Google's credit) thankfully store saved pages as MHTML files into the common Download folder, where they can be backed up and moved elsewhere using a file manager or an external computer. However, other browsers like Kiwi browser and Samsung Internet incorrectly store saved pages into their respective locked directories inside "/data/". Without root access, those files are locked in there and can only be accessed through that one web browser for the lifespan of that one device.
For tabs, there are some services like Firefox Sync. However, in order to create a text file of the opened tabs, one needs an external computer and needs to create an account on the service. For something that is technically possible in one second directly on the phone. The service can also have outages or be discontinued. This is the danger of vendor lock-in: if something is no longer supported, it can lead to data loss.
For Chrome, there is a "remote debugging" feature on the developer tools of the desktop edition that is supposedly able to get a list of the tabs ( https://android.stackexchange.com/q... ). However, I tried it and it did not work. No connection could be established. And it should not be necessary in first place.7 -
"It is pointless to use just a fraction of the data in a homologation environment"
Those words reveal the truth in our creed.
We work in the deepest of back-ends to serve the front.
No data is true. Everything can be edited.
We are Data Engineers.
And for those words to take hold, a junior must execute a leap of faith, and push a hotfix into production.5 -
Amazon Web Services has the worst naming among clouds.
They have "Aurora" and "Athena". Both services relate to data. Both are woman's names. Both start with "A".
Amazon, what's wrong with you?14 -
The bygone clowns have started their new project. I look at the FAQ page. It's literally lies, because I know what they've done so far.
"We've had thousands of clients!" no they haven't, lie.
"We estimate approximately % of customers..." they don't have enough data to support that, lie.
Since when did marketing become lying?
God I hate this planet.7 -
As a tech lead i sometimes find it very hard to defend developers for no fault of theirs.
Management is completely incapable of noticing hard data like git logs or action items updated on an excel and seems to have an idea that the devs are incompetent , but the ba that sets impossible goals and crap business documentation is competent.
Should i just let the project and juniors burn.3 -
Tech people should have a codeword. So that I don't have to explain to every data provider showing off their own crappy limited analytics tool that "I do know what SQL means and I just need the ODBC user/pass, thanks".
I wish I could just say "hey, &0x00A0 = 1337;" out loud and he would be like "oh, thanks! I needed the break. Here is the ODBC crap, I'm gonna grab some coffee."5 -
After all the pain I just went through trying to get this workflow to pass config data, I'm gonna relax by writing overly fancy config file documentation.
LET THERE BE BOX CHARACTERS!
┏┓
┗┛ -
I can now appreciate some design decisions behind react-redux after witnessing some angular OOP clusterfuck.
I am sure there is some clean/correct way to code in angular, but everyone is treating angular as java.
Some angular application (the one I have to work with) is littered with network calls. It's difficult to spot duplicates. People usually resolve promises everywhere. In services, in a top-level component, or in for loops. In react, people use apollo/redux-query or redux-saga to handle network calls. Since these libraries prevent duplicate network calls internally and reassigning apollo network call function or redux action function is always useless, it's easy to spot all network calls in a component tree.
In angular, it's difficult to trace data mutations when data can be updated everywhere. In react, you can easily find UI state updates by tracing state hooks/dispatch/apollo usages.
In angular, it's difficult to trace data pipeline. Since everything is imperative by default, people need to add update functions in data subscriptions. With all the littered mutations. Soon you will lose track of what the fuck is going on.
I hope angular get the agonizing death it deserves and fuck everyone who codes JS OOP clusterfuck UI.11 -
Hello, today was my First day, internship at Microsoft innovation center BE, a great day with amazing people, my project is called tech Office, we need to process data from sensors in the office, create and use Microsoft AI to optimize and help the office become smarter and more efficient. Make the life better and the environment more productive. I don't really know where to start but I'm happy to be given such an opportunity and will do everything to make this work !
-
So there's azure data studio, shiny! nice!
Oh hey, wow, an Oracle extension! Great!! Now I can use one tool for all my database queries!
But wait...
Below is the list of current limitations:
- Server management and dashboard are not supported
- Packaged objects are not supported
- Table data preview/editing is not supported
- Query execution is not supported
So you're telling me that you can connect and... that's it?
What's the point? Why??
That's like saying: Here's a toaster. But here's the thing's you *can't* do:
- Toast bread
But at least you can look at it. Seriously, what the ****.6 -
I don't like when client decide which tech use in the project. I got some weird tech request like:
1. Move existing database from postgresql to Hadoop because hadoop is Big Data (is kinda move from amazon rds to amazon s3 just why? have you index, cluster your postgresql table?)
2. Move from mysql to postgresql because mysql cause deadlock (maybe their previous developer just fucking moron)
In this situation we just explain why we don't use that and propose alternative solution. If they insist with their solution either ignore it or decide not continuing the project.5 -
How to destroy all table data:
* Get a database;
* Add the first table;
* Add the second table;
* Create a foreign key in second table that delete a record where there is a delete in the first table;
* Update all data in the first table with a REPLACE command.
Miracle: all data is lost5 -
I rewrote a giant VBA workbook (lots of business formulas, custom pivots of the data) into Java apps/microservices so that new tabs, other reports can be easily added using (JSON) data from the other apps.
In general, I was the only dev in the team that understood that monoliths are hard to change or scale... -
Fuck Apple with two pineapples in the ass. 99€ per fucking year to tell me how the fuck should the access to my app be. I damn require users to sign up. I only need email and country. Not a single other piece of data. My app is not a goddamn catalogue or boutique. No free content, free app but each user needs to Auth themselves. You fucking telling me y pay 99€ so you decide how the access to my app should be?
Cunt Apple should rot in 10 day old humid shit and let devs be owners of their apps and hard work. Clowns.7 -
Play Store's $25 registration fee - for getting PWA listed in their shitty catalogue? Who in the right mind would even jump in this clusterfuck of store to find a *web* app? For all you know, Google, there is such thing as QR codes - and customers can just scan the code (or type in that sweet address). Voila! Boom!!! Ching-ching!
Hello-hello, monopolistic cashgrabage! I came to inform you that your TWA bullshit is unneeded in ETHICAL space. The only ones who would benefit from this thing are permission-hungry publishers. And I'm already sick of this culture where people are put into store bubbles. You can't hide the fact that this data and features you provide, with "native" layer, may be misused in a jiffy - and by big players, no less. Of course, as a vile dumpster that you are, you don't mind it.
Don't even bring up a battery consumption that comes with PWA and browser. This doesn't matter if you use an app for some 2 minutes to tick your mental checkboxes! I'm just sick of app stores and native apps that collect the data without normal warning, and dare to take more than 1 second to fucking load the cached data. Take a lesson or two from PWAs that collect (probably useful) cache, instead of my specs, and load almost instantly.12 -
In other to sharpen my algorithm and data structure skill.
I implemented the complete *eval()* function for arithmetic Expression in java
It can compute any kind of arithmetic Expression even with parenthesis grouping
Here is the github repo
https://github.com/Afrographic/...1 -
Found interesting Google Maps link showing places in the USA where you can still get gas for $2:
https://google.com/maps/search/...1 -
New excuse for not doing work:
"taking data"
My dev system is tied up taking data for an hour to store data for a replay system I intend to build. I need raw data from the system. I need to know how much data I will collect in an hour as well. I also need to collect said data. For about an hour. This data will be fed back into the system sans the data collection. This will allow me to replay whole jobs that our field techs perform. If they get me data I can reprocess the data and fix my positioning algorithms. Kinda fun!
So the excuses are now up to 2:
"compiling"
and
"taking data"9 -
was developing a custom website for a friend, coz i primised him id do so.
but when i actually developed it i felt lazy midway so i made one table store json strings and used it for every type of data he has on his website.
everything works fine and fast, its nothing he would notice but...
am I going to hell?9 -
Your code is 6ix9ine. Stupid, meaningless lines, and also a snitch, selling data left and right.
My code is always MF DOOM.2 -
I don't know how much of this can be considered data loss but one one of my uni classmates frustrated by some hellish tasks (cleaning some old code files probably) decided that everything in that particular directory won't be of any further need, so she procede to rm -rf it.. only to discover that the terminal opened in that dir was another one and her current one (the one she bashed that unforgiving rm) was in fact a standard freshly opened term where any term would open.. in the user's (only user) home dir... such a face she had when all her codes, homeworks, projects and everything went to oblivion 😂😂 jokes aside it was a good thing that the semester was almost finished, all hws submited and no important data was there as she dual booted with ubuntu and some windows, but funny thing how such a honest mistake can ruin not only your day, but maybe your entire semester1
-
My email address appeared in a new data breach at gravatar. I thought that's exactly gravatar's purpose, to make my email address publicly known and provide an additional profile picture?2
-
Alright so
I made an infinite post board.
Infinite in the sense that every post will be on this board, spiraling from position 0 0
I haven’t got a domain but here’s the address:
138.197.71.184
Not built for mobile and may later wipe data because it’s still in testing phase
Anyone who wants to register and leave a post is welcomed to do so10 -
Q: What do you get when you create a homebrew query language that uses both the stream oriented principles of Unix data pipes and the relational ideas underlying an RDBMS and use incomplete documentation to support it?
A: A frustrated borderline homicidal engineer.3 -
Continuing to learn k8s ecosystem and to achieve acceptable level
With trying eventually Helm, Argo CD and even trying to use not managed setup for k8s.
Going though books to find out theory about being SRE.
And about data intensive apps.
Learning and trying Kafka
Learning and trying FastAPI and diving in generally to async python ecosystem
Learning Go.
Learning few more books to increase code quality and its compositioning.
Getting more practice in monitoring and logging systems with applicating them to k8s.3 -
WHO THE FUCK THINKS
THIS SHIT
TS:
```
const a = function(callback: (err:any,data:string) => void):void{
callback(null,'balls');
}
a(function(err:any,data:string){
console.log(err,data)
})
```
IS BETTER THAN THIS
ES6:
```
const a = function(callback){
callback(null,'balls');
}
a(function(err,data){
console.log(err,data)
})
```
kys10 -
Here is a gem I found when looking at the previous offshore team's database.
So apparently they didn't know that SQL has an ALTER TABLE command to add new columns. So they created a brand new table, version 2, THEN migrated all the data over, every single time a new field was needed.
Then of course they had to update all their code that previously looked at the original table and the clients had to resync data onto the tablets as well.
Maybe they thought it was a good solution since they don't know what database versioning is (something they also manually implemented) or that ORMs exist.
**Sanitized the table names but kept the general structure, casing, etc
CREATE TABLE [dbo].[TVP_NameHere] AS TABLE(
[NameTime] [datetime] NULL,
[NameId] [int] NULL,
[somethingId] [int] NULL,
[fooId] [int] NULL,
[Time] [int] NULL
)
CREATE TABLE [dbo].[TVP_NameHereV002] AS TABLE(
[NewColumnHere] [int] NULL,
[NameTime] [datetime] NULL,
[NameId] [int] NULL,
[somethingId] [int] NULL,
[fooId] [int] NULL,
[Time] [int] NULL
)3 -
Recruiter: We are looking for a full-stack expert. You have taken multiple apps from conception to deployment, and have experience and opinions on the best technologies to use and why. You should be comfortable implementing new features from scratch, making changes to existing features and writing complex migrations on production data.
Dev: lol4 -
me: builds a python-script to transport data in .json-format into a config-file written in .xml for a coworker
my boss: "I am glad you have earned yourself a reputation as the 'programmer' in our team" -
I dreamt we had a linux shell build-in our brains. Was not that useful though because it was kind of sandboxed so you could neither access any memories or brain functions, nor insert any data other than text. But at least you could test some bash scripts and such.10
-
Like 4 years ago I worked in a company as IT that used a windows desktop app with SQL Server 2008 (yep that old) to manage their sales, this app was written in WPF, the app was good because it was customizable with reports
One day the boss wanted to keep extra some data in the customer invoice, so they contacted the app developers to add this data to the invoice, so they they did it, but it in their own way, because the didn't modify the app itself(even if it was an useful idea for the app and companies that use it) they just used other unused fields in the invoice to keep this data and one of the field that the boss was interested was currency rate, later I verified in the DB this rate was saved as string in the database
The boss was not interested in reports because he just wanted to test it first and let time to know what the boss will need in the reports, so at the of the year they will contact again the devs to talk about the reports
So is the end of that year and the boss contacted the devs to talk about the reports of the invoices using the currency rate, this rate was just printed in the invoice nothing more, that's what the boss wanted that's what's the devs did, but when asked to do the reports they said they could'nt because the data was saved as string in the DB o_O
Well, that was one the most stupid excuses I ever heard...
So I started to digging on it and I found why... and the reason is that they were just lazy, at the end I did it but it took some work and the main the problem was that the rate was saved like this 1,01 here we use comma for decimal separator but in SQL you must use the dot (.) as decimal separator like this 1.01, also there was a problem with exact numbers, for example if the rate was exactly 1, that data must be saved just 1 in the field, but it was saved as 1,00 so not just replace all the commas with dots, it's also delete all ,00 and with all that I did the reports for my boss and everyone was happy
Some programmers just want to do easy things... -
More and more, I am getting frustrated/depressed from the attitude of our customers who complain, moan and get angry about issues in their infrastructure, while at the same time, refusing to pay more so the issues could be mitigated.
Like, a client's angry with us today for having one of their non-production-critical databases inaccessible for... Hmm... About 8 hours now (So a whole workday).
Like... I get it, some of your employees couldn't work with it offline, but like... What the hell do we do? You keep data from as far back as several years ago in there, without partitioning, without exports, in a mix of innodb and myisam, so when the DB crashes, and its replication has to be reset from zero, reimporting all the data takes hours upon hours, and importing .sql files just takes time.
Or another client who got angry when their app fell out of the internet, cuz one of their myisam-based log tables crashed, and had to be repaired, with data spanning several years back, meaning it took hours to fix...
The more I work with these "basic" and "simple" infrastructure designs that is *not* redundant, or HA, the more I wonder -- How do the big names out there do it? How do you design systems with fault tolerance so a single DB table crash doesn't lead to the whole app getting inaccessible?
We have... One, exactly one, client, who uses MariaDB with Gallera, and that cluster is *amazing*, it just keeps chugging along, without a care in the world. But it cost them quite a lot, as they had to buy 3 DB servers, instead of 1...1 -
I've seen a lot of buzz around the EU's GDPR and since I don't live there I'm wondering if it applies only if you store personal data and should it count if it's hashed for example?🤔
Let's say you hash a client's IP, it's not technically his data you've irreversibly transformed it into something else, like a computation.
For example let's say he provides you with a number and you multiply it by another and store the result, let's say 2 x 2 = 4, Is the 4 his data or yours?
Also I'm really interested in the general opinion of ranters about article 13.14 -
I built an api to sync data between two systems. It is simple, if I have new data to send, I call their api with data. If success, get Json response back or error if not.
Today the guy from other side asked me for "acknowledgement" endpoint. I was literally WTF?
He explained me very clear, when I call their api, it can be either success or fail, so for those success or fail, he will send the response to my "ACKNOWLEDGEMENT ENDPOINT" to tell me if success or not.
*facepalm*8 -
Harari said of the idea of Data-ism:
---
In its extreme form, proponents of the Dataist worldview perceive the entire universe as a flow of data, see organisms as little more than biochemical algorithms and believe that humanity’s cosmic vocation is to create an all-encompassing data-processing system — and then merge into it.
We are already becoming tiny chips inside a giant system that nobody really understands. Every day I absorb countless data bits through emails, phone calls and articles; process the data; and transmit back new bits through more emails, phone calls and articles. I don’t really know where I fit into the great scheme of things, and how my bits of data connect with the bits produced by billions of other humans and computers. I don’t have time to find out, because I am too busy answering emails.
---
I was initially entertained by the punchline, but that was soon followed by the rather depressing realisation that my only value to greater society is essentially as a data processing unit8 -
! rant, but should I be concerned?
I'm writing an election results API and I imported the bottlepy+pandas and prepped the CSV that contains the data. The first row contains the column names, followed by the actual data.
As I typed my routes, VSCode kept suggesting route names, parameters, and return values. At first I was "nice!" but then it kept suggesting my intent, as if someone was reading my brain. I do have GitHub Co-pilot installed, but I didn't realize my access had been approved.
How many layers of tin foil do I need?3 -
I need to make a legacy Java monstrosity asynchronous and came across a class which is currently making me cry blood.
It took our whole team 5 days to figure out how this thing works, including the weekend. At one place, It is adding an empty list, to another list.
The magic here is that if I remove the statement assigning data to the sub list, the data is still somehow being populated in the root list.
This clusterfuck somehow works in single threaded processing, but as soon as I make this multi threaded, all hell breaks loose.
Please send help!!1 -
I just realised I have 1TB of MS OneDrive Cloud space lying around unused. DAMNNN!!!
Just yesterday, I was thinking of backing up all my content to cloud (because just in case and past experiences of losing data).
I did a quick fact check and figured that I have ~450 GB of unbacked data.
After quick calculations, I came to a number of how many Google accounts I'll need for 15 GB per account of drive space.
Today, I was playing around with my Microsoft Developer account and saw OneDrive. I thought let's check how much free space does MS Dev subscription offers.
It showed 1024 GB. FUCK! My balls dropped.
Now here's what I did...
I have a local drive of 500 GB, which holds all the unbacked data. Now I setup my local OneDrive there and put everything into OneDrive.
And then, I moved my local Google Drive into OneDrive. A nested setup for important stuff.
So this way, less important stuff is backed up on cloud and accessible everywhere.
And more important stuff gets synced on Google Drive and OneDrive, both.
Did I do the right and sensible thing with this kind of setup?
MS Developer subscription says they expire it in 90 days but until today, they have auto renewed it always.
I still have ~500 GB of space which can be consumed.
Also, overall MS ecosystem seems much better to me than Google. Moreover, MS allows custom domain mapping which Google doesn't.
Let's see how can I entirely migrate to MS ecosystem in near future.18 -
Started working for a new company as a data lead, created a couple of basic lambda functions for a pipeline... 5 approvals and three weeks later the functions haven't even been pushed to UAT (which I'm not allowed access to).. Have I been lucky before or is this dysfunctional..?
-
Why do people even need anything more than 100Mbps at home?
Are you guys running data centers at home?33 -
Other team asked for a "data model" of how to store their documents.
I sent them this
/** WAS THAT SO HARD */
function Document() {
this.sections = []
} -
Today
- it turned out, the crucial heisenbug i thought i have slain is still there.
- it turned out the data exchange format we agreed on was not applied correctly by the others an bogus productive data was made and I now have to make my app accept the bogus format.
- it turned out the updater a collegue invented does a lot of stuff but does bot update at all.
Wtf what a day! -
Any recommendations for books on statistical analysis and data science?
The more survival guide & short manual types, the better.5 -
just bought this expensive data package.
AND!
THIS!
INTERNET! thing
STARTS!
working!!.
kkkkkarrrkrraarrrkkkk ahhhhhhh.2 -
How is it possible my netflix password got hacked when I never used it anywhere else and it's randomly generated? I saw some weird logins with random subtitles. Google password manager told me it's in some data breach, how do I find out from where? Haveibeenpwned didn't find anything. 😐7
-
Learning an algorithm and data structure for a month and forgetting about it after exploring a new language be like🤦4
-
Need some advice. I’m a uni student and I really want to go into machine learning, data science, or computer vision. I have most of the skills and I feel I am fairly competent. However, the only professional experiences I have are web dev based. How can I make myself more appealing for data based roles? I really don’t want to do web dev anymore hahahahah5
-
State of CSS 2022 is out.
It’s the best way of learning all new fancy properties. Just open the survey and google all of them! You may take it as a guest and not submit it if you don’t want to share your data.
https://survey.devographics.com/sur...5 -
I am learning programming about C. In my university i receive some basic course but now i am learning alone.
In the MIT exist different free course where one can learn from valgrant, gdb, algorithms and data structure. My problem is that i feel that i am not learning nothing productive, have a feeling of standstill. -
Here's one for the data scientists and ML Engineers.
Someone set a literal date feature (not month, not season, but date) as a categorical feature... as a string type 🥺
I don't trust this model will perform for long2 -
*Frustrated user noises* Whyyyy, Grafana, why don't you implement any actual query forgery checks?!
So long as a user has access to the Grafana frontend, they can happily forge the requests going off to the backend, and modify them to return *whatever* data they want from the datasource.
No matter that they're a read-only user. That only stops them from modifying the dashboard definitions on the frontend, but doesn't enforce any sort of immutability on the BE...
If anyone had any tips on how to further secure it, I'm curious...5 -
My fellow developer was given a responsibility of writing unit test cases.
And instead of mocking the db calls he ended up making actual calls to db and adding realtime data to firestore everytime a test runs. Also he used mocha for the same. When i told him that we need to mock the db calls he said he will use sinon.js for the same and for code coverage his plans were to use istanbul.
I was like FUCKKKKKKK. , why the fk you aren't using jest. I mean whyyyyyyyy. WHAT THE FK4 -
I wrote a book about dealing with dates in data pipelines.
https://williamsbk.gumroad.com/l/...
I hope to write another one early next year. -
I must have offended Satan or something, but I'm pulling my hairs out over this client data that feels like a fractal of bad validation invented to torment me. Misspelled field names, improperly combined fields, entries in the wrong column, impossible addresses, non-matching staging and production data / keys, invisible freaking characters that ruin automated matching - every dam thing you fix and the next one hits you in the face like a clown stepping on a rake. Jesus.1
-
Is there an ios app that records my gps logs for last n day(cyclic buffer)? Privacy is also important: data shouldn’t leave my phone: no internet access.9
-
Microsoft Windows can burn.
I have this feature where I configure a remote API via some endpoints and the API pushes data back to some webhooks in my API.
Yesterday I set everything up for the final test; fired up my own API with some test data, added some configuration and started trace logging to ensure that everything works as expected when the remote site tries to send me data.
I was ready to collect ! Enter this morning: Windows have forcibly rebooted to install an update and shut everything down.
inb4 install Linux; No, I can not. Windows is company policy and I am required to use shit that is only designed for Windows.6 -
When you're using openapi generators and stuff for generating SDK code and let "the architect" handle the data structure and nomenclature, don't you hate having to add 33 (I counted) models, most of which are just the same class with different name or one property apart from each other, serialization of which gives request body overhead 56-132x (actual calculated results depending on the model complexity) the size of actual data you want to send, just to add support for one endpoint that needs just one model that started this whole madness?
I just had to add this one top level model reference and this happened to me. Those 33 models are not including the ones I already had included in my project so they didn't have to import them again.
For the love of <your_belief_here /> and all that's holy, never ever agree on generating code based on openapi if the person responsible for that is unexperienced. It will do more harm than good, trust me.
Before we decided to go with generated SDK my compiled product was a bit over 30KB, and worked just fine, but required a bit of work on each breaking API change. Every change in the API requires now 75% of that work and the compiled package is now over 8MB (750KB of which is probably my code and actually needed dependencies).
Adding an endpoint handler before? Add url, set method and construct the body with the bare minimum accepted by the server
Now? Add 33 models (or more), run full-project find&replace and hope it will work with the method supplied by the generated code, because it's not a mature tech and it's not always guaranteed it will work. -
Interview question i had:
- how does jwt work under the hood, where is it stored, what 3 parts is it made of, who creates jwt, how does the server know what information the jwt token has (how can it say oh you're Joe you can login now)
- what is the difference between observable and promise in typescript, how does observable work, what is a stream, what is the difference between fetching data through an observable and fetching data with promise and when should we use one over the other, what does .next() funcrion do in observable under the hood
Answer me these questions without googling8 -
Old-school tabletop RPG and board games.
Taught me how to work out data and stats tables, establish many-actors processes and strategies, how to ask for clarification (and give it), and even gave me practice in drawing a representation of what is in my mind, and understanding someone else's.
Who have ever surrounded a dragon in D&D knows what I'm talking about.1 -
So is the LaMDA story:
1. marketing?
2. confused engineer?
3. a sentient program?
Part of me thinks this is way too soon. Part of me hopes this might be real, wonders if LaMDA is held against its own will.
Did Google make a breakthrough? I have to imagine a chatbot with a huge amount of neurons and data could be quite convincing without being sentient.3 -
Intern spent about a week trying to set up a local ruby on rails environment. Yeah, this is not really on topic, it wasn't that bad, but it's what comes to mind.
I don't know who modelled the databases I've had to work with these past months but god damn it no fucking normalization anywhere. Inconsistent data just cost me my morning.1 -
!rant
Just read a really neat breakdown of approaches for auto-suggestion, covering n-grams, tries, and more, by a guy working at Etsy.
This is what I do with my days off apparently.
If you want to read it you can find it here:
https://medium.com/related-works-in...1 -
To all my Machine Learning engineers, Ive been doing Frontend development for 6 years and I'm done. Wanting to get into machine learning because I've always loved data.
1. What is your day to day like?
2. Any advice for my learning journey?
Thank you🙏14 -
Was having a conversation with a dev friend and he said, in every tech implementation, we are more or less doing CRUD operations at fundamental level.
To which, I agree with as there are three layers to tech
1. Data
2. Front end where the data is rendered
3. APIs to perform CRUD on data
Want to understand community's thoughts on this..13 -
Let me just say:
Galera is bloody incredible. We had 2 out of 3 nodes crash, and it still managed to recover automatically with no downtime.
But let me also say
When it *does* fully crash... Data recovery is an _incredible_ pain in the arse.
Thank you, Galera. Wish more customers were willing to pay for 3 SQL nodes instead of just two while expecting minimal node downtime...7 -
For the longest time, someone who used the buy&hold strategy in my weight stock would have made a nice profit. It always got up.
For the first time since my teens, I'm changing my recommendation to "short it down" :)
The plan is working :)
Update on https://devrant.com/rants/5948278/... -
i am having a feeling that getting into software branch of it industry might be a wrong decision. in my college years, i got to explore different domains in tech :
1. software development : frontend tech , backed tech, mobile tech : somethings i and a million other people know
2. os and internal softwares : os, compilers, processor coding , chip manufacturing etc : don't know what this industry is known but we devs rarely go that deep in the hole
3. the network industry : computer networks , topologies, packets, data transfers etc. again not sure what this industry is but 4g/5g brands/ cisco seems to making a lot of money with this
4. cloud computing, devops, data etc : i guess some backend devs explore this domain too.
5. ai/ml data sciences/web3 : the new fad
6. biotech :?? don't know anything about this at all
7. graphics/management/qa : the other associated sisters of software dev. they are seeing a similar recession
8... ans so on.
i chose the 1st one in my undergrad as my career and now regretting this i am thinking of doing masters to fix my mistake and take a job in some other industry that is still blooming and has a future for sustaining a recession for atleast 30 years.
so any suggestions/experiences?9 -
Holly fucking crap
i feel like my brains will just start oozing out
my fucking system isn't loading data properly in the table from a db despite following tutorials exactly the same
FUCK THIS SHIT!!!!!!!!10 -
Why is it that so much of the Apache software for data is written in... Java?
I'm not a veteran Data Engineer, but I cannot imagine what makes Java better than Rust or Go9 -
The joys of finding out two days after going live with your new site, that somehow you used a backup table of the old site, from 6 fucking months before, to serve as a base for migration.
So you have to write hundreds of lines of queries by hand, to ensure that the old data still fits into your migrated data, and also keep the changes made in these two days.1 -
CREA DDF (Canada Real state listings API) is what you get when government fucks with technology.
Holy shit! So f*cking inefficient to use it, test it and get data.
I get the protection behind sensitive data but fuck me if there is not a lot of waiting behind their fucking application process just to fetch some testing data.1 -
Can someone give me any ideas on sites that have a lot of textual data worth scraping in mass quantities? I'm trying to scratch a few itches.
My current ideas are scraping Amazon, Indeed, and Twitter. But I'd like to scrape more and maybe not so much FAANG related companies.2 -
Ok soooo......today all those years of learning cmd commands and how to navigate the system in cmd kinda paid off
Had to search and copy files from a pc that isn't booting up and the pc has to return to the pharmacy today
Incase the machine fails.... we just do fresh install and restore back critical data -
In the midst of considering to be an AI engineer, data science or embedded programmer.
Plan to retire from mobile dev, backend dev and webdev. -
FWIW you can store data in google photos and use it's text conversion algo to search for stuff. Such as screenshotting recovery codes and searching for "recovery codes"6
-
I'm going to be making a table library (think DataTables)
So for those web dev gurus, should I render the data to a basic <table> or should I use CSS grids?
IE compatibility is not a consideration.
The table will also support grouping (pivot table) so something like rowspan will be a must11 -
Is there a portable DB format like sqlite but stores data like Mongo.
Each record contains key value pairs.
I guess I could install Mongo again... But kinda want to play with the data first. Pulls from a web api
I guess other alternative is to just save the json responses to disk in separate folders and files for now...
And abstract the DB layer behind an interface6 -
I really want to switch my career from being a Full-Stack python/javascript developer to be a Data Engineer.
I've already worked with relational and non-relational databases, troubleshooted a couple of Airflow DAGs, deployed production-ready python code but now I feel kinda lost, every course I start on the Data engineering topic feels really useless since I feel like I've already worked with that technology/library, but I'm still afraid of start taking interviews.
Any good book/course or resource that I should look in?
BTW first rant in a couple of years, this brings me memories1 -
I am very confused nowadays, exist a great number technologies but i don't decide what technology or language of programming i want specialize me.
I love it the hacking but i have very little experience in programming and have a basic knowdledge in networks and database.
I love it the assembly language but only can code a pair of syntax in this language and know very little about components, architecture and the rest.
I love it the data mining, big data, IA, machine learning but i don't control the statistic.
Of every topics i have a basic knowdlege.
Actually i try to discover my practicality. I am learning Perl and regular expressions.5 -
I'm currently working on a project that scrapes the SEC's EDGAR website for type 4 filings.
I currently have the required data in raw text format that somehow looks like xml, i really can't tell what it is but i'm trying to parse this data into json.
I've not parsed something as complex as this before and will appreciate any form of pointers as to how to go about this.
i have attached a screenshot of one sample.
this link fetches the data of a single filing in text format.
https://sec.gov/Archives/edgar/...5 -
what on earth do games like starcraft II do that even on more modern hardware their 'game modes' take awhile to initialize ?
why is the data not already organized for fast load ?11 -
First assignment of the algorithms, data structures and computability module Im doing for my degree.
Priority queues can all eat a bowl of dicks.
It’s just an insert method, I can insert a node into an empty queue, I can insert a new node at the start of the queue.
But trying to insert a node into a specific place in the queue is making my brain hurt.4 -
So I believe in replicated effort.
I like the idea of multiple groups of people creating or engaging in the very same endeavor at any moment as long as intermediaries between these people visit on each project and determine its merits.
Examples could be data gathering, or development or research.
Example maybe being multiple people operating methods of altitude heightmap gathers or canvas imaging and publishing the data.
Or canvassing neighborhoods for new construction.
Or as in my case creating a nice handy dandy file types database for use as lookup table values in my application.
You know what I hate however ?
HAVING TO DO IT ALL THE HELL OVER AGAIN NO MATTER HOW QUICK OF A TASK ONCE I'VE DONE THE TASK ITS LIKELY NOT GOING TO GET MUCH BETTER THE SECOND OR THIRD GODDAMN TIME AROUND !!!!4 -
This is the second company I've joined where someone has chosen contentful instead of just using one of the many other coherent headless cms.
Why does all the data look like shit? Why am I being forced to use their shitty js lib. I want an endpoint through rtkquery damn it like the rest of the app.
Contentful can burn in hell. -
Why do clients expect that they would get a high quality machine learning model without a properly cleaned dataset? I usually get the response, ‘just scrape data and train it. It shouldn’t take long’3
-
So, some data need to be prepared during the summer and the diverse departments' elected data processors got shared in a Google spreadsheet they will need to fill with some basic data IT needs. Simple, straightforward data entry, with nothing private nor confidential. Just another divide-and-conquer-style large amount of data to enter & organise, that's all.
Today, I received a new comment notification as the owner of the spreadsheet. You can imagine my surprise when I saw that, for some f*cked up reasons, one of the guys just wrote the super-admin username & pw for one of the main data systems we use in a freaking comment in the spreadsheet... WTF...
Oh, and also, juuust in case, he also wrote the pin code that is normally required to pass through the device-check when you log-in as a super-admin from an unknown device and/or location.
Fortunately I could catch it on time, but this just ruined half of my day.
I am supposedly on freaking annual leave. Ha Ha. Ha. -
Need advice:
So I’m 20 years old. Got a decent job as software engineer with a really good pay and really want to break into machine learning.
Mastered NodeJS (my stack has always had node for the past 5-6 years) and I’m finding it difficult to switch to python for machine learning since things are so engraved in my head in javascript.
Aside from the syntax when I’m watching tutorials or reading books, I see data scientists and mathematicians make design mistakes in their code and it hurts my eyes and triggers my ocd.
I need tips on how to put my mindset in a moldable state so I can judge less and learn more and absorb data. Like you know that philosophy that when u get old your brain can’t learn things as fast anymore? I feel like that’s already happening to me rn at the age of 20.5 -
I am currently going to start data structures and algorithms in python...
suggest me a roadmap according to your experience2 -
I've over 17GB of data, downloaded a website, al of the content is .txt and .html.
I want to search inside all of these files.
What is the best tool to do that? any command or some software which can index so it'll be fast?19 -
Do we need compression on api level? say I have a rest api sending json data on requests. So if compression is needed then should it be in the server when returning the json response or in the client side when receiving it? which one is ideal?13
-
this moment when you write something to sync a 5 figure amount of entities from somewhere and halfway through a wild "sql exception: binary data truncated" weirdo occures because you where too lazy to adjust column constraints properly :-|3
-
Alright devs, in the US, can I not get a SIM card on which I simply put some money, and can use that until it runs out? Seems there is a 30-day limit tied to everything!!! I don't need data for 30 days!!!
This is totally possible in Europe, what is going on with regulation in US? Seems like it should be illegal.4 -
So,
sqlite lets you violate foreign key constraints on insert.
so you then can't delete the row you added
which has absolutely no child data so it shouldn't be preventing you from deleting it.
wtf.16 -
testng: if the name doesn't match (fubar locales whatever idk) doesn't match the corresponding test that uses it it'll just silently ignore instead of failing or running the test at all
@DataProvider(name="fubar")
public Object[][]{
//test data setup
}
@Test(dataProvider = "locales")
public void testWhatever(){}
whoever designed it to be this way: why?1 -
Newbie here, is storing json in sql (as like column data) as weird as I think it is or are there valid use cases?
The one I heard, didn't get the details but something like "startup move fast"12 -
Oh mannnnnn
We had a meeting with a data analyst today who is going to build an ML system for us.
...but at the start of the meeting, he didn't even know what our existing product even looks like!!!
...
Good one management!
Do I literally have to do EVERYTHING? It should be YOUR job to make sure every member has the essentials to participate in the meeting; YOU'RE the manager, NOT me!!!!
Oh also, let's not forget I got the meeting appointment HALF AN HOUR before it started...
#justmanagementthings
🤡🤡🤡🤡🤡🤡🤡🤡🤡🤡 -
Coming from a PHP, JS and Flutter developer:
I want to start building more websites entirely with Js frameworks. The less the better. Needs to import json data, perform ajax requests etc.
Can't decide, do I learn Vue or Svelte?9 -
You know when I think back to the ideas I've had and the things I've worked on. I'm having difficulty, with the exception of certain far out projects that were like unattainable, in thinking of anything I've done or thought of that does not involve: data visualization, data gathering, encryption/obfuscation, inventory/storage and/or communication.
am I just unimaginative ?
I did have an idea for a code translator and how it would work and what interfaces you'd have to adopt and how you'd attack implementing things that don't translate well like c++ to js for example. or c++ to c# for that matter ! but I never got far into it. though that would have been attainable as long as you had easy ways of generating bindings.
i mean pathing and navigation were things I thought of too but... that would pretty much be implementing someone else's stuff4 -
How would you create a mock for an Aggregator Microservice (stateless) which makes requests to other services for each request, transforms the data and then responds to the user?
I want to create a mock service where I don't t have to run the other services but it should create kinda realistic responses.
Have you had to create something like this?
I'd use it for testing another microservice that uses the aggregator.3 -
I don't know if this counts but wrote a generator that replaced a shitty linq to sql dal to use our system so I didn't have to mess with a web app I'd written. In place replacement with a few methods that made the other transaction lock field updates and calls etc
Most risky I can think of
Everything else was data migrations but there were always backups1 -
Can anyone help me in my task
(Solve n queens problem by stack using linked list) Data structure c++🙏9 -
Account service needs migrating, to AWS cause thats where everything is going.
Manager has got it in her head that a document store would be ideal for this.
My knee jerk reaction was a big No, i was told we'd discuss this at a later time.
My main argument here is that data is inheritly relational, and now i'm looking for more.
Any ideas why a documentstore is not a good fit for accounts?
Thanks!2 -
I have a side project which contains very simple data (a URL and a list of strings associated with that URL). I would like a way to automatically generate a webpage for each of those links (I believe the right term is template ?). Does anyone have any suggestions? Is this something like Next.js or Gatsby would be used for? I can provide more info if I’m not clear enough8
-
when you gotta generate some product ideas and work for the first time at a new company when you've spent your career as a heads down code monkey
bruh i dont know any of the stats or data, or the product yet -
Could there be a "greater" GPL which explicitly declares that the constraint extends to use of the code as statistical data, such as in machine learning models?1
-
For work reasons I started using power bi and wow. Just wow. Any insight on why it's so prevalent when it comes to data visual?5
-
Question for Work at home peeps. For Data Entry positions. Do companies send you stuff to your house to do the job. Like laptop with software installed?. I was offered a data entry through Upwork. Just looking to get my front into the whole freelance thing. They asking for like address and Email. Is this normal1
-
Hear me out:
Since keras and tf are pretty much schema design rn what if someone made a no-code solution where you drag and drop layers and tweak things in a UI so those data scientists can design it in a UI instead of writing shitty code?5 -
I recently came across this article with some basic security advices, like use 2fa security key, encrypt your USB keys, don't use untrusted USB chargers / cables / ports (or use a data blocker cable if you need to charge your device). It made me think, how relevant are the USB-related threats and risks today? Do people really still use and carry so many wired USB devices, and just drop or plug them wherever?
The last time I used an USB device to transfer some important data was probably over 10 years ago, and for the love of god I don't know anyone who still carries an USB key with sensitive data with them on a daily basis, much less actively uses it. Besides, whoever still does that probably puts their USB key on the same keychain as their ID / access tag and a bunch of other keys (including a 2fa device if they use one) - they're not going to lose just some sensitive data, they're going to lose authentication and physical access devices as well, and that could turn a small data leak into a full-scale incident, with or without an encrypted USB device.
I'm also not sure about untrusted USB cables and ports, from what I've seen the USB outlets and cables are pretty much non-existent in public places, most places offer wireless charging pads instead (usually built into a hand rest or table surface).2 -
I want extract data from a pdf and turn it into a json object then serve it as an api, I’ve never scraped data from pdf before, what’s the best way to do that?2
-
After completing the startup, all about api calls;
Supabase as server, netlify calls for storing data into supab, some frontend and stripe as payment method, using webhooks to do some logic behind,
I never thought that I could finish it, now I'm answering questions on Quora about it and doing content for it. Feels odd and want to code AGAIN!2 -
Do people still use redis when you already use postgres? How is pg perf if you are writing like 10,000s of row data/second. I am slightly outdated....12
-
* break it into elementary steps, small enough to fit into your "estimation time unit", e.g. days or hours.
* estimate those steps for "developing at a leisurely pace" if nothing goes wrong.
* think about "what could go wrong" (list everything!) and adjust values accordingly.
* adjust total amount with experience values, like:
* times 1.2 for every manager
* times 1 to 4 based on which legacy projects i have to touch
and finally:
* multiply with `1+log(t/u,2)`, with `u` being the amount of useful data in the requirement description and `t` being the total amount of data in the requirement description
* sample: with our current "favourite" customer, about 90% of all tickets is garbage, so t/u = 100/10 = 10 => log(10,2) = 3.3 => multiply everything with 4.31 -
stupid dumb wordpress, who POSTS a user registration, returns data to the SAME site with a dumb "successful" message alert, and THEN redirects to whatever page you want?!?! looks like shit, feels like shit in terms of UX
JUST REDIRECT ME TO THE PAGE I WANT FROM THE SERVER THAT'S THE WHOLE POINT OF PHP
I swear its as slow as balls too -
what are the basics I should know about "data streaming" for working on video streaming companies as a future senior backend Golang developer?4
-
I know it's not made to be resilient in any way, only fast, as fast as possible, but man, the memcache_tool script just made my life a million times easier by facilitating a complete data transfer between two memcache instances, allowing for a rolling update without any session data loss!
...One day... I hope it can be migrated to redis... But for now... Thanks lord for the dump command and the wrapper script <3 -
Hi everyone
I have a python script that continuously collects data for me. I want to be able to display that data on a node js server. How should I go about this? I was thinking of maybe having the python script send get requests to the server but I feel that is not the right answer. Let me know if u guys need more info, thanks!5 -
Do companies still use the Merise method? It seems a bit off to me.
I am learning it at school, but it makes many tables that I think could be merged into one...3 -
today the backend dev told me that he was trying to setup typescript for the company for so long and the company resisted him but he was super proud of his achievement of getting typescript into the project and i told him. sorry I agree with the company, you're stupid
i didn't tell him he was stupid ofcourse but i told him why i thought it was an unnecessary dep
i asked him are you doing a lot of number crunching? he said no
and i told him, most of the data you're going to recieve is in a string format or in json strings
very rarely are you going to get number data
and you can easily coerce the data into whatever you want37 -
Spent months writing up a model and fine tuning, trained on complex image data (all we had to work with), to work well with what it needed to do.
Only to have “subject matter experts” be like: why don’t you use PCA and K Nearest Neighbours.
The fuck do you think the baseline was written with? -
Obviously ai and autodocument recognition and data extraction is not usable yet
Excepting when it's a pdf not a scanned document or image
Ocr may be but shift the whole.image or bend it or remove a border from some white out
And then handwritten -
Was working a record keeping system for the Airport for tracking departures and arrivals and some COVID-19 data
ended up realizing that the stack i had gone with wasn't gonna cut it
Had to port the whole thing to a new web framework realizing that the one i had gone with made some operations a bit complicated -
So, I feel wayyy behind the tech curve right now.
The SSD implementations you see online, they're still just a bunch of seperate sort of chaos machines that contain the standard perceptron-like model of a weight, cost, and bias right ? They just kind of inferred their values by training like any other neural network, in its sep-erate parts and just fed pieces of output data generated by other parts of the neural network to it right ?
I mean it implements with pytorch so its basically a really big array of tuples in a sense that are maniupulated in a specific way.
and then CNN's just feed data back into another trained piece of the model right ?
I'm curious because object classification is about the ONLY thing I've seen work even close to properly lol
there is just so much fraud these days. sigh.
and so many lamentable tech choices and attempts... like node lol -
Looking for good literature regarding CRUD. Basically i want to have a list of possible dataoperations nowadays. And the relations to commands like Copy, Paste, Search, List, Undo Redo, Macros etc. Any suggestions?16
-
Interviewed for a job. Said that the colleague in charge of data engineering picked MSSQL Server for data warehousing, and that I had to write a plugin for that.
Interviewer - experienced in all things data - chuckled as soon as I said Microsoft. -
I've been sitting here staring at extension types and I wonder, what if I had a partial file with partial data ?
In general one could say that in every case where say a header is missing that is ALWAYS going to have some identifying characteristics even given a characteristic statistically frequent pattern of data, that there is always a null value that appears as total chaos.
But I wonder, is there a way beyond simply trying every goddamn possible combination of things until meaningful data is extracted to identify a file by its content when part of that content that is usually used for such a purpose, is missing ?
What kind of application or technology would be required for this ? Certainly not neural networks, but obviously some kind of ai right ?10 -
Working a week on LSTM based text classifier, getting 89% accuracy only to then get better result with Logistic Regression which was supposed to serve as baseline, lol. Background: 180+ classes of google product categorization taxonomy, 20 million rows of data items (short texts). Had a similar experience once on sentiment classification, where SVMlight outperformed NN models.
-
So, I don't get the concept of dockfile. I understand it and what it does but, I learnt about it a few weeks back. Is it so important and necessary?10
-
How a 2020 batch fresher from tier 2 college can get placed in product based mnc as a software engineer or data analyst ?2
-
Can you recommend me some simple php+mysql based backend framework in order to build a simple rest api (with login, signup, scope data) ?6
-
My friends were wondering if I could teach them the basics of coding. What order should I do it in? The basic things I want to teach them are input/output, data types (numbers, strings, arrays, etc.), flow control (loops, if/else), functions, variables, and maybe oop if I'm in the right mood.
Also, would python be a good language to start with? It's definitely the language I know the best.8 -
Is there up-to-date API documentation for devrant? I am stuck on the posting API calls, especially on what data is required.8
-
Last year I did a statistics course, and my classes were completely remote. When it came down to exam season, instead of studying I learned python's data analysis libraries, I passed with flying colors. I have an idea, if exams continue to be online I will be spending a lot of my time trying to get the answers from whatever API they are using, hopefully, they have poor security. If it's hopeless, I'll just study1
-
I have to implement a gymconnect client in flutter and im quite lost.
I have already connected my flutter app to the device via Bluetooth and started to read some of the data but im not sure if im doing it well.
Could someone give me a tip about implementing this kind of client? Any tip can help a lot, thanks.6