Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
-
The teacher told me to describe the transformations, so I did. I wonder why she didn't accept it...
10 -
Writing more infrastructure than product.
Look, my application requests and transforms data from a single external API endpoint, it's just one GET request...
But I made an intelligent response caching middleware to prevent downtime when the parent API goes down, I made mocks and tests for everything, the documentation is directly generated from the code and automatically hosted for every git branch using hooks, responses are translated into JSONschema notation which automatically generate integration tests on commit, and the transformations are set up as a modular collection of composable higher order lenses!
Boss: Please use less amphetamine.5 -
One day after the release of the website of a medium sized travel company, I made a big mistake by accidentally taking it offline for 1 hour during peak usage (~150 simultaneous visitors).
Turns out deleting the wrong image transformation cache folder in production can hang up the PHP process for taking too much load on regenerating image transformations.
The designer of PHP probably took a big load too while creating the first draft.9 -
I accidentally created something adorable. See, what happened was I wanted to figure out how to programmatically create blobcats for different countries, since I’d done it by hand for every country represented at my company and wanted a quick way to catch up once we add people from different countries.
Enter Cloudinary!
Wrote a transform that would take the image, shrink it to 150 px wide, then add the blobcat emoji as the underlayer behind and above it, then add the little arms I drew die holding the flag on top.
And that worked! I had a folder of flags and the frontend could query them using the transformations and the country blobcats were born! I even added a preset so I could view the little blobcats with their flags when I looked at the uploaded flags
But then I’m like “Eh, each transform adds a thing against my limit and I really don’t want to go over if this gets popular (why do I ever think _that’s_ a possibility?). I know! I’ll just add the transform on upload and reupload all the flags to a different folder! Then I’ll just need to request the image instead of having to do the transform per request.
And that worked!
But I forgot I still had my preset transform on.
So in addition to the successful country blobcat, I had a preview of what I like to call papa blobcat with kitten blobcat.
4 -
EoS1: This is the continuation of my previous rant, "The Ballad of The Six Witchers and The Undocumented Java Tool". Catch the first part here: https://devrant.com/rants/5009817/...
The Undocumented Java Tool, created by Those Who Came Before to fight the great battles of the past, is a swift beast. It reaches systems unknown and impacts many processes, unbeknownst even to said processes' masters. All from within it's lair, a foggy Windows Server swamp of moldy data streams and boggy flows.
One of The Six Witchers, the Wild One, scouted ahead to map the input and output data streams of the Unmapped Data Swamp. Accompanied only by his animal familiars, NetCat and WireShark.
Two others, bold and adventurous, raised their decompiling blades against the Undocumented Java Tool beast itself, to uncover it's data processing secrets.
Another of the witchers, of dark complexion and smooth speak, followed the data upstream to find where the fuck the limited excel sheets that feeds The Beast comes from, since it's handlers only know that "every other day a new one appears on this shared active directory location". WTF do people often have NPC-levels of unawareness about their own fucking jobs?!?!
The other witchers left to tend to the Burn-Rate Bonfire, for The Sprint is dark and full of terrors, and some bigwigs always manage to shoehorn their whims/unrelated stories into a otherwise lean sprint.
At the dawn of the new year, the witchers reconvened. "The Beast breathes a currency conversion API" - said The Wild One - "And it's claws and fangs strike mostly at two independent JIRA clusters, sometimes upserting issues. It uses a company-deprecated API to send emails. We're in deep shit."
"I've found The Source of Fucking Excel Sheets" - said the smooth witcher - "It is The Temple of Cash-Flow, where the priests weave the Tapestry of Transactions. Our Fucking Excel Sheets are but a snapshot of the latest updates on the balance of some billing accounts. I spoke with one of the priestesses, and she told me that The Oracle (DB) would be able to provide us with The Data directly, if we were to learn the way of the ODBC and the Query"
"We stroke at the beast" - said the bold and adventurous witchers, now deserving of the bragging rights to be called The Butchers of Jarfile - "It is actually fewer than twenty classes and modules. Most are API-drivers. And less than 40% of the code is ever even fucking used! We found fucking JIRA API tokens and URIs hard-coded. And it is all synchronous and monolithic - no wonder it takes almost 20 hours to run a single fucking excel sheet".
Together, the witchers figured out that each new billing account were morphed by The Beast into a new JIRA issue, if none was open yet for it. Transactions were used to update the outstanding balance on the issues regarding the billing accounts. The currency conversion API was used too often, and it's purpose was only to give a rough estimate of the total balance in each Jira issue in USD, since each issue could have transactions in several currencies. The Beast would consume the Excel sheet, do some cryptic transformations on it, and for each resulting line access the currency API and upsert a JIRA issue. The secrets of those transformations were still hidden from the witchers. When and why would The Beast send emails, was still a mistery.
As the Witchers Council approached an end and all were armed with knowledge and information, they decided on the next steps.
The Wild Witcher, known in every tavern in the land and by the sea, would create a connector to The Red Port of Redis, where every currency conversion is already updated by other processes and can be quickly retrieved inside the VPC. The Greenhorn Witcher is to follow him and build an offline process to update balances in JIRA issues.
The Butchers of Jarfile were to build The Juggler, an automation that should be able to receive a parquet file with an insertion plan and asynchronously update the JIRA API with scores of concurrent requests.
The Smooth Witcher, proud of his new lead, was to build The Oracle Watch, an order that would guard the Oracle (DB) at the Temple of Cash-Flow and report every qualifying transaction to parquet files in AWS S3. The Data would then be pushed to cross The Event Bridge into The Cluster of Sparks and Storms.
This Witcher Who Writes is to ride the Elephant of Hadoop into The Cluster of Sparks an Storms, to weave the signs of Map and Reduce and with speed and precision transform The Data into The Insertion Plan.
However, how exactly is The Data to be transformed is not yet known.
Will the Witchers be able to build The Data's New Path? Will they figure out the mysterious transformation? Will they discover the Undocumented Java Tool's secrets on notifying customers and aggregating data?
This story is still afoot. Only the future will tell, and I will keep you posted.6 -
YES FUCKING FINALLY. After 8 years of programming professionally, I got to use linear transformations and algebra while coding with javascript in real life.4
-
Oh I have quite a few.
#1 a BASH script automating ~70% of all our team's work back in my sysadmin days. It was like a Swiss army knife. You could even do `ScriptName INC_number fix` to fix a handful of types of issues automagically! Or `ScriptName server_name healthcheck` to run HW and SW healthchecks. Or things like `ScriptName server_name hw fix` to run HW diags, discover faulty parts, schedule a maintenance timeframe, raise a change request to the appropriate DC and inform service owners by automatically chasing them for CHNG approvals. Not to mention you could `ScriptName -l "serv1 serv2 serv3 ..." doSomething` and similar shit. I am VERY proud of this util. Employee liked it as well and got me awarded. Bought a nice set of Swarowski earrings for my wife with that award :)
#2 a JAVA sort-of-lib - a ModelMapper - able to map two data structures with a single util method call. Defining datamodels like https://github.com/netikras/... (note the @ModelTransform anno) and mapping them to my DTOs like https://github.com/netikras/... .
#3 a @RestTemplate annptation processor / code generator. Basically this dummy class https://github.com/netikras/... will be a template for a REST endpoint. My anno processor will read that class at compile-time and build: a producer (a Controller with all the mappings, correct data types, etc.) and a consumer (a class with the same methods as the template, except when called these methods will actually make the required data transformations and make a REST call to the producer and return the API response object to the caller) as a .jar library. Sort of a custom swagger, just a lil different :)
I had #2 and #3 opensourced but accidentally pushed my nexus password to gitlab. Ever since my utils are a private repo :/3 -
// Stupid JSON
// Tale of back-end ember api from hell
// Background: I'm an android dev attempting to integrate // with an emberjs / rails back-end
slack conversation:
me 3:51pm: @backend-dev: Is there something of in the documentation for the update call on model x? I formed the payload per the docs like so
{
"valueA": true,
"valueB": false
}
and the call returns success 200 but the data isn't being updated when fetching again.
----------------------------------------------------------------------------------------
backend-dev 4:00pm: the model doesn't look updated for the user are you sure you made the call?
----------------------------------------------------------------------------------------
me 4:01pm: Pretty sure here's my payload and a screen grab of the successful request in postman <screenshot attached>
----------------------------------------------------------------------------------------
backend-dev 4:05pm: well i just created a new user on the website and it worked perfectly your code must be wrong
----------------------------------------------------------------------------------------
me 4:07pm: i can test some more to see if i get any different responses
----------------------------------------------------------------------------------------
backend-dev 4:15pm: ahhhhhh... I think it's expecting the string "true", not true
----------------------------------------------------------------------------------------
me 4:16: but the fetch call returns the json value as a boolean true/false
----------------------------------------------------------------------------------------
backend-dev 4:18pm: thats a feature, the flexible type system allows us to handle all sorts of data transformations. android must be limited and wonky.
----------------------------------------------------------------------------------------
me 4:19pm: java is a statically typed language....
// crickets for ten minutes
me 4:30pm: i'll just write a transform on the model when i send an update call to perform toString() on the boolean values
----------------------------------------------------------------------------------------
backend-dev 4:35: great! told you it wasn't my documentation!
// face palm forever4 -
For shit's sake, data stream processing really is only for people with high throughput looking to do transformations on their data; not for people aggregating <10Gb/day of data.
Fuck me DSP is going to be the new buzzword of 2020 and I'm not looking forward to it. I've already got stakeholders wondering if we can integrate it when we dont have the need, nor the resources or funds.10 -
Just happened today!
So since this morning we've been trying to get our website ready for UAT deployment Monday next week, even though we only were told of it yesterday. Since we had some critical merge conflicts to unscrew on our dev branch for promoting to UAT, we sent a warning to everyone on our hipchat group
Dev team: @all please don't commit anything to the repo for an hour or so while we get the branch good for dev and uat build
Tech lead: ok
That should be enough warning, right? Surely our tech lead, who has been piling up our scope creep trying to please our stakeholders, understands well enough not to do a single goddamn thing on our repo until we sort it out, right?
Nope.
10 minutes later our tech lead pushes several changes that not only break our builds but also remove all our configuration transformations. I just stormed out of the office to avoid sending her on a one-way ticket to slapsville and fuckyoutown. Geez goddamn louise. -
You know that experience when you update a modern desktop app that uses a zillion abstraction layers and the first time you do anything it freezes for a little bit while the heavily deferred metaprogramming and asset transformations are executed and cached?
I always imagine polystyrene balls flying all over the place.1 -
I hate looking for sources to cite in my thesis.
Either it's a book for 100€ or a paper thst I need to subscribe to IEEE xplore to..
And most of the stuff on Springer Link (which I get free access through uni) is behind a pay wall anyway...
On that note, does anyone happen to know decent sources on basic signal processing and image processing?
I.e. DCT, DFT transformations and so on19 -
My apologize to everyone I told that functional programming is declarative.
It's actually imperative. Thank you @AndSoWeCode for figuring that out. I spent the whole day thinking about it.
Lisp is imperative. It's just different way to define the exact data transformations, and that's quite imperative.
On the other hand, HTML, CSS, config files and markup languages are declarative.
But writing the imperative program which is configured with declarative configs seems like great idea. Consider Apache web server and others.3 -
I just posted dump of my notes on linear transformations, any feedback is appreciated! ◾️▫️
https://monades.dev/notes-on-linear...
2 -
How capable are mobile gpus? I made a small game with godot, and all of the animations are done with shaders, and i get a really good performance, but i want to try a 3d game next and not sure if throwing 3d transformations on a mobile gpur is a good idea.5
-
Question about PR best practices.
I work for an analytics company and often have to implement new ETL steps. The data transformations in these steps can be complex and the major changes are usually 500+ lines up to around 2000 (the last one had 765 lines just for schemas).
What's the best way to split up the changes into multiple PRs, bearing in mind that it isnt guaranteed that a file won't change as the change is built up? -
I swear I touched some weird and complex programming shit in over a decade of programming.
I interfaced myself through C# to C++ Firmware, I wrote Rfid antennas calibration and reading software with a crappy framework called OctaneSDK (seems easy until you have to know how radio signal math and ins and outs work to configure antennas for good performance), I wrote full blown, full stack enterprise web portals and applications.with most weird ass dbs since the era of JDBC, ODBC up to managed data access and entity framework, cloud documental databases and everything.
Please, please, please, PLEASE I BEG YOU, anyone, I don't even have the enough life force to pour into this, explain me why the hell Jest is still a thing in javascript testing.
I read on the site:
"Jest is a delightful JavaScript Testing Framework with a focus on simplicity."
Using jest doesn't feel any delightful and I can't see any spark of focus and simplicity in it.
I tried to configure it in an angular project and it's a clustefuck of your worst nightmares put togheter.
The amount of errors and problems and configurations I had to put up felt like setting up a clunky version of a rube goldberg's machine.
I had to uninstall karma/jasmine, creating config files floating around, configure project files and tell trough them to jest that he has to do path transformations because he can't read his own test files by itself and can't even read file dependencies and now it has a ton of errors importing dependencies.
Sure, it's focused on simplicity.
Moreover, the test are utter trash.
Hey launch this method and verify it's been launched 1 time.
Hey check if the page title is "x"
God, I hate js with passion since years, but every shit for js I put my hands on I always hope it will rehab its reputation to me, instead every fucking time it's worse than before. -
The number of concurrent transformations impacting more than half of the codebase in Orchid surpassed 4, so instead of walking the reference graph for each of these I'm updating the whole codebase, from lexer to runtime, in a single pass.
In this process, I also got to reread a lot of code from a year ago. This is the project I learned Rust with. It's incredible, not just how much better I've gotten at this language, but also how much better I've gotten at structuring code on general.
Interestingly though my problem-solving ability seems to be the same. I can tell this by looking at the utilities I made to solve specific well-defined abstract problems. I may have superficial issues with how the code is spelled out in text, but the logic itself is as good as anything I could come up with today.2 -
Honest question. When do you consider yourself a "Big data engineer"?
Today I managed to create a system that collects historical metrics from monitoring tools every 5 minutes and do all sorts of crazy transformations to make them ingestible by grafana Mimir in OTLP protocol. Doing 600gb a dat, millions of active time series, .... And I still feel it's, "small"
Thoughts?5 -
Okay. Here's the ONLY two scenarios where automated testing is justified:
- An outsourcing company who is given the task of bug elimination in legacy code with a really short timeframe. Then yes, writing tests is like waging war on bugs, securing more and more land inch after inch.
- A company located in an area where hiring ten junior developers is cheaper than hiring one principal developer. Then yes, the business advantage is very real.
That's it. That's the only two scenarios where automated testing is justified. Other such scenarios doesn't exist.
Why? Because any robust testing system (not just "adding some tests here and there") is a _declarative_ one. On top of already being declarative (opposed to the imperative environment where the actual code exists), if you go further and implement TDD, your tests suddenly begins to describe your domain area, turning into a declarative DSL.
Such transformations are inevitable. You can't catch bugs in the first place if your tests are ignorant of entities your code is working with.
That being said, any TDD-driven project consists of two things:
- Imperative code that implements business logic
- Declarative DSL made of automated tests that also describes the same business logic
Can't you see that this system is _wet_? The tests set alone in a TDD-driven project are enough to trivially derive the actual, complete code from it.
It's almost like it's easier to just write in a declarative language in the first place, in the same way tests are written in TDD project, and scrap the imperative part altogether.
In imperative languages, absence of errors can be mathematically guaranteed. In imperative languages, the best performance (e.g. the lowest algorithmic complexity) can also be mathematically guaranteed. There is a perfectly real point after which Haskell rips C apart in terms of performance, and that point happens earlier on than you think.
If you transitioned from a junior who doesn't get why tests are needed to a competent engineer who sees value in TDD, that's amazing. But like with any professional development, it's better to remember that it's always possible to go further. After the two milestones I described, the third exists — the complete shift into the declarative world.
For a human brain, it's natural to blindly and aggressively reject whatever information leads to the need of exiting the comfort zone. Hence the usual shitstorm that happens every time I say something about automated testing. I understand you, and more than that, I forgive you.
The only advice I would allow myself to give you is just for fun, on a weekend, open a tutorial to a language you never tried before, and spend 20 minutes messing around with it. Maybe you'll laugh at me, but that's the exact way I got from earning $200 to earning $3500 back when I was hired as a CTO for the first time.
Good luck!6 -
Do you recommend hiring junior or mid-level dev for a python role that involves mostly data transformations with pandas, growth and marketing projects like social media bots, consuming APIs for data and some experience with Azure SQL db? I’m worried if we hire too senior then they will leave as the role doesn’t involve any advanced software engineering, like caches, web apps, rest apis, etc. It’s more of a handyman that can automate and hack a solution to a business problem: for example, learning openCV to automatically crop thousands of images extracting only the text3
Top Tags
Weekly Rant
View