Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "map reduce"
-
Hi, I am a Javascript apprentice. Can you help me with my project?
- Sure! What do you need?
Oh, it’s very simple, I just want to make a static webpage that shows a clock with the real time.
- Wait, why static? Why not dynamic?
I don’t know, I guess it’ll be easier.
- Well, maybe, but that’s boring, and if that’s boring you are not going to put in time, and if you’re not going to put in time, it’s going to be harder; so it’s better to start with something harder in order to make it easier.
You know that doesn’t make sense right?
- When you learn Javascript you’ll get it.
Okay, so I want to parse this date first to make the clock be universal for all the regions.
- You’re not going to do that by yourself right? You know what they say, don’t repeat yourself!
But it’s just two lines.
- Don’t reinvent the wheel!
Literally, Javascript has a built in library for t...
- One component per file!
I’m lost.
- It happens, and you’ll get lost managing your files as well. You should use Webpack or Browserify for managing your modules.
Doesn’t Javascript include that already?
- Yes, but some people still have previous versions of ECMAScript, so it wouldn’t be compatible.
What’s ECMAScript?
- Javascript
Why is it called ECMAScript then?
- It’s called both ways. Anyways, after you install Webpack to manage your modules, you still need a module and dependency manager, such as bower, or node package manager or yarn.
What does that have to do with my page?
- So you can install AngularJS.
What’s AngularJS?
- A Javascript framework that allows you to do complex stuff easily, such as two way data binding!
Oh, that’s great, so if I modify one sentence on a part of the page, it will automatically refresh the other part of the page which is related to the first one and viceversa?
- Exactly! Except two way data binding is not recommended, since you don’t want child components to edit the parent components of your app.
Then why make two way data binding in the first place?
- It’s backed up by Google. You just don’t get it do you?
I have installed AngularJS now, but it seems I have to redefine something called a... directive?
- AngularJS is old now, you should start using Angular, aka Angular 2.
But it’s the same name... wtf! Only 3 minutes have passed since we started talking, how are they in Angular 2 already?
- You mean 3.
2.
- 3.
4?
- 5.
6?
- Exactly.
Okay, I now know Angular 6.0, and use a component based architecture using only a one way data binding, I have read and started using the Design Patterns already described to solve my problem without reinventing the wheel using libraries such as lodash and D3 for a world map visualization of my clock as well as moment to parse the dates correctly. I also used ECMAScript 6 with Babel to secure backwards compatibility.
- That’s good.
Really?
- Yes, except you didn’t concatenate your html into templates that can be under a super Javascript file which can, then, be concatenated along all your Javascript files and finally be minimized in order to reduce latency. And automate all that process using Gulp while testing every single unit of your code using Jasmine or protractor or just the Angular built in unit tester.
I did.
- But did you use TypeScript?37 -
I AM TIRED
warning: this rant is going to be full of negativity , CAPS, and cursing.
People always think and they always write that programming is an analytical profession. IF YOU CANNOT THINK IN AN ANALYTICAL WAY THIS JOB IS NOT FOR YOU! But the reality could not be farther from the truth.
A LOT of people in this field whether they're technical people or otherwise, just lack any kind of reasoning or "ANALYTICAL" thinking skills. If anything, a lot of of them are delusional and/or they just care about looking COOL. "Because programming is like getting paid to solve puzzles" *insert stupid retarded laugh here*.
A lot of devs out there just read a book or two and read a Medium article by another wannabe, now think they're hot shit. They know what they're doing. They're the gods of "clean" and "modular" design and all companies should be in AWE of their skills paralleled only by those of deities!
Everyone out there and their Neanderthal ancestor from start-up founders to developers think they're the next Google/Amazon/Facebook/*insert fancy shitty tech company*.
Founder? THEY WANT TO MOVE FAST AND GET TO MARKET FAST WITH STUPID DEADLINES! even if it's not necessary. Why? BECAUSE YOU INFERIOR DEVELOPER HAVE NOT READ THE STUPID HOT PILE OF GARBAGE I READ ONLINE BY THE POEPLE I BLINDLY COPY! "IF YOU'RE NOT EMBARRASSED BY THE FIRST VERSION OF YOU APP, YOU DID SOMETHING WRONG" - someone at Amazon.
Well you delusional brainless piece of stupidity, YOU ARE NOT AMAZON. THE FIRST VERSION THAT THIS AMAZON FOUNDER IS EMBARRASSED ABOUT IS WHAT YOU JERK OFF TO AT NIGHT! IT IS WHAT YOU DREAM ABOUT HAVING!
And oh let's not forget the tech stacks that make absolutely no fucking sense and are just a pile of glue and abstraction levels on top of abstraction levels that are being used everywhere. Why? BECAUSE GOOGLE DOES IT THAT WAY DUH!! And when Google (or any other fancy shit company) changes it, the old shitty tech stack that by some miracle you got to work and everyone is writing in, is now all of a sudden OBSOLETE! IT IS OLD. NO ONE IS WRITING SHIT IN THAT ANYMORE!
And oh my god do I get a PTSD every time I hear a stupid fucker saying shit like "clean architecture" "clean shit" "best practice". Because I have yet to see someone whose sentences HAVE TO HAVE one of these words in them, that actually writes anything decent. They say this shit because of some garbage article they read online and in reality when you look at their code it is hot heap of horseshit after eating something rancid. NOTHING IS CLEAN ABOUT IT. NOTHING IS DONE RIGHT. AND OH GOD IF THAT PERSON WAS YOUR TECH MANAGER AND YOU HAVE TO LISTEN TO THEM RUNNING THEIR SHITHOLE ABOUT HOW YOUR SIMPLE CODE IS "NOT CLEAN". And when you think that there might be a valid reason to why they're doing things that way, you get an answer of someone in an interview who's been asked about something they don't know, but they're trying to BS their way to sounding smart and knowledgable. 0 logic 0 reason 0 brain.
Let me give you a couple of examples from my unfortunate encounters in the land of the delusional.
I was working at this start up which is fairly successful and there was this guy responsible for developing the front-end of their website using ReactJS and they're using Redux (WHOSE SOLE PURPOSE IS TO ELIMINATE PASSING ATTRIBUTES FOR THE PURPOSE OF PASSING THEM DOWN THE COMPONENT HIERARCHY AGIAN). This guy kept ranting about their quality and their shit every single time we had a conversation about the code while I was getting to know everything. Also keep in mind he was the one who decided to use Redux. Low and behold there was this component which has THIRTY MOTHERFUCKING SEVEN PROPERTIES WHOSE SOLE PURPOSE IS BE PASSED DOWN AGAIN LIKE 3 TO 4 TIMES!.
This stupid shit kept telling me to write code in a "functional" style. AND ALL HE KNOWS ABOUT FUNCTIONAL PROGRAMMING IS USING MAP, FILTER, REDUCE! And says shit like "WE DONT NEED UNIT TESTS BECAUSE FUNCTIONAL PROGRAMMING HAS NO ERRORS!" Later on I found that he read a book about functional programming in JS and now he fucking thinks he knows what functional programming is! Oh I forgot to mention that the body of his "maps" is like 70 fucking lines of code!
Another fin-tech company I worked at had a quote from Machiavelli's The Prince on EACH FUCKING DESK:
"There is nothing more difficult to take in hand, more perilous to conduct, or more uncertain in its success, than to take the lead in the introduction of a new order of things."
MOTHERFUCKER! NEW ORDER OF THINGS? THERE 10 OTHER COMPANIES DOING THE SAME SHIT ALREADY!
And the one that got on my nerves as a space lover. Is a quote from Kennedy's speech about going to the moon in the 60s "We choose to go to the moon and do the hard things ..."
YOU FUCKING DELUSIONAL CUNT! YOU THINK BUILDING YOUR SHITTY COPY PASTED START UP IS COMPARABLE TO GOING TO THE MOON IN THE 60S?
I am just tired of all those fuckers.13 -
Two big moments today:
1. Holy hell, how did I ever get on without a proper debugger? Was debugging some old code by eye (following along and keeping track mentally, of what the variables should be and what each step did). That didn't work because the code isn't intuitive. Tried the print() method, old reliable as it were. Kinda worked but didn't give me enough fine-grain control.
Bit the bullet and installed Wing IDE for python. And bam, it hit me. How did I ever live without step-through, and breakpoints before now?
2. Remember that non-sieve prime generator I wrote a while back? (well maybe some of you do). The one that generated quasi lucas carmichael (QLC) numbers? Well thats what I managed to debug. I figured out why it wasn't working. Last time I released it, I included two core methods, genprimes() and nextPrime(). The first generates a list of primes accurately, up to some n, and only needs a small handful of QLC numbers filtered out after the fact (because the set of primes generated and the set of QLC numbers overlap. Well I think they call it an embedding, as in QLC is included in the series generated by genprimes, but not the converse, but I digress).
nextPrime() was supposed to take any arbitrary n above zero, and accurately return the nearest prime number above the argument. But for some reason when it started, it would return 2,3,5,6...but genprimes() would work fine for some reason.
So genprimes loops over an index, i, and tests it for primality. It begins by entering the loop, and doing "result = gffi(i)".
This calls into something a function that runs four tests on the argument passed to it. I won't go into detail here about what those are because I don't even remember how I came up with them (I'll make a separate post when the code is fully fixed).
If the number fails any of these tests then gffi would just return the value of i that was passed to it, unaltered. Otherwise, if it did pass all of them, it would return i+1.
And once back in genPrimes() we would check if the variable 'result' was greater than the loop index. And if it was, then it was either prime (comparatively plentiful) or a QLC number (comparatively rare)--these two types and no others.
nextPrime() was only taking n, and didn't have this index to compare to, so the prior steps in genprimes were acting as a filter that nextPrime() didn't have, while internally gffi() was returning not only primes, and QLCs, but also plenty of composite numbers.
Now *why* that last step in genPrimes() was filtering out all the composites, idk.
But now that I understand whats going on I can fix it and hypothetically it should be possible to enter a positive n of any size, and without additional primality checks (such as is done with sieves, where you have to check off multiples of n), get the nearest prime numbers. Of course I'm not familiar enough with prime number generation to know if thats an achievement or worthwhile mentioning, so if anyone *is* familiar, and how something like that holds up compared to other linear generators (O(n)?), I'd be interested to hear about it.
I also am working on filtering out the intersection of the sets (QLC numbers), which I'm pretty sure I figured out how to incorporate into the prime generator itself.
I also think it may be possible to generator primes even faster, using the carmichael numbers or related set--or even derive a function that maps one set of upper-and-lower bounds around a semiprime, and map those same bounds to carmichael numbers that act as the upper and lower bound numbers on the factors of a semiprime.
Meanwhile I'm also looking into testing the prime generator on a larger set of numbers (to make sure it doesn't fail at large values of n) and so I'm looking for more computing power if anyone has it on hand, or is willing to test it at sufficiently large bit lengths (512, 1024, etc).
Lastly, the earlier work I posted (linked below), I realized could be applied with ECM to greatly reduce the smallest factor of a large number.
If ECM, being one of the best methods available, only handles 50-60 digit numbers, & your factors are 70+ digits, then being able to transform your semiprime product into another product tree thats non-semiprime, with factors that ARE in range of ECM, and which *does* contain either of the original factors, means products that *were not* formally factorable by ECM, *could* be now.
That wouldn't have been possible though withput enormous help from many others such as hitko who took the time to explain the solution was a form of modular exponentiation, Fast-Nop who contributed on other threads, Voxera who did as well, and support from Scor in particular, and many others.
Thank you all. And more to come.
Links mentioned (because DR wouldn't accept them as they were):
https://pastebin.com/MWechZj912 -
EoS1: This is the continuation of my previous rant, "The Ballad of The Six Witchers and The Undocumented Java Tool". Catch the first part here: https://devrant.com/rants/5009817/...
The Undocumented Java Tool, created by Those Who Came Before to fight the great battles of the past, is a swift beast. It reaches systems unknown and impacts many processes, unbeknownst even to said processes' masters. All from within it's lair, a foggy Windows Server swamp of moldy data streams and boggy flows.
One of The Six Witchers, the Wild One, scouted ahead to map the input and output data streams of the Unmapped Data Swamp. Accompanied only by his animal familiars, NetCat and WireShark.
Two others, bold and adventurous, raised their decompiling blades against the Undocumented Java Tool beast itself, to uncover it's data processing secrets.
Another of the witchers, of dark complexion and smooth speak, followed the data upstream to find where the fuck the limited excel sheets that feeds The Beast comes from, since it's handlers only know that "every other day a new one appears on this shared active directory location". WTF do people often have NPC-levels of unawareness about their own fucking jobs?!?!
The other witchers left to tend to the Burn-Rate Bonfire, for The Sprint is dark and full of terrors, and some bigwigs always manage to shoehorn their whims/unrelated stories into a otherwise lean sprint.
At the dawn of the new year, the witchers reconvened. "The Beast breathes a currency conversion API" - said The Wild One - "And it's claws and fangs strike mostly at two independent JIRA clusters, sometimes upserting issues. It uses a company-deprecated API to send emails. We're in deep shit."
"I've found The Source of Fucking Excel Sheets" - said the smooth witcher - "It is The Temple of Cash-Flow, where the priests weave the Tapestry of Transactions. Our Fucking Excel Sheets are but a snapshot of the latest updates on the balance of some billing accounts. I spoke with one of the priestesses, and she told me that The Oracle (DB) would be able to provide us with The Data directly, if we were to learn the way of the ODBC and the Query"
"We stroke at the beast" - said the bold and adventurous witchers, now deserving of the bragging rights to be called The Butchers of Jarfile - "It is actually fewer than twenty classes and modules. Most are API-drivers. And less than 40% of the code is ever even fucking used! We found fucking JIRA API tokens and URIs hard-coded. And it is all synchronous and monolithic - no wonder it takes almost 20 hours to run a single fucking excel sheet".
Together, the witchers figured out that each new billing account were morphed by The Beast into a new JIRA issue, if none was open yet for it. Transactions were used to update the outstanding balance on the issues regarding the billing accounts. The currency conversion API was used too often, and it's purpose was only to give a rough estimate of the total balance in each Jira issue in USD, since each issue could have transactions in several currencies. The Beast would consume the Excel sheet, do some cryptic transformations on it, and for each resulting line access the currency API and upsert a JIRA issue. The secrets of those transformations were still hidden from the witchers. When and why would The Beast send emails, was still a mistery.
As the Witchers Council approached an end and all were armed with knowledge and information, they decided on the next steps.
The Wild Witcher, known in every tavern in the land and by the sea, would create a connector to The Red Port of Redis, where every currency conversion is already updated by other processes and can be quickly retrieved inside the VPC. The Greenhorn Witcher is to follow him and build an offline process to update balances in JIRA issues.
The Butchers of Jarfile were to build The Juggler, an automation that should be able to receive a parquet file with an insertion plan and asynchronously update the JIRA API with scores of concurrent requests.
The Smooth Witcher, proud of his new lead, was to build The Oracle Watch, an order that would guard the Oracle (DB) at the Temple of Cash-Flow and report every qualifying transaction to parquet files in AWS S3. The Data would then be pushed to cross The Event Bridge into The Cluster of Sparks and Storms.
This Witcher Who Writes is to ride the Elephant of Hadoop into The Cluster of Sparks an Storms, to weave the signs of Map and Reduce and with speed and precision transform The Data into The Insertion Plan.
However, how exactly is The Data to be transformed is not yet known.
Will the Witchers be able to build The Data's New Path? Will they figure out the mysterious transformation? Will they discover the Undocumented Java Tool's secrets on notifying customers and aggregating data?
This story is still afoot. Only the future will tell, and I will keep you posted.6 -
Map/filter/reduce in a tweet:
map([🌽, 🐮, 🐔], cook)
=> [🍿, 🍔, 🍳]
filter([🍿, 🍔, 🍳], isVegetarian)
=> [🍿, 🍳]
reduce([🍿, 🍳], eat)
=> 💩 -
Rant/Love.
I am sitting in this Big data class next to a girl.(I don't really talk to her so much but things are about to change ) A guy is asking her to share her code base for some insights/ideas on homework and she advices
Don't look at the code base else you'll get same ideas I got and would never come with an original solution. Then she further goes on advising on solving map reduce problems and giving me some tips to be careful about.
I turned amazed. It was like deja vu. I said the same thing to my friend some time last week.
My eyes glittered and suddenly I am like
Where were you all these days ???
Nothing is more attracting then a girl talking about code.
Am I the only one ??6 -
How do I become good with functional features(map,reduce,filter,zip,flatmap) in Javascript and Python?
It feels so alien. I'm so used to writing plain old loops.
Reading and undersranding this kind of style in other people's code is really hard for me, especially if all this is happeining on the same line.4 -
Perhaps as a tip for the junior devs out there, here's what I learned about programming skills on the job:
You know those heavy classes back in college that taught you all about Data Structures? Some devs may argue that you just need to know how to code and you don't need to know fancy Data Structures or Big o notation theory, but in the real world we use them all the time, especially for important projects.
All those principles about Sets, (Linked) lists, map, filter, reduce, union, intersection, symmetric difference, Big O Notation... They matter and are used to solve problems. I used to think I could just coast by without being versed in them.. Soon, mathematics and Big o notation came back to bite me.
Three example projects I worked in where this mattered:
- Massive data collection and processing in legacy Java (clients want their data fast, so better think about the performance implications of CRUD into Collections)
- ReactJS (oh yes, maps and filters are used a lot...)
- Massive data collection in C# where data manipulation results are crucial (union, intersection, symmetric difference,...)
Overall: speed and quality mattered (better know your Big o notation or use a cheat sheet, though I prefer the first)
Yes, the approach can be optimized here, but often we're tied to client constraints, with some room if we're lucky.
I'm glad I learned this lesson. I would rather have skills in my head and in memory than having to look up things and try to understand them all the time.5 -
Forgetting every day tasks...
So I can use map, reduce, filter in my sleep, have memorised huge chunks of valuable programming information.
Today I went to the gym and laced up my jogging pants, looked at the vogue knot I'm supposed to tie to prevent them from falling down and my brain just said:
"Fucked if I know how to tie that!" -
What's your opinion on functional shortcuts and 'hacks' in many languages, like map/reduce/filter, ternary operator, lambdas,inner/anonymous classes?
Imo they can make development faster and more efficent but they make the code very unreadable, especially if someone else has to read it, Therefore I try to use them only when it's appropriate. My dev friends use them too much and it makes reading their code a hellish experience, especially in Javascript with Rx.3 -
Having a hard time thinking the alternates to if statements is a good idea. I was genuinely curious how this was done. The examples I am finding seem to just spread the logic everywhere across multiple objects. To me this makes the logic objectively less clear. I didn't understand the obsession with objects until I saw the examples that creates a fuckton of boiler plate objects. How someone can say this is preferred over a few if statements boggles my mind. I actually am trying to understand the functional mindset as well. It is not going well for me. I can sorta see some value in using a map. Technically a lookup could be faster. But again it spreads the code all around adding more boilerplate.
https://blog.bitsrc.io/reduce-if-el...
https://dev.to/phouchens/...
Is it because these are contrived examples? I initially searched to find ways of reducing ifs in a functional approach. I did find it in the second example. I was however hoping to find that by lazy eval or something. I see people making references to how one you "get it" functional logic is easier to understand and evaluate. I cannot tell if this is straight up gaslighting or my brain is just too fucking imperative.11 -
When ur pm just attended session on map-reduce and starts using it evrywhr.
Development manager: this will take a month to complete this module.
Project manager: hmm! We need to get this done by 2morrow and I hv a plan. -
The forEach in JavaScript makes no sense. It looks like a map /filter/reduce but doesn’t actually return the array. Can you please loop like a normal language?8
-
Thinking to reduce google foot print by replacing google map with here map... Any suggestion before I do?10
-
Do you think, that its a good idea, to add FP-features like Map,Filter,Reduce to Stack or Queue datastructures, in the way, that they pop all elements?6
-
hello folks, any help would appreciated :). serious question about designing/developing a rest backend.
here is a little insight: I want to reduce the endpoints for many CRUD operations as I can. So for that approach I defined a set of "dynamic" routes like /:moduleName/list, /:moduleName/update and so on.
Now I want to also reduce hardcoding as much stuff as I can for the front end. like I want forms/view/components to know which fields can be sent in the "/:moduleName/xxx" endpoints from above. So I'm thinking to make some /:moduleName/list/map, /:moduleName/update/map endpoints that tells the frontend which fields/keys can be sent for X or Y operation.
regarding design/security concerns Is that a good approach? do you know any other approach that's like to what I want to achieve?6 -
Made a proof of concept combination of React + Highland.js + Recompose: https://codepen.io/hedgepig/pen/...
It's scrappy now, but the idea is a streaming alternative to redux/mobx whatever. This nice thing is one can treat events as a function over time, meaning one can map, pipe, reduce (scan), zip etc.
Going to try it on a side project (potentially Hive Sim: https://devrant.io/collabs/975778) and see how it goes. -
Just something I've been thinking on for a while:
How could programming be done if we couldn't use ordinary if-statements (but functional set operations such as map, filter, with an if- in the lambda function etc. is alright).
Could it work? Also would it be possible to reduce the amount of while loops by using functions for most of the "loop situations" as well?4 -
So on my new position I get to work on Spark jobs. Never had to work with the infamous big data technologies. I never thought this would get SO frustrating for all the wrong reasons.
I'm currently trying to introduce integration tests for some Spark job I wrote. This isn't trivial though, as the data comes from several HBase tables. Mocking everything simply isn't feasible. So why not use the integrated HBaseTestingUtility? With it you can start a mini cluster that runs all nessecary services in the scope of your test.
Sounds great, eh? WRONG. Firstly the used mapr dependencies get in the way. The baked in configuration tries to automatically authenticate with your local cluster through Kerberos. Of course this doesn't work. And of course there is no way to reconfigure this as it happens IN A FUCKING STATIC BLOCK. AHHHH.
Ok. So after calming down I "simply" had to exclude all mapr dependencies and replace them with vanilla ones. After two days of dependency hell it FINALLY works!
...or does it? Well now we need test data. For that we got a map reduce algorithm that can import dumps. Sounds again, great, eh? WROOOONNNG.
The fucking map reduce mini cluster can't start, as it tries to write a symlink. Now take a wild guess what the sys admin here blocked. Yepp. TWO DAYS OF WORK RENDERED USELESS, BECAUSE OF SOME FUCKING SECURITY SETTING.
This is fine.