Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "derived data"
-
Woohoo!!! I made it to 1000++s :) Now I feel less newbie-like around here :)
So... I don't want to shit-post, so in gratitude to all you guys for this awesome community you've built, specially @trogus and @dfox, I'll post here a list of my ideas/projects for the future, so you guys can have something to talk about or at least laugh at.
Here we go!
Current Project: Ensayador.
It's a webapp that intends to ease and help students write essays. I'm making it with history students in mind, but it should also help in other discipline's essay production. It will store the thesis, arguments, keywords and bibliography so students can create a guideline before the moment of writting. It will also let students catalogue their reads with the same fields they'd use for an essay: that is thesis, arguments, keywords and bibliography, for their further use in other essays. The bibliography field will consist on foreign keys to reads catalogued. The idea is to build upon the models natural/logical relations.
Apps: All the apps that will come next could be integrated in just one big app that I would call "ChatPo" ("Po" is a contextual word we use in my country when we end sentences, I think it derived from "Pues"). But I guess it's better to think about them as different apps, just so I don't find myself lost in a neverending side-project.
A subchat(similar to a subreddit)-based chat app:
An app where people can join/create sub-chats where they can talk about things they are interested in. In my country, this is normally done by facebook groups making a whatsapp group and posting the link in the group, but I think that an integrated app would let people find/create/join groups more easily. I'm not sure if this should work with nicknames or real names and phone numbers, but let's save that for the future.
A slack clone:
Yes, you read it right. I want to make a slack clone. You see, in my country, enterprise communications are shitty as hell: everything consists in emails and informal whatsapp groups. Slack solves all these problems, but nobody even knows what it is over here. I think a more localized solution would be perfect to fill this void, and it would be cool to make it myself (with a team of friends of course), and hopefully profit out of it.
A labour chat-app marketplace:
This is a big hybrid I'd like to make based on the premise of contracting services on a reliable manner and paying through the app. "Are you in need of a plumber, but don't know where to find a reliable one? Maybe you want a new look on your wall, but don't want to paint it yourself? Don't worry, we got you covered. In <Insert app name> you can find a professional perfect to suit your needs. Payment? It's just a tap away!". I guess you get the idea. I think wechat made something like this, I wonder how it worked out.
* Why so many chat apps? Well... I want to learn Erlang, it is something close to mythical to me, and it's perfect for the backend of a comms app. So I want to learn it and put it in practice in any of these ideas.*
Videogames:
Flat-land arena: A top down arena game based on the book "flat land". Different symmetrical shapes will fight on a 2d plane of existence, having different rotating and moving speeds, and attack mechanics. For example, the triangle could have a "lance" on the front, making it agressive but leaving the rest defenseless. The field of view will be small, but there'll be a 2d POV all around the screen, which will consist on a line that fills with the colors of surrounding objects, scaling from dark colors to lighter colors to give a sense of distance.
This read could help understand the concept better:
http://eldritchpress.org/eaa/...
A 2D darksouls-like class based adventure: I've thought very little about this, but it's a project I'm considering to build with my brothers. I hope we can make it.
Imposible/distant future projects:
History-reading AI: History is best teached when you start from a linguistic approach. That is, you first teach both the disciplinar vocabulary and the propper keywords, and from that you build on causality's logic. It would be cool to make an AI recognize keywords and disciplinary vocabulary to make sense of historical texts and maybe reformat them into another text/platform/database. (this is very close to the next idea)
Extensive Historical DB: A database containing the most historical phenomena posible, which is crazy, I know. It would be a neverending iterative software in which, through historical documents, it would store historical process, events, dates, figures, etc. All this would then be presented in a webapp in which you could query historical data and it would return it in a wikipedia like manner, but much more concize and prioritized, with links to documents about the data requested. This could be automated to an extent by History-reading AI.
I'm out of characters, but this was fun. Plus, I don't want this to be any more cringy than it already is.12 -
dev, ~boring
This is either a shower thought or a sober weed thought, not really sure which, but I've given some serious consideration to "team composition" and "working condition" as a facet of employment, particularly in regard to how they translate into hiring decisions and team composition.
I've put together a number of teams over the years, and in almost every case I've had to abide by an assemblage of pre-defined contexts that dictated the terms of the team working arrangement:
1. a team structure dictated to me
2. a working temporality scheme dictated to me
3. a geographic region in which I was allowed to hire
4. a headcount, position tuple I was required to abide by
I've come to regard these structures as weaknesses. It's a bit like the project management triangle in which you choose 1-2 from a list of inadequate options. Sometimes this is grounded in business reality, but more often than not it's because the people surrounding the decisions thrive on risk mitigation frameworks that become trickle down failure as they impose themselves on all aspects of the business regardless of compatibility.
At the moment, I'm in another startup that I have significantly more control over and again have found my partners discussing the imposition of structure and framework around how, where, why, who and what work people do before contact with any action. My mind is screaming at me to pull the cord, as much as I hate the expression. This stems from a single thought:
"Hierarchy and structure should arise from an understanding of a problem domain"
As engineers we develop processes based on logic; it's our job, it's what we do. Logic operates on data derived from from experiments, so in the absence of the real we perform thought experiments that attempt to reveal some fundamental fact we can use to make a determination.
In this instance we can ask ourselves the question, "what works?" The question can have a number contexts: people, effort required, time, pay, need, skills, regulation, schedule. These things in isolation all have a relative importance ( a weight ), and they can relatively expose limits of mutual exclusivity (pay > budget, skills < need, schedule < (people * time/effort)). The pre-imposed frameworks in that light are just generic attempts to abstract away those concerns based on pre-existing knowledge. There's a chance they're fine, and just generally misunderstood or misapplied; there's also a chance they're insufficient in the face of change.
Fictional entities like the "A Team," comprise a group of humans whose skills are mutually compatible, and achieve synergy by random chance. Since real life doesn't work on movie/comic book logic, it's easy to dismiss the seed of possibility there, that an organic structure can naturally evolve to function beyond its basic parts due to a natural compatibility that wasn't necessarily statistically quantifiable (par-entropic).
I'm definitely not proposing that, nor do I subscribe to the 10x ninja founders are ideal theory. Moreso, this line of reasoning leads me to the thought that team composition can be grown organically based on an acceptance of a few observed truths about shipping products:
1. demand is constant
2. skills can either be bought or developed
3. the requirement for skills grows linearly
4. hierarchy limits the potential for flexibility
5. a team's technically proficiency over time should lead to a non-linear relationship relationship between headcount and growth
Given that, I can devise a heuristic, organic framework for growing a team:
- Don't impose reporting structure before it has value (you don't have to flatten a hierarchy that doesn't exist)
- crush silos before they arise
- Identify needed skills based on objectives
- base salary projections on need, not available capital
- Hire to fill skills gap, be open to training since you have to pay for it either way
- Timelines should always account for skills gap and training efforts
- Assume churn will happen based on team dynamics
- Where someone is doesn't matter so long as it's legal. Time zones are only a problem if you make them one.
- Understand that the needs of a team are relative to a given project, so cookie cutter team composition and project management won't work in software
- Accept that failure is always a risk
- operate with the assumption that teams that are skilled, empowered and motivated are more likely to succeed.
- Culture fit is a per team thing, if the team hates each other they won't work well no matter how much time and money you throw at it
Last thing isn't derived from the train of thought, just things I feel are true:
- Training and headcount is an investment that grows linearly over time, but can have exponential value. Retain people, not services.
- "you build it, you run it" will result in happier customers, faster pivoting. Don't adopt an application maintenance strategy
/rant2 -
This is the third part of my ongoing series "The Ballad of the Six Witchers and the Undocumented Java Tool".
In this part, we have the massive Battle of Sparks and Storms.
The first part is here: https://devrant.com/rants/5009817/...
The second part is here: https://devrant.com/rants/5054467/...
Over the last couple sprints and then some, The Witcher Who Writes and the Butchers of Jarfile had studied the decompiled guts of the Undocumented Java Beast and finally derived (most of) the process by which the data was transformed. They even built a model to replicate the results in small scale.
But when such process was presented to the Priests of Accounting at the Temple of Cash-Flow, chaos ensued.
This cannot be! - cried the priests - You must be wrong!
Wrong, the Witchers were not. In every single test case the Priests of Accounting threw at the Witchers, their model predicted perfectly what would be registered by the Undocumented Java Tool at the very end.
It was not the Witchers. The process was corrupted at its essence.
The Witchers reconvened at their fortress of Sprint. In the dark room of Standup, the leader of their order, wise beyond his years (and there were plenty of those), in a deep and solemn voice, there declared:
"Guys, we must not fuck this up." (actual quote)
For the leader of the witchers had just returned from a war council at the capitol of the province. There, heading a table boarding the Archpriest of Accounting, the Augur of Economics, the Marketing Spymaster and Admiral of the Fleet, was the Ciefoh Seat himself.
They had heard rumors about the Order of the Witchers' battles and operations. They wanted to know more.
It was quiet that night in the flat and cloudy plains of Cluster of Sparks and Storms. The Ciefoh Seat had ordered the thunder to stay silent, so that the forces of whole cluster would be available for the Witchers.
The cluster had solid ground for Hive and Parquet turf, and extended from the Connection River to farther than the horizon.
The Witcher Who Writes, seated high atop his war-elephant, looked at the massive battle formations behind.
The frontline were all war-elephants of Hadoop, their mahouts the Witchers themselves.
For the right flank, the Red Port of Redis had sent their best connectors - currency conversions would happen by the hundreds, instantly and always updated.
The left flank had the first and second army of Coroutine Jugglers, trained by the Witchers. Their swift catapults would be able to move data to and from the JIRA cities. No data point will be left behind.
At the center were thousands of Sparks mounting their RDD warhorses. Organized in formations designed by the Witchers and the Priestesses of Accounting, those armoured and strong units were native to this cloudy landscape. This was their home, and they were ready to defend it.
For the enemy could be seen in the horizon.
There were terabytes of data crossing the Stony Event Bridge. Hundreds of millions of datapoints, eager to flood the memory of every system and devour the processing time of every node on sight.
For the Ciefoh Seat, in his fury about the wrong calculations of the processes of the past, had ruled that the Witchers would not simply reshape the data from now on.
The Witchers were to process the entire historical ledger of transactions. And be done before the end of the month.
The metrics rumbled under the weight of terabytes of data crossing the Event Bridge. With fire in their eyes, the war-elephants in the frontline advanced.
Hundreds of data points would be impaled by their tusks and trampled by their feet, pressed into the parquet and hive grounds. But hundreds more would take their place. There were too many data points for the Hadoop war-elephants alone.
But the dawn will come.
When the night seemed darker, the Witchers heard a thunder, and the skies turned red. The Sparks were on the move.
Riding into the parquet and hive turf, impaling scores of data points with their long SIMD lances and chopping data off with their Scala swords, the Sparks burned through the enemy like fire.
The second line of the sparks would pick data off to be sent by the Coroutine Jugglers to JIRA. That would provoke even more data to cross the Event Bridge, but the third line of Sparks were ready for it - those data would be pierced by the rounds provided by the Red Port of Redis, and sent back to JIRA - for good.
They fought for six days and six nights, taking turns so that the battles would not stop. And then, silence. The day was won, all the data crushed into hive and parquet.
Short-lived was the relief. The Witchers knew that the enemy in combat is but a shadow of the troubles that approach. Politics and greed and grudge are all next in line. Are the Witchers heroes or marauders? The aftermath is to come, and I will keep you posted.4 -
The first fruits of almost five years of labor:
7.8% of semiprimes give the magnitude of their lowest prime factor via the following equation:
((p/(((((p/(10**(Mag(p)-1))).sqrt())-x) + x)*w))/10)
I've also learned, given exponents of some variables, to relate other variables to them on a curve to better sense make of the larger algebraic structure. This has mostly been stumbling in the dark but after a while it has become easier to translate these into methods that allow plugging in one known variable to derive an unknown in a series of products.
For example I have a series of variables d4a, d4u, d4z, d4omega, etc, and these are translateable now, through insights that become various methods, into other types of (non-d4) series. What these variables actually represent is less relevant, only that it is possible to translate between them.
I've been doing some initial learning about neural nets (implementation, rather than theoretics as I normally read about). I'm thinking what I might do is build a GPT style sequence generator, and train it on the 'unknowns' from semiprime products with known factors.
The whole point of the project is that a bunch of internal variables can easily be derived, (d4a, c/d4, u*v) from a product, its root, and its mantissa, that relate to *unknown* variables--unknown variables such as u, v, c, and d4, that if known directly give a constant time answer to the factors of the original product.
I think theres sufficient data at this point to train such a machine, I just don't think I'm up to it yet because I'm lacking in the calculus department.
2000+ variables that are derivable from a product, without knowing its factors, which are themselves products of unknown variables derived from the internal algebraic relations of a product--this ought to be enough of an attack surface to do something with.
I'm willing to collaborate with someone familiar with recurrent neural nets and get them up to speed through telegram/element/discord if they're willing to do the setup and training for a neural net of this sort, one that can tease out hidden relationships and map known variables to the unknown set for a given product.17 -
WASTED MY ENTIRE DAY AJD ENTIRE NIGHT.
MY flutter app isn’t running on iPhone simulator!!! I cleaned and deleted derived data and cleaned thousand times but it crashes at startup without error! What the fuckkkkk do I do!!????
Oh and my iPhone is not being recognised by any IDE Idk why. :((((((((( pls m so miserable pls dnt :(4 -
As I started learning React, I found the allure of declarative style of programming appealing. I try to avoid maintaining multiple state variables for data that can be derived from the base state itself that's stored in the redux store. It works wonders when I have to change something; as I just need to make changes to one function in the utils folder and that change is implemented across the whole app, rather than change the instances everywhere as was the case when I initially started working on this project after the previous dev left.
But I see myself redefining a lot of computed values everywhere, and if I just try to define them in the root component, I'll end up with a huge list of props being passed to a couple of components. Shifting it to the utils folder helps a bit, but then I find myself defining even the simplest of array filtering methods to the utils folder.
Is this need to define computed values everywhere a trade-off that you need to accept when you write declarative code, or is there a workaround/solution I am missing? As of now, the code-base is much better than how it used to be when they had a literal Java dev work on React with their knowledge of Java patterns being used in a framework that is the polar opposite of OOP, but I still feel like there's room for improvement in this duplication of computed values.2 -
So, I just (few hours ago)made a new variable that's either brilliant or innately flawed... not sure yet. It's an oddly unique var...
__bs__
So far I only made it in python and windows env (i script like the methodology of css).
I bet you're wondering how I've defined __bs__ and the practicality of it.
__bs__ is derived from a calculated level of bullshit that annoys me to tolerate, maintain, etc. as well as things that tend to throw nonsensical errors, py crap like changing my strings to ints at seemingly random times/events/cosmic alignments/etc or other things that have a history of pulling some bs, for known or unknown reasons.
How/why did this come about now?
Well I was updating some symlinks and scripts(ps1 and bat) cuz my hdd is so close to death I'm wondering if hdd ghosts exist as it's somehow still working (even ostream could tell it should be dead, by the sound alone).
A nonsense bug with powershell allowing itself to start/run custom ps1scripts with the originating command coming from a specific batch script, which worked fine before and nothing directly connected to it has changed.
I got annoyed so took an ironic break from it to work on python crap. Python has an innately high level of bs so i did need to add some extra calculations when defining if a py script or function is actually __bs__ or just py.
The current flavour of py bs was the datetime* module... making all of my scripts using datetime have matching import statements to avoid more bs.
I've kept a log of general bs per project/use case. It's more like a warning list... like when ive spent hours debugging something by it's traceback, meticulous... to eventually find out it had absolutely nothing to do with the exception listed. Also logged aliases i created, things that break or go boom if used in certain ways, packages that ive edited, etc.
The issue with my previous logging is that it's a log... id need to read it before doing anything, no matter how quick/simple it should be, or im bound to get annoyed with... bs.
So far i have it set to alert if __bs__ is above a certain int when i open something to edit. I can also check __bs__ fot what's causing the bs. I plan to turn it into a warning and recording system for how much bs i deal with and have historical data of personal performance vs bs tolerance. There's a few other applications i think ill want to use it for, assume it's not bs itself.
*in case you prefer sanity and haven't dealt with py and datetime enough, here's the jist:
If you were to search any major forum like StackOverflow for datetime use in py, youd find things like datetime.datetime.now() and datetime.now() both used, to get the same returned value. You'll also find tons of posts for help and trying to report 'bugs', way more than average. This is because the datetime package has a name conflict... with itself. It may have been a bug several years ago, but it beeb explicitly defined as intentional since.2 -
Persisterising derived values. Often a necessary evil for optimisation or privacy while conflicting with concerns such as auditing.
Password hashing is the common example of a case considered necessary to cover security concerns.
Also often a mistake to store derived values. Some times it can be annoying. Sometimes it can be data loss. Derived values often require careful maintenance otherwise the actual comments in your database for a page is 10 but the stored value for the page record is 9. This becomes very important when dealing with money where eventual consistency might not be enough.
Annoying is when given a and b then c = a + b only b and c are stored so you often have to run things backwards.
Given any processing pipeline such as A -> B -> C with A being original and C final then you technically only need C. This applies to anything.
However, not all steps stay or deflate. Sum of values is an example of deflate. Mapping values is an example of stay. Combining all possible value pairs is inflate, IE, N * N and tends to represent the true termination point for a pipeline as to what can be persisted.
I've quite often seen people exclude original. Some amount of lossy can be alright if it's genuine noise and one way if serving some purpose.
If A is O(N) and C reduces to O(1) then it can seem to make sense to store only C until someone also wants B -> D as well. Technically speaking A is all you ever need to persist to cater to all dependencies.
I've seen every kind of mess with processing chains. People persisting the inflations while still being lossy. Giant chains linear chains where instead items should rely on a common ancestor. Things being applied to only be unapplied. Yes ABCBDBEBCF etc then truncating A happens.
Extreme care needs to be taken with data and future proofing. Excess data you can remove. Missing code can be added. Data however once its gone its gone and your bug is forever.
This doesn't seem to enter the minds of many developers who don't reconcile their execution or processing graphs with entry points, exist points, edge direction, size, persistence, etc.2