Interviewing a Data Scientist

Wow Amazon how could I say no to a personalized deal like this.......😒

"Thank you for choosing Microsoft!"

No Microsoft, I really didn't choose you. This crappy hardware made you the inevitable, not a choice.

And like hell do I want to run your crappy shit OS. I tried to reset my PC, got all my programs removed (because that's obviously where the errors are, not the OS, right? Certified motherfuckers). Yet the shit still didn't get resolved even after a reset. Installing Windows freshly again, because "I chose this".

Give me a break, Microshaft. If it wasn't for your crappy OS, I would've gone to sleep hours ago. Yet me disabling your shitty telemetry brought this shit upon me, by disabling me to get Insider updates just because I added a registry key and disabled a service. Just how much are you going to force data collection out of your "nothing to hide, nothing to fear" users, Microsoft?

Honestly, at this point I think that Microsoft under Ballmer might've been better. Because while Linux was apparently cancer back then, at least this shitty data collection for "a free OS" wasn't yet a thing back then.

My mother still runs Vista, an OS that has since a few months ago reached EOL. Last time she visited me I recommended her to switch to Windows 7, because it looks the same but is better in terms of performance and is still supported. She refused, because it might damage her configurations. Granted, that's probably full of malware but at this point I'm glad she did.

Even Windows 7 has telemetry forcibly enabled at this point. Vista may be unsupported, but at least it didn't fall victim to the current status quo - data mining on every Microshaft OS that's still supported.

Microsoft may have been shady ever since they pursued manufacturers into defaulting to their OS, and GPU manufacturers will probably also have been lobbied into supporting Windows exclusively. But this data mining shit? Not even the Ballmer era was as horrible as this. My mother may not realize it, but she unknowingly avoided it.

Oh, so it wasn't that complicated after all !

"we gave you (the students) a choice if you want to learn data mining in matlab, java, python or c. most of you chose python. the results were horrible, so we decided to teach you in matlab instead"
- professor

ARE U FCKING KIDDING ME

I wrote pagerank algorithm in python for data mining course but my teacher told me to write it in R because according to him python can't be used as data mining tool.

Today I uninstalled WhatsApp, I mean I haven't had a look at it in over a month and nobody texts me anways.

Writing this makes me realize how sad this is

I don't want to support Facebook or any data mining company, instead I want to support (free) / open source software like Signal

I've found and fixed any kind of "bad bug" I can think of over my career from allowing negative financial transfers to weird platform specific behaviour, here are a few of the more interesting ones that come to mind...

#1 - Most expensive lesson learned

Almost 10 years ago (while learning to code) I wrote a loyalty card system that ended up going national. Fast forward 2 years and by some miracle the system still worked and had services running on 500+ POS servers in large retail stores uploading thousands of transactions each second - due to this increased traffic to stay ahead of any trouble we decided to add a loadbalancer to our backend.

This was simply a matter of re-assigning the IP and would cause 10-15 minutes of downtime (for the first time ever), we made the switch and everything seemed perfect. Too perfect...

After 10 minutes every phone in the office started going beserk - calls where coming in about store servers irreparably crashing all over the country taking all the tills offline and forcing them to close doors midday. It was bad and we couldn't conceive how it could possibly be us or our software to blame.

Turns out we made the local service write any web service errors to a log file upon failure for debugging purposes before retrying - a perfectly sensible thing to do if I hadn't forgotten to check the size of or clear the log file. In about 15 minutes of downtime each stores error log proceeded to grow and consume every available byte of HD space before crashing windows.

#2 - Hardest to find

This was a true "Nessie" bug.. We had a single codebase powering a few hundred sites. Every now and then at some point the web server would spontaneously die and vommit a bunch of sql statements and sensitive data back to the user causing huge concern but I could never remotely replicate the behaviour - until 4 years later it happened to one of our support staff and I could pull out their network & session info.

Turns out years back when the server was first setup each domain was added as an individual "Site" on IIS but shared the same root directory and hence the same session path. It would have remained unnoticed if we had not grown but as our traffic increased ever so often 2 users of different sites would end up sharing a session id causing the server to promptly implode on itself.

#3 - Most elegant fix

Same bastard IIS server as #2. Codebase was the most unsecure unstable travesty I've ever worked with - sql injection vuns in EVERY URL, sql statements stored in COOKIES... this thing was irreparably fucked up but had to stay online until it could be replaced. Basically every other day it got hit by bots ended up sending bluepill spam or mining shitcoin and I would simply delete the instance and recreate it in a semi un-compromised state which was an acceptable solution for the business for uptime... until we we're DDOS'ed for 5 days straight.

My hands were tied and there was no way to mitigate it except for stopping individual sites as they came under attack and starting them after it subsided... (for some reason they seemed to be targeting by domain instead of ip). After 3 days of doing this manually I was given the go ahead to use any resources necessary to make it stop and especially since it was IIS6 I had no fucking clue where to start.

So I stuck to what I knew and deployed a $5 vm running an Nginx reverse proxy with heavy caching and rate limiting linked to a custom fail2ban plugin in in front of the insecure server. The attacks died instantly, the server sped up 10x and was never compromised by bots again (presumably since they got back a linux user agent). To this day I marvel at this miracle $5 fix.

Fucking windows! I am so fucking done with this microsoft bullshit!

Hear me out here, i am a gamer. I need windows because it has the games (and software to aid those games) unlike any other platform. But windows 10 is basically already phishing andmalware at this point. I stuck to win 7 because it had a start menu and didn't totally drive me up the wall.

Just a short list of their bullshits: ads in the explorer window, ads in your taskbar reminders, data mining like it is nobodies business and trying to hide it, sharing my wifi access with friends (wtf), the fucking retarded new start menu, the crappy fullscreen apps which have less functionality than the actual proper desktop applications that you need to config what you want, and even then pushing multiple updates that simply broke peoples pc's. Fuck that, ill stick to 7.

They are making win10 worse by the week making it unlikely i will ever join that hell, and they are also aiming to force me there. Making windows store exclusives and dx12 only games. What am i supposed to do against that?! The current releases don't bother me much but fuck i figure it is a matter of time until the newest katamari game is their exclusive and i nanananana katamari damacy all over their platform.

And well all alternative os's are just out of the question unless vulkan rendering gets the upper hand. Then i'd switch to whatever stable distro and learn about our new penguin based overlords languages.

For now i will just stick to win7, suck on my thumb while in fetal position and hope it just all goes away.

When you search rbf on Google for radial basis function(for data mining) and it answers with 'R'esting 'B'itch 'F'ace!

Several rants ago I promised to drop a bombshell about Android. What took me so long was my research.

I wanted to measure the extent of Google’s background data mining. I put Android at a significant disadvantage — it was Redmi 6, a device with a 5-year-old half-dead battery that was heavily used by my partner. The only change was me installing Lineage OS + microG — a private, degoogled combo that has no quality of life ramifications. Google Play Store opens, apps download. MicroG emulates Google Play Services — maps, banking and other Play Services-dependent apps work flawlessly. This made a huge difference.

Before degoogling, this phone lasted one day tops on standby. Now, with Wi-Fi connection enabled, apps auto-update working (one game I had installed auto-updated during the test), and no battery saver engaged, I was able to pull ELEVEN DAYS on full charge. Battery saver promised even more uptime, but I considered that cheating.

Modern phones have modern screens that drain battery quickly. Yet, they also have 4000+ mAh batteries. If your Android smartphone performs worse than mine in a test like this that doesn't use screen, kiss your privacy goodbye.

I fucking hate the Safari browser to death.
This piece of disk space waste is 50% of the reason why I have to spend hours to find out what the fuck is incorrectly displaying or not working on W.I.P. websites.
The other 50% is Edge, IE <=11 and Firefucks.

Just for piss smelling Safari, I need to either run a VM with Crapple OS X for debugging or borrow a Crapbook from $randomPerson.
Is it fucking not possible to compile Safari blowser to run on Linux or Winblows?

Eventhough I'm disgusted about Google and its privacy and data mining policies, Chrome is the most decent browser there is on the whole digital world. It only happens extremely rarely that something is not working/displaying as expected during development.

Most browser developers seem to be useless pubes eaters and like to shower with curd soap.

Dogecoin hit USD $0.40 recently, which means it's time for the Crypto Rant.

TL;DR: Dogecoin is shit and is logically guaranteed to eventually fall unless it is fundamentally changed.

===========================

If you know how Crypto works under the hood, you can skip to the next section. If you don't, here's the general xyz-coin formula:

Money is sent via transactions, which are validated by *anybody*.

Since transactions are validated by anybody, the system needs to make sure you're not fucking it up on purpose.

The current idea (that most coins use today) is called proof-of-work. In short, you're given an extremely difficult task, and the general idea is you wouldn't be willing to do that work if you were just going to fuck up the system.

For validating these transactions, you are rewarded twofold:

1) You are given a fixed-size prize of the currency from the system itself. This is how new currency is introduced, or "minted" if you prefer.
2) You are given variable-size and user-determined prize called "transaction fees", but it could be more accurately called a "bribe" since it's sole purpose is to entice miners to add YOUR transaction to their block.

This system of validation and reward is called mining.

===========================

This smaller section compares the design o f BTC to Dogecoin - which will lead to my final argument

In BTC, the time between blocks (chunks of data which record transactions and are added to the chain, hence blockchain) is ten minutes. Every ten minutes, BTC transactions are validated and new Bitcoins are born.

In Dogecoin, the time between blocks is only one minute. In Theory, this means that mining Dogecoin is about ten times easier, because the system expects you to be able to solve the proof of work in an average of one minute.

The huge difference between BTC and Doge is the block reward (Fixed amount; new coins minted). The block reward for BTC is somewhat complicated compared to Doge: It started as 50 BTC per block and every 4 years it is halved ("the great halving"). Right now it's 6.25 BTC per block. Soon, the block reward will be almost nothing until BTC hits it's max of 21 million bitcoins "minted".

Dogecoin reward is 10,000 coins per block. And it will be that way for the end of time - no maximum, no great halving. And remember, for every 1 BTC block mined, 10 Doge blocks are mined.

===========================

Bitcoin and Dogecoin are now the two most popular coins in pop culture. What makes me angry is the widespread misunderstanding of the differences between the two. It is likely that most investors buy Dogecoin thinking they're getting in "early" because it's so cheap. They think it's cheap because it isn't as popular as Bitcoin yet. They're wrong. It's cheap because of what's outlined in section two of this rant.

Dogecoin is actually not very far off Bitcoin. Do the math: there's a bit over 100 billion Dogecoin in circulation (130b). There's about 20 million BTC. Calculate their total CURRENT values:

130b * $0.40 = 52b
20m * $60k = 1.2t

...and Doge is rising much, much faster than BTC because of the aforementioned lack of understanding.

The most common thing I hear about Doge is that "nobody expects it to reach Bitcoin levels" (referring to being worth 60k a fucking coin). They don't realize that if Doge gets to be worth just $10 a coin, it will not just reach Bitcoin levels but overtake Bitcoin in value ($1.3T).

===========================

It's worth highlighting that Dogecoin is literally designed to fail. Since it lacks a cap on new coins being introduced, it's just simple math that no matter how much Doge rises, it will eventually be worthless. And it won't take centuries, remember that 100k new Doge are mined EVERY TEN MINUTES. 1,440 minutes in a day * 10K per minute is 14.4 million new coins per day. That's damn near every Bitcoin to ever exist mined every day in Dogecoin

WanBLowS, you data hungry piece of shit! WHAT IN THE NAME OF FUCK MAKES YOU THINK THAT I WANT TO SIGN IN?!! Motherfuckers!!!

Not sure why but every time I open that featureful piece of Zuccshit called Facebook, it never fails to make my blood boil to even higher levels than it did last time... Fucking curse of the internet, data mining, Zuccy piece of featureful trash!

Actually I'm pleasantly surprised about Windows' stability nowadays. It's capable for running for up to a week with no stability issues, whereas systemd on the other hand.. let's just say that my Arch containers could do better right now.

Data mining aside, damn man.. Microsoft is improving for once! Is this the so-many'th unusable/somewhat stable switch? I mean, it's not like we haven't seen that happen yet! Windows 98, shit! Windows 2000, kinda alright! Windows Me, shit! Windows XP, kinda alright! Windows Vista, oh don't even get me started on that pile of garbage! Windows 7, again kinda okay! Windows 8, WHERE THE FUCK DID THAT START MENU GO YOU MOTHERFUCKERS?!!! Windows 10, well at least that Start menu got fixed. Then it got into some severe QA issues, which now seem to have gotten somewhat fixed again.

I'm starting to see a pattern here! 🤔

When you forget to normalize or standardize your data.

Iran:
Job offer ==>
We need full stack developer for Java, c#, python, angular, flutter, ..
Data mining experience
Project and team management experience for 7+ years

Salary is 500$ per month

My god the wall looks really punchable right now. Let me tell you why.

So I’m working on a data mining project, and I’m trying to get data from google trends. Unfortunately, there have been a lot of roadblocks for what should have been an easy task.

First it won’t give a raw search volume, only relative “interest”.
Fortunately it lets me compare search terms, which would work for my needs however it will only let me compare a few at a time. I need to compare 300.

So my solution is simple: compare all the terms relative to one term. Simple enough, but it would be time consuming so I figured I’d write a program to get the data.

But then I learned that they don’t have an official api. There’s a node module for this very thing based on a python module that reverse engineers the api endpoints. I thought as long as it works I’d use it.

It does work... But then I discovered that google heavily rate limits the endpoints.

So... I figured I’d build a system to route the requests through different tor nodes to get around the rate limit. Good solution right? Well like a slap to the face, after spending way to much time getting requests through tor working, I discovered that THEY FUCKING BLOCKED TOR IPS.

So I gave up, and resigned to wait 5 hours for my program to get the data... 1 comparison at a time... 60s interval between requests. They, of course, don’t tell you the rate limit threshold, so this is more or less a guess (I verified that 30s interval was too short and another person using the module suggested 60s).

Remember when I said the discovery that the blocked tor came like a slap to the face? This came as a sledge hammer to the face: for some reason my program didn’t dump the data at the end. I waited 5 fucking hours to get nothing.

I am so mad right now. I am so fucking mad.

I write a thesis about some data mining project.
I need to process 15 million tracking points - today I finally finished coding my algorithm. I was pretty excited and pressed run:
Now, after 2,5 hours of processing, my program got already through a third of all tracking points, as I realized:
I gave my algorithm a non-valid output path.
And that thouht stuck me, as I was already on my way home.
Now I have to go tomorrow (on my day off) to work, to fix and run it again.

What you are expected to learn in 3 years:

power electronics,
analogue signal,
digital signal processing,
VDHL development,
VLSI debelopment,
antenna design,
optical communication,
networking,
digital storage,
electromagnetic,
ARM ISA,
x86 ISA,
signal and control system,
robotics,
computer vision,
NLP, data algorithm,
Java, C++, Python,
javascript frameworks,
ASP.NET web development,
cloud computing,
computer security ,
Information coding,
ethical hacking,
statistics,
machine learning,
data mining,
data analysis,
cloud computing,
Matlab,
Android app development,
IOS app development,
Computer architecture,
Computer network,
discrete structure,
3D game development,
operating system,
introduction to DevOps,
how-to -fix- computer,
system administration,
Project of being entrepreneur,

and 24 random unrelated subjects of your choices

This is a major called "computer engineering"

I was just writing a long rant about how my rant style changed, and how I could fix anything that annoys me in a heartbeat by just putting my mind to implementing a change. Then YouTube once again paused the synth mix that was playing on my laptop in the background, with that stupid "Video paused. Continue watching?" pop-up. I even installed an add-on for it in Firefox to make it automatically click that away. I guess that YouTube did yet another bullshit update to break that, for "totally legitimate user interface improvements" or whatever. Youtube-dl faces similar challenges all the time, and it's definitely not alone in that either. I also had issues with that on Facebook when I wanted to develop on top of that, where the UI changes every other day and the API even changes every other week. And as far as backwards compatibility goes, our way or the highway!

So I did the whole "replace and move on" type of thing. I use youtube-dl often now to get my content off YouTube into a media player that doesn't fuck me over for stupid reasons like "ad fraud" (I use an ad blocker you twats, what ads am I gonna fraud against), or "battery savings" (the damn laptop is plugged in and fully topped up for fucks sake, and you do this crap even on desktop computers). Gee I wonder why creators are moving on to Floatplane and Nebula nowadays, and why people like yours truly use "highly illegal" youtube-dl. Oh and thank you for putting me in Saudi Arabia again. Pinnacle of data mining, machine learning and other such wank could not do GeoIP. for a server that used to be in a datacenter in Italy for years, and recently has been moved to another hosting provider in Germany. It's about as unchanging and static, and as easy to geolocate as you can possibly get. But hey, kill off another Google+ when?

Like seriously, yes I'm taking your Foobar challenges and you may very well be the company I end up working for. But if anything it feels like there's a shitton of stuff to fix. And the challenges themselves still using Python 2.7 honestly feels like the seldom seen tip of the iceberg.

#10 year challenge is basically data set father for new ai which will predict how X looks after 10 years

Data mining at its best

When company assigned me, an internship student, as only developer for a project that needs data mining on connected hives, back-end alert management, front-end difficult animation management with Microsoft technologies when I was learning C#

OMG! the ancient one... amazon sells data mining book from year where not even computer existed.

FYI - if it's typo then not fixed for months

Anyone who has watched Black Mirror and thought what shit it is and technology can never go that far but wonder what if it does and we all are fucked.

Or we are already there but mini version of it. I see people glued on there phone, screen, and even I am guilty of that. But the amount of data we create everyday which can be treated as huge asset by the AI and Data mining enthusiast, I dont doubt we will be at a breaking point, which is coming sooner than we think.

I normally just have nightmares about the projects I'm working on, especially when I struggle with a bug for days. Those are usually about just me stressing out about it. However, I have a lot of dreams about computers/technology, not necessarily coding-related:

- datacenters were just potato fields. If you go work the field, you'd go data mining

- in Biology, when being taught how having children works, you only tell that "parenting is only chmod-ing the rights of your children until they become the owners themselves"

- IP addresses with emojis instead of numbers were a standard now and they actually managed to replace IPv4, because everyone was so into emojis. They named it IPvE

- I witnessed a new Big Bang when the 32-bit Unix time overflown in 2038, and we were all quantum bits

Wow! Just updated my devRant android and it has search! Opens the floodgates to data mining the rapidly growing content. Thanks!!!

Ok so I have done some work with crypto currency mining pools and recently a client requested for me to make a splash page that showed data from multiple instances of these pools APIs. I went to find some documentation for this open source api and to my surprise there is none. I thought of querying the public API from the clients side and it worked, however it's so slow that the data shows up roughly 20 seconds after the page loads.

Easy fix right? Make a PHP server get the data every 5 seconds, cache it and serve the data with the page and use a websocket for live updates! Until I found out that there is no practical way in this garbage framework to get the damn API data without making an HTTP request or mutilating the original source code. I'm so done with this garbage framework. It literally loads pages based on a page and action parameter on the index.php. I quit.

The worst thing about university is having to listen to lecturer say DATA MINING IS GOOD at every single lecture there is.

Learning Image Processing,Deep Learning,Machine Learning,Data modeling,mining and etc related to and also work on them are so much easier than installing requiremnts, packages and tools related to them!

Did the simplest corridor gen I could think of. The tile that makes up corridors is different than the tile that makes up the rooms. They are drawn the same here though. In the floor data structure they are different types. This will allow me to easily place doors the like. The dots are potential door placements.

Now that I have simple room gen working I can work on filling it with 3d models to make up walls, doors, etc.

Most of the time the rooms connect on the whole map. But once in a while they do not. I like this as I will incorporate mining. The final map will be much bigger. This is 32x32 and I want 256x256. I will need to figure out how to determine room density versus grid size.

I need to spend some time cleaning up the code and try and generalize the code. I will need to allow for pregenned rooms as well with defined entry points. The entry points on these rooms is all random. It will probably be tricky to do random room to pregenned room corridors. Proximity seems to work. So prox to a predefined door location should work.

When a client wants java, c, PHP, Ruby, Javascript, html, css, data mining, AI, bot development, Android development, iPhone development, Amazon services, masonry, Kung Fu, cooking, driving, flying, deployment, mechanic, electrician, plumbing as required skills.

I got no sentiment for sentiment analysis today 😒

Was reading the voice command rant and got curious, what do you guys think about the brain-machine-interface? Will it ever happen?

I would love it but also have some fears. Being able to control devices just by thought would be a huge time saver. But data mining is getting out of hand and that is scaring me. Companies would be able to analyze a lot.
Now they 'just' know what I am buying and which sites I am visiting.
If they know what I am thinking, Amazon would suggest me rubber gloves, body bags and whisky.

So today in school I decided on my career choice. So I've decided on becoming a data scientist because I can still use my programming and I very much do like looking at data and researching things as well as program obviously :) so if there are any data scientists or data miners out there do you have advice?

Fucking Hetzner, I am definitely not sending you copy of my ID after I read the privacy policy. Data-hungry cunts. I just hope you send back that 20 euros, you requested on account creation.

So I sign up for this fancy pants website were developers rant about shit. They ask me bunch of data mining personal questions upfront like a marketing bitch so I go through and fill out my profile. Ok it's a social site, whatever and nobody gives a fuck about me anyway. I hit save, continue and go to click the email that injects my lies into their database. But I figure I might go back and fix some of the lies I wrote about myself just in case Google craws it and somebody I meet in the future calls me liar. So I spend 10 minutes trying to navigate the whole website to find a way to get back there to edit the lies or even perhaps find some site help relating to profiles. Of course the profile page itself does not do this profile edit stuff and bugger me if there is no help at all on how to use this website. So I did the only thing I could and wrote this rant in the hope somebody knows how the fuck I cover my tracks on devrant.com

Hey guys, how would you rate iOS vs Android for privacy? I read recently that Android pulls up to 10x more data off of your phone (for data mining) than iOS and that overall, iOS is really the mobile OS of choice when privacy and control of your data is a priority.

Anyone have any hard evidence to support this?

Start learning Data Mining

Since i'm clearly not swamped with things to do already, school, work, family.. I thought i should pick up some sideprojects. Where to start?

I was wondering if it is allowed to crawl all posted rants on devrant to do some fancy data mining stuff while learning python. Any clue?

Just playing with Google Allo's Assistant when suddenly..

I just disabled 2 recommendation notifications from Google Map and Photos...

My first thought is Google is turning into Facebook. My next read Google knows too much... But what can I do...

It's also very convenient... when it's not data mining the data it stores...

I want to study programming in Germany, some good web dev. But all the unis(for masters) I find online gimme data mining, machine learning, automation etc. I have only done 3 years of college, the counsellors here say you need 4. Anyone know a uni in Germany with my requirements?