The highest data transfer rate today - 256 gigabytes per second - was achieved when the cleaner's vacuum cleaner accidentally sucked the flash drive in from the floor.

I just visited a data center for the first time in my life.

I'm in love 😍

Solving crashes via *big data*

Interviewing a Data Scientist

BIG DATA reality..

Data Structures...

Where are the BIG data guys ?

Data Manipulation...

Source: Analytics Vidhya

"I am still cleaning the data"
- A data scientist's equivalent of 'It's compiling'

Swag of a Data Scientist 😎

I’ve been thinking lately, what is it that devRant devs do?
So after a couple of days of pulling data from devRants API's, filtering through the inconsistent skill set data of about 500 users (seriously guys, the comma is your friend) i’ve found an interesting set of languages being used by everyone.

I've limited this to just languages, as dwelling into frameworks, libraries and everything else just grows exponentially, also ive only included languages with at least 5 users out of the pool.
sorry you brainfuck guys.

Professor essentially doing a dance to show data visualisation:
“Hey pay attention! I’m trading my dignity for your education”

"My teacher always said that data redundancy is a really bad thing."

Why would a keyboard store 155 MB data?
Oh wait! It's Google's keyboard

Why data sanitization is important.

Was recently phone interviewed by a recruiter that asked "So do you know data structures and algorithms?" I replied "That's like asking someone if they know Mathematics - can you be a bit more specific?"

PM used his wife's data in testing against legacy data.
Discovered she had different social number before they met.
Apparently she was also a he.....

Might be a interesting evening for him.

Data structures.... The easy way....

When you have to figure out whether it's better to transfer 85GB of data via internet or to load it onto a harddrive and drive it 600 km/ 370 miles in your car.

!rant
Yesterday i updated my LinkedIn title to Data scientist.
Today morning i got 3 phone calls for job offers.

At the data restaurant:
Chef: Our freezer is broken and our pots and pans are rusty. We need to refactor our kitchen.
Manager: Bring me a detailed plan on why we need each equipment, what can we do with each, three price estimates for each item from different vendors, a business case for the technical activities required and an extremely detailed timeline. Oh, and do not stop doing your job while doing all this paperwork.
Chef: ...
Boss: ...

Some time later a customer gets to the restaurant.
Waiter: This VIP wants a burguer.
Boss: Go make the burger!
Chef: Our frying pan is rusty and we do not have most of the ingredients. I told you we need to refactor our kitchen. And that I cannot work while doing that mountain of paperwork you wanted!
Boss: Let's do it like this, fix the tech mumbo jumbo just enough to make this VIP's burguer. Then we can talk about the rest.

The chef then runs to the grocery store and back and prepares to make a health hazard hurried burguer with a rusty pan.
Waiter: We got six more clients waiting.
Boss: They are hungry! Stop whatever useless nonsense you were doing and cook their requests!
Cook: Stop cooking the order of the client who got here first?
Boss: The others are urgent!
Cook: This one had said so as well, but fine. What do they want?
Waiter: Two more burgers, a new kind of modern gaseous dessert, two whole chickens and an eleven seat sofa.
Chef: Why would they even ask for a sofa?!? We are a restaurant!
Boss: They don't care about your Linux techno bullshit! They just want their orders!
Cook: Their orders make no sense!
Boss: You know nothing about the client's needs!
Cook: ...
Boss: ...

That is how I feel every time I have to deal with a boss who can't tell a PostgreSQL database from a robots.txt file.
Or everytime someone assumes we have a pristine SQL table with every single column imaginable.
Or that a couple hundred terabytes of cold storage data must be scanned entirely in a fraction of a second on a shoestring budget.
Or that years of never stored historical data can be retrieved from the limbo.
Or when I'm told that refactoring has no ROI.
Fuck data stack cluelessness.
Fuck clients that lack of basic logical skills.

Devrant api for data warehouse school assignment, yay or nay?

Did you know?
This rant is a part of the 3 quintillion bytes of data that the world generated today.

I've dreamed of learning business intelligence and handling big data.

So I went to an university info event today for "MAS Data Science".

Everything's sounded great. Finding insights out of complex datasets, check! Great possibilities and salery.

Yay! 😀

Only after an hour they've explained that the main focus of this course is on leading a library, museum or an archive. 😟 huh why? WTF?

Turns out, they've relabled their librarian education course to Data science for getting peoples attention.

Hey you cocksuckers! I want my 2 hours of wasted life time back!
Fuck this "english title whitewashing"

"Big data" and "machine learning" are such big buzz words. Employers be like "we want this! Can you use this?" but they give you shitty, ancient PC's and messy MESSY data. Oh? You want to know why it's taken me five weeks to clean data and run ML algorithms? Have you seen how bad your data is? Are you aware of the lack of standardisation? DO YOU KNOW HOW MANY PEOPLE HAVE MISSPELLED "information"?!!! I DIDN'T EVEN KNOW THERE WERE MORE THAN 15 WAYS OF MISSPELLING IT!!! I HAD TO MAKE MY OWN GODDAMN DICTIONARY!!! YOU EVER FELT THE PAIN OF TRAINING A CLASSIFIER FOR 4 DAYS STRAIGHT THEN YOUR GODDAMN DEVICE CRASHES LOSING ALL YOUR TRAINED MODELS?!!
*cries*

So today I made a mock exam of data science.

$50,000+ Computer Science fucking diploma, yet here I am on a Sunday evening doing data entry coz there is no one else to do it!

Hey guys
So, why so many rants about Facebook now?
I already knew that Facebook sells our data for a few years.... Didn't anyone else knew before that stupid #deletefacebook campaign?

The public seems to be worried a lot on the Facebook "data breach" yet doesn't bat an eye on a bigger website that has already been selling private data for more than a decade.

Google

Our Service Oriented Architecture team is writing very next-level things, such as JSON services that pass data like this:

<JSON>
<Data>
...
</Data>
</JSON>

So, What should it be pronounced, 'Data' or 'Data' ?
😕

Happy data privacy day!

I just deleted 3.5TB of junk data from S3, effectively saving my company about 88 dollars.

I feel so fucking good.

Think I'm going to ask for a raise😂

Hope there is no Data Scientist here...

340 MB in 9 days, that's quite a bit of data for a therapeutic app 😀

Let me repeat this out and loud so that we are clear before another idiot starts pitching to me on building their "world class machine learning algorithm":
NO DATA, NO MACHINE LEARNING!

Data Engineering cycle of hell:

1) Receive an "beyond urgent" request for a "quick and easy" "one time only" data need.

2) Do it fast using spaghetti code and manual platforms and methods.

3) Go do something else for a time period, until receiving the same request again accompanied by some excuse about "why we need it again just this once"

4) Repeat step 3 until this "only once" process is required to prevent the sun from collapsing into a black hole

5) Repeat steps 1 to 4 until it is impossible to maintain the clusterfuck of hundreds of "quick and simple" processes

6) Require time for refactoring just as a formality, managers will NEVER try to be more efficient if it means that they cannot respond to the latest request (it is called "Panic-Driven Development" or "Crappy Diem" principle)

7) GTFO and let the company collapse onto the next Data Engineering Atlas who happens to wander under the clusterfuck. May his pain end quickly.

WHAT THE FUCK IS WRONG WITH YOU APPLE?! THATS A SIXTH OF MY MOBILE DATA! WHERE IS THIS MUCH USAGE COMING FROM?????

WHY DO PEOPLE USE STRINGS ON THEIR BINARY CLASSIFICATION DATA LIKE IS IT HARD TO USE 0 AND 1?!?!

OK I live in lithuania, small country, my grandparents live in silute, super small city, the internet is shit here, I need to use my mobile data to program, next day I wake up to this graph explaining me how I lost all of my fucking data😤

Coworker: so once the algorithm is done I will append new columns in the sql database and insert the output there

Me: I don't like that, can we put the output in a separate table and link it using a foreign key. Just to avoid touching the original data, you know, to avoid potential corruption.

C: Yes sure.

< Two days later - over text >

C: I finished the algo, i decided to append it to the original data in order to avoid redundancy and save on space. I think this makes more sense.

Me: ahdhxjdjsisudhdhdbdbkekdh

No. Learn this principal:
" The original data generated by the client, should be treated like the god damn Bible! DO NOT EVER CHANGE ITS SCHEMA FOR A 3RD PARTY CALCULATION! "

Put simply: D.F.T.T.O
Don't. Fucking. Touch. The. Origin!

"We use WSDL and SOAP to provide data APIs"
- Old-fashioned but ok, gimme the service def file
(The WSDL services definition file describes like 20 services)
- Cool, I see several services. In need those X data entities.
"Those will all be available through the Data service endpoint"
- What you mean "all entities in the same endpoint"? It is a WSDL, the whole point is having self-documented APIs for each entity format!
"No, you have a parameter to set the name of the data entity you want, and each entity will have its own format when the service return it"
- WTF you need the WSDL for if you will have a single service for everything?!?
"It is the way we have always done things"

Certain companies are some outdated-ass backwater tech wannabees.
Usually those that have dominated the market of an entire country since the fucking Perestroika.

The moment I turn on the data pipeline, those fuckers are gonna be overloaded into oblivion. I brought popcorn.

Connect a pen drive, format it successfully. Connect to a new machine to copy data and see the data exists.

Crap! which drive did I formatted :(

Windows: Would you like me to give all your data to MS or somewhat less data?
Me: Well .. I guess .. I'll choose the Basic option..
Windows: Here is a fun fact .. I'll send all the data regardless of your choice :)

This is what's happening in the background, I think. I mean there isn't even a phuqing option to turn this thing off.

User: I need you to extract all the invoice data for us.

Me: What invoice data in particular, what are filters you require. This is a massive database with millions of transactions.

User: JUST EXTRACT THE TABLE!

Me: Right.....(this is a database with 3000+ tables and hundreds of joins)

Wonder which data structure they're using

data science is just a sexy word for statistics.

1% programming skills
99% practically all science thats not computer science

The lights in the data center broke.
Everything else is working.

Oh shit i watched youtube with mobile data and i used 459mb... 31mb left to use it for 12 days... Shit

That feeling when the business wants you to allow massive chunks of data to simply be missing or not required for "grandfathered" accounts, but required for all new accounts.

Our company handles tens of thousands of accounts and at some point in the past during a major upgrade, it was decided that everyone prior to the upgrade just didn't need to fill in the new data.

Now we are doing another major upgrade that is somewhat near completion and we are only just now being told that we have to magically allow a large set of our accounts to NOT require all of this new required data. The circumstances are clear as mud. If the user changes something in their grandfathered account or adds something new, from that point on that piece of data is now required.
But everything else that isn't changed or added can still be blank...
But every new account has to have all the data required...

WHY?!

Everytime people talk about big data/ operations research

When you forget to normalize or standardize your data.

FaceBook, over the years you’ve proven that you can’t be trusted and you still have the nerve to ask for this type of personal data?

Talk about data protection, I am fucking furious!! A hotel i stayed in recently has sent an email with a scan of my passport and credit card. Do I have any legal rights to fuck them up the arse? The hotel is in france.

This is how U became a Data Scientist 😆

Recently found an artist called "Big Data". His song titles describe what Big Data actually is really well:

!rant

Did you ever have that feeling of "what the fuck - how did I do THAT" - I just had one; I was fine with the algorithm taking like 15/25 minutes to fetch, process, save all data, now it shreds through 10 times of the previous amount in like 7 minutes.

I don't get all the amazement etc people show when they find out that I don't use services like WhatsApp, Instagram, Gmail, Outlook, Google etc, why would it be 'weird' to want control over who has/owns your data? That's not being fucking paranoid, that's being conscious about who you give your data to.

Apply for a data engineer role.
Get invited for a data science interview.
HR says they're building AI and I were to supervise another person writing its algorithm.

It's a media company.
*Risitas intensifies*

pushing sensitive data to GitHub repo...not realising for days...and then going through the painful process of removing it again 😑

"The client is complaining that the data takes a long time to load."
Me, after investigating: "Does the client really need [not an exaggeration, actual figures] 130,000 rows of data?"
"I will show them these stats but they said they need it"

Ok this is freaking creepy. I searched for information about login systems on other websites for some time. Not once did I touch Facebook. Then Facebook show me targeted ads when I hoped on 30 minutes later. How the heck does Facebook get that data in the first place? I'm starting to get pissed that my data is just handed out to everyone, even by search engines.

"What is Data Science?" The title of the first slide of way too many data science talks.

Trying to explain data structures in Java to my Dad! ...he just nods his head. Love you, Dad!

OH MY FUCKING GOD!!!!! FUCK YOU SPRING-DATA-NEO4J YOU ARE DRIVING ME CRAZY YOU SHIT FUCK !!! FUCK YOU AND YOUR DELETE ON SAVE BULLSHIT!!!! OMFG!!!!!!! EVERYTIME IS SOME FUCKING SHIT THAT DELETES OTHER SHIT THAT SHOULD NOT HAVE BEEN DELETED!!! JUST FUCK YOU ALREADY IM GONNA REWRITE ALL THIS SHIT!!!!!!!

#10 year challenge is basically data set father for new ai which will predict how X looks after 10 years

Data mining at its best

The real big data

I need more mobile data now that I found devRant ._.

Devrants mobile data usage is so low. Really love don't needing a good connection and don't using to much of my mobile data when using it :)

Also: Grammarly STOP auto-correcting devrant to servant ALL THE TIME...

Forgot to turn off mobile hotspot when I connected my phone with a Windows PC. Fucking windows used all my data to download 'security' updates. Bhenchod.

Big data text book

I accidently lost a whole tables data today. Lucky me I also created a XML-file on the server which was for printing.
I parsed the files back into a simplexmlobject and recovered most of the lost data. 💪

Got myself into Facebook's Graph-API...
...everything is so easy and well optimized.

NOT! Now I have to optimize the request to reduce loadtime. Why is there data so fractured?

Data Science in Python is mind-blasting.

When you hear the data engineers talk about ambari, storm, sqoop, kafka & phoenix and you are not sure if they are talking about frameworks or pokemons..

Data Disinformation: the Next Big Problem

Automatic code generation LLMs like ChatGPT are capable of producing SQL snippets. Regardless of quality, those are capable of retrieving data (from prepared datasets) based on user prompts.
That data may, however, be garbage. This will lead to garbage decisions by lowly literate stakeholders.
Like with network neutrality and pii/psi ownership, we must act now to avoid yet another calamity.

Imagine a scenario where a middle-manager level illiterate barks some prompts to the corporate AI and it writes and runs an SQL query in company databases.
The AI outputs some interactive charts that show that the average worker spends 92.4 minutes on lunch daily.
The middle manager gets furious and enacts an Orwellian policy of facial recognition punch clock in the office.
Two months and millions of dollars in contractors later, and the middle manager checks the same prompt again... and the average lunch time is now 107.2 minutes!
Finally the middle manager gets a literate person to check the data... and the piece of shit SQL behind the number is sourcing from the "off-site scheduled meetings" database.
Why? because the dataset that does have the data for lunch breaks is labeled "labour board compliance 3", and the LLM thought that the metadata for the wrong dataset better matched the user's prompt.

This, given the very real world scenario of mislabeled data and LLMs' inability to understand what they are saying or accessing, and the average manager's complete data illiteracy, we might have to wrangle some actions to prepare for this type of tomfoolery.

I don't think that access restriction will save our souls here, decision-flumberers usually have the authority to overrule RACI/ACL restrictions anyway.
Making "data analysis" an AI-GMO-Free zone is laughable, that is simply not how the tech market works. Auto tools are coming to make our jobs harder and less productive, tech people!
I thought about detecting new automation-enhanced data access and visualization, and enacting awareness policies. But it would be of poor help, after a shithead middle manager gets hooked on a surreal indicator value it is nigh impossible to yank them out of it.

Gotta get this snowball rolling, we must have some idea of future AI housetraining best practices if we are to avoid a complete social-media style meltdown of data-driven processes.
Someone cares to pitch in?

Oh fuck off Google with your data grabbing slimy ethics...

Saw a reddit thread earlier asking about the most unsettling shit that people have found out Google has on them by downloading their data. I saw a bunch of comments about people finding voice recordings that Google had taken. After reading these, I was wondering what I could find from downloading my data. Decided to download my data, and on the page for it I saw that apparently I had disabled location history, audio activity, and device information.

Knowing companies like Google, I wouldn't be surprised if they didn't stop recording that stuff, just that they're not providing it to me. There were zero voice recordings, but there was location history up until about the beginning of 2017.

Another thing they have is all the pictures from all of my hangouts chats. Apparently there had a good amount of older pictures of myself in there. Going back to probably 2-3 years ago, when I had my emo hair. Just a bit of a throwback. One picture I saw was from last January, when my hair was reaching my chest. Made me really miss my hair.

Other than that, nothing that interesting. Just something I thought I'd share.

A lot of people give Google, Facebook, Microsoft, etc. shit for "selling" user data although in my opinion acting as a matchmaker between advertisers and users does not really constitute selling data.

In contrast there seem to be a lot of companies that actually do sell user data that I never hear anyone here talking about.

Did GDPR actually make ANY difference to how our data is handled?

Anyone who has watched Black Mirror and thought what shit it is and technology can never go that far but wonder what if it does and we all are fucked.

Or we are already there but mini version of it. I see people glued on there phone, screen, and even I am guilty of that. But the amount of data we create everyday which can be treated as huge asset by the AI and Data mining enthusiast, I dont doubt we will be at a breaking point, which is coming sooner than we think.

Girl (trying to turn things on): people from my country are great in bed.
Guy: what data points do you have ?
*Girl walks out of the room*

My God is map development insane. I had no idea.

For starters did you know there are a hundred different satellite map providers?

Just kidding, it's more than that.

Second there appears to be tens of thousands of people whos *entire* job is either analyzing map data, or making maps.
Hell this must be some people's whole *existence*. I am humbled.

I just got done grabbing basic land cover data for a neoscav style game spanning the u.s., when I came across the MRLC land cover data set.

One file was 17GB in size.

Worked out to 1px = 30 meters in their data set. I just need it at a one mile resolution, so I need it in 54px chunks, which I'll have to average, or find medians on, or do some sort of reduction.

Ecoregions.appspot.com actually has a pretty good data set but that's still manual. I ran it through gale and theres actually imperceptible thin line borders that share a separate *shade* of their region colors with the region itself, so I ran it through a mosaic effect, to remove the vast bulk of extraneous border colors, but I'll still have to hand remove the oceans if I go with image sources.

It's not that I havent done things involved like that before, naturally I'm insane. It's just involved.

The reason for editing out the oceans is because the oceans contain a metric boatload of shades of blue.
If I'm converting pixels to tiles, I have to break it down to one color per tile.
With the oceans, the boundary between the ocean and shore (not to mention depth information on the continental shelf) ends up sharing colors when I do a palette reduction, so that's a no-go. Of course I could build the palette bu hand, from sampling the map, and then just measure the distance of each sampled rgb color to that of every color in the palette, to see what color it primarily belongs to, but as it stands ecoregions coloring of the regions has some of them *really close* in rgb value as it is.

Now what I also could do is write a script to parse the shape files, construct polygons in sdl or love2d, and save it to a surface with simplified colors, and output that to bmp.

It's perfectly doable, but technically I'm on savings and supposed to be calling companies right now to see if I can get hired instead of being a bum :P

I use my data in exponents of 2

Such useful data...

Oh my fucking god. Austria wants to sell the data of it's citizens to schools, universities museums, and: Companies with enough money. What the fuck?

The data contains shit from the central register of residents, information about name, date of birth, sex/gender, nationality, recidence, health data (!), education, social security/insurance, tax data, E-Card/ELGA Data (system where your doctor visits, prescribed medicines/drugs, all these things, are saved), and other shit.

Welcome to 2018, where you can try as hard as you want to keep your privacy, and then your government sells all the shit you are not able to remove. Fucking bullshit.

I hate debugging document oriented data types..
Can't even sysout easily like primitive data types.
Need a debugging duck.

One thing when working with a ton of data:
If there is a slight, infinitesimal probability that something will be wrong, then it will 100% be wrong.

Never make assumptions that data is consistent, when dealing with tens of gigabytes of it, unless you get it sanitized from somewhere.

I've already seen it all:
* Duplicates where I've been assured "these are unique"
* In text fields that contain exclusively numeric values, there will always be some non-numeric values as well
* There will be negative numbers in "number sequences starting with 1"
* There will be dates in the future, and in the far far future, like 20115 in the future.
* Even if you have 200k customers, there will be a customer ID that will cause an integer overflow.

Don't trust anything. Always check and question everything.

Opened my PC today...found my local disk empty...170gigs of data lost...
Feel like crying...hope Recuva works

it was me

I caused the data bug

joy!

I think I just miiiight have found a new job, but before, some comments about the state of the data engineering industry:

- Sooooooo many people outsource it. Man, outsourcing your data teams is like seeing the world through an Apple Vision Pro fused to your skull. Fine if it is working well, but you will go blind of your subscription expires. Or if Apple decides to ban you. Or if they decide to abandon the product... you are entirely dependent on their whims. In retrospect this is par for the course, I guess.
- Lots of companies think data engineering *starts* with an SQL database. Oh, honey, I have some bad news.
- Quite a few expect MS POWER BI will be able to deliver REAL TIME DASHBOARDS summarizing TERABYTES of data sourced from SQL SERVER (or similar). Facepalm.
- Nearly all think the handling of data engineering products is just like that of software engineering. Just try. I dare you.
- Why people think that "familiarity in several SQL dialects" is something to brag about?
- Shit, startups. Startups are dead, boomers. Deader than video rental physical stores.

That's all. On to the next round of interviews!

Wow! Just updated my devRant android and it has search! Opens the floodgates to data mining the rapidly growing content. Thanks!!!

When your company has data integrity issues and they expect hacky workarounds instead of fixing the data

The kind of data I like to see.

Holly shit, they will never understand....

If you come to dev, minding his buissness, writing some code, beeing focused, with headphones on, etc. You absolutely destroy his cache of things in mind, you stay next to him gazing at his screen and throwing away madly words that you need some data extracted from database NOW and you will stay standing next to him gazing when he quickly types few lines of ultra-wide screen of SQL querry with all the fucking joins and shit you wanted with exac aliases you understand and makes one typo but query executes and spits out some data. He didn't notice that something is wrong with it becouse he didn't memorize database's data, and he sends it to you.

Now you are coming back to him pissed and in general fighting mood becouse he did one fucking typo when you was actively pressurizing him to make fucking query faster while you absolutely destroyed his flow of work (in meaning he now needs to dive back into code, figure out why he was editing given file and what was idea for further structure)

Now you are standing again next to him and absolutely pissing him off that he fucked up and made one fucking typpo that unfortnately database didn't say "nope" but it instead spit out wrong data.

If you can relate to "you" stated above, sincerly fuck you.
If you can relate to "he" stated above, I feel ya man, its fucking annoying, isn't it?

The worst thing about university is having to listen to lecturer say DATA MINING IS GOOD at every single lecture there is.

Am I the only person here who thinks data science is awesome? 😬

You know that the Algorithm & Data Structure exam is coming, when you start traversing binary search trees in your sleep

For shit's sake, data stream processing really is only for people with high throughput looking to do transformations on their data; not for people aggregating <10Gb/day of data.

Fuck me DSP is going to be the new buzzword of 2020 and I'm not looking forward to it. I've already got stakeholders wondering if we can integrate it when we dont have the need, nor the resources or funds.

Labelling almost 20 TB of Data in few Minutes

How many of you use the right data structures for the right situations?

As seasoned programmer and mentor Simon Allardice said: "I've met all sorts of programmers, but where the self-taught programmers fell short was knowing when to use the right data structure for the right situation. There are Arrays, ArrayLists, Sets, HashSets, singly linked Lists, doubly linked Lists, Stacks, Queues, Red-Black trees, Binary trees,.. and what the novice programmer does wrong is only use ArrayList for everything".

Most uni students don't have this problem though, for Data Structures is freshman year material. It's dry, complicated and a difficult to pass course, but it's crucial as a toolset for the programmer.

What's important is knowing what data structures are good in what situations and knowing their strengths and weaknesses. If you use an ArrayList to traverse and work with millions of records, it will be ten-fold as inefficient as using a Set. And so on, and so on.

Husband looking into online schools for CS. Anything data science related. He loves math (and is freaking good at it) and teaches himself R for fun.

I put 0 thought into my own schools (terrible, I know, but not likely to change any time soon). Any suggestions for good online data Science programs, with a math minor potentially?

It's for his bachelor's.

What container data structures do all of you actually use?

For me, I'd say: dynamically sized array, hashtable, and hash set. (And string if you count that.)

In the past 8 months, I've used another container type maybe once?

Meanwhile, I remember in school, we had all these classes on all these fancy data structures, I havent seen most of them since.

i was learning neural networks, started with keras and was on the first tutorial where they started by importing pandas
so i switched to learning data analysis using pandas in Python where they started by importing matplotlib and i realized data visualization is also important and now I'm reading matplotlib docs...🙄

I wonder, are some apps on purpose made to be small size, but then later when you open them force you to download the actual data made with converting people that are low on space or dont want heavy apps to actual download numbers? since if they now after discovering that, delete the app - they already have boosted the downloads

I can understand pulling offline databases or game updates from own servers to get the newest data, but sometimes you come across apps that have no need in any of those and could just e.g. package the audio which will never be updated

A few minutes ago, I was going crazy over a bug caused due to data mutation in Rails.

Basically, user1 was creating a post record and it's stored in the database normally however, on page reload for a second user, user2, the post (that user1 created) was update to belong to user2. This is because, on page reloads, I was using `<<` method call to append user2 and user1 posts together! Apparently, `<<` not only mutates the array, it also performs a database update.

Kill me please!!

Also, data immutability seems a more reasonable feature in languages now.

Why is python supposedly something big data people use ? Sounds like r and stats and well I don’t see the adoption of that though python is used somewhat I note in a lot of Linux apps and utilities

Just seems strange that an interpreted language would be used that way to me or am I an idiot ?

Am I Data Engineer or Software Development Engineer ?

I design the infrastructure for analytics data, and I build the infra entirely including an development. Except making reports out of the data.

What I'm supposed to be called ?
Data Engineer ?
Software Development Engineer ?

Definitely not an Data Scientist. Official designation given by company is Data Engineer II. But what I'm ?

Confused, someone help me please.

One of my colleagues who thinks he knows all about "big data": "I try to put everything in hadoop, that is my philosophy".

We don't even have a hadoop cluster.

truth about big data 😂

Customer: You say you connect to XYZ service, but my data is not getting synced!

*checks their account*
Me: I see you have not integrated your two accounts. Go to Settings > Integrations > Connect to XYZ.

C: Why? But you already connect XYZ! You should already know all of my data!

what's your expirience with business intelligence / data science? Do your company has a own department for it?
If so how does it affect you?

Zomato was data breached!

I guess the devs at rdocumentation need to work on how they "analyze data" 🤣

Went to a Big Data workshop, now I know why there's a elephant as hadoop's logo and how it came. Still no clue how it works.

Fellow ranters, always remember:
It's not the size of yout data that matters. It's how you use it

I decided to try and see if I can live on my 3GB mobile data plan for work and study today. For 3 hours, I just burnt 200MB by ssh, whatsapp, telegram, browsing forums to fix ticket, and reading manuals.

Guess I have to rely on my customer's restrictive WiFi. Sigh.

I've just spent the last hour or so banging my head against a brick wall trying to figure out why I'm unable to retrieve some data via AJAX even though I know data is being returned as I can see it in my error log.

Turns out the permission system I wrote a few days ago actually works and because I didn't specify a permission it automatically denied my user from retrieving the data. One thing I forgot to add was an error message to tell me when I don't have sufficient permission to do something. Adding a message could have just saved me a lot of time :/

Recipe for reverse engineering data structures / binary formats:

1℅ understanding the theory.
1℅ expections about what you will find.
3% luck.
45% trial and error.
50℅ persistence.

Sending email to client (the following is a short version of it)

"
Dear Data Evaluation Team,

Here is a link with the password to the data export for the questionaire.

You will find in there 4 sections:

1. Utilization Report
2. Question List
3. User Responses to questionnaire
4. Summary of responses
"

Email from client
"
Thank you data team.
I see that the user responses have some ids for the questions. Can you please give me the full question text, where is it?
"
My response
"Section 2, Question List"

Like really? Did you just not f*ing read the email and just jumped into the data export blindly. I wrote some fucking docs for a reason.

To solve an issue of slow data retrieval on an app, I suggested pre-loading the data once a week and storing it in a spreadsheet ready for instant download.
"Great idea. If more data gets added to the database after we download the data, will the spreadsheet we downloaded also update?"
Seriously, I understand that not everybody is technical but fuck me how do you expect us to update a file on your computer?

shit data... always with the shit data.

2018: Data Scientist = Stack overflow copy pasting: "I followed a 12-hour 'DS' course on Lynda!"

lm() # science

Learning Data Science to predict the possible ending(s) of GOT.
Hell Yeah

Hey guys any advice/study plan for a 22 year old C++ Server developer to transition into data science ?

P.s. currently based in London

What is it about robot collected data that makes researchers so anal? Like, dude, it's not even personal data. It's literally robot's joint motor recordings. It's not nuclear data, so why the fuck do you protect it like your life and your country depend on it?

I hope you get fisted by that data every night and how it will end up in oblivion sooner because you didn't publish it. You asshole.

"It is pointless to use just a fraction of the data in a homologation environment"
Those words reveal the truth in our creed.
We work in the deepest of back-ends to serve the front.
No data is true. Everything can be edited.
We are Data Engineers.

And for those words to take hold, a junior must execute a leap of faith, and push a hotfix into production.

We specified a very optimistic setup for a data science platform for a client....

Minimum one machine with a 16 core CPU with 64GB RAM to process data.....

Client's IT department: Best we can do is an 8 core 16GB server.

Literally what I have on my laptop.

Data scientist doesn't use any out-of-memory data processing framework, e.g. Dask, despite telling him it's the best way to be economical on memory; ipykernel kills the computation anyway because it runs out of memory.

Data scientist has a 64GB machine himself so he says it's fine.

Purpose of the server: rendered pointless.

The rainbow scale is the comic sans of data visualization.

Imagine an annual $50k+ enterprise software package that didn't distinguish between a null and an empty string in valuing critical data. Not noticed for years - wtf?

Can anyone recommend good resources for learning how to design NoSQL (document) data models?

I'm interested in stuff that talks about how to make the choices about distributing data across collections, etc.

When to have a single collection, when to split data across different collections, when to duplication data, etc,

I need cyclic data structures but algebraic data types are my first love and tying the knot is impossible with the eagerness F# has. The interfaces and classes I abandoned C# for are the ones I am now writing in F#. What a job well done on my part in avoiding mutability :(

I’ve been reading this book “Designing Data Intensive Applications” by Martin Kleppman. The concepts are really well explained !!

When Data Engineers deals with web backend, this happens!

Let’s just face it. Once the majority of people understand, data science (machine learning on custom behaviour mostly) will be the most disgusted job in the world. Mind boggling! You get shot right between the eye by doing it.

Sometimes I just wish Core Data would die.

I logged the managed object context of each object being touched and they're all identical. 😑😑😑😑😑😑

When someone from your data team claims that 250k worth of data doesn't exist and when the shit hits the fan it magically appears.

Need help, Day-ta vs. Dat-uh. Which one is it?(data)

Perhaps as a tip for the junior devs out there, here's what I learned about programming skills on the job:

You know those heavy classes back in college that taught you all about Data Structures? Some devs may argue that you just need to know how to code and you don't need to know fancy Data Structures or Big o notation theory, but in the real world we use them all the time, especially for important projects.

All those principles about Sets, (Linked) lists, map, filter, reduce, union, intersection, symmetric difference, Big O Notation... They matter and are used to solve problems. I used to think I could just coast by without being versed in them.. Soon, mathematics and Big o notation came back to bite me.

Three example projects I worked in where this mattered:
- Massive data collection and processing in legacy Java (clients want their data fast, so better think about the performance implications of CRUD into Collections)
- ReactJS (oh yes, maps and filters are used a lot...)
- Massive data collection in C# where data manipulation results are crucial (union, intersection, symmetric difference,...)
Overall: speed and quality mattered (better know your Big o notation or use a cheat sheet, though I prefer the first)
Yes, the approach can be optimized here, but often we're tied to client constraints, with some room if we're lucky.

I'm glad I learned this lesson. I would rather have skills in my head and in memory than having to look up things and try to understand them all the time.

Anybody have any great tutorials about web scraping with python? The data science courses I took only covered maipulauting and visualizing data not getting it.

I want to do something data-science-y.
Gimme project ideas, and where can I get the data for it?

Also, not looking for machine learning, just basic data analysis stuff.

I'm bored.

Big Data Analysis

Question & Android

Is there a way on android to limit what apps are allowed to use mobile data and what are only allowed to use wifi?

!rant
Started Data Science course on big data universuty. The outcomes are heavily dependent on domain experts/stakeholders!!! Since all the answers are false positives and need to decide what make sense with the help of domain experts. And most of the Data scientists are not from programming background, they are domain experts who turned into Data scientists. Thoughts if I should continue with learning big data/data science, knowing that I have knowledge in information retrieval and search engines.

Best way to pass a DB dump with sensitive data: zip with password and email. :|

Hello data structure!
I faild becuase of you.
Wait for me😁

Data Structures professor enters class, today's topic being RBTs. He opens slides, forgot to redact the name and university from where he blatantly copied it from. Continues anyways. *Facepalm*

Having fun with GitHub data. Plotted 8 days GitHub data.

I received the following e-mail today:
Hey, XYZ! Could you please check the following in your web application. The data do not show correctly. Could it be a bug?
[insert attached screenshot with said "bug"]

My reply:
Hey, ABC! It is not a bug. You uploaded the data into the wrong table. 😊
[insert attached screenshot with the incriminating evidence]
-----------------------------
I felt a bit savage and I liked it. 🔥

Fucked a clients wordpress website up by editing a saved option thats saved as serialised php data. Tried using the row data from a backup and updating it in the database and it still loads the fallback theme settings. What do?

So I get this email from google for my development account about these new general data protection regulations and what they're doing with admob and all that good stuff.

I didn't dive too deep and there's nothing crazy in it but it definitely feels like it's spawned on by this "selling you data to advertisers thing."

We live in such a weird society where it's like outrage after outrage. I've never known anybody who has NOT known that their statistics and data was sold to third party for marketing for EVERYTHING they do on the computer or phone. For a DECADE or longer. It always seemed to be such a second hand thought but now out of nowhere everyone has their panties in a wad for something they ALREADY knew.

Are we like that miserable/bored/no hobbies/unsatisfied with our first world life that we have to just flip out about dumb crap all the time?

The gap of data science in industry and academia is so large. As a data scientist in a large financial company, I see that people are still using traditional models such as linear regression and SVM, while people in academia keep inventing new concepts and techniques such as deep learning.

I am not saying that we should completely embrace deep learning, or stick to classic methods. But I just feel so surprised that the gap is so large...Sometimes I am even thinking whether I am doing the right "data science"...

Seriously, Clojure is a beautiful language.
Especially for data manipulation

if AI is so smart why doesn't it clean its own data, eh?!

If there is no data, it's kind of hard to return it isn't it?????

So I cant resize the window and I can't scroll down anymore... Where the fuck is "below" so I can turn "off" data collection 😤

Real question:

If I save your data in the cloud and it's raining outside, will I lose my data? or Am I safe?

*-- There's something kind of child like and adorable about working for a client who spends THOUSANDS of dollars on their data infrastructure, yet finds it ever so difficult to provide ONE user to help reconcile and test the new data warehouse.

Dirty data? More like dirty laundry! And don't even get me started on explaining complex models to non-techies. It's like trying to teach a cat to do calculus. Furr-get about it!

Why is everyone into big data? I like mostly all kind of technology (programming, Linux, security...) But I can't get myself to like big data /ML /AI. I get that it's usefulness is abundant, but how is it fascinating?

So we are gonna have a presentation offshore in an international event, in some hours from now.

Boss asked me to update some data along with some that a coworker was working on.

I asked about said data to coworker and she replied: they are in dropbox since yesterday.
And I was like OK.

AND NOW I'M FUCKING AWAKE AT 3 AM TO UPDATE IT AND 80% OF THE FUCKING DATA ISN'T ON THE FUCKING DROPBOX!!!! WTFFFF!!!!!

I see bad data and thought to myself, "I'll be able to fix this with a simple regex." A month later, I'm still finding new data patterns. Never give users the ability to store all of their data in a huge textarea box where they can make stuff up.

So today in school I decided on my career choice. So I've decided on becoming a data scientist because I can still use my programming and I very much do like looking at data and researching things as well as program obviously :) so if there are any data scientists or data miners out there do you have advice?

So we finished our requirement ( barely) for a new client. Next is data modelling and system design.

We started with data modelling. Unfortunately the lead developer does not know the difference between database and data modelling.

me: hey bro, we'll do the database and stuff later, now let's focus on data modelling.

him: (acting like he knows) yeah I have developed a sample design for the "data model".

me: no this is database design.

him: what's the difference?

me: dude, they're totally different. Okay, simple explanation data model is what you want to store, whereas DB design is how you store it.

him: So, if I am not wrong, it's implied that you know what to store if you are talking about how to store it.

me: but you don't know what it is you want to store yet. And one of them precedes the other.

him: Okay, let's start with DB design.

me: What?????? you want to build a house without a plan??? That's it for me I am done !!!

I left the project yesterday, later I heard that, the team members are coders, who think that developing a software is all about coding and fixing errors.

When you wanna be a Data Scientist and always land into internships where you are assigned with web development...
Learnt Node, Flask, Spring frameworks across different internshis...

I took a project. Wild mix of php and html including db stuff and data processing. About 200 files, some 3000+ lines long with if else cases processed in another template/logic behemoth...

I wrote a js file included it at the footer of the monster and update dom + data via ajax on my own api implementation because I'm too afraid to write in any of those files.

I've been told its quality code and well documented

Not sure what I wanna do ---

Backend Development or Data Analyst

Even in backend development confused between nodejs, PHP, Python though I don't have proper knowledge of all of them.

For Data Analyst I am looking the way from where I should start.

I am in 3rd year of my college
Can anyone please help me with this?

I am excited about Websocket getting more popular. I just completed a project using it. It is so good for sending market data to the front.

What's your favourite Data analytics/ visualisation tool? and why ?

Saw a movie related to Data Security and Data privacy. The movie ended 1 hour ago and i am now terrified how my data is going to end up somewhere where it can be misused .Frantically removed all app permissions from my mobile. Wonder how many days it will last. But now after hearing such gory details , i wonder how i can keep my interests safe in this world. I am now even afraid to give my laptop for changing its battery.. Thinking of wiping all possible compromisable data. But dont know how to.

How will technologies like blockchain affect this ? Will it make it worse or is it trying to make it better..?

Can anyone explain to me why facebook got pulled over for selling user data and google hasn't yet?

Data scientists should be charged with animal trafficking and animal abuse because they import Pandas 🐼 and feed them to Python 🐍

Just dumped one month's GitHub public data or my local machine. Ah boy that was too much data. About 80 GB with 30.9 million documents in mongodb. Too much for local machine. :sigh:

I hate data structures. I try to work out their algorithms in my head but they're completely counterintuitive. lol.

I've recently learned how committing of the Save Data to file works in my project.

The file is updated w/ _each change_ made to the settings.

Worse yet - the file is updated even when _no actual change_ is made due to the setting already being at its highest / lowest value possible.
/*
e.g. 5 is maximum sound volume.
- You try increasing the sound volume.
- Setting can't get any higher, so remains unchanged.
- *Update the Save Data*.
*/

What kind of abusive masochist would do that?

// Yes... there's always blame.

Has anyone played around with kaggle.com to get into data science?

I’m good with data, good with analytics and good with programming but I need to combine those skills and learn whatever I’m missing.

Kaggle came up and it seems to have some nice data sets and challenges to do.

Thought?