Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "python-requests requests"
-
So a few days ago I felt pretty h*ckin professional.
I'm an intern and my job was to get the last 2003 server off the racks (It's a government job, so it's a wonder we only have one 2003 server left). The problem being that the service running on that server cannot just be placed on a new OS. It's some custom engineering document server that was built in 2003 on a 1995 tech stack and it had been abandoned for so long that it was apparently lost to time with no hope of recovery.
"Please redesign the system. Use a modern tech stack. Have at it, she's your project, do as you wish."
Music to my ears.
First challenge is getting the data off the old server. It's a 1995 .mdb file, so the most recent version of Access that would be able to open it is 2010.
Option two: There's an "export" button that literally just vomits all 16,644 records into a tab-delimited text file. Since this option didn't require scavenging up an old version of Access, I wrote a Python script to just read the export file.
And something like 30% of the records were invalid. Why? Well, one of the fields allowed for newline characters. This was an issue because records were separated by newline. So any record with a field containing newline became invalid.
Although, this did not stop me. Not even close. I figured it out and fixed it in about 10 minutes. All records read into the program without issue.
Next for designing the database. My stack is MySQL and NodeJS, which my supervisors approved of. There was a lot of data that looked like it would fit into an integer, but one or two odd records would have something like "1050b" which mean that just a few items prevented me from having as slick of a database design as I wanted. I designed the tables, about 18 columns per record, mostly varchar(64).
Next challenge was putting the exported data into the database. At first I thought of doing it record by record from my python script. Connect to the MySQL server and just iterate over all the data I had. But what I ended up actually doing was generating a .sql file and running that on the server. This took a few tries thanks to a lot of inconsistencies in the data, but eventually, I got all 16k records in the new database and I had never been so happy.
The next two hours were very productive, designing a front end which was very clean. I had just enough time to design a rough prototype that works totally off ajax requests. I want to keep it that way so that other services can contact this data, as it may be useful to have an engineering data API.
Anyways, that was my win story of the week. I was handed a challenge; an old, decaying server full of important data, and despite the hitches one might expect from archaic data, I was able to rescue every byte. I will probably be presenting my prototype to the higher ups in Engineering sometime this week.
Happy Algo!8 -
I believe this is why companies look for Junior Developers who actually know enough to be a mid or senior developer.
One day, a company that doesn't have the technical chops to know the difference between python and ruby hire a developer who is still in school. That developer doesn't know what he's worth, so the company gets him for pretty cheap. He does amazing things, takes last minute requests, learns some along the way, but eventually leaves because he just got contacted by a recruiter telling him how much he's really worth. He leaves, but the company needs to fill his spot. The company asks the former rockstar all the technologies he used to accomplish his job and throw that into a job description. The company could only really afford the junior so they keep all the stuff about being a junior, but because they need to maintain all the hodge podge stuff the previous developer put in, they need someone with experience enough to jump in.6 -
I'm 54 y.o.
I think I'm completely outdated in my skill, as in the last 14 years, I worked on a specific business problem, with an old technology: a JSP application + javascript + postgres.
I do understand software development, agile, web application development, linux server, basic/moderate AWS skills, etc.
Now they laid me off instead of including me in the evolution of version 2 of the software. Maybe covid, company had almost no cash-flow. Well they have now...So basically they fired me to find money to rewrite the application.
I feel without hope at my age.
I'm a generalist.
I can understand fairly well everything you'll throw at me, reactnative, angular, nosql, python, but I have little first-hand experience.
I don't have a lot of management skills, even if I've given frequent presentations to C-roles and board, and I implemented a whole agile methodology in my team.
I don't know what to do.
The amount of technology to study is huge nowadays. When I was younger I could get away with some php and java.
Full-stack developer is a big word for me. Maybe I could handle a full stack web application, but not from scratch.
I feel at my age, I'll compete with 20-something guys with better skills and lower salary requests.
I don't think I can pull a night anymore.
I'm trying to shoot high to management positions with no much success.
I'd like to go on developing, I know that there are 50-something developer out there, but who managed to find a new position at 55? at 60?
As soon as I finish the few money I spared, I'll be on the street, I'l be the "website for food" guy.49 -
So, i tried to demonstrate my roommate how many people push their credentials to github by searching for "password remove" commits.
I decided to show him the file and noticed something interesting. A public IP, and mysql credentials.
I visit the IP and what do i see there, a directory listening with a python script, with injects the database into a webpage (???) and a log of all http requests. Lots of failed attacks aiming at the PHP CGI. Still wondering how they failed on a python server 🤔🤔🤔
Edit phpmyadmin to connect to the mysql database. Success.
Inserted a row telling him the his password is on github. Maybe i should also have told him how to actually remove it. 😅
Yes, root can login from %
This is how far i can get with my current abilities.
------------------------------
Scary how insecure this world is.4 -
So. A while ago I was on OkCupid, trying to find the Pierre to my Marie Curie (without the whole brain getting crushed under a horse carriage wheel obviously) and I decided the best way was to have my profile lead with my passion for technology. It turned out pretty unique, if I do say so myself.
At the end of it, I amassed some interesting and unique messages:
- A Java pickup line (that I never responded to. Yes I'm a very basic Devranter)
- A request to turn the man's software into hardware (to which I politely informed him that this was scientifically impossible unless a reader proves me wrong)
- Another impossible request to turn his floppy disk into a hard drive (how outdated too, why not HDD to SSD for faster speed amirite? That was awful don't mind me)
- A sincere request to help troubleshoot a laptop (Honestly I would've helped with help requests but this is a dating site...)
- A sincere request to help debug a student project followed with a link to a GitHub repo
- Another sincere request with studying for a computer exam
- And lastly, my favourite: a sincere job offer by a guy who went from flirtatious to desperate for a programmer in a minute. He was looking for *insert python, big data, buzzwords here* and asked me for a LinkedIn. I proceeded to inquire exactly what he wanted me to do. He then asks me to WRITE a Python tutorial and that he would pay a few cents per word written so he could publish it. Literally no programming involved.
Needless to say I went to look elsewhere.26 -
Coolest thing I’ve built solo?
Damn, there’s been a lot of things over the years, but I guess the most used one I’ve made would be my voice activated tv remote - yes it’s real.
So in essence it’s a google home... yea I know spyware and all, but look it was free so I’m going to make use of it... err where was I, oh yea.
An IFTTT account which taps into the google assistant API and creates a webhook, although the authentication side of things is 0 to none, so had to put a api-key into the requests to at least have some layer of auth.
This webhook then hits a raspberry pi containing a PHP API to accept and authenticate the request in, digest this into KEY commands for the TV, and drops this into a Python script to connect to the TV over a web socket connection ( I found python more stable for this ) and sends the pre made key requests, it can even do multiple keys at a time... that was a pain.
So after all that, the end game becomes about a second from saying “hey google, change the tv channel to xxx”
This sick and twisted contraption is finished and the tv is my little bitch.
This has been built out to handle channels by name, number, volume up/down, sources switching to hdmi, tv, vga and a bunch of other things.
The things we do when we can’t find a tv remote for days....
Next up, getting it to launch Netflix app and going to a specified show / episode.. but may be to adventurous. -
So I need to let off some steam, let me know if you think I need to calm down. Personally I'm just having a hard time understanding my team lead.
So I've been trying to update our codebase for the past two months so we run tests against the latest versions of each respective major browser. I've also been trying to cleanup our code and split it into logical modules.
Need I add, according to Bitbucket, I've written over 80% of our code on our 4 projects with 4 team members including myself.
He's out for a week, so I decide it's fine time to get some work done -- which is ridiculous in itself. I finish, add unit tests for crap I missed because he kept shutting down my PRs for shit he couldn't understand.
He tells me on Friday, when he got back, that he'll be declining my pull requests because my code is too complex -- my team lead -- thinks list comprehension and OOP in Python is too complex. Doesn't understand why we need to have pep8 lint tests, or why we can't just export one giant monolithic client package with over 3k lines of code.
Is it worth arguing or should I just let my department head know I can't work on this team anymore? He won't get talked to or fired, he's been at my company for 6 years and he's in the inner circle.6 -
New developers. Tip: There is no silver bullet.
If you like Python, please understand GIL's behavior before making a system that handles thousands of requests.
If you like Java, know that "Write once, run anywhere" is a fallacy. Even application servers don't like the same WAR.
If you like PHP, understand the life cycle of a request before connecting to the database from all corners.
If you like C#, don't make it a small command-line application that will be used on FreeBSD.
If you like C, meet valgrind.
If you like C++, templates are cool, but don't overdo it. And take the opportunity to meet valgrind.
Never use the same tool to do everything. Elect the language and framework for the given need with rationality.
Every time I see a "Java Man", a "C++ Chad" or anything like that, it comes to mind that if he were a carpenter, he would be tightening screws with hammers.
Every lock-in is bad.11 -
My god the wall looks really punchable right now. Let me tell you why.
So I’m working on a data mining project, and I’m trying to get data from google trends. Unfortunately, there have been a lot of roadblocks for what should have been an easy task.
First it won’t give a raw search volume, only relative “interest”.
Fortunately it lets me compare search terms, which would work for my needs however it will only let me compare a few at a time. I need to compare 300.
So my solution is simple: compare all the terms relative to one term. Simple enough, but it would be time consuming so I figured I’d write a program to get the data.
But then I learned that they don’t have an official api. There’s a node module for this very thing based on a python module that reverse engineers the api endpoints. I thought as long as it works I’d use it.
It does work... But then I discovered that google heavily rate limits the endpoints.
So... I figured I’d build a system to route the requests through different tor nodes to get around the rate limit. Good solution right? Well like a slap to the face, after spending way to much time getting requests through tor working, I discovered that THEY FUCKING BLOCKED TOR IPS.
So I gave up, and resigned to wait 5 hours for my program to get the data... 1 comparison at a time... 60s interval between requests. They, of course, don’t tell you the rate limit threshold, so this is more or less a guess (I verified that 30s interval was too short and another person using the module suggested 60s).
Remember when I said the discovery that the blocked tor came like a slap to the face? This came as a sledge hammer to the face: for some reason my program didn’t dump the data at the end. I waited 5 fucking hours to get nothing.
I am so mad right now. I am so fucking mad.4 -
I've kinda ghosted DevRant so here's an update:
VueJS is pretty good and I'm happy using it, but it seems I need to start with React soon to gain more business partnerships :( I'm down to learn React, but I'd rather jump into Typescript or stick with Vue.
Webpack is cool and I like it more than my previous Gulp implementation.
Docker has become much more usable in the last 2 years, but it's still garbage on Windows/Mac when running an application that runs on Symfony...without docker-sync. File interactions are just too slow for some of my enterprise apps. docker-sync was a life-saver.
I wish I had swapped ALL links to XHR requests long ago. This pseudo-SPA architecture that I've got now (still server-side rendered) is pretty good. It allows my server to do what servers do best, while eliminating the overhead of reloading CSS/JS on every request. I wrote an ES6 component for this: https://github.com/HTMLGuyLLC/... - Frankly, I could give a shit if you think it's dumb or hate it or think I'm dumb, but I'd love to hear any ideas for improving it (it's open source for a reason). I've been told my script is super helpful for people who have Shopify sites and can't change the backend. I use it to modernize older apps.
ContentBuilder.js has improved a ton in the last year and they're having a sale that ends today if you have a need for something like that, take a look: https://innovastudio.com/content-bu...
I bought and returned a 2019 Macbook pro with i9. I'll stick with my 2015 until we see what's in store for 2020. Apple has really stopped making great products ever since Jobs died, and I can't imagine that he was THAT important to the company. Any idiot on the street can you tell you several ways they could improve the latest models...for instance, how about feedback when you click buttons in the touchbar? How about a skinnier trackpad so your wrists aren't constantly on it? How about always-available audio and brightness buttons? How about better ports...How about a bezel-less screen? How about better arrow keys so you can easily click the up arrow without hitting shift all the time? How about a keyboard that doesn't suck? I did love touch ID though, and the laptop was much lighter.
The Logitech MX Master 3 mouse was just released. I love my 2s, so I just ordered it. We'll see how it is!
PHPStorm still hasn't fixed a couple things that are bothering me with the terminal: can't reorder tabs with drag and drop, tabs are saved but don't reconnect to the server so the title is wrong if you reopen a project and forget that the terminal tabs are from your last session and no longer connected. I've accidentally tried to run scripts locally that were meant for the server more than once...
I just found out this exists: https://caniuse.email/
I'm going to be looking into Kubernetes soon. I keep seeing the name (docker for mac, digitalocean) so I'm curious.
AWS S3 Glacier is still a bitch to work with in 2019...wtf? Having to setup a Python script with a bunch of dependencies in order to remove all items in a vault before you can delete it is dumb. It's like they said "how can we make it difficult for people to remove shit so we can keep charging them forever?". I finally removed almost 2TB of data, but my computer had to run that script for a day....so dumb...6 -
So here I am testing some python code and writing to a file. No big deal. But damn is it taking a long time to get data back from this API. Ah it's fine I'll let it work in the background.
40 minutes later.
Oh! The requests timed out. No big deal. I'll just cut out the parts that are already done.
1st request in.
I wonder what the file is looking like.
Only showing 1 request.
waitaminute.jpeg
I should have more than that.
*Suddenly realizes that I was writing to the file and not appending.
Fuuuuuuuuuuuuuuuuuuck2 -
HTTPS requests in most languages:
Import a couple of libraries, you may need to install a few as well. It's possible that you will need to initialize and set up the socket. Be sure to specify SSL settings. Create the connection, provide it a URL, and attempt the connection. Read the response, usually in chunks. You may need to manually create a buffer of fixed size, depending on if the language has buffer helper classes or not. You will probably need to convert the input stream response to a string to do anything with it. Close the connection and clean up any buffers used.
HTTPS requests in Python:
import urllib
urllib.YEET()6 -
requests lib - HTTP for humans (python)
It really is a breeze to work with! Clean and intuitive design, loving it!2 -
It's still in development. It often says the opposite from what is expected. Try Retoor1b chatbot at https://llm.molodetz.nl
This was result after building bot + chat website from scratch including training with embeddings. Design is generated by GPT, I tried my own but all ugly.
It's quite cool huh? Ask it to write some code for you. It's absolutely terrible. If it's down, try again in 5 minutes. I'm still working on it.
What's the result? I finally have a toolkit to make good/serious bots. Code could be bit better, but that's for other day.
Stack: self written webserver (and yes, you can post a gb to it or ddos it. Not sure if it survives the first one. I should limit requests to one mb anyway. Http headers may officially not be more than 4096 in total) since I know http protocol from my head anyway. Python websockets module. Asyncio, chromadb.
It could have xss issues. Don't care.
Let me know what you think42 -
When your egghead boss (who is a dev, BTW) fails miserably in understanding that JavaScript fetch does not behave like the default synchronous nature of requests in Python.
After failing to make him learn about the asynchronous nature of JavaScript promises, he ends the discussion by saying "that's why python is better than js"
*facepalm*2 -
I’ve spent 2 weeks trying to simply automate logging into my damn school’s blackboard but this ducking popup won’t freaking let me access it. I’ve tried selenium. I’ve used beautifulsoup and requests. I’ve even tried a tool called mechanize with python
But I’ve now realized I simply have no damn idea what I’m doing. I’ve read and tried way too many stack overflow articles and I’m just sick of this damn popup
If I can’t figure it out by the end of the upcoming thanksgiving break I’m dropping this damn project until I learn enough to utilize the blackboard API’s. I’m a little sure those will help16 -
***ILLEGAL***
so its IPL(cricket) season in india, there is a OTT service called hotstar (its like netflix of india), the cricket streams exclusively on hotstar..
so a quick google search reveals literally thousands of emails & passwords, found a pastebin containing 500 emails&passwords ...but those are leaked last year most of passwords are changed & many of them enabled 2FA.. after looking through them we can find some passwords are similar to their emails , some contains birth year like 1975,1997 etc, some passwords end with 123 ..so after trying a few different versions of the passwords like
1) password123 -> password@123, password1234
2) passwordyear -> password@year
2) for passwords similar to emails, we can add 123 ,1234, @ etc
created a quick python script for sending login requests
so after like 30-40 mins of work, i have 7 working accounts
*for those who have basic idea of security practices you can skip this part
lessons learnt
1) enable 2FA
2) use strong passwords, if you change your password , new password should be very different from the old one
there are several thousands of leaked plaintext passwords for services like netflix,spotify, hulu etc, are easily available using simple google search,
after looking through & analysing thousands of them you can find many common passwords , common patterns
they may not be as obvious as password ,password123 but they are easily guessable.
mainly this is because these type of entertainment services are used by the average joe, they dont care about strong passwords, 2FA etc6 -
Should i be posting on devRant to hire a fellow devRanter for a project in my company? (temporarily but may become constant if we click well)
More info:
Looking to hire a mostly frontend react dev.
The project is about graph visualisation and traversal.
Our stack is python (flask, appbuilder, SQLalchemy), react, react d3-graph, jest.
Must know how to use git and pull-requests.
(The software is based on Apache superset)
The code is a bit of a mess. But let's be honest, which big project isn't?12 -
tldr: my classmates suck and I hate them
We study cs in school, and my classmates are super dumb.
Here is an example from today:
The task: build an http server in python, using sockets.
My classmates: writes everything in the main function, uses try-expect for everything and every error possible, nothing works, nothing worked after a week.
Me: properly separated to different functions, used goddam regrx to get data from requests, used asyncio to make sure it can handle multiple requests at the same time, everything worked after 2 hours.
But, and here is the problem, after I finish they ask me a bunch of dumb, 'Just Google it dude' questions and they call me condescending because I get mad after the second hour of teaching them the same thing.
Once they told me:"you think you are a better programmer then us" and I just want to say this out loud: I AM A BETTER PROGRAMMER THEN THEM, THEY ARE THE PERFECT EXAMPLE OF HOW YOU SHOULDN'T DO ANYTHING AND I HATE THEM.
That's it, I'm done. I feel much better now.
PS: it's okay to suck at programming, but please stop thinking that everyone who's better than you is condescending.4 -
The fucking worst part of my job is when I get thrown an error on a bug I'm trying to fix and it doesn't even contain an error message. For example, I'll try to verify a self signed certificate but I get an error and literally the only error thrown is [Error 0], no fucking message. If you try to track down the error in the docs they don't even fucking list errors. Times like these make me wish I could castrate people with my1
-
I have until Friday to write a service that's going to consume our backend api.
I'm doing it with Python, the back end is in Node. It's going to be the first pyhon service in our company, so everybody is looking at it to see if we could/should do more Python!
Under pressure! -
The project my company agreed to work with 1 year ago is totally shitty.
We basically use Python 2, Tornado, MySQL and driver for it from 2005, custom made "ORM" where you have to write SQL in strings in Python, custom asynchronous service that runs jobs and all such...
The rest of the team writes code really badly and only after a year of fucking with this shit we made them do pull requests.
I became totally neurotic because of the shit I have to go through daily at work. I do not develop myself, no new things I have learned in the past year or so. What do I do, devrant? What is your advice?1 -
I just learned the hard way not to recklessly daemonize stuff; I ran into a case where my Python script crashes Python itself.
The issue was that one of the query that “requests” package makes for OS was not fork safe, thus causing a segfault. Since Python drops dead as soon as it receives SIGSEGV, all I had was macOS crash log (which is oftentimes hard to decipher). I spent like good one hour before I found “faulthandler” package which enabled me to log the stack trace to stderr and see what the f*<k was going on.
I mean, I’ve seen quite a lot of occurrences of thread safety issues, but now it’s fork safety!?
Maybe I should be sticking to Docker or something unless the situation *really* requires me to daemonize something lol2 -
(Note: I got a bit carried away while writing this, so the end result is a lot longer than I expected. Apologies for the long post!)
The beginning of my programming journey started with a book.
This was back in 7th grade. I had some basic exposure to BASIC (pun maybe intended?) from our school curriculum, but it was nothing too interesting as our teachers never really treated it as anything important. They would stress a lot on those Microsoft Office chapters (yes, we actually studied Microsoft Office as part of our computer science course at school) and mostly ignore the programming chapters because I dare say many of them struggled with it themselves. So although I had been exposed to *some* programming, it was mostly memorizing the syntax without actually understanding what was going on.
Then one day there was this book fair thing going on at this local Carrefour (for those of you who've no idea, it's a pretty famous hypermarket chain) in this mall, and for some reason my mother and I were in that mall on that day. Now the interesting thing is that this usually never happens -- I usually visit malls with my dad or my friends, this is the only instance I remember where I had actually visited one with just my mom. This turned out to be fortuitous. My father is the kind of person who's generally not amenable to any kind of extraneous shopping requests. My mother, on the other hand, was and remains pliable.
So I basically saw this book -- Sams' Teach Yourself JavaScript in 24 Hours -- being sold at half price. I vaguely remembered having read somewhere that JavaScript is a good introductory programming language (and it helped that this was the time when I was getting into a Google-craze -- I basically saw some photos of Google Zurich and went all HOLY SHIT THAT'S WHERE I NEED TO WORK WHEN I GROW UP (for those of you who haven't seen it, I recommend googling it. That office is the bomb) -- and I'd also read that you need programming skills to join Google). So I begged and begged my mum to buy that book, and thankfully she did.
Back home I returned with my new prize under my arm. Dad took one look at it and scoffed that I'll never actually use it. Pretty much entirely out of spite (to prove him wrong), I attacked the book with a zeal. I still remember how I felt when I wrote my very first JavaScript program (printing the current system date in an h1 tag) and marveling at the output. I guess that was when something struck -- the realization that this was probably what I wanted to do in life.
Fast forward to today, and I've never looked back and wondered what it would be like to have done something else.
PS: for all you beginners out there, JavaScript is a horrible language. Please start with something like Python. Also there are better resources than Sams' Teach Yourself JavaScript in 24 Hours available, that I just didn't know of back then. I'd recommend Eloquent JavaScript any day. -
When I show a school mate how to send http requests using the requests library, instead of urllib in python
-
I'm working on my own smarthome project (how rare nowdays...:) ). I'm using microsoft's git repo. I'm not make my project open source, not because I'm a douchebag, but because I'm a very bad coder. My biggest fear is not the fact, that it would be commented as wtf is this code, but receiving pull requests from pros and I wouldn't even understand their code.
This is my "training" project (python, flask, apscheduler, sqlalchemy, bootstrap2, mqtt, micropython)
I'm thinking to make it public without accepting external pulls, just for me to learn more.
I'm just wondering your thoughts on this.
Thoughts?1 -
Damnit I am an idiot. I am making a downlader for talkpython lectures and ive managed to get the "user_tpt" (auth key) well when I'm set up the request I sent it in the header when its supost to be in the cookies. I couldent figure out why it wasn't working so I left it for 2 days and now just when I open it I see my mistake1
-
PHP: Laravel, Phalcon, Composer
JavaScript: Node.js, Express.js, Restify, Sequelize, AngularJS, React, npm, bower
Python: Django, Flask, Requests, pip, SQLAlchemy
Java: Spring, Dropwizard, Hibernate
DB: MySQL, MariaDB, PostgreSQL, Cassandra1 -
Stuck in debugging a python script (using 'requests' library to achieve 'curl' type function) for the last 2 hours
Worked fine yesterday in Python REPL.. Throws exception when put in a long exisitng .py script.. Works fine again when put in Python REPL
Found out that when in REPL, I am careful to import 'requests' library every time but ignored when typing in .py script
(Feeling stupid)
Lesson learned: Don't use "generic exceptions"!! They never let you know what the real problem is.1 -
Hi everyone
I have a python script that continuously collects data for me. I want to be able to display that data on a node js server. How should I go about this? I was thinking of maybe having the python script send get requests to the server but I feel that is not the right answer. Let me know if u guys need more info, thanks!5