So python, who knew it could automate a lot of crappy tasks 😎
Actually I did, but god it’s so quick and easy to implement than expected.

I have a new formed respect for python devs after my day of web crawling and ftp transfers and outlook mining 😘

In your country web crawling can send you to jail?

The main reason I moved from Linux to macOS was that I grew up. If we count not just Linux experiments but prolonged usage, I was an avid Crunchbang fan. After it died, I moved to elementaryos.

What I want to say is, Linux can be very fun and educational when you're still in the uni. You have all the energy in the world, and you can afford to diverge from your daily routine for an hour to debug GPU drivers.
Now, the backbone of my life is keeping a very tight sleep schedule, taking meds on time, avoid infohazards, avoid scrolling on the web, all to remain in a very fragile state of balance that keeps the bipolar disorder away. I'm in the middle of all this, earning derealization (yes, I'm also autistic) every time I design a data model. All I want from my computer is to be treated like a careless, regular user, not like someone with a CS degree.

I use Sublime Merge instead of command line Git. I use Postico to explore PostgreSQL databases, not psql from my terminal. By the way, my terminal is not iTerm, Alacritty or some other such thing, my terminal is whatever came with my Mac, with whatever default settings.

Linux is crawling into a non-street-legal racecar's cockpit and strapping yourself in, ready to blast off. MacOS is your chauffeur, holding your old shaking hand as he helps you into your Maybach's backseat. They're different, and that's okay.
Can Maybach race? Well, it has a 621 HP V12, so if _you_ can race, it probably can too, but we all know it's not a racecar.

Windows? Windows is an SS officer, wearing the all too familiar Windows logo for swastika, throwing you into a gaswagen.

Imposter syndrome.
A question guys, I'm a web dev since 2012, started with php, then shifted to frontend, for 3 years my main was PHP and basic HTML CSS, in 2017 I shifted to / did courses on vuejs, angular and react (loved angular the most) also laravel. Have also dabbled a bit in python, for crawling and mining. The problem is I've never worked with a team or for a full fledged Dev company, so I'm unsure as to how to judge my growth and whether I'm moving in the right direction. I feel like I need a lot better understanding of Linux usage and server control, or should I learn nativescript etc.

What do you suggest? Should I simply look for a mentorship program, if yes any clue where?

Any web Crawling enthusiasts out there

Took a challenge to start with a new role as Full Stack Web Developer leaving a job as a Banker for more than 6 years. It hurts when things you develop doesn't work also you are not able to complete a task for weeks.
Just crawling to get something done.
Need some encouragement dose.

It goes back two years ago, i was writing web crawlers with scrapy. i don't remember how long i worked, but i think it took a full day

why: because, web crawling is so much fun and also i was young and stupid

Recently we got a new project assigned and as always you are hyped, really really hyped...........

We were supposed to find all kind of driver updates (especially bios ones) for all devices the company owns. So first of all we thought:

EAAAASY! A little bit of web crawling, regex, etc.
.
.
.
.
B
U
U
U
U
T
!

We were sooooo soooo wrong these fucking manufacturer websites are absolutely awful to crawl or parse and nowadays there are no proper FTP Servers or something else anymore you could use to get the information. Every subsite is little bit different...
While coding and literally brute forcing possible urls (there was some kind of vague pattern) we learned AGAIN to appreciate proper developed and designed websites. Especially by devs who may have some more usage scenarios in mind for their site than simple human clients.

So thank you to all of you awesome web developers who design proper websites and web tools!

All in all it took us 2 weeks to come up with a proper solution (by the way we are a smal team of 3 devs) which somewhat works reliable and can deal with site changes etc.

Should I switch to Chrome Headless/Puppeteer for webpage header+footer scraping or stick with Express+Cheerio?

How can I able to create a dynamic Site map for my website on 3 parameter
- Update on Daily Basis
- update on Monthly Basis
- update on a yearly basis

And the website for which I am asking is Study24x7, in this website on daily basis 100's of people update content, and for better crawling, I need site map of these content