Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
![](/static/devrant/img/pipeless-devrant-banner-white.png)
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "all the cores"
-
The stupid stories of how I was able to break my schools network just to get better internet, as well as more ridiculous fun. XD
1st year:
It was my freshman year in college. The internet sucked really, really, really badly! Too many people were clearly using it. I had to find another way to remedy this. Upon some further research through Google I found out that one can in fact turn their computer into a router. Now what’s interesting about this network is that it only works with computers by downloading the necessary software that this network provides for you. Some weird software that actually looks through your computer and makes sure it’s ok to be added to the network. Unfortunately, routers can’t download and install that software, thus no internet… but a PC that can be changed into a router itself is a different story. I found that I can download the software check the PC and then turn on my Router feature. Viola, personal fast internet connected directly into the wall. No more sharing a single shitty router!
2nd year:
This was about the year when bitcoin mining was becoming a thing, and everyone was in on it. My shitty computer couldn’t possibly pull off mining for bitcoins. I needed something faster. How I found out that I could use my schools servers was merely an accident.
I had been installing the software on every possible PC I owned, but alas all my PC’s were just not fast enough. I decided to try it on the RDS server. It worked; the command window was pumping out coins! What I came to find out was that the RDS server had 36 cores. This thing was a beast! And it made sense that it could actually pull off mining for bitcoins. A couple nights later I signed in remotely to the RDS server. I created a macro that would continuously move my mouse around in the Remote desktop screen to keep my session alive at all times, and then I’d start my bitcoin mining operation. The following morning I wake up and my session was gone. How sad I thought. I quickly try to remote back in to see what I had collected. “Error, could not connect”. Weird… this usually never happens, maybe I did the remoting wrong. I went to my schools website to do some research on my remoting problem. It was down. In fact, everything was down… I come to find out that I had accidentally shut down the schools network because of my mining operation. I wasn’t found out, but I haven’t done any mining since then.
3rd year:
As an engineering student I found out that all engineering students get access to the school’s VPN. Cool, it is technically used to get around some wonky issues with remoting into the RDS servers. What I come to find out, after messing around with it frequently, is that I can actually use the VPN against the screwed up security on the network. Remember, how I told you that a program has to be downloaded and then one can be accepted into the network? Well, I was able to bypass all of that, simply by using the school’s VPN against itself… How dense does one have to be to not have patched that one?
4th year:
It was another programming day, and I needed access to my phones memory. Using some specially made apps I could easily connect to my phone from my computer and continue my work. But what I found out was that I could in fact travel around in the network. I discovered that I can, in fact, access my phone through the network from anywhere. What resulted was the discovery that the network scales the entirety of the school. I discovered that if I left my phone down in the engineering building and then went north to the biology building, I could still continue to access it. This seems like a very fatal flaw. My idea is to hook up a webcam to a robot and remotely controlling it from the RDS servers and having this little robot go to my classes for me.
What crazy shit have you done at your University?9 -
I have a whatsapp group with my friends, none of which are techies. A while ago one of them was looking for a phone to buy, so he started looking at models, specs and all that, but got pretty confused and asked a pretty well-informed question to the group:
"Guys, what is that quad core thing?
And what is a RAM? Is it something like the processor of the phone or what? "
OK, pretty typical stuff up until this point. The guy knows nothing about this sort of things, I wouldn't criticize him or insult him or anything like that. No, that's not the problem. The problem is the person that responded to him. This... This melted my brain so much I will never forget:
"Don't worry about that, you only have to look at how many gigahertz does the processor run at. Don't worry about the number of cores or ram. The GHz are the result of the amount of ram and cores, so the more the gigahertz, the better the phone."
PD: "Also take a look at how many megapixels does the camera have if you want to take photos".
Some people just talk out of their ass and pretend like they're experts on any topic they've read about for 5 minutes on the Internet7 -
This rant is particularly directed at web designers, front-end developers. If you match that, please do take a few minutes to read it, and read it once again.
Web 2.0. It's something that I hate. Particularly because the directive amongst webdesigners seems to be "client has plenty of resources anyway, and if they don't, they'll buy more anyway". I'd like to debunk that with an analogy that I've been thinking about for a while.
I've got one server in my home, with 8GB of RAM, 4 cores and ~4TB of storage. On it I'm running Proxmox, which is currently using about 4GB of RAM for about a dozen VM's and LXC containers. The VM's take the most RAM by far, while the LXC's are just glorified chroots (which nonetheless I find very intriguing due to their ability to run unprivileged). Average LXC takes just 60MB RAM, the amount for an init, the shell and the service(s) running in this LXC. Just like a chroot, but better.
On that host I expect to be able to run about 20-30 guests at this rate. On 4 cores and 8GB RAM. More extensive migration to LXC will improve this number over time. However, I'd like to go further. Once I've been able to build a Linux which was just a kernel and busybox, backed by the musl C library. The thing consumed only 13MB of RAM, which was a VM with its whole 13MB of RAM consumption being dedicated entirely to the kernel. I could probably optimize it further with modularization, but at the time I didn't due to its experimental nature. On a chroot, the kernel of the host is used, meaning that said setup in a chroot would border near the kB's of RAM consumption. The busybox shell would be its most important RAM consumer, which is negligible.
I don't want to settle with 20-30 VM's. I want to settle with hundreds or even thousands of LXC's on 8GB of RAM, as I've seen first-hand with my own builds that it's possible. That's something that's very important in webdesign. Browsers aren't all that different. More often than not, your website will share its resources with about 50-100 other tabs, because users forget to close their old tabs, are power users, looking things up on Stack Overflow, or whatever. Therefore that 8GB of RAM now reduces itself to about 80MB only. And then you've got modern web browsers which allocate their own process for each tab (at a certain amount, it seems to be limited at about 20-30 processes, but still).. and all of its memory required to render yours is duplicated into your designated 80MB. Let's say that 10MB is available for the website at most. This is a very liberal amount for a webserver to deal with per request, so let's stick with that, although in reality it'd probably be less.
10MB, the available RAM for the website you're trying to show. Of course, the total RAM of the user is comparatively huge, but your own chunk is much smaller than that. Optimization is key. Does your website really need that amount? In third-world countries where the internet bandwidth is still in the order of kB/s, 10MB is *very* liberal. Back in 2014 when I got into technology and webdesign, there was this rule of thumb that 7 seconds is usually when visitors click away. That'd translate into.. let's say, 10kB/s for third-world countries? 7 seconds makes that 70kB of available network bandwidth.
Web 2.0, taking 30+ seconds to load a web page, even on a broadband connection? Totally ridiculous. Make your website as fast as it can be, after all you're playing along with 50-100 other tabs. The faster, the better. The more lightweight, the better. If at all possible, please pursue this goal and make the Web a better place. Efficiency matters.9 -
Okay, story time.
Back during 2016, I decided to do a little experiment to test the viability of multithreading in a JavaScript server stack, and I'm not talking about the Node.js way of queuing I/O on background threads, or about WebWorkers that box and convert your arguments to JSON and back during a simple call across two JS contexts.
I'm talking about JavaScript code running concurrently on all cores. I'm talking about replacing the god-awful single-threaded event loop of ECMAScript – the biggest bottleneck in software history – with an honest-to-god, lock-free thread-pool scheduler that executes JS code in parallel, on all cores.
I'm talking about concurrent access to shared mutable state – a big, rightfully-hated mess when done badly – in JavaScript.
This rant is about the many mistakes I made at the time, specifically the biggest – but not the first – of which: publishing some preliminary results very early on.
Every time I showed my work to a JavaScript developer, I'd get negative feedback. Like, unjustified hatred and immediate denial, or outright rejection of the entire concept. Some were even adamantly trying to discourage me from this project.
So I posted a sarcastic question to the Software Engineering Stack Exchange, which was originally worded differently to reflect my frustration, but was later edited by mods to be more serious.
You can see the responses for yourself here: https://goo.gl/poHKpK
Most of the serious answers were along the lines of "multithreading is hard". The top voted response started with this statement: "1) Multithreading is extremely hard, and unfortunately the way you've presented this idea so far implies you're severely underestimating how hard it is."
While I'll admit that my presentation was initially lacking, I later made an entire page to explain the synchronisation mechanism in place, and you can read more about it here, if you're interested:
http://nexusjs.com/architecture/
But what really shocked me was that I had never understood the mindset that all the naysayers adopted until I read that response.
Because the bottom-line of that entire response is an argument: an argument against change.
The average JavaScript developer doesn't want a multithreaded server platform for JavaScript because it means a change of the status quo.
And this is exactly why I started this project. I wanted a highly performant JavaScript platform for servers that's more suitable for real-time applications like transcoding, video streaming, and machine learning.
Nexus does not and will not hold your hand. It will not repeat Node's mistakes and give you nice ways to shoot yourself in the foot later, like `process.on('uncaughtException', ...)` for a catch-all global error handling solution.
No, an uncaught exception will be dealt with like any other self-respecting language: by not ignoring the problem and pretending it doesn't exist. If you write bad code, your program will crash, and you can't rectify a bug in your code by ignoring its presence entirely and using duct tape to scrape something together.
Back on the topic of multithreading, though. Multithreading is known to be hard, that's true. But how do you deal with a difficult solution? You simplify it and break it down, not just disregard it completely; because multithreading has its great advantages, too.
Like, how about we talk performance?
How about distributed algorithms that don't waste 40% of their computing power on agent communication and pointless overhead (like the serialisation/deserialisation of messages across the execution boundary for every single call)?
How about vertical scaling without forking the entire address space (and thus multiplying your application's memory consumption by the number of cores you wish to use)?
How about utilising logical CPUs to the fullest extent, and allowing them to execute JavaScript? Something that isn't even possible with the current model implemented by Node?
Some will say that the performance gains aren't worth the risk. That the possibility of race conditions and deadlocks aren't worth it.
That's the point of cooperative multithreading. It is a way to smartly work around these issues.
If you use promises, they will execute in parallel, to the best of the scheduler's abilities, and if you chain them then they will run consecutively as planned according to their dependency graph.
If your code doesn't access global variables or shared closure variables, or your promises only deal with their provided inputs without side-effects, then no contention will *ever* occur.
If you only read and never modify globals, no contention will ever occur.
Are you seeing the same trend I'm seeing?
Good JavaScript programming practices miraculously coincide with the best practices of thread-safety.
When someone says we shouldn't use multithreading because it's hard, do you know what I like to say to that?
"To multithread, you need a pair."18 -
Dad handed over this beast to me
Casio PB-100
RAM: 1KB
ROM: 10KB
CPU: 455KHz
Display: 12 characters Monochrome
It's hard to believe that someone used this to code back in those days. Suppose to have been one of the first personal handheld computers. And people here complain about having 32gb RAM with 8 cores shame on you all10 -
Prof: So yeah this is going to be difficult. We're going to make the scalable math library. Then we have to make a functional finite elements library using that. Then make a multiphysics engine using that library. This could easily take your entire PhD. Are you prepared for that?
Me: May I show you something?
Prof: Sure, sure.
Me, showing him: We can use moose to code in the multiphysics. It's built atop libmesh for the finite elements. Which can be built with a petsc backend. Which we can run on GPUs and CPUs, up to 200k cores. All of this has been done for us. This project will, at worst, take a couple months.
Prof: ...
Guys, libraries. Fucking. Libraries. Holy fucking shit.5 -
Doing linguistic research where I need to parse 2000 files of a total of 36 GB. Since we are using python the first thing I thought was to implement multi threading. Now I changed the total runtime from three days to like one day and a half. But then when I checked the activity monitor I saw only 20 percent of the CPU usage. After a searching process I started to understand how multi threading and multi processing works. Moral of the story: if you want to ping a website till they block you or do easy tasks that will not use up all power of one core, do multi thrading. If you need to do something complicated that can easily consume all the powers of a single CPU core, split up the work and do multi processing. In my case, when I tried to grab information from a website, I did multi thrading since the work is easy and I really wanted to pin the website 16 times simultaneously but only have 4 cores. But when it come to text processing which a single file will take 80 percent of cpu, split it up and do multi processing.
This is just a post for those who are confused with when to use which.12 -
I know it is 1 week too late but i dont care. Im aware that my workbench looks like shit but in reality it is a "creative mess"
@Condor Do you envy me now? I have 1054z that is hacked and has all its features unlocked! I have desoldering station and crappy (very crappy) soldering station. In addition to that i got simple sine/square generator that goes to 1Mhz (old communist poland tech - indestructible and great)
Situation with my pc is complicated. My main pc (with the curved monitor)
-i5 3350p
-12 Gb or ram (2*4Gb and another 4 that friend gave me)
-radeon 7800hd (*RIP* cant play games anymore 😥)
-256Gb sata SSD
-2Tb WD blue HDD
In addition to that i have a laptop
i5 4 cores, 8Gb ram, 1Tb HDD, GTX1050
I use my lap for gaming now, i even connected that monitor as main lap.
(double monitors yay!) sometimes i use my old pc for minor things but i use rdp for that, it is great experience! (my lap displaying rdp from my old pc on second monitor that was my old pcs main one 😂 i find it a tiny bit funny)15 -
Client : We have performance issues on <this> page.
US : Testing with their data on : Xeon : All good, Core i7 : all good, Core i5 : all good, A smartphone : all good, core i5 UM version from 5 years ago (You know, 1.8Ghz, 2 cores, not 4) : all good.
US : Could you share your screen and show us ?
Client : Sure ! Shares his screen. Chrome is already opened. Refreshes the page, after 15 seconds, chrome : Would you like to stop script on this page ?”
US with mic on mute : “WTF?! It cannot happen here. It’s only a list of about 500 rows!”.
US : “Hello, could you open a tasks manager?”
Client : “ Sure, but I need to do it through here”
Some random VID interface popping up. Opens tasks manager with a beautiful “48 vCPUs detected”.
I have no idea how their virtualization is settled, but it seems to be messed up. Tomorrow going to be fun. I have a call at 9am to explain them their problem. Their IT will hate me forever.
If anyone has tips on Chrome through virtualization, please tell me.
EDIT : Yes 48. Not 4-8. It's not an error6 -
I've never used Windows in my day-to-day life. No kidding.
When I got my father's first computer, I used an old distribution called BBC Linux. I didn't have any computer knowledge, it was my first contact with a computer, so I went to a friend's house and asked for a CD to install on my computer. I don't know if this friend ended up making a "gotcha" and thought I'd give up, but I just read the manuals and fell in love. That was year 2000.
Then I used Conectiva Linux, then I went to Red Hat 9, then Slackware, then in 2007 I started using Solaris. And I stayed on Solaris (Solaris 10, Solaris Nevada and OpenSolaris) until 2011.
In 2011 I bought a Mac. I stayed at Apple until 2020, when I couldn't stand Apple forcing me to buy new computers (I still don't understand how a 2011 iMac, i5 (4 Hyper Thread cores) with 16GB of RAM, 1TB SSD only runs up to High Sierra).
Then I bought a Dell. It came with Windows 10, the first thing I did was install WSL2. I could not stand it, the system is bad, sorry. I installed OpenSuse and have been using it for two years.
It's just that every day someone tells me "how can you use this"? "There is no alternative to Windows, do you want to be different?"
I know that my story was the reverse of the "mainstream", so I'm going to talk about my vision of Windows, that in my brain it is actually the "alternative".
- Having a file explorer without "tabs" in 2022 is unthinkable for me.
- I love terminal. And the Windows terminal is very limited. "ps ... | awk ... | xargs ..." is a must for me. "find ./ -name '...' -exec ..."... these things on Windows are totally "different" and have the "powershell way" while all other operating systems keep the same form. And cygwin is not an option. As Wine for serious work is also not.
- Dragging a file into the terminal, and having it write its path, is so natural, that when Windows didn't do it, I was dismayed.
- I've always used StarOffice, OpenOffice and now LibreOffice. All the people in my story received my documents and reports as a PDF and no one complained. Until a coworker saw me editing in LibreOffice and said "oh I want it in word format". As long as he didn't know, everything was fine, right?
- Windows is paid. And is there advertising? I don't understand. And I refuse. If you want to display advertising, then excuse me. I have no problem paying, I'm not an opensource shiite. It's just that paying and not working bothers me much more than an opensource that I can fix or expect a fix knowing the good will of the people involved.
- Hyper-V is a joke. QEMU/KVM is better, and Bhyve on FreeBSD which is a very young project, is already a million times better than Hyper-V.
- Developing in C/C++ for Windows is only possible in two ways: Either you've always lived in Windows and your brain is conditioned, or you compile with MSYS2 (CLang or GCC).
- There is no significant evolution of the windows desktop since 95.
- Multiple workspace support with multiple monitors, not ready. It's another joke.
- REGEDIT does not need any comment.
- The system loses performance over time. I still don't know how Windows achieves this.
- I've seen people complain about desktop fragmentation on Unix and Linux. Many DEs end up leaving applications with different themes (like running a Qt application in Gnome and GTK in KDE), but to be quite honest, the lack of Windows standard bothered me much more. Even Microsoft's own software is completely different: Control Panel, Calculator, Paint and Office, To-Do, and Settings, have horrible style differences and look-and-feel fragmentation.
- Dark mode has not been implemented. It's another joke. Many applications are white while everything else is dark. Sorry, even on Linux which is a mess, this has been resolved. And well resolved.
- NTFS? Serious?
- C:, D:.. It doesn't convince me since DOS.
- Bloatware.
- News "biased" in the search bar is a lack of respect for those who use the computer to work.
And that. For me, Windows is the alternative operating system. I can't take Windows seriously, for me it's an experimental one like Haiku or ReactOS. It's good to play.
About market share, it doesn't convince me to use it. But convinces me to sell. I've always developed applications to run on Windows. And when I need it, I turn on a VM to compile the project. But in everyday life? Impractical.15 -
God damn it lastpass, how the hell do you get a memory leak and an infinite loop in a fucking browser extension?! Using 7GB of RAM and all 8 cores @ 3,4GHz!!9
-
So I'm writing some multithreaded shit in C that is supposed to work cross-platform. MingW has Posix threads for Windows, so that saved already half of the platform dependency. The other half was that these threads need to run external programs.
Well, there's system(), right? Uhm yes, but it sucks. It's incredibly slow on Windows, and it looks like you can have only one system() call ongoing at the same time. Which kinda defeats the multithreaded driver. Ok, but there's CreateProcessA(), and that doesn't suck.
Fine, now for Linux. The fork/exec hack is quite ugly, but it works and is even fast. Just never use fork() without immediate exec(). First try under Cygwin... crap I fork bombed my system! What is this shit? Ah I fucked up the path names so that the external executable couldn't be run.
Lesson learnt: put an exit() right after the exec() in the path for child process. Should never be reached, but if it goes there, the exit() at least prevents a fork bomb.
Well yeah, sort of works under Cygwin, but only with up to 3 threads. Beyond that, it seems like fork() at some point gives two processes the same PID, and then shit hangs.
Even slapping a mutex around the fork and releasing it only in the parent process didn't help. Fork in Cygwin is like a fork in the ass. posix_spawn() should work better because it can be mapped more easily to the Windows model, but still no dice.
OK, testing under real Linux. Yeah, no issues with that one! But instead, I get some obscure "free(): invalid size" abort. What the fuck would that even mean?! Checking my free() calls: all fine.
Time to fire up GDB in the terminal! Put a catch on the abort signal, mh got just hex data. Shit I forgot to compile with -O0 and -g. Next try. Backtrace shows the full call trace, back to the originating line in my program - which is fclose() on a file.
Ahhh I remember! Under Linux, fclosing a file that is already closed makes the program crash. So probably I was closing it twice. Checking back.. yeah that's where it was.
Shit runs fast on several cores now!8 -
School gave me 3 DigitalOcean droplets to try out Kubernetes in the cloud, awesome!
Wrote an Ansible script to not only simply install docker and add users but also add kubernetes, nice!
Oh wait, error?! Well I should've known this wasn't going to be easy... ah well no problem. Let's see... Ansible is cryptic as always, it can't connect to the API server? Is it even running?
Let's ssh to the master, ah nothing is running, great. Let's try out kubeadm init and see what happens, oh gosh, my Docker version has not been validated! No problem, let's just downgrade!
How do I do that? Oh I know, change the version in the role! Wait that version doesn't exit? Let's travel to Docker's website and see what versions exist of docker-ce, oh I see, it needs a subversion, no problem.
Oh that errors too? Wait then what... Oh I need a ~ and a ubuntu and a 0 somewhere, my mistake!
Let's run it again! Fails!
Same ssh process, oh wait...
Oh god no...
Kubernetes requires 2 cores and these things only have 1...
Welp, time to ask the teachers to resize my droplet by a small amount tomorrow, hopefully I'll get a new error!
----------------------------------------------
My adventure so far with Kubernetes. I'm not installing it for any serious/prod reason, just for educational purposes. K8s seems like 'endgame' to me, like one of the 'big guys' that big enterprises use so I'm eager to throw stuff at a droplet and see what happens.
Going further down the rabbit hole tomorrow!
Wish me luck :3
(And yes, I could've figured this all out beforehand with documentation, but this is more fun in my opinion)8 -
Finally, I can play around with a proper server.
HP ProLiant DL380 G6 = dual 8-core Xenons @ 2.4GHz with 32GB RAM and 12TB / RAID1-0 of WD Purples (we happened to have them for some reason).
Already pissed at HP because they don't support JBOD and already pissed at myself for using CentOS, but other than that, enjoying the hell out of it!
And it's ALL MINE! ... Well, technically it's the org's, but it won't go into production for half a year and I'm the only one with the root access so, for now, it's MINE! 😅13 -
Me: *pulls down the quick settings drawer and taps Bluetooth*
phone: *forgets everything about having multiple CPU cores and threads - the process responsible for all user input and drawing the ui grinds to a halt*
me: *sighs* oh, fuck, not this shit again *sets phone down*
*several minutes pass*
*watchdog decides that the UI is stuck and kills it, forcing the phone to soft-reboot*
phone: *boots up after a minute of loading*
me: *checks if Bluetooth is on*
*Bluetooth still off*
me: *tries to tap Bluetooth again*
*the procedure repeats*11 -
Can someone help me understand?
I subscribed to a nifty IT-releated magazine, and on its back, there's an ad for "Dedicated root server hosting", nothing unusual at a first glance, but after I read the issue, I decided to humor them and see what it is that they offered, and... It just... Doesn't make sense to me!
An ad for "Dedicated Root Server" - What is a dedicated root server first of all? Root servers of any infrastructure sound pretty important.
But, the ad also boasts "High speed performance with the new Intel Core i9-9900K octa-core processor", that's the first weird thing.
Why would anyone responsible enough want to put an i9 into a highly-reliable root server, when the thing doesn't even support ECC? Also, come on, octa-core isn't much, I deal with servers that have anywhere between 2 and 24 cores. 8 isn't exactly a win, even if it has a higher per-core clock.
Oh, also, further down the ad has a list of, seeming, advantages/specs of the servers, they proclaim that the CPU "incl. Hyper-Threading-Technology"... Isn't that... Standard when it comes to servers? I have never seen a server without hyperthreading so far at my job.
"64 GBs of DDR4 RAM" - Fair enough, 64 gigs is a good amount, but... Again, its not ECC, something I would never put into a server.
"2 x 8 TB SATA Enterprise Hard Drive 7200 rpm" - Heh, "enterprise hard drive", another cheap marketing word, would impress me more if they mentioned an actual brand/model, but I'll bite, and say that at least the 7200 rpm is better than I expected.
"100 GBs of Backup Space" - That's... Really, really little. I've dealt with clients who's single database backup is larger than that. Especially with 2x8 TB HDD (Even accounting for software raids on top)
This one cracks me up - "Traffic unlimited"
Whaaaat?! You are not gonna give me a limit to the total transferred traffic to the internet for my server in your data center? Oh, how generous of you, only, the other case would make the server just an expensive paperweight! I thought this ad was for semi-professionals at least, so why mention traffic, and not bandwidth, the thing that matters much more when it comes to servers? How big of a bandwidth do I get? Don't tell me you use dialup for your "Dedicated Root Server"s!
"Location Germany or Finland" - Fair enough, geolocation can matter when it comes to latency.
"No minimum contract" - Oooh, how kiiiind of you, again, you are not gonna charge me extra for using the server only as long as I pay? How nice!
"Setup Fee £60" - I guess, fair enough, the server is not gonna set itself up, only...
The whole ad is for "monthly from £55.50", that's quite the large fee for setup.
Oh, and a cherry on top, the tiny print on the bottom mentions: "All prices exclude VAT and are a subject to..." blah blah blah.
Really? I thought that this sort of almost customer deceipt is present only in the common people's sphere!
I must say, there's being unimpressed, and then... There's this. Why, just... Why? Anyone understands this? Because I don't...12 -
We'd just finished a refactor of the gRPC strategy. Upgraded all the containers and services to .Net core 3, pushed a number of perf changes to the base layer and a custom adaptive thread scheduler with a heuristic analyzer to adjust between various strategies.
Went from 1.7M requests/s on 4 cores and 8gb ram to almost 8M requests/s on the same, ended up having to split everything out distributed 2 core instances because we were bottlenecking against 10gb/e bandwidth in AWS.2 -
Is your code green?
I've been thinking a lot about this for the past year. There was recently an article on this on slashdot.
I like optimising things to a reasonable degree and avoid bloat. What are some signs of code that isn't green?
* Use of technology that says its fast without real expert review and measurement. Lots of tech out their claims to be fast but actually isn't or is doing so by saturation resources while being inefficient.
* It uses caching. Many might find that counter intuitive. In technology it is surprisingly common to see people scale or cache rather than directly fixing the thing that's watt expensive which is compounded when the cache has weak coverage.
* It uses scaling. Originally scaling was a last resort. The reason is simple, it introduces excessive complexity. Today it's common to see people scale things rather than make them efficient. You end up needing ten instances when a bit of skill could bring you down to one which could scale as well but likely wont need to.
* It uses a non-trivial framework. Frameworks are rarely fast. Most will fall in the range of ten to a thousand times slower in terms of CPU usage. Memory bloat may also force the need for more instances. Frameworks written on already slow high level languages may be especially bad.
* Lacks optimisations for obvious bottlenecks.
* It runs slowly.
* It lacks even basic resource usage measurement.
Unfortunately smells are not enough on their own but are a start. Real measurement and expert review is always the only way to get an idea of if your code is reasonably green.
I find it not uncommon to see things require tens to hundreds to thousands of resources than needed if not more.
In terms of cycles that can be the difference between needing a single core and a thousand cores.
This is common in the industry but it's not because people didn't write everything in assembly. It's usually leaning toward the extreme opposite.
Optimisations are often easy and don't require writing code in binary. In fact the resulting code is often simpler. Excess complexity and inefficient code tend to go hand in hand. Sometimes a code cleaning service is all you need to enhance your green.
I once rewrote a data parsing library that had to parse a hundred MB and was a performance hotspot into C from an interpreted language. I measured it and the results were good. It had been optimised as much as possible in the interpreted version but way still 50 times faster minimum in C.
I recently stumbled upon someone's attempt to do the same and I was able to optimise the interpreted version in five minutes to be twice as fast as the C++ version.
I see opportunity to optimise everywhere in software. A billion KG CO2 could be saved easy if a few green code shops popped up. It's also often a net win. Faster software, lower costs, lower management burden... I'm thinking of starting a consultancy.
The problem is after witnessing the likes of Greta Thunberg then if that's what the next generation has in store then as far as I'm concerned the world can fucking burn and her generation along with it.6 -
At the institute I did my PhD everyone had to take some role apart from research to keep the infrastructure running. My part was admin for the Linux workstations and supporting the admin of the calculation cluster we had (about 11 machines with 8 cores each... hot shit at the time).
At some point the university had some euros of budget left that had to be spent so the institute decided to buy a shiny new NAS system for the cluster.
I wasn't really involved with the stuff, I was just the replacement admin so everything was handled by the main admin.
A few months on and the cluster starts behaving ... weird. Huge CPU loads, lots of network traffic. No one really knows what's going on. At some point I discover a process on one of the compute nodes that apparently receives commands from an IRC server in the UK... OK code red, we've been hacked.
First thing we needed to find out was how they had broken in, so we looked at the logs of the compute nodes. There was nothing obvious, but the fact that each compute node had its own public IP address and was reachable from all over the world certainly didn't help.
A few hours of poking around not really knowing what I'm looking for, I resort to a TCPDUMP to find whether there is any actor on the network that I might have overlooked. And indeed I found an IP adress that I couldn't match with any of the machines.
Long story short: It was the new NAS box. Our main admin didn't care about the new box, because it was set up by an external company. The guy from the external company didn't care, because he thought he was working on a compute cluster that is sealed off behind some uber-restrictive firewall.
So our shiny new NAS system, filled to the brink with confidential research data, (and also as it turns out a lot of login credentials) was sitting there with its quaint little default config and a DHCP-assigned public IP adress, waiting for the next best rookie hacker to try U:admin/P:admin to take it over.
Looking back this could have gotten a lot worse and we were extremely lucky that these guys either didn't know what they had there or didn't care. -
Every time I think I've seen the worst there's someone to prove me wrong...
GRANT ALL ON ALL TABLES/SEQUENCES
To web user on production
if (x == 1) y = 1;
else y = x;
loop through a collection and get 'few' relations using ORM - 1000s of queries and not a single join - but don't worry, "The ITs will just add more RAM and some CPU cores to the server"
4th day off and I already miss this2 -
If you are feeling a little chilly in your office, try building a docker container with a Ruby install inside...
All four cores of this i7 are pegged and the air is keeping my coffee warm. -
I already wrote a rant about this yesterday, but since I'm a sysadmin trying to convert to dev.. I dunno, maybe it's not a bad idea to muddy the waters a bit and talk about why not to be a sysadmin.
Personally I think it's that the perceived barrier to entry is just too high, while it isn't. You don't need a huge Ceph cluster and massive servers when you're just starting out. Why overbuild an appliance like that if it's gonna start out at maybe 5 requests a minute?
Let's take an example - DNS servers! So there's been this guy on the bind-users mailing list asking how to set up a DNS server on 2 public servers, along with a website. Nothing special I guess - you can read the thread here: https://0x0.st/ZY-d. Aside from the question being quite confusing, there was advice to read RFC's, get a book, read the BIND ARM, etc etc. And the person to deny this? No one less than Stephane Bortzmeyer, one of the people who works for nic.fr (so he maintains the .fr TLD) and wrote some of those RFC's as part of the DNSOP working group in the IETF. As for valid reasons to set up a DNS server? Could just be to learn how the DNS works, or hell even for fun. As far as professional DNS servers go.. this (https://0x0.st/ZYo9) is the nugget that powers the K root server, one of the 13 root servers that power the root zone of the internet, aka the zone apex. 2 RJ45 connections, and a console connection. The reason why this is possible is the massive recursor networks that ISP's, Google DNS, Cloudflare DNS, Quad9, etc etc provide. Point is, you don't need huge infrastructure to run a server!
Or maybe your business needs email. How many thousands of emails per second are you gonna need to build your mail server against? How many millions will you need to store? If your business has 10 employees and all of those manage about 10k emails total.. well that's easy, 100k emails total. Per second? Hundreds of emails per second per employee? Haha, of course not. Maybe you'll see an email a minute at most. That is not to say that all email services are like this - it is true that ISP's who offer email to their customers, and especially providers like Microsoft and Google do need massive mail servers that can handle thousands of emails per second. But you are not Microsoft or Google. So yeah, focus on the parts of email that are actually hard.. and there is plenty.
Among sysadmins you have this distinction between "professional" sysadmins and homelabbers. I don't mind the distinction itself but I think both augment each other. If you've started out by jumping into a heap of legacy at an established company, you will have plenty of resources, immediately high complexity, and probably a clusterfuck right away. But you will have massive amounts of resources. If you start out with a homelab, you will have not many resources, small workloads, and something completely new for you to build and learn with. And when running a server like that, you'll probably find that the resources required are quite small, to provide you with your new services. My DHCP servers take 12MB memory each. My DNS servers hover around the 40MB mark. The mail server.. to be fair that one consumes around 150. But if you'd hear the people saying that you need huge servers.. omg you need at least a TB of RAM on your server and 72 cores, massive disks and Ceph!1!
No you don't. All that does is scaring people away and creating a toxic environment for everyone. Stop it.1 -
- All the 6 cores at 100ºC ~25% of the time
- Trying to figure out wtf
- aahhh, it's my self-service automation script launching ~300 curls in parallel every few seconds to monitor the environment! I guess the temps are alright then...
Moral: even the devil is not that bad when you know the whole context.7 -
Question: What was the worst mistake you made in Linux?
So... Because I've finally upgraded my PC (rip money on bank account) I can now run a VM with Linux all the time that isn't slow as a snail.
I installed Linux mint, with 4Gb of Ram and 6 cores, and it runs like a brize, while I play on windows and stuff. BTW I'll be using the VM for programming stuff, since I'm finally at home (homesick because of burn out), when I'm better I'll finally have the patience and memory to learn new stuff and get my projects up and going.
And because I've never really used Linux I'm watching YouTube videos about Linux, and found a Perl I've watched before, #Linux Sucks
And It's great... I get so many laughs, but also, learn stuff I didn't know, like, how Linux Pros make mistakes that Windows users can't even do, like breaking the OS.
So... I would love to know, what was the worst mistakes you ever done on Linux? How did you brake you're system?
BTW this would also be great for noobs like me to not make them... I hope. Since I'll be moving full Linux when I'm comfortable.
BTW @dfox this would be a great wk ...18 -
Android Studio is by far worst development environment I've ever seen. Slow as fuck, messed up and cumbersome, not much better than eclipse. Menus are hilariously fucked up. On top of that, clear new setup (on Windows) may as well melt up your PC if you have that shit running overnight: after a weekend I came to the office and my MBP is fucking screaming and breathing hot air... I was WTF is that, and when I logged in, this fucking garbage android studio was taking 100% of all 8/16 cores ... WHAT A FUCKING HIDEOUS PILE OF SHIT FROM GOOGLE. Google: please kill this project, no need to improve or fix it, it's garbage. Start over.26
-
My 2009 ThinkStation has been running loud for a couple of years now, reaching temps in the 70s-80s at max load (40s idle) with stock settings, fans spinning like jet engines to try to cool it. Only recently (today) did I consider that maybe there's dust stuck somewhere, so I took the fan off my cooler and began the hunt there.
IT WAS FUCKING COVERED. Half of the fin intake area behind the fan was completely clogged up with dust. I was starving the poor thing and I never figured out why until now.
I deep-cleaned the entire system, and now it's running a gentoo install with all cores maxed running compiler tasks... Fans are much quieter, barely above idle noise, the main difference is just pitch of the noise because of the higher RPMs.
I dont have the firmware installed to measure temps, but I will update once I get that data.
Specs, in case anybody is curious:
> Xeon W3520 4c/8t 2.67Ghz
> 8GB DDR3 1066MHz
> RX 460 2GB (my student-budget upgrade)
> Dual 500/750GB HDD3 -
Am I the only one excited for the RTX 3000 series not because of gaming, but because of all those sweet, fast CUDA cores? Never mind framerates, that should make training those models way faster.4
-
If you have a 13900k and you have random BSODs and application crashes, use XTU and turn down your P-Cores in Performance Per Core Turning to 54x. There is some kind of bug in the turbo boost and going past the 5.4GHZ mark just doesn't work.
I've basically built this computer twice now. I replaced the motherboard because I destroyed it, replaced the RAM because I thought I had the wrong type, and now the processor which was the actual root of all my problems.4 -
If ever there was something like dark ages, then we are living it: which programmer still knows what his statement does through the whole software stack down to the CPU (and could also account for what a modern CPU does with all its cores, caches, pipelines and -1, -2, -3 rings). Piled higher and deeper. I know nothing. So it's like being a cargo cult sorcerer, conjuring copy&pasted spells from SO to invoke Bjarne's, Linus' or whomevers forlorn spirit, so this shit won't break.2
-
Is it normal, that certain cores on linux stay on 100% load whilst the others do nothing, this all while being idle.8
-
This night I had a fever and seen a weird dream. So, scientists discovered that our whole perspective is wrong, and the notion that anything is but a sum of its parts is now completely irrelevant. One can’t disassemble a thing and know how it works and what it consists of. Basically you want to build a computer, you buy all the parts you need, but you can’t assemble it anymore — it just doesn’t work, and nobody knows why. Same is true for cars, industrial machines, software and pretty much anything that have something to do with engineering.
So, earth went into riots, massive layoffs occured, world economy collapsed, and the forces of what used to be USA, China and Russia joined to tackle the problem. A research was started and we found out that we now have complexity cores (!) that distribute emergence (!!) in an ephemeral way (!!!) and that is what makes things work. Theory of New Complexity (!) emerged, and all the engineers were required to go back to universities to attend lectures about how complexity works and how to make things in that new reality7 -
In last episode of "How SystemD screwed me over", we talked about Systemd's PrivateTMP and how it stopped me from generating SSL certificates.
In today's episode - SystemD vs CGroups!
Mister Pottering and his team apparently felt that CGroups are underused (As they can be quite difficult to set up), and so decided to integrate them into SystemD by default. As well as to provide a friendlier interface to control their values.
One can read about these interactions in the manual page "systemd.resource-control"
All is cool so far. So what happened to me today?
Imagine you did a major system release upgrade of a production server, previously tested on a standalone server. This upgrade doesn't only upgrade the distribution however, it also includes the switch from SysVInit to SystemD. Still, everything went smooth before, nothing to worry now then, right? Wrong.
The test server was never properly stress-tested. This would prove to be an issue.
When the upgrade finishes, it is 4 AM. I am happy to go to bed at last. At 6 AM, however, I am woken up again as the server's webservices are unavailable, and the machine is under 100% CPU load. Weird, I check htop and see that Apache now eats up all 32 virtual cores. So I restart it, casting it off to some weird bug or something as the load returns to normal.
2 hours later, however, the same situation occurs. This time, I scour all the logs I can, and find something weird - Many mentions that Apache couldn't create a worker thread? That's weird.
Several hours of research and tinkering later, I found out the following:
1 - By default, all processes of a system that runs SystemD are part of several CGroups. One of these CGroups is the PID CGroup, meant to stop a runaway process from exhausting all PIDs/TIDs of a system.
This limit is, by default, set to a certain amount of the total available PIDs. If a process exhausts this limit, it can no longer perform operations like fork().
So now, I know the how and why, but how should I solve this? The sanest option would be to get a rough estimate of just how many threads the Apache webserver might need. This option, though, is harder, than apparent. I cannot just take the MaxRequestsWorkers number... The instance has roughly double the amount of threads already. The cause being, as I found out, the HTTP/2 module, which spawns additional threads that do not count towards this limit. So I have no idea what limit to set.
Or I could... Disable the limit for just the webserver via the TasksAccounting switch. I thought this would work. And it did seem to... Until I ran out of TIDs again - Although systemctl status apache2.service no longer reported the number of tasks or a task limit of the process, the PID CGroup stayed set to the previous limit. Later I found out that I can only really disable the Task Accounting for all the units of a given slice and its parents.
This, though, systemctl somewhat didn't make apparent (And I skimmed the manual, that part was my fault)
So... The only remaining option I had was to... Just set the limit to infinite. And that worked, at last.
It took me several hours to debug this issue. And I once again feel like uninstalling systemd again, in favor of sysvinit.
What did I learn? RTFM, carefully, everything is important, it is not enough to read *half* the paragraph of a given configuration option...
Oh, and apache + http/2 = huge TID sink. -
TLDR; My laptop's CPU temperature becomes +90 C when I compile stuff.
I have two different laptops that have the exact same configs and OS. I use one as a desktop and the other one is just for school stuff and homework. As you might think, the one that I use as a desktop is looking like a mini desktop. I was into DIY stuff while ago so I made a custom case and separate the LCD as well. Don't ask me why though.
Today I was working on a personal project that has relatively complex build config. Since the compilation always took 3 to 5 mins I went to the kitchen for some coffee. Bumm. My laptops fans are working in a way that one can think they're in the airport. Seriously. All 8 cores are +90 C when I checked them.
The next thing I did was compiling the same project on the other laptop which I used for school projects. It took like 20 mins to compile but the max temperature was like 50 C.
So, in the end, I'm still trying to find the reason for that behaviour.4 -
Turns out the reason the selection of RA cores on the PS3 sucks so much dick is that it has a half-complete OpenGL implementation.
Fuck it, we're doing it live. I've got CFW, i'll dynarec if I damn well please. If it won't compile i'll fucking make it compile, I don't give a fuck about "sanity restraints" and "needing OpenGL 2" and "LV1 required" I can fucking replace all of LV1 and LV2 if I fucking want FUCK YOU
The Cell processor is pretty beefy, I could prolly do it all-CPU if I wanted.8 -
I've been a user of NetBeans IDE for PHP for quite sometime. I use it both at work and at home for personal projects. Lately I haven't done many personal projects so I fired it up on my Mac only to find it utilising 100% of all my CPU cores. This has always been an issue with NetBeans for years and it just seems like they can't be bothered to fix it.
So now I'm on the search for a replacement. At work now I use PHPStorm, but the $200 yearly license for small personal projects is a bit much. So for the time being I'm going to be trying out Atom with the ide-php package. It's a shame really as NetBeans is one of the good IDEs out there, but it's such a resource hog that I'm not sure how people can do any work on it.4 -
Im ranting in progress of the issue so i dont get the urge to do any of the things not seem as acceptable to fix this issue.
Issue: yesterday i activated a device i havent had any (even prepaid) service on in years, and had a 'new'(to me) number assigned...
Today, after being sick so muting nuisances immediately for rest, i check, 3missed calls from the same, less spammy looking number. I havent use this number for even a txt code verification at all... aside from 1 call to comcast (for the blissful irony of seeing if its an option (they need to survey physically) since im suing my current isp who didnt take my VERY NICE and explictly required in their business t&c, refund for the issue's duration.. after months of tryjng to directly get a message (not using my not technically hacking expertise like just scrubbing for email formatting and popped up in their inbox (calling them is more frowned upon)...
Their conclusion as to "why" (they nvr solved the issue... dhcpv6 was in aggressive lease mode(no response per lease(NOT batches) of about 60 for about 20 devices which i ofc use my /28 static ipv4 block... not ipv6 (they also claimed there was no logs til i dug and found verbose, long history high/med high debug level logs in their prop. dev's gui... which they forced me to use, has 2 separate cores/stacks which is done for 1 reason only... constant simultaneous ipv4 and ipv6 (so ofc was auto enabled)...
Basically it was spamming do to a config issue with their scripts, and their WAN6 dev/script's config. Have found a single person who knows what ipv6 (or v4) or wan6 device actually means... their conclusion from multiple "specialist departments " ..."we dont support ipv6 so if u had issues caused by using something we dont support it's your fault... sooooo ludacris.
.... ok back to main point.
callback options
1 schedule a call back for "later"
2 dont schedule and hang up/try some other time
3. cancel callback and join the end of the cue(from previous message it told me a callback in 6-10m or lose your place in line and go to the end... hours later no call and they definitely have the number as it reiterated -.-
...
answer to wait in line>
experiencing extremely high wait time
>your current wait time 31-60m
2.5sec later.. let me connect you to a rep ...etc (identical as in callback options intro)
> your current wait time is 30sec
waiting nearly 25min whilst typing this.(i did make sweet potato stuff, propagated a rose, fed JSON some of his new, in closure buffet of things he previously never encounted and bought a literal ton of rubber mulch)40min to a rep 5more to solve (last guy at same position didnt know this option exited, despite me decribing it verbosely to him.
Everything the automated syst asks is about account numer... there is none ive never even had a burner that was at&t brand.
Wzf.3 -
Amazon prime days sale...
I find a Fire 7 for $30 instead of $50. I think that would be great to put books on. I am thinking Kindle is an Android type device. Even some searches for Android tablets bring up Kindles on Amazon and web.
I get my kindle and like it. I signed up for trial of Kindle Unlimited. There is almost no selection for Kindle Unlimited for technical books. So I think I can just put the Paktpub app on the Kindle. No app for Kindle. That is okay, I can just put the Play store on there. Technically you can, if you side load it, but it will stop functioning after a day. Not an officially licensed Android device so cannot use Google services.
At this point I am not happy with the Kindle. I got it to read technical books and the selection of technical books is poor. At least on Kindle Unlimited. So I start looking at tablets on Amazon.
I find that there is a serious price breakpoint on Android tablets (cannot get Paktpub app for Windows tablets). For $100 (US) they are not very good. At > $150 they start getting really good feature wise. I end up buying a Samsung tablet for $200. It has 2GB ram and 8 cores at 1.6GHz.
I have been using the tablet for a few days now and am happy with what I can do with it. Now I have to wonder if Kindle is actually an upsell product rather than a serious product. I might not have went for a $200 tablet unless I had not had issues with the Kindle. Not sure there. Amazon made out for both product sales as I just gave the Kindle to the kids.
In the end I am very happy. Paktpub has all the tech books I can handle at the moment. Will probably not consider Kindle Unlimited again. This tells me that competition is good in the book sector. Good for the end user.5 -
I hate the feeling when the processes maxing out all my cpu cores are processes I thought were long since terminated. I guess even when I rm -f I don't really let go and still have the tar.gz in the back of my mind somewhere, and somehow zcat pipes those seemingly tidy archives all over my cwd at the worst possible times like some systemd transient timer that I can't recall the syntax to check... This is when the shell becomes unresponsive and I can't cd away, or even ps aux | grep -i 'the bad thoughts' to get their pid to figure out why this is happening again. Is it really time to hold down the power button? I'm so afraid of loosing unsynced data, I'll wait a little longer...
-
Gah, I just received this Ubuntu 18.04 VM with 8 cores and 8 gigs of ram, and since it'll be a production server both serving public and "private" networks (yes, shout at me, but projects won't be about hosting sensitive information, I wouldn't put all that on one server), and I'm struggling between my options.
Docker, or not docker?
The server's main use is to host our growing blog and install Varnish, which will hog some ram after a while. I use Laradock for my dev projets, it's really easy to develop with it, but I am unsure if it fits a production environment with performance, security and traffic load in mind :(
I read Docker has stability issues (in 2016-2017), and can bring the machine down with it, I don't know if I should just install the software (nginx, apache, percona/mysql/maria) without "containerizing" it and go for it
I'm lost xD7 -
Fucking hate to explain basic shit to computer illiterate. Usually I don't mind, but right know I working on the project, want to automate one thing I need to do every morning, put two numbers to web page(I will explain details maybe in next rant). So I am only one who fix, buys computers, printer(for some problems I call for other repair man.). Generally speaking working as IT guy. Firm has like 50 computers, some of them has SCADA software. Some computers have Win 7, some win 8 and others win 10, can't upgrade those computers, not enough money(I can deal with this problem). And yes, computer buying is not the fastest, easiest thing too. Because is public firm, I need to do public buying(I don't know how to translate to english), and most of the time wins the lowest price, I am ok with that. But I can't on item specification write I want that model pc or it components. Example: I can't write I want intel processor, however I can write number of cores, frequency. But it's not that bad, usually i have template for all things I buy. One of the worst thing is this, our firm bought new bookkeeping software version, old version was using visual foxpro framework. Good thing I didn't initiate the purchase, because right know I would be jobless, not because I would be fired, but because our senior accountant would drive me crazy. In fact accountants drive me crazy, but I can handle it for now. As I wrote before our form has about 120 workers, major part of workers are old, like my parents age. (I am 28 btw. Mom is 55.). As you all know what happens if you say you work with computers. So our accountants are like 60 years old, got new program, don't know how to work with it, and they ask me how to do certain things. if I don't know how to I ask program's support, every question is like 90 Eur. So in short accountants expect I should know their work and how program works. If I try say something they don't like, they try to make my day hard. Next thing is our billing program. Man that worked before me done some payments import. And when I came everyone expect me to do that. Ok I did that because that people working with billing program would probably fuck it up. And I semi automated that, so I don't mind that much. Sometimes that program fucks up, like it happened yesterday, it send email invoices attachment without filename. Example: people got this attachment ".pdf"(no filename, only extension), And if you save it you need do OPEN WITH command and then select pdf reader or rename file (I don't know what easier). And surprise surprise our firm, customer support redirects all phone calls, emails to me. But I did explain to customer support what to say to people. Still they redirect it to me.
PS: This is my first job after school. I work as part time.
TL;DR Thinking my life, carrier choices. accountants are not the nicest people.8 -
Working all my cores in parallel; working on multiple projects at the same time without finishing any project.2