Manager: our file IO is slow, any suggestions to make it faster?

Code: multithread writing to a few hundred small (temp) files then single thread combine to one big file and delete the temp files.

Eyes: bleeding

Just redid a small work script in Golang to test it out.

Honestly, speed matters little for what it is, the original was in Python.

By heavens Golang is one ugly ass looking language.

I like using it tho, its easy to understand and performant for networking, file io and shit like that.

But man....shit is ugly to look at from a distance.

I do think that most langs look ugly af tho, so shit is allright.

Syntactic whitespace is still shit btw.

In fact I'm a sinful dev, so that I can't easily decide which one is worst. From indenting with tabs, or using nano instead of vim/emacs, to hardcoding database credentials on server, to many hacks and workarounds I use as actual "fixes" when the deadline is upon me and I've tried all I could. But it always led only to my own regret. For instance, my latest sin was that I prefered Debian over Arch and used proprietary graphic drivers to speed up my new setup. But ended up with a curse from St. Ignucius. (check my last rant)

But my worst sin probably goes to when I was "printf-debugging" some issue for a GSM controller on a raspberry pi. I forgot to remove one little print line and deployed the new "fixed" version. I didn't follow that project after that for like a month or so, when the client posted back the device and said that "it just doesn't work anymore". It seemed that raspbian didn't boot beacause the sd card was curroptted. I dd'ed through the card and I noticed that there are billions of lines of "DEBUG:: reading stream from 192.some.shitty.ip", took almost all over the 32G sdcard. Just as I suddenly remembered the cursed line I just added a month ago, I declared the sd card dead with no hesitation, dunce-commented the line (so the history would remember), implemented a time out for the thread containing it, setup a journald unit for my service and removed the redirection of process output to a log file, found a new sd card and installed everything again, and finally posted back the new "fix" to the client.

Moral: Never comfort yourself for the sins you have commited in the past kids, they certainly will come back to you. And also not to do any io especially write to a file on an SD card with ext fs, in a potentially infinite loop with no timeout.

P.S: I'd posted my last rant just before the new week rant last nigh. I really liked the St. Ignucius meme so decided to create a new one. He's very adorable :)

A few weeks ago, I was kept up until the wee hours of the morning trying to figure out how in the hell the Monty Hall problem works. After finally getting it (I'm slow, okay?), I decided to write a program to run simulations of it.

First incarnation of program took user input. User enters what door they choose (1, 2, or 3), then is told what door Monty opens, then given the decision of staying with the door they originally chose or switching, then informed how that worked out for them.

Second incarnation of program ran on a loop. At the start of each loop, a random door is picked for the user guess. Then the door Monty opens is calculated from the remaining doors (excludes user guess and prize door). Then user switches doors (choosing the door that was not their original door or the door Monty opened). At the end of each loop, if the door they switched to was the prize door, it would increment a win counter, else increment a loss counter. After running the loop 1000000000 times, it printed to console `You always switched doors, resulting in ${wins} wins and ${losses} losses`.

THEN I decided to write a variation to run a while loop on the outside of the loop to increase the number of total doors until the point where the decision to switch doors hurt more often than it helped. At this point, I decided to incorporate file I/O and write to a file rather than a console. And that was neat!

And then I decided it would be cool to go back to the three door variation, printing on each loop the original door, the door Monty opened, the door that was switched too, the result of the switch (win or lose) and what the prize door was.

But for the life of me, I couldn't seem to get the file to write properly. It would, like, always crash my terminal. I tried open + append, I tried append. I tried createWriteStream. Still just failure.

And then I changed it to an appendFileSync and happened to look at one of the files that I was writing to. "Huh, over a gig seems a lot."

"Well, how much are you writing each loop? Did you forget to keep in mind how many bytes that would be?"

TLDR: If you're going to write a program that's going to write data to a file on a loop, you might want to figure out how much it's going to end up writing .... before trying to run it. And running a loop 1000000000 times may be a little excessive.

*face palm*

Guys, I think I am addicted and I have problems... I tried to write a simple SudokuSolver program in C# (one file) but I ended up writing an ExtensionMethods class, a Puzzle class, a Solver class, an IO class, a Program class...

That's why I hate working on big side-projects for too long

The guy who wrote a ton of legacy code at my company apparently had no idea that you need to close a file stream. And also that IO may throw exceptions.

Wooo

Today I spent 9 hours trying to resolve an issue with .net core integration testing a project with soap services created using a third party soap library since .net core doesn't support soap anymore. And WCF is before my time.

The tests run in-process so that we can override services like the database, file storage, basically io settings but not code.

This morning I write the first test by creating a connected service reference to generate a service client. That way I don't need to worry about generating soap messages and keeping them in sync with the code.

I sent my first request and... Can't find endpoint.

3 hours later I learn via fiddler that a real request is being made. It's not using the virtual in-process server and http client, it's sending an actual network request that fiddler picks up, and of course that needs a real server accepting requests... Which I don't have.

So I start on MSDN. Please God help me. Nope. Nothing. Makes sense since soap is dead on .net core.

Now what? Nothing on the internet because above. Nothing in the third party soap library. Nothing. At this point I question of I have hit my wall as a developer.

Another 4 hours later I have reverse engineered the Microsoft code on GitHub and figured out that I am fucked. It's so hard to understand.

2 more hours later I have figured out a solution. It's pure filth..I hide it away in another tooling project and move all the filth to internal classes :D the equivalent of tidying your room as a kid by shoving it all under the bed. But fuck it.

My soap tests now use the correct http client with the virtual server. I am a magician.

I remember the day I used Linux to enforce size quotas on user directories oh what a day
I was surrounded by horrible monsters wearing manho suits that made them appear superficially human... though the sounds screeching forth from their many toothed mouths in whining jarring tones would suggest otherwise and I used this wonderful feature : you can call most of the file systems commands on files !

did a dd to create a disk image specified to the correct size made an fs inside and mounted it straight into the home directory adding entries to the fstab to auto mount and setting the io permissions
It was so wonderful and the little bastards refused to use the server !

Programmer looking for a new language

I have been a JavaScript developer for a few years now (non professionally) and I really like the language. I mainly program for execution with NodeJS rather than web, because I feel like I get more freedom (e.i. the ability to use a computer file system).

I normally never use other people's libraries and instead either write my own library/ies for the specific task or use an old one. I only ever use someone else’s if I need a quick frame work to test an idea, never for something I will actually use.
I prefer to work object / class orientated.

I have worked on distributed servers with NodeJS before, however trying to distribute a load across one computer across it's multiple threads has proved problematic due to the heavy delays of standard io transfer speeds.

Why do I want to switch?:
•Because JavaScript is not at all created with multithreading in mind, and pretty much any multithreading solution is a bodge and allot of the time it is more efficient to work single threaded.
•Also, I get the sense that JavaScript + NodeJS is not used often in the programming industry comparison to other languages like; ruby, python, and I don't want to get stuck in a nesh language of which would decrease my employment chances heavy.

Side Note: I have been working on a pet project to have a distributed database (made with nodejs), and so far, there are no language specific problems, but I feel like it would be more efficient if I used a programming language designed more to cater for multi threading.

This is just me throwing out my thoughts from the past few weeks.

edit: this is long

> Working on a C# project. its going well Its teaching me a lot about SQLite and file IO. I'm having a lot of fun with it, even the debugging as much I want to slam my head on the wall but I'm not asking for help so far and I'm very proud of myself because it feels so much better. like I don't mind asking for help but its so much more rewarding and I learn more from it.

> I need portfolio of software I can show off to employers and the current project I'm working on is the first programs in the portfolio. The place I want to apply to uses C#, but I still wanted a few other programs in other languages such as Python or JS just to show what I'm capable of.

> I was looking at what ASP.NET Core offers and it impresses the fuck out of me, and confuses me. The parts that confuse me, like for example the normal asp webapp is a very impressive hello world app. and it has so many different files and such but how or what do they expect me to add? how am I supposed to work with it? and if I delete any files I don't need (the premade js, bootstrap, jquery, html, and css) it produces errors because of the project files are pointing to those. and i know I can use the empty project (I do) but does that question my ability as a dev since I don't want to use it for my projects?

> On that note I love using Intellisense and debuggers and auto complete and I can go without them I just don't want to rely on them. idk I've just been a little more stressed these past few weeks.

Bloody fucking Android! Updates, updates and more updates! My development Nexus 5X won't allow me to sideload apps since it updated... Hello, printf debugging! Goodbye, profiler and debugger!

My hate for Android grows with each version after 4.0.$something... 2 was shit, I missed 3, 4 was OK, and since then it's going steeply down.

And don't get me started on Material Design...! Good luck figuring out what's a button and what's a label...

And what's up with the "let's keep all apps running all the time to save a few ms on start" philosophy!? Who thought that is a good idea!? Yeah, System.exit(0) works, but... Is it so hard to determine when it's not needed anymore (has no services running etc.)? Why should a web browser (for example) stay in memory after I quit? Minimize is a thing (Home button), why make it so confusing?

Another thing - feedback-less async tasks - why? I like to know when it is working in the background... How the hell am I supposed to find out if it is supposed to do this or if it is frozen?

And Android deciding to kill your process whenever it pleases without any callback... Happened to me once with an Activity in the foreground (no exceptions anywhere in my app, it just quit). How do you do IO properly? It seems you can't guarantee some file or socket or something that must be closed doesn't stay open (requiring to restart Bluetooth 'cause the socket wasn't closed, for example)...

One of the worst practices in programming is misusing exceptions to send messages.

This from the node manual for example:

> fsPromises.access(path[, mode])

> fsPromises.access('/etc/passwd', fs.constants.R_OK | fs.constants.W_OK)
> .then(() => console.log('can access'))
> .catch(() => console.error('cannot access'));

I keep seeing people doing this and it's exceptionally bad API design, excusing the pun.

This spec makes assumptions that not being able to access something is an error condition.

This is a mistaken assumption. It should return either true or false unless a genuine IO exception occurred.

It's using an exception to return a result. This is commonly seen with booleans and things that may or may not exist (using an exception instead of null or undefined).

If it returned a boolean then it would be up to me whether or not to throw an exception. They could also add a wrapper such as requireAccess for consistent error exceptions.

If I want to check that a file isn't accessible, for example for security then I need to wrap what would be a simple if statement with try catch all over the place. If I turn on my debugger and try to track any throw exception then they are false positives everywhere.

If I want to check ten files and only fail if none of them are accessible then again this function isn't suited.

I see this everywhere although it coming from a major library is a bit sad.

This may be because the underlying libraries are C which is a bit funky with error handling, there's at least a reason to sometimes squash errors and results together (IE, optimisation). I suspect the exception is being used because under the hood error codes are also used and it's trying to use throwing an exception to give the different codes but doesn't exist and bad permissions might not be an error condition or one requiring an exception.

Yet this is still the bane of my existence. Bad error handling everywhere including the other way around (things that should always be errors being warnings), in legacy code it's horrendous.

4 hours wasted due to a rather strange "scope poisoning" bug... it was really fucking strange.
Had to move the 5 lines of code, above where the error happened, to a private function.
The 5 lines just called an XLSX library which read a given file and returned an array. I don't know how but the thing somehow messed with my Soap client call one line below. The culprit must have been some IO resource handle which didn't get destroyed when in the same scope due to persisting object references..