Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "extremely large files"
-
I fucking HATE all those extremely high level abstractions, IT IS ALMOST IMPOSSIBLE to find anything low level, especially for ARM... IT CANT BE THAT HARD TO JUST FUCKING FIND SOMETHING THAT DOES NUT USE 100000 HEADER FILES, and stupid large frameworks. I feel like everyone is fucking retarded, I want to learn the real stuff, but everything is bloated with high level stuff, and some kind of cult that gets a horny from using extremely easy bullcrap, that completely takes away the interesting parts of processors and embedded systems, IVE Been searching for days to FIND SOMETHING FUCKING USEFULL, even an MOTHERFUCKING 'LOW LEVEL' book GOES AND USE A BILLION HEADER FILES, and STUPID IDE's from which you learn absolutely nothing, IF i wanted to do nothing and learn nothing I WOULD USE ARDUINO IDE, but no i wont, I want to learn something, and I dont have access to university or anything, and it literally is impossible to find anything usefull, every idiot uses library's for everything, and builds their crap on frameworks as large as the mount everest.. Fuck me, why cant this be different ?13
-
! exactly dev
I'd ditched Windows and spent a while exploring the Linux ecosystem for content creation. And I have to say, it was not a nice experience.
As much as I respect the Linux mantra of "free as in freedom" and "you need to roll up your sleeves and figure out stuff on your own", it just isn't good enough for non-dev work. Sorry guys, but I need software that gets out of my way and at least does what it's supposed to do. I can't stand a horrible UI or delays and random crashes, which is exactly what happens with most things under Linux.
To replace my Windows workflow I used the following:
1. Windows -> elementaryOS (because Debian/Ubuntu repositories seem to have the best software support, and elementaryOS is the least horrible looking thing that supports that) and then Arch, because, well, Arch.
2. Blender + Maya -> Blender + Maya on Linux.
3. Reaper + FL Studio -> Ardour + LMMS.
4. Photoshop -> GIMP + Krita + Inkscape.
5. ZBrush -> nothing :(
As you can see, my use cases are pretty much all over the spectrum.
Firstly, installing and configuring stuff. A pleasure on Windows, an absolute pain on Linux. Everything just worked on Windows, I had to wrestle with library versions and patches and unstable audio layers (Linux audio just sucks, except for JACK) on Linux.
Out of these, Blender and Maya were the best experience. But even then, both would suffer from random crashes that just didn't happen on Windows.
Ardour is actually really nice when it works. Its use of JACK for routing makes it really really flexible, but it just isn't stable enough to depend on. LMMS is utter crap. I'm sorry, but I just hate the UI. Can't stand it.
GIMP, Krita, and Inkscape can't beat Photoshop, even when you consider them together. Adobe software workflow is just so much better and more intuitive.
Blender 3D sculpting is not bad, but it's nowhere as good as ZBrush.
Also, if you're a C++ dev like me, nothing beats Visual Studio 2017. Nothing. That IDE just blows everything else out of the water. Even VSCode. And it's not slow at all, it handled a fairly large project (PBRTv3) just fine on my Windows development VM. Yes, a VM.
So...I ditched Linux and went back to Windows, but I keep Linux as a VM for when I actually want to mess with Blender or Ardour. Or some dev stuff which Windows sucks at (which is becoming less frequent because of WSL).
Out of all the above, the only one I'd consider ready for production use would be Blender. Developers of open source software, please learn from Blender. Kickass UI and user friendly operation is extremely important, you can't make a random window with GTK buttons and text boxes and arcane config files and expect people to use it for serious work.
Also, Windows beats Linux hands down as an everyday OS. It's always been rock solid, if you take care of it properly (and that goes for any OS). Updates hardly take any time because I run it on a SSD. As for all the advertising and marketing bullshit, you can block a large amount of stuff. And for what can't be blocked, well, I just have to live with it, because the alternative is compromising on my creative output, which is too much for me.
I still run Linux on my server, though. And on my embedded devices (Pi, BeagleBone, etc.). It absolutely rocks there.
I realize that Linux software is not going to improve unless we do something about it, so I'll be contributing fixes and code (the joys of being a C++ dev, yay). Still, I feel that the platform and software as a whole is just not mature enough.18 -
Buffer usage for simple file operation in python.
What the code "should" do, was using I think open or write a stream with a specific buffer size.
Buffer size should be specific, as it was a stream of a multiple gigabyte file over a direct interlink network connection.
Which should have speed things up tremendously, due to fewer syscalls and the machine having beefy resources for a large buffer.
So far the theory.
In practical, the devs made one very very very very very very very very stupid error.
They used dicts for configurations... With extremely bad naming.
configuration = {}
buffer_size = configuration.get("buffering", int(DEFAULT_BUFFERING))
You might immediately guess what has happened here.
DEFAULT_BUFFERING was set to true, evaluating to 1.
Yeah. Writing in 1 byte size chunks results in enormous speed deficiency, as the system is basically bombing itself with syscalls per nanoseconds.
Kinda obvious when you look at it in the raw pure form.
But I guess you can imagine how configuration actually looked....
Wild. Pretty wild. It was the main dict, hard coded, I think 200 entries plus and of course it looked like my toilet after having an spicy food evening and eating too much....
What's even worse is that none made the connection to the buffer size.
This simple and trivial thing entertained us for 2-3 weeks because *drumrolls please* none of the devs tested with large files.
So as usual there was the deployment and then "the sudden miraculous it works totally slow, must be admin / it fault" game.
At some time it landed then on my desk as pretty much everyone who had to deal with it was confused and angry, for understandable reasons (blame game).
It took me and the admin / devs then a few days to track it down, as we really started at the entirely wrong end of the problem, the network...
So much joy for such a stupid thing.18 -
MTP is utter garbage and belongs to the technological hall of shame.
MTP (media transfer protocol, or, more accurately, MOST TERRIBLE PROTOCOL) sometimes spontaneously stops responding, causing Windows Explorer to show its green placebo progress bar inside the file path bar which never reaches the end, and sometimes to whiningly show "(not responding)" with that white layer of mist fading in. Sometimes lists files' dates as 1970-01-01 (which is the Unix epoch), sometimes shows former names of folders prior to being renamed, even after refreshing. I refer to them as "ghost folders". As well known, large directories load extremely slowly in MTP. A directory listing with one thousand files could take well over a minute to load. On mass storage and FTP? Three seconds at most. Sometimes, new files are not even listed until rebooting the smartphone!
Arguably, MTP "has" no bugs. It IS a bug. There is so much more wrong with it that it does not even fit into one post. Therefore it has to be expanded into the comments.
When moving files within an MTP device, MTP does not directly move the selected files, but creates a copy and then deletes the source file, causing both needless wear on the mobile device' flash memory and the loss of files' original date and time attribute. Sometimes, the simple act of renaming a file causes Windows Explorer to stop responding until unplugging the MTP device. It actually once unfreezed after more than half an hour where I did something else in the meantime, but come on, who likes to wait that long? Thankfully, this has not happened to me on Linux file managers such as Nemo yet.
When moving files out using MTP, Windows Explorer does not move and delete each selected file individually, but only deletes the whole selection after finishing the transfer. This means that if the process crashes, no space has been freed on the MTP device (usually a smartphone), and one will have to carefully sort out a mess of duplicates. Linux file managers thankfully delete the source files individually.
Also, for each file transferred from an MTP device onto a mass storage device, Windows has the strange behaviour of briefly creating a file on the target device with the size of the entire selection. It does not actually write that amount of data for each file, since it couldn't do so in this short time, but the current file is listed with that size in Windows Explorer. You can test this by refreshing the target directory shortly after starting a file transfer of multiple selected files originating from an MTP device. For example, when copying or moving out 01.MP4 to 10.MP4, while 01.MP4 is being written, it is listed with the file size of all 01.MP4 to 10.MP4 combined, on the target device, and the file actually exists with that size on the file system for a brief moment. The same happens with each file of the selection. This means that the target device needs almost twice the free space as the selection of files on the source MTP device to be able to accept the incoming files, since the last file, 10.MP4 in this example, temporarily has the total size of 01.MP4 to 10.MP4. This strange behaviour has been on Windows since at least Windows 7, presumably since Microsoft implemented MTP, and has still not been changed. Perhaps the goal is to reserve space on the target device? However, it reserves far too much space.
When transfering from MTP to a UDF file system, sometimes it fails to transfer ZIP files, and only copies the first few bytes. 208 or 74 bytes in my testing.
When transfering several thousand files, Windows Explorer also sometimes decides to quit and restart in midst of the transfer. Also, I sometimes move files out by loading a part of the directory listing in Windows Explorer and then hitting "Esc" because it would take too long to load the entire directory listing. It actually once assigned the wrong file names, which I noticed since file naming conflicts would occur where the source and target files with the same names would have different sizes and time stamps. Both files were intact, but the target file had the name of a different file. You'd think they would figure something like this out after two decades, but no. On Linux, the MTP directory listing is only shown after it is loaded in entirety. However, if the directory has too many files, it fails with an "libmtp: couldn't get object handles" error without listing anything.
Sometimes, a folder appears empty until refreshing one more time. Sometimes, copying a folder out causes a blank folder to be copied to the target. This is why on MTP, only a selection of files and never folders should be moved out, due to the risk of the folder being deleted without everything having been transferred completely.
(continued below)29 -
Wouldn't call it a feature. More like worst practice. Data manager (and my boss at the time) kept using our website as a way to host large files 3rd party vendors/partners could download instead of using one of the many secure transfer methods out there to send them data. This was sometimes extremely sensitive data. No authentication or security that I could find. I went ballistic on him after seeing that.
-
So a couple of months ago I had some stability issues which seems to have caused Baloo go crazy and create an 1.7 exabyte index file. It was apparently mainly empty as zfs compressed it down to 535MB
Today I spent some time trying to reproduce the "issue" and turns out that wasn't that hard.
So this little program running on FreeBSD with a compressed (lz4) zfs dataset creates an 1.9 Exabyte large file, nicely compressed down to 45KB :)
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/limits.h>
int main(int argc, char** argv) {
int fd = open("bigfile.lge", O_RDWR|O_CREAT, 0644);
for (int i = 0 ; i < 1000000000; i++) {
lseek(fd, INT_MAX, SEEK_CUR);
}
write(fd, " ",1);
close(fd);
}3