I’ve been bashing my head against a project for the past 8 weeks. The project creates a PDF pulling data from multiple APIs, scrapes and private DBs and plots charts using Plotly. We built it with Python, wkhtmltopdf and Celery+Redis. The input is an excel with a list of up to 5 influencers to analyse and compare. Runs on demand on a Linux machine and each report takes around 20 mins to generate. The project has no unittests so the only way I can check everything works is by running a bunch of different inputs. Even though you test 10 inputs (taking you more than a day), there is a high chance something goes wrong on the 11th input. I’m thinking that the only way to fix this mess is to go back to the drawing board and plan yet another refactoring to add unittests everywhere. What do you guys think?

test driven development ftw
but you gotta change your mindset from how to make it work to how to break it. or find someone who thinks this way

Make the scraping and plotting to pdf part at least two things. That way you can take the scraping output and generate pdfs more often. Maybe also separate out the excel parsing part.

Also maybe cache some stuff if possible to make scraping repeately aster.

@LinusCDE @Sm4o about speed: do you compile your python?

@bad-frog You can do it. You can turn all files from .py to .pyc which is the some intermidate language and also created in the __pycache__/ directory.

You can then just do "python main.pyc" and even remove the py files completely.

Pros:
- May make reversing a bit harder (didn't find a python 3 decompiler afaik)
- Can catch some common syntax errors that may only have gotten caught when a components gets actually imported. For example in a ci system
- Maybe some initial speed boost.

Cons:
- Compatibility is probably really limited in that state. Probably only useful if you also know the python interpreter version
- Will probably not be faster when running longer

I used to do that for some work thing. Was more to check in ci and make reversing a bit harder. But we had our own firmware with python and everything anyway. Never went into production. I replaced it with rust since I hated the runtime errors that would creep up and it was only getting more complex.

@bad-frog Btw I made a script that recursed a directory, compiled all .py files to .pyc and remove all sources afterwards.

I used a builtin ast library or similar which has a compile function.

This seems promising as well though:
https://docs.python.org/3/library/...

@LinusCDE yeah, python really isnt the best for the wilder shenanigans.

id say write it in C or C++, but rust might be really better especially coming from a python background.

C# could be nice too i think. seems it has a simpler syntax too

but then de-assembling back into script? wth? didnt know it was a thing...

@LinusCDE also why not use shell?

@bad-frog The python case is really only some byte representation afaik.

The project involved a lot of hardware access as well as hosting an web server with an api. So shell script would be totally unsuited for it. C probably also.

@LinusCDE yeah, C is a thing of the past.
c++ maybe would have the necessary libraries, but i doubt it.
there's not much web development made in c++ after all:)
i mentionned them more for OP's project parsing part. thats what i would have used, bc i know and love these languages. and C-like can be lightning fast especially when dealing with parsing.

but you dont have memory managed for you. and its a bitch and a half to get the hang of it.

c# is a little bloated for my liking (boundary detection even when compiled).
i dont know rust but the syntax seems harder.

but what i dont understand is how shell would be unsuitable to automate the compiling part.
a quick googling told me it can be done using a command(tool) that comes with python it seems...

@LinusCDE @bad-frog Thanks both for the tips!

@bad-frog For compiling, a shell script should do well. I misunderstood the purpose of it.

C++ would've probably been suited for the project as well. I just personally have not done anything beyond one project involving classes with it. I mainly use stuff from the linux kernel or other unix libraries for the hw stuff. So C++ would've been probably a bit faster there.

I agree that Rust is hard to learn. Felt to me like learning my first programming language again. But it is really rewarding once you understand it. It is also not really possible to mess up memory management there and it doesn't need a garbage collector for that.

@bad-frog I think C# would've been a little hard to do interaction to some more niche linux systems with. I personally also have more knowledge of Java but I don't really find good usecases for it nowadays.

@LinusCDE isnt it that java has an enormous library collection for everything web-driven?
or maybe you mean java (full stop)?

@bad-frog I mean Java. Not JS. 😂

@LinusCDE lol. yeah, java is dead to such an extent that when people say java they mean js.

but then i just googled it and saw that 90% of fortune 500 companies reportedly still use it and it is actively maintained.

i know some industries, especially banking tends to lag behind technology-wise (i heard some still use cobol. in 2021).

but back to java, why would that be?

@bad-frog It was my first language. The reason was Minecraft Server Plugin Development. I think I used Java a good 3/4 years for that before proceeding with other Java Stuff (mainly Swing UIs and TCP Networking related stuff).

After that I learned python, a tiny bit C, php (only one project or really simple things), bash and last year finally Rust.

I think I also learned Kotlin and did a university project in Java with Spring. But Java is dead to me personally.

Ofc, I also learned html, css, js and later NodeJS (mainly Vue and Angular) along the way.

So yeah. As soon as I saw other languages, Java is usually in a weird middleground for me that I don't need.

Especially using IntelliJ over eclipse makes new projects a bigger hassle and not worth it for me where I could just create a simple python file.

@LinusCDE thanks for the enlightenment.
most useful info for me. im completely green in what concerns the IOT.
and i have the project of building a service, backend/frontend, with a friend.
thanks to you instead of a stab in the dark and falling on my face until i learn to walk, now i have a roadmap: java bad, everybody uses nodejs, and python has gone a long way since i actually used the language.
thanks again:)

@bad-frog I personally liked Java for the time I used it. But my preferences and needs changed so I didn't find much use in it anymore.

I would personally say: Try a few other languages. Then just use a language you find most fitting for personal projects. There is always a language specialized in a task you need. That doesn't necessarily mean that more general languages are bad either.

@LinusCDE i will do exactly that.
i think i will begin with python since im familiar with it. i think it would reduce the cognitive overhead while getting familiar with the more frontish side of things.

i mean the terminology and concepts mainly. then i think js and nodejs because it seems to be an industry standard.

also because i found it quite confusing (it was some time ago, and i "leveled up" since then) and, finally php. well, i have already dabbled in the latter, and find its syntax to be somewhat similar to python, so it shouldnt be too hard for me: thence i save it for last.

and java somewhere in between since you say it does have value, even if it is outdated.

my rationnale for that order would be: if i know exactly what i need, it would become a good "point of entry" to learn a new language...

you think its a good order of things?

(sorry if i bore you with my questions, last one i promise:)

@bad-frog Though I think C# and Java have pretty similar usecases. If you already know C#, I think that'll be fine. Java is pretty much the older of the two.

@LinusCDE i wouldnt exactly say that i "know" C# (surely not to the extent i know C), but that is very encouraging:).
ngl, of all these languages java is a big unknown for me...

@bad-frog Maybe try a hello world in it. Otherwise I don't think that it'll be a tradedgy to now know it.

The minecraft community uses it for mods and plugins. Coorperations probably have older or maybe even newer projects with it. The enterprise area especially around Spring and Beans/Dependency Injections is not small. Also probably for native android apps (Kotlin can be seen as an alternate grammar with some extra goodies).

Maybe a small app might be an interesting Project to get the jist of it.

But I think that C# has succeeded Java in many things. Maybe some areas still use it more but that probably depends on your region.

Not speaking of experience here. I'm still studieng. Heard a couple of things from the YouTuber "Fredrik Christenson" though. He seems to be a Recreuiter/Manager nowadays but has a lot of videos describing his careeer or opinions about Java/C#/NodeJS and some other languages:
https://youtu.be/t4SDKErrQ5U

@LinusCDE lol im a student too. however im 32, so i dont think ill ever do anything else than freelance and personnal projects. and, tbh, it suits me perfectly. (but i coded various projects in python, then C for years by now. i know stuff that i use in my projects, and since i now have access to education, i have the occasion to learn all the rest)

and since what the industry wants is somewhat a secondary consideration for me,
i think java will become a new favourite of mine.

you are absolutely right. java's syntax is pretty much the same as C#. its gonna be a breeze to learn. and i love the fact that its absolutely portable. it will allow for more freedom with my project:)
also that it is "semi-compiled".

yeah, java has just jumped several echelons up in my priority list. its gonna make my life so much easier!