Often I hear that one should block spam email based on content match rather than IP match. Sometimes even that blocking Chinese ranges in particular is prejudiced and racist. Allow me to debunk that after I've been looking at traffic on port 25 with tcpdump for several weeks now, and got rid of most of my incoming spam too.

There are these spamhausen that communicate with my mail server as much as every minute.
- biz-smtp.com
- mailing-expert.com
- smtp-shop.com

All of them are Chinese. They make up - rough guess - around 90% of the traffic that hits my edge nodes, if not more.

The network ranges I've blocked are apparently as follows:
- 193.106.175.0/24 (Russia)
- 49.64.0.0/11 (China)
- 181.39.88.172 (Ecuador)
- 188.130.160.216 (Russia)
- 106.75.144.0/20 (China)
- 183.227.0.0/16 (China)
- 106.75.32.0/19 (China)
.. apparently I blocked that one twice, heh
- 116.16.0.0/12 (China)
- 123.58.160.0/19 (China)

It's not all China but holy hell, a lot of spam sure comes from there, given how Golden Shield supposedly blocks internet access to the Chinese citizens. A friend of mine who lives in China (how he got past the firewall is beyond me, and he won't tell me either) told me that while incoming information is "regulated", they don't give half a shit about outgoing traffic to foreign countries. Hence all those shitty filter bag suppliers and whatnot. The Chinese government doesn't care.

So what is the alternative like, that would block based on content? Well there are a few solutions out there, namely SpamAssassin, ClamAV and Amavis among others. The problem is that they're all very memory intensive (especially compared to e.g. Postfix and Dovecot themselves) and that they must scan every email, and keep up with evasion techniques (such as putting the content in an image, or using characters from different character sets t̾h̾a̾t̾ ̾l̾o̾o̾k̾ ̾s̾i̾m̾i̾l̾a̾r̾).

But the thing is, all of that traffic comes from a certain few offending IP ranges, and an iptables rule that covers a whole range is very cheap. China (or any country for that matter) has too many IP ranges to block all of them. But the certain few offending IP ranges? I'll take a cheap IP-based filter over expensive content-based filters any day. And I don't want to be shamed for that.

24th, Christmas: BIND slaves decide to suddenly stop accepting zone transfers from the master. Half a day of raging and I still couldn't figure out why. dig axfr works fine, but the slaves refuse a zone update according to tcpdump logs.

25th, 2nd day: A server decides to go down and take half my network with it. Turns out that a Python script managed to crash the goddamn kernel.

Thank you very much technology for making the Christmas days just a little bit better ❤️

At least I didn't have anything to do during either days, because of the COVID-19 pandemic. And to be fair, I did manage to make a Telegram bot with fancy webhooks and whatnot in 5MB of memory and 18MB of storage. Maybe I should just write the whole thing and make another sacred temple where shitty code gets beaten the fuck out of the system. Terry must've been onto something...

I added some boards to this fucking Beowolf of a fucking Raspberry Pi!

Pi with 4GB RAM, 2TB SSD, 8 USB ports, 2 Ethernet ports, and a sense hat.

Gonna put this between my modem and router and see what fun I can get up to.

Would like to build a web portal that tracks my family's data usage with the tcpdump to graph approach, and probably a little weather widget to go with it using the sense hat.

At the institute I did my PhD everyone had to take some role apart from research to keep the infrastructure running. My part was admin for the Linux workstations and supporting the admin of the calculation cluster we had (about 11 machines with 8 cores each... hot shit at the time).

At some point the university had some euros of budget left that had to be spent so the institute decided to buy a shiny new NAS system for the cluster.

I wasn't really involved with the stuff, I was just the replacement admin so everything was handled by the main admin.

A few months on and the cluster starts behaving ... weird. Huge CPU loads, lots of network traffic. No one really knows what's going on. At some point I discover a process on one of the compute nodes that apparently receives commands from an IRC server in the UK... OK code red, we've been hacked.

First thing we needed to find out was how they had broken in, so we looked at the logs of the compute nodes. There was nothing obvious, but the fact that each compute node had its own public IP address and was reachable from all over the world certainly didn't help.
A few hours of poking around not really knowing what I'm looking for, I resort to a TCPDUMP to find whether there is any actor on the network that I might have overlooked. And indeed I found an IP adress that I couldn't match with any of the machines.

Long story short: It was the new NAS box. Our main admin didn't care about the new box, because it was set up by an external company. The guy from the external company didn't care, because he thought he was working on a compute cluster that is sealed off behind some uber-restrictive firewall.

So our shiny new NAS system, filled to the brink with confidential research data, (and also as it turns out a lot of login credentials) was sitting there with its quaint little default config and a DHCP-assigned public IP adress, waiting for the next best rookie hacker to try U:admin/P:admin to take it over.

Looking back this could have gotten a lot worse and we were extremely lucky that these guys either didn't know what they had there or didn't care.

Samsung Smart TV becomes Samsung Dumb TV.

Welcome back dear readers, to the next installment of my Raspberry Pi / Pi Hole / MitM box adventure!

For those of you who are new to this story, I'm a long experience programmer who knows very little about his home network or networking in general and has constantly been going over his 250GB data plan because 'rona, and thus, wants answers to "where is the data going".

So, I got the Pi, codenamed Mini-Beowolf, positioned between the modem and router... worked some fuckin systemd.networkd magic (which was sort of easy... but was hard cause I'm new to it) and viola, this son of a bitch passes through the ethernet and doesn't even show up on the router. Fu-King Beastly, I love it.

Now to static IP all my devices so I fire up my trusty TP-Link admin portal. I should add here... I've visited this admin about a total of 10 minutes prior to this when I set this wifi router up and just let it do DHCP.

So I'm getting to know my admin portal... I've got most of my devices connected to reserved IPs... and I find this one fuckin device reporting as "localhost".

Now, I've got a MAMP install... but it hasn't been running. But still I thought for sure it was just MAMP run a bit amok.

But no... it was my fucking Samsung "Smart" TV. That piece of shit is, and apparently has been reporting its device name as, sure as shit, fucking "localhost"... PROBABLY FOR YEARS.

Now, IDK how that didn't cause me any major problems over the years, and I read quite a few forums about people who it did mess up their network. So I resolved to rename the Samsung TV device.

I found the spot in the network settings of the TV... I changed the name from the pick list of rooms in a house like "Living Room" and "Bed Room", then I tried entering my own device name. But no matter what I picked, or no matter how many times I restarted/reset that TV the network name is ALWAYS "localhost".

Even though somehow my network survived this long... I'm not standing for that shit.

My Samsung TV is now blocked COMPLETELY at the router level. (After I ran one last factory reset and update)

The kicker? That Pi I built has a Samsung SSD... so I'm blocking Samsung WITH FUCKING SAMSUNG.

Needless to say, these are likely among my last Samsung purchases.

Join me next time when I FINALLY try to turn Pi Hole on and then get a tcpdump (or some other lesser output from the tcp stream) going.

*laughing maniacally*

Okidoky you lil fucker where you've been hiding...

*streaming tcpdump via SSH to other box, feeding tshark with input filters*

Finally finding a request with an ominous dissector warning about headers...

Not finding anything with silversearcher / ag in the project...

*getting even more pissed causr I've been looking for lil fucker since 2 days*

*generating possible splits of the header name, piping to silversearcher*

*I/O looks like clusterfuck*

Common, it are just dozen gigabytes of text, don't choke just because you have to suck on all the sucking projects this company owns... Don't drown now, lil bukkake princess.

*half an hour later*

Oh... Interesting. Bukkake princess survived and even spilled the tea.

Someone was trying to be overly "eager" to avoid magic numbers...

They concatenated a header name out of several const vars which stem from a static class with like... 300? 400? vars of which I can make no fucking sense at all.

Class literally looks like the most braindamaged thing one could imagine.

And yes... Coming back to the network error I'm debugging since 2 days as it is occuring at erratic intervals and noone knew of course why...

One of the devs changed the const value of one of the variables to have UTF 8 characters. For "cleaner meaning".

Sometimes I just want to electrocute people ...

The reason this didn't pop up all the time was because the test system triggered one call with the header - whenever said dev pushed changes...

And yeah. Test failures can be ignored.
Why bother? Just continue meddling in shit.

I'm glad for the dev that I'm in home office... :@

TLDR: Dev changed const value without thinking, ignoring test failures and I had the fun of debunking for 2 days a mysterious HAProxy failure due to HTTP header validation...

Probably the one where we had an error, because the service from a thirdparty we needed to install used a fourthparty service that was behind a proxy. Due of internal reasons we needed to use our own tomcat instead of the standard tomcat. We made a meeting because we didn't found out at that point that the problem was the firewall that dropped the packages from the application. We replayed it to them (at that point it was in my musclememory) and after a month i got the idea to use tcpdump to see if the server is calling another webservice, which was denied by the developers.

On my way home from work using public transport and I want to enjoy some rants.
But thanks to this fucking bad mobile network access I'm not able to load a single one.

AHHHHHHHHHHGGGH

I HATE VPN SETUP

- Trying OpenSwan
Installing open swan on a Debian machine.. setting up the config.
Restarting openswan. Syntax error. No syntax error to be found.
Different tutorial.. it starts! Try to connect.. I can’t connect. Look at the logs. No errors.
Tcpdump. My traffic is coming through.. all fine.. try to connect again.. it works! (Nothing changed!)
Try to ping somewhere else.. no connectivity.
Try to ping an IP in the same network.. works fine. So I have connectivity, just no internet.
Spend an hour finding out about traffic directions of which no one seems to know what they really mean.
Boss tells me to stop using openswan because it’s deprecated and replaced by strong swan..

- Strongswan
Reinstall Debian machine, install strongswan. Copy openswan config. Oh, they’re incompatible? Look up strong swan config, and the service starts.
Connect to the VPN.. it works! Again, no internet, just connectivity in the same network. Spend 2h debugging the config, disable firewalls everywhere, find an ancient bug in the Debian package related to my issues.. ok, let’s try compiling from source.. you know what, let’s not. I’ll throw this Debian machine away and try something completely different.

- pfSense
Ok, this looks easy enough! Let’s just click through the initial setup, change some firewall rules, create an L2TP VPN with a simple wizard.
Try to connect to VPN. First, it times out. Maybe a firewall issue? Turn off firewall.. ah, something happens now. I get an error message right after trying to connect to the VPN. Hmm, the port doesn’t even get opened when I enable the firewall.. this implementation seems a bit buggy.. let’s try their OpenVPN module.

Configure OpenVPN. Documentation isn’t that clear.. apparently a client isn’t actually a client but a user is a client.. ok, there’s a hidden checkbox somewhere.
Now where do I download my certificate? Oh, I need a plug-in for that.. ok, interesting. Able to download the certificate, import it, connect and.. YES!!! I can ping! But, I have no DNS..
Apparently, ICMP isn’t getting filtered but all outbound ports are.. yet the firewall is completely disabled. Maybe I need outbound NAT? Oh. There’s no clear documentation on where to configure it. Find some ancient doc, set it up, still no outbound connectivity.

AHAHAHAHHHHHHHHHHG

Then I tried VyOS. I had a great L2TP VPN working in less than 15 mins. Thank you VyOS for actually providing proper docs and proper software.

So I was wondering why my tcpdump was displaying more than 10 packets per second between my local dev machine and my server, until I realized that I was running tcpdump in the ssh session...