Like most people I needed some extra cash during uni, so I proceeded to learn CSS + Photoshop (yeah, I know). Followed by PHP and WordPress.

It can be a very shitty platform until you realize that you can stop combining plug-ins from all over the place with dubious code quality and roll your own.

Anyhow I kept at it until I was able to join a niche company doing a quite popular caching plug-in for WP (yeah, W3 Total) when I suddenly became *very* interested in anything and everything performance.

This landed me a very cozy consulting gig in the Nordics - they were using WP for an elephant-traffic website and had run into a myriad of perf issues.

Fixing them and breaking the monolith awarded me with skills in nodejs, linux, asynchronous caching among others.

I was soon in charge with managing the dev boxes for the entire team, and when the main operations dude left, I was promoted to owning the entire platform. (!) Tinkering with Linux for most of my life really came in handy here. (remember Debian potato?)

Used saltstack + aws cloudformation to achieve full parity between all environments. Learned myself some python and all various tips and tricks which in the end amounted to 90% reduction in time-to-first-byte and considerable cost savings.

By the end of the 2yr contract I had turned myself into a fullstack systems engineer and never looked back.

Lawyers not getting along resulted in us having to abandon NewRelic, so I got to learn and deploy the ELK stack as a homegrown replacement, which was super-fun.

Now I work in the engineering effectiveness department of a Swedish fintech unicorn where all languages under the Sun are an option (tho we prefer Python), so the tech stack is unlimited. Infinite tools and technologies, but with strong governing principles and with performance always in mind so as to pick the right tool for the job.

It's like that childhood feeling when you've just dumped a ton of Lego on the floor and are about to build something massive.

I guess the morale here is however disappointed you feel by your current stack - don't. Always strive to make things better, faster, more decoupled, easier to test, etc. and always challenge yourself to go outside the comfort zone.

Fuck this Kibana shit and give me back my old grep (or even better: ripgrep). In 2008, I used to find shit in my fucking logfiles. Now I have an ELK stack that smells like liquid shit.

Today at work I accidentally redeployed our ELK instance without taking a backup of all of Kibana's saved objects...

I didn't realize Elastic stores all indexes including the Kibana's in it's app folder by default.

Tomorrow will be fun.... I can't decide what to do first... Recreating all the charts and tables from memory... Or fixing the deployment script to change the data dir path...

If you ever want to know how NOT to create documentation, try installing Elastic Search for a server.

Is it only me, or is reducing production code size by #ifdef'ing sh*t out sexy as F?

So I was setting up ELK (Elasticsearch, Logstash and Kibana) all in one EC2 on AWS today for demo purposes. I had everything prepared. Elastic IP, correct security group rules, etc.

I figured I would just do quick test before writing filters and templates if I can access Kibana. So I started service for it and tried to open it with Chrome.

Timeout.

Checked config file. Compared it to documentation. Seemed good but changed some things just for sake of change. Restarted service.

Timeout.

Reverted changes I've made in config. Restarted service. Curl on localhost. It work... OK. 😐

It took me half an hour but finally I figured it out after I took my phone and opened it from there. It was working from the beginning. Stupid company network was for some reason blocking this connection. Fuck! 😡And I was restarting that poor service like crazy trying to fix something that wasn't broken.

Spent another half day learning ELK and how to programatically query and run aggregations against the data that's now collected.

So I can feed it into a testing framework for releases.

I sorta feel like I'm dragging everyone else into the light...

Like "you see what you've been missing all these years? This is how it's supposed to be these days..."

Data data data... Useful data.... This is what you can do when have structured and searchable logs rather then huge messy text files ...

Interviewer: What is ELK stack?
Me: It's a data structure which follows first in last out.
Interviewer: .......

WTF!!!! I come back from a 1 week vacation and nothing has been done and some things seem to have gone to shit...

I transferred the responsibility of running and supporting a report thats supposed to go live to someone else. I show up today and check and well none of the reports for the last 2 weeks were run (first report was already late).

I sent out a few emails asking for feedback on a new JSON log I wanted to add so it can be used by ELK. The people I was asking (a senior dev on a sister team that shares ownership) never replied like he said he would.

So following my previous post, the issue happened again. And actually for background what I've been telling my boss, for years, we need ELK setup and integrated into all our APIs ASAP.

I think it's a punishable crime if any program is released into prod at a tech company with out real time logging/monitoring built in?

So issue still happening, user sent us the request details. So now need to find the actual now that handles the request and look into it's logs to see the details.

Now he's doing it the hard way.... Just finished took 1hr, and the best answer her can come up with is "I think .... Maybe ..."

And if course this is based on infinite data. He stopped after finding a "probably cause"

I have a script that is like promotion ELK, downloads all looks and parsed then so I can run queries to pinpoint the exact call and which log it's in. And can see what's happening around it.

We'll see what my way find but definitely does not take more than 1hr...

Loading data maybe but that's because it needs to download the logs and parse them all...

On a side note, guess I'm Beck on devrant as I have something to rant about. Though it's the same something that I was wanting about years ago... Monkeys...

I setup ELK for our team and went live with it on Production VM.

I'm the only one that knows how it works, is setup... Because no one else cares or wants to know as long as it works...

And well if it doesn't, let's just say they hope that I'm around...

On a side note, I think I'll leave a bit early today since I cut or main projects build process time by 50%.

Root cause: SONAR complains if you implement that using if else to match each field... it is pretty ugly...

And can use Lombok to clean it up, last rant.

So shaved off 10 minutes in each build... And well I'm like seriously? No one else bothered to figure this out for the last year or 2?

I mean I've been pretty busy too but the team had like 20 ppl and at least 4 senior devs and well u don't even need to be senior? Just inquisitive and proactive?

Using grafana together with tinc+promotheus, has been a blast.

Initially I wanted to get into ELK with Kibana and all that, but that required 8G of ram, the instructions to get it running in the open source "mode" was nearly non-existent, together with all the ready docker compose stacks out there simply not working or the images being broken.

I'm sure I could've managed around most of those issues, but the fact it is as hungry as gitlab, made it a literal no-go for the usual server resources my clients host or my own scaled down server recently.

Thankfully I remembered that there's grafana and me having experimented some time ago with tinc, so I can have very lightweight beat'esque prometheus agents deployed listening on tinc local net only, with the typical nginx auth and some whitelists to all of the servers I host and all those of my clients.

The dashboard creation was especially great in grafana (tbf promotheus does actually most of it), literally what I always wanted out of those "complicated" solutions, that do it all, but have no proper query language, complex documentation, heavy collectors with no properly named data points, expensive resource runtimes, ..

with grafana I can just easily put dashboards into folders, create users to look only at certain stats or even dashboards (opened up some interesting contracts actually, because now I can also offer proper monitoring for all things delivered), easily drag and drop around stuff to fit more information (most others fix you to a small 3x2 grid, a too big grid for a TV or simply non resizable tiles, making that one counter take up an entire row) and resize to my hearts desire

tinc of course allows me to easily create private networks that are resistant to failure across any region and the routing is done for me, so I don't have to run around it all that much either

P.S: a damn tiny fly went into one of my now 4 monitors and died right in the middle, because I thought it's just some dirt and I pressed it in while trying to wipe it off, so that monitor now serves as the top most on a vesa mount

What's a good book to learn ELK hands-on?

I have an instance set up and working but want to need some more advanced features? like mapping, index templates, querying remotely for aggregations.

The Elastic docs feel very high level and maybe assumes u can read their minds... A lot of snippets I just go "uhm.. where do I put this in the file, which file?"

It's been a good month where honestly I had nothing to rant about. Pretty much doing my own project setting up ELK.

But last few days I had to return to the reality called teammates....

It where it ok... I mentored one of them, then did the code review yesterday

And that's when the shit hit the fan.

I told them to do X but then they did Y instead thinking that they were smart.

In hindsight they seem to have no idea wtf they were doing, inexperienced and couldn't even use console.log and JSON.stringify to debug object states...

Which course now reminded what's wrong with this team, you got people jumping around stacks and projects so they're all mediocre on all of them. Rather than having specific people being good at one of them (aka more experienced than a noob).

And if course this morning, manager asked me to look into something on a program I haven't support in a while (there are a free people that are more experienced and know the current state better). And he said this is quick and urgent... And actually when he said that I'm like uh.... don't think so....

And last thing is we had to rerun a report in production so needed the shipper ten to do it. Asked them look yesterday, users were waiting.

Today... Still not done. And well I actually can run the report myself locally.. takes 5mins but in production they need to reload the data but that should take at most 20mins... Either way... Nothing was done.

Oh and I just remembered I raised a request to it SA group to have some not script installed... That not done either.

And this is why relying on others it at least these people is a bad idea..... Unless your are capable of firing them...

Is there some sort of Query Builder for ElasticSearch?

I have ELK setup and in Kibana can generate all the aggregation visualizations but now I want the data to be usable in a program so it can generate reports like who are our top users.

But the aggregation queries seem to be very verbose... not sure how anyone can generate or understand it by hand vs telling Kibana I want a chart with X and Y axes using these terms.

IDeally I'd like to have Kibana then tell me what's the actual JSON/Elastic query it used to generate that but can't seem to find something like that.

Agree or disagree?

In algo terms ELK is like going from O(n^3) to O(ln(n))

I actually just said that to my boss.... after finishing running my version of analyzing the issue...

Follow up sorta...

So I got pulled into a support issue on a day off. Some system was facing timeouts on our servers so had to investigate.

Over the weekend as part of the release, I released the ELK stack I built and today I used that to help.

Pretty much immediately pinpointed which machine was hanging though still had to investigate and confirm so split between KQL and checking the server logs.

One thing I've always griped about is how no one created schema docs for it mongo collections so can't easily figure out what they do or your to get the document needed.

Well guess it's my turn.... Because only I know the schema :)

Where can find an all in one guide to set up/configure a ELK environment manually from the tars (not yum, apt-get, docker image)?

I am following the component docs from Elastic but not sure how each component integrates with each other or how to set the mongo connection (DB is not local)

Hi fellow devRanters, I need some advice on how to detect web traffic coming from bad/malicious bots and block them.

I have ELK (Elastic) stack set up to capture the logs from the sites, I have already blocked the ones that are obviously bad (bad user-agent, IP addresses known for spamming etc). I know you can tell by looking at how fast/frequently they crawl the site but how would I know if I block the one that's causing the malicious and non-human traffic? I am not sure if I should block access from other countries because I think the bots are from local.

I am lost, I don't know what else I can do - I can't use rate limiting on the sites and I can't sign up for a paid service cause management wants everything with the price of peanuts.

Rant:

Someone asked why I can't just read through the logs (from several mid-large scale websites) and pick out the baddies.

*facepalm* Here's the gigabytes log files.

How the fuck do you use and make a fields.yml for dynamic filebeat indexes?

Aka what if i don't want all the fields?