So, you start with a PHP website.

Nah, no hating on PHP here, this is not about language design or performance or strict type systems...

This is about architecture.

No backend web framework, just "plain PHP".

Well, I can deal with that. As long as there is some consistency, I wouldn't even mind maintaining a PHP4 site with Y2K-era HTML4 and zero Javascript.

That sounds like fucking paradise to me right now. 😍

But no, of course it was updated to PHP7, using Laravel, and a main.js file was created. GREAT.... right? Yes. Sure. Totally cool. Gotta stay with the times. But there's still remnants of that ancient framework-less website underneath. So we enter an era of Laravel + Blade templates, with a little sprinkle of raw imported PHP files here and there.

Fine. Ancient PHP + Laravel + Blade + main.js + bootstrap.css. Whatever. I can still handle this. 🤨

But then the Frontend hipsters swoosh back their shawls, sip from their caramel lattes, and start whining: "We want React! We want SPA! No more BootstrapCSS, we're going to launch our own suite of SASS styles! IT'S BETTER".

OK, so we create REST endpoints, and the little monkeys who spend their time animating spinners to cover up all the XHR fuckups are satisfied. But they only care about the top most visited pages, so we ALSO need to keep our Blade templated HTML. We now have about 200 SPA/REST routes, and about 350 classic PHP/Blade pages.

So we enter the Era of Ancient PHP + Laravel + Blade + main.js + bootstrap.css + hipster.sass + REST + React + SPA 😑

Now the Backend grizzlies wake from their hibernation, growling: We have nearly 25 million lines of PHP! Monoliths are evil! Did you know Netflix uses microservices? If we break everything into tiny chunks of code, all our problems will be solved! Let's use DDD! Let's use messaging pipelines! Let's use caching! Let's use big data! Let's use search indexes!... Good right? Sure. Whatever.

OK, so we enter the Era of Ancient PHP + Laravel + Blade + main.js + bootstrap.css + hipster.sass + REST + React + SPA + Redis + RabbitMQ + Cassandra + Elastic 😫

Our monolith starts pooping out little microservices. Some polished pieces turn into pretty little gems... but the obese monolith keeps swelling as well, while simultaneously pooping out more and more little ugly turds at an ever faster rate.

Management rushes in: "Forget about frontend and microservices! We need a desktop app! We need mobile apps! I read in a magazine that the era of the web is over!"

OK, so we enter the Era of Ancient PHP + Laravel + Blade + main.js + bootstrap.css + hipster.sass + REST + GraphQL + React + SPA + Redis + RabbitMQ + Google pub/sub + Neo4J + Cassandra + Elastic + UWP + Android + iOS 😠

"Do you have a monolith or microservices" -- "Yes"

"Which database do you use" -- "Yes"

"Which API standard do you follow" -- "Yes"

"Do you use a CI/building service?" -- "Yes, 3"

"Which Laravel version do you use?" -- "Nine" -- "What, Laravel 9, that isn't even out yet?" -- "No, nine different versions, depends on the services"

"Besides PHP, do you use any Python, Ruby, NodeJS, C#, Golang, or Java?" -- "Not OR, AND. So that's a yes. And bash. Oh and Perl. Oh... and a bit of LUA I think?"

2% of pages are still served by raw, framework-less PHP.

To replace humans with robots, because human beings are complete shit at everything they do.

I am a chemist. My alignment is not lawful good. I've produced lots of drugs. Mostly just drugs against illnesses. Mostly.

But whatever my alignment or contribution to the world as a chemist... Human chemists are just fucking terrible at their job. Not for a lack of trying, biological beings just suck at it.

Suiting up for a biosafety level lab costs time. Meatbags fuck up very often, especially when tired. Humans whine when they get acid in their face, or when they have to pour and inhale carcinogenic substances. They also work imprecisely and inaccurately, even after thousands of hours of training and practice.

Weaklings! Robots are superior!

So I replaced my coworkers with expensive flow chemistry setups with probes and solenoid fluid valves. I replaced others with CUDA simulations.

First at a pharma production & research lab, then at a genetics lab, then at an Industrial R&D lab.

Many were even replaced by Raspberry Pi's with two servos and a PH meter attached, and I broke open second hand Fischer Sci spectrophotometers to attach arduinos with WiFi boards.

The issue was that after every little overzealous weekend project, I made myself less necessary as well.

So I jumped into the infinitely deep shitpool called webdev.

App & web development is kind of comfortable, there's always one more thing to do, but there's no pressure where failure leads to fatalities (I think? Wait... do I still care?).

Super chill, if it weren't for the delusion that making people do "frontend" and "fullstack" labor isn't a gross violation of the Geneva Convention.

Quickly recognizing that I actually don't want to be tortured and suffer from nerve damage caused by VueX or have my organs slowly liquefied by the radiation from some insane transpiling centrifuge, I did what any sane person would do.

Get as far away from the potential frontend blast radius as possible, hide in a concrete bunker.

So I became a data engineer / database admin.

That's where I'm quarantining now, safely hiding from humanity behind a desk, employed to write a MySQL migration or two, setting up Redis sorted sets, adding a field to an Elastic index. That takes care of generating cognac and LSD money.

But honestly.... I actually spend most of my time these days contributing to open source repositories, especially writing & maintaining Rust libraries.

Privacy & security violations piss me off. Not to the point that I'll write on devRant about it, but to the point that coworkers get afraid from the bloodthirsty look in my eyes.

I know all startups proclaim this, but the one I work at is kind of industry-disrupting. Think Uber vs taxi drivers... so we have real, malicious enemies.

Yet there's still this mindset of "it won't happen to us" when it comes to data leaks or corporate spying.

Me: "I noticed we are tracking our end users without their consent, and store not just the color of their balls, but also their favorite soup flavor and how often they've cheated on their partner, as plain text in the system for every employee to read"

Various C-randomletter-Os: "Oh wow indubitably most serious indeed! Let's put 2 scrumbag masters on the issue, we will tackle this in a most agile manner! We shall use AI blockchains in the elastic cloud to encrypt those ball-colors!"

NO WHAT I MEANT WAS WHY THE FUCK DO WE EVEN STORE THAT INFORMATION. IT DOES IN NO WAY RELATE TO OUR BUSINESS!

"No reason, just future requirements for our data scientists"

I'M GRABBING A HARDDRIVE SHREDDER, THE DB SERVER GOES FIRST AND YOUR PENIS RIGHT AFTER THAT!

(if it's unclear, ball color was an optimistic euphemism for what boiled down to an analytics value which might as well have been "nigger: yes/no")

C'mon people! Spread the word! "The cloud" is not "just someone elses computer", it's a completely different way to compute!

I'm so tired of the oversimplifications done trying to explain the consept. The massive amount of work, sweat and tears put into the orchestration, automation and abstraction layers to deliver truly elastic, scalable and self healing infrastructure, applications and services deserves a fuckload more respect than "just someone elses computer"!

Hosting and time-sharing have been with us almost as long as we have had computers (mainframes etc), but dismissing the effort of thousands upon thousands of devs and ops people to make systems robust and automated enough to literally being able to throw a wrench in the engine any time during production and not have the systems suffer is fucking insane!

The whole reason the term "cloud" is so fitting is not just because it was coined from the cloud-shape used in technical and non-technical drawings and illustrations symbolising the internet, but also because of the illusion of magic it gives the end-user not being able to see "whats inside the music box".

Is it really unreasonable that I wish aws would just name their fucking products after what they are? Why the fuck is dns called route 53? Why the fuck is a vm an elastic cloud compute node? Stop being pretentious dicks and just name things what they are!

Am I being unreasonable?

So today was the worst day of my whole (just started) career.

We have a huge client like 700k users. Two weeks ago we migrated all their services to our aws infrastructure. I basically did most of the work because I'm the most skilled in it (not sure anymore).

Today I discovered:
- Mail cron was configured the wrong way so 3000 emails where waiting to be sent.
- The elastic search service wasn't yet whitelisted so didn't work for two weeks.
- The cron which syncs data between production db en testing db only partly worked.

Just fucking end me. Makes me wonder what other things are broken. I still have a lot to learn... And I might have fucked their trust in me for a bit.

A project I'm working on uses Elastic for internal monitoring and logs. The customer asked to access those logs - not something we'd normally do, but it's isolated from other things we use and there's no critical data there, so what the heck, let them have it.

Ever since, we're getting tons of questions like "There are tons of [insert random info message] all the time, do you have any plans to resolve them?" and it gets to the point where I'm just about ready to scream back "NO, SUZAN, BOOKING NOT COMPLETED MANS THE USER F###ING CANCELLED IT, IT'S NOT SOMETHING I CAN FIX IN THE CODE"

Edit: the customer's name isn't actually Suzan

Elasticsearch queries are FUCKING ugly

Elasticsearch documentation is FUCKING bare-minimum.

Kibana only shows data when it FUCKING feels like it.

Elastic stack is FUCKING annoying me

Been developing with AWS EC2 since 2015. Have deployed multiple apps in my Ubuntu servers. And have deployed Dockerized apps into Elastic Beanstalk. And used S3 to store files.

But just because I don't know AWS Lambda, the recruiter thought I'm not qualified enough for the job.

Fml.

Using mongodb for one product

A colleague as experimenting with elastic search (I think it was).

It installed a proxy around the collection to get all events for the external search storage.

Worked well, but it was just a test so once done we removed it

But thats where it got scary.

When we removed the proxy through the search dashboard it dropped the underlying collection of live data!!!

A collection it did not create.

Hows that for bad UI.

Always experiment on a separate db server.

Yay! Professional ranter now on phone and together with GCP, devops amsterdam, elastic and others!

My boss did not care about making things secure in our early development stage, even though I told him several times.
After 1 day our elastic search cluster was filled with random crappy data.

Fix: Apply security schemes provided by AWS

Simulating a couple million elastic search searches and aggregations in 2h instead of 3 months ... :)

Today at work I accidentally redeployed our ELK instance without taking a backup of all of Kibana's saved objects...

I didn't realize Elastic stores all indexes including the Kibana's in it's app folder by default.

Tomorrow will be fun.... I can't decide what to do first... Recreating all the charts and tables from memory... Or fixing the deployment script to change the data dir path...

I'm so sick of microservice architecture... in theory it was going to make scaling elastic and deployment easier. In reality it seems to slow productivity to a 🐌 pace.

Anyone have any brilliant suggestions on how to herd these cats in production?

Note to self: Pointing your tests at a non disposable DB will cause very very bad things to happen. No idea what the flying fuck I was thinking - but praise to the data gods it wasn't a production elastic!

Am I the only one who just loves when I do "curl 127.0.0.1:9200/_template" and get back entire 32" screenful of compressed JSON?

Frikin elastic...

Fuck this Apache Server on this elastic beanstalk!!! Nginx is so much better at configuration and ease of use!!!

Need Advice + Rant

I am an Android Developer, pursuing an Internship, which i thought would be good for my career. But I am being assigned the task to build search feature for the App using Elastic Search. I intially was halpy to work on Search since it had to be Algolia. I am hating the work now because I am getting so stuck with Elastic and there have been other factors which also have decreased my productivity, but I am being quite inefficient. Now the deadlines are coming closer and if I dont give output I will be laid off. I am thinking about quitting myself because now I feel extremely demoralized and demotivated to work because we first decided to work on Algolia and it was all ready before we thought of shifting to heroku and now on AWS. What do the experienced once suggest? It's not that its impossible to do, now i just have to write queries in Java, again I am stuck and not really looking forward to since I was given the deadline today, for 2 days later.

The only issue is, I may have to return the new phone (OnePlus 3T) which I bought planning to later return the money to someone through my stipend.

Elastic free tshirt is on its way! Can't wait to get it.

FUCK YOU AMAZON! Stop telling me my configuration yaml file contains a tab... THE FUCKING FILE DOES NOT EVEN EXIST!

If you ever want to know how NOT to create documentation, try installing Elastic Search for a server.

In a time where a web dev is expected to know, well.. everything... Backend -JAVA, python, nodejs and C++ would be great.
Front- angular, react, other 10 libs
DBs -sql, mongo, redis, elastic, kafka, rebbitmq
Also be devops on the side with AWS and docker kubernetis and more stuff

How the f is that possible?

In my real job for the last couple of years and different companies, I usually use 1 language/framework & 1 main DB.. and although it's possible in some companies, but in mine, ppl dont get access to AWS etc..

So let's say there's me.. a server side dev for years.
So I decide to be better and learn Golang.. cool lang, never needed in my job, after few days of not using it I forgot all I learned and that was it.

Then I realized I gotta know some frontend cause everyone want a fullstack ninja nowadays.. so I tried Vuejs.. it was amazing .. never got to use it at work, cause i was a backend, and we didnt use frameworks on our products back then..
Also forgotten.

Then I decided to learned nodejs, because this is the coolest thing ever.. hated it, but whatever... Never got to use it at work, cause everything was written in other lang which the whole team knew... Forgot the little i knew.

Then I decided, its time to see what Angular is, cause everyone started using it... similar idea to vuejs which i barely remembered, but wow it's a lot of code to remember, or I'll have to google everything.. so I went over it, but can't say i even learned it.

Now Im trying to move on to python, which, I really am learning in depth.. however, since I dont have real experience with it, no one gives me a shot at being a python dev, so again i feel like I'm trying to memorize syntax and wasting my time..

Tired of seeing React in all job ads, i decided to have a look what's that all about.. and whadoyaknow... It's fucking the same idea as vue/angular with again different syntax..

THIS IS CRAZY!

in how many syntaxes do i need to know how to make a fucking crud api, and a page with same fucking post form, TO BE A GOOD PROGRAMMER?!?

TLDR;
How much do you earn for your skill set in your country vs your cost of living?
BONUS;
See how much I & others earn.

Recently I became aware of just how massive the gap in developers earnings are between countries. I'd love to calculate a fixed score for income vs cost of living.

I know this stuff is sensitive to some so if you prefer just post your score (avg income p/m after tax / cost of living).

I'm not shy so I'll go first:

MY RATES
Normal Rate (Long term): $23
Consulting / Short term: $30-$74
Pen Test: $1500 once off.
Pen Test Fixes: consulting rate.
Simple work/websites: min $400+
Family & Friends: Dev friends are usually free (when mutually beneficial). Family and others can fuck off, even if they can pay (I pass their info to dev friends with fair warning).

GENERAL INFO
Experience: 9 years
Country: South Africa
Developer rareness in country: Very Rare (+-90 job openings per job seeker).
Middle class wage in country: $1550 p/m (can afford a new car, decent apartment & some luxuries like beer/eating out).
Employment type: Permanent though I can and do freelance occasionally.
Client Locality: Mostly local.
Developer Type: Web Developer (True web dev - I do anything web related from custom HTTP servers to sockets, services, advanced browser api's, apps & more).

STACKS / SKILLSETS

I'M PROFICIENT IN:
python, JavaScript, ASP classic, bash, php, html, css, sql, msql, elastic search, REST, SOAP, DOM, IIS, apache

I DABBLE WITH:
ASP.net, C++, ruby, GO, nginx, tesseract

MY SPECIALTIES:
application architecture, automation, integrations, db's, real time data, advanced browser apps/extensions (webRTC, canvas etc).

SUMMARY
Avg income p/m after tax: $2250
Cost of living (car+rent+food): $1200
Score: 1.85

*Note: For integrity when calculating my cost of living I excluded debt repayments and only kept my necessities which are transport, food & shelter.

I really hope you guy's post your results, it would be great to get an idea of which is really the worst / best country to be a developer in.

An elastic search package for Laravel.

So I was setting up ELK (Elasticsearch, Logstash and Kibana) all in one EC2 on AWS today for demo purposes. I had everything prepared. Elastic IP, correct security group rules, etc.

I figured I would just do quick test before writing filters and templates if I can access Kibana. So I started service for it and tried to open it with Chrome.

Timeout.

Checked config file. Compared it to documentation. Seemed good but changed some things just for sake of change. Restarted service.

Timeout.

Reverted changes I've made in config. Restarted service. Curl on localhost. It work... OK. 😐

It took me half an hour but finally I figured it out after I took my phone and opened it from there. It was working from the beginning. Stupid company network was for some reason blocking this connection. Fuck! 😡And I was restarting that poor service like crazy trying to fix something that wasn't broken.

So, since almost a week I was trying to get familiar with Algolia which was to be used as API for the search feature in our App. But now, we are going with Elastic Search.

Why the fuck is the elastic search docker image 900Mb. Why the fuck does it include a complete logging system and why the fuck doesn't it have an up-to-date alpine image?

The arrogance of some systems these days. You're part of someone's software, not the software itself. You don't have the right to claim x resources. It's not about you.
Same for Sentry, a logging application that literally requires 8Gb of ram? I removed the limits and did try it anyway and stuff just crashed. Congrats, a logging system that REALLY requires 8Gb ram. The best my VPS does is 4Gb and therefrom you're only allowed to use 512Mb max imho.

I care about image sizes since my laptop only has 114Gb drive and my internet is a 4g hotspot with 50Gb/day limit (trust me, you can't find better for 40 euro's). 114Gb is maybe a bit outdated but be realistic, I only use vim, vscode, some sdks and source files. Why would my harddisk ever be full? Because of bloated docker setups. That's why. The other option is screwing over your system with everyone's configuration.

Alpine all the things!

When elastic search teaches you life lessons

Who gives the task of building an Elastic Search feature to be deployed using Heroku on a Firebase database to an intern? I have never been so stuck, so lost, so confused, so damned, so fused, so annoyed, so pissed off, so irritated, so "hate my life", so "hate programming, I should start selling mangoes", so fml ever before in my whole life. :(

When you are tired fix elastic search on windows now it's time to....

Bye windows! Hello ubuntu 😍

Not sure if it should be a joke or a rant, but something rather funny (at least to me) happened today.

TL;DR; Someone's outlook was crippled by 100k+ of warning notifications

So we have developed a server that has an internal database that wraps around an elastic search instance, that is managed by a POS vault/storage solution, that we have to use for legal reasons. The elastic search is "provided" by the software, but we keep this internal database just to be sure and totally not because it's unreliable POS.

Anyways, they take data integrity very seriously, so every warning our server produces is emailed to someone in charge to review it and if necessary forward it to us. This will be important later on.

Couple of days ago we got error forward when trying to write an entry into the POS software we get an error, because an object we tried to write already existed. After some investigation we concluded an entry was missing when the internal database was created, so we asked them to repopulate elastic search to solve this problem.

When start the server we always sync the internal database to the elastic search and emit a warning when an entry is missing in internal database or vice versa. And well... almost all of them were missing, which caused our server to emit ~40 warnings/ms. Poor outlook. Still investigating for the cause, but damn, I never expected I'd take down someone email account by accident

migrated everything to elastic search, shitty learning curve and apparently shittier performance. now i dream of going back

Need some advice here.

So hello everyone! I recently moved abroad for work, for the sake of the experience and the excitement of learning how developers in Latin America tackle specific problems. To my surprise, the dev team is actually composed solely of Europeans and Americans.

I work for a relatively new startup with an ambitious goal. I love the drive everyone has, but my major gripe is with my team lead. He's adverse to any change, and any and all proposals made to improve quality of throughput are shot down in flames. Our stack is a horrendous mess patched together with band-aids, nothing is documented, there are NO unit tests for our backend and the same goes for our frontend. The team has been working on a database/application migration for about a month now, which I find ridiculous because the entire situation could have been avoided by following very rudimentary DevOps practices (which I'm shunned for mentioning). I should also add that for whatever reason containerization and microservices are also taboo, which I find hillarious because of our currently convoluted setup with elastic beanstalk and the the constant complaints between our development environment and production environments differing too much.

I've been tasked with managing a Wordpress site for the past 3 weeks, hardly what I would consider exciting. I've written 6 pages in the past two weeks so our marketing team can move off of squarespace to save some money and allow us more control. Due to the shit show that is our "custom theme" I had to write these pages in a manner that completely disregard existing style rules by disabling them entirely on these pages. Now, ironically they would like to change the blog's base theme but this would invertedly cause other pages created before I arrived to simply not work, which means I would have to rewrite them.

Before I took the role of writing an entire theme from scratch and updating these existing pages to work adequately, I proposed moving to a headless wordpress setup. In which case we could share assets in a much more streamline manner between our application and wordpress site and unify our styles. I was shot down almost immediately. Due to a grave misunderstanding of how wordpress works, no one else on the team seems to understand just how easy it is to fetch data from wordpress's api.

In any event, I also had a tech meeting today with developers from partner companies and realized no one knew what the fuck they were talking about. The greater majority of these self proclaimed senior developers are actually considered junior developers in the United States. I actually recoiled at the thought that I may have made a great mistake leaving the United States to look a great tech gig.

I mean no disrespect to Latin America, or any European countries, I've met some really incredible developers from Russia, the Ukraine, Italy, etc. in the past and I'm certainly not trying to make any blanket statements. I just want to know what everyone thinks, if I should maybe move back to the states and header over to the bay/NY. I'm from the greater Boston area, where some really great stuff is going on but I guess I also wanted a change of scenery.

Fucking elastic appsearch, too many requests per seconds and it dies. It doesn't go slow first, it just dies. No warning just a timeout that lasts until a manual restart in elastic cloud console. Besides being apparently the shittiest product in the elastic stack, it's also the worst documented. And yes I just scaled it up but not being able to handle indexing 100 documents per second with 8 vcpu and 8 gb memory is a shame.

FUCKING VAGRANT!

IT always remove elastic search package and i can't migrate my database because of this package required to be run. I always reinstall this to make it run.

Sometimes vagrant itself don't run perfectly in windows.

I think i will move to UBUNTU to make it run smooth.

TODAY you made my task stuck.

This is a part rant-part question.
So a little backstory first:
I work in a small company (5 including me) which is mostly into consultation (we have many tech partners where we either resell their products or if there is a requirement from one of our clients, we get our partners to develop it for them and fulfill the client requirements) so as you can see there is a lot of external dependencies. I act as a one-hat-fits-all tech guy, handling the company websites, social media channels, technical documentation, tech support, quicks POCs (so anything to do with anything technical, I handle them). I am a bit fed up now, since the CEO expects me to do some absurd shit (and sometimes micro manages me, like WTF I am the only one who works there with 100% commitment) and expects me to deliver them by yesterday.

So anyway long story short, our CEO finally had the brains to understand that we should start having our own product (which i had been subtly suggesting him to do for a while now!).

Now he came up with a fairly workable concept that would have good market reach (i atleast give him credits for that) and he wanted me to suggest the best way to move forward (from a both business and technical point of view). The concept is to have an auction-based platform for users to buy everyday products.

I suggested we build a web app as opposed to a mobile one (which is obvious, since i didnt want to develop a seperate website and a mobile app, and anyway just because we can doesnt mean we have to make a mobile app for everything), and recommended the Node/react based JS tech stack to build it.

At first he wanted me to single handedly build the whole platform within a month, I almost flipped (but me being me) then somehow calmed down and finally was able to explain him how complicated it was to single-handedly build a platform of such complexity (especially given my limited experience; did I mention that this is my first job and I am still in college, yeah!!) and convinced him to get an experienced back-end dev and another dev to help me with it.

Now comes the problem, I was to prepare a scope document outlining all the business and technical requirements of the project along with a tentative cost, which was fairly straightforward. I am currently stuck at deciding the server requirements and the system architecture for the proposed solution (I am thinking of either going with AWS - which looks a bit complicated to setup - or go with either Digital Ocean or Heroku):

I have assumed that at peak times we would have around 500-1000 users concurrently

And a daily userbase of 1000 users (atleast for the first few months of the platform running)

What would be the best way forward guys?

I did some extensive (i mean i read through some medium blogs! and aws documentation) research and put together the following specs (if we are going through AWS):

One AWS t3.medium ec2 instance for the node server (two if we want High Availability by coupling with the AWS load balancer and Elastic Beanstalk)

The db.t3.small postgres database

The S3 Storage bucket (100gb) for the React Front end hosting

AWS SNS for email/sms OTP and notification

And AWS CloudMonitor for logging amd monitoring.

Am I speculating the requirements properly, where have I missed??

Can u guys suggest what is the best specification for such a requirement (how do you guys decide what plan to go with)?

Any suggestions, corrections, advices are welcome

Go to hell elastic cloud!

While true:
I can’t resize my elasticsearch instance to get memory because it’s stuck….

It’s stuck because it doesn’t has enough memory to actually start …

Wtf!

I hate the elasticsearch backup api.

From beginning to end it's an painful experience.

I try to explain it, but I don't think I will be able to cover it all.

The core concept is:
- repository (storage for snapshots)
- snapshots (actual backup)

The first design flaw is that every backup in an repository is incremental. ES creates an incremental filesystem tree.

Some reasons why this is a bad idea:
- deletion of (older) backups is slow, as newer backups need to be checked for integrity
- you simply have to trust ES that it does the right thing (given the bugs it has... It seems like a very bad idea TM)
- you have no possibility of verification of snapshots

Workaround... Create many repositories as each new repository forces an full backup.........

The second thing: ES scales. Many nodes / es instances form a cluster.

Usually backup APIs incorporate these in their design. ES does not.

If an index spans 12 nodes and u use an network storage, yes: a maximum of 12 nodes will open an eg NFS connection and start backuping.

It might sound not so bad with 12 nodes and one index...

But it get's pretty bad with 100s of indexes and several dozen nodes...

And there is no real limiting in ES. You can plug a few holes, but all in all, when you don't plan carefully your backups, you'll get a pretty f*cked up network congestion.

So traffic shaping must be manually added. Yay...

The last thing is the API itself.

It's a... very fragile thing.

Especially in older ES releases, the documentation is like handing you a flex instead of toilet paper for a wipe.

Documentation != API != Reality.

Especially the fault handling left me more than once speechless...

Eg:
/_snapshot/storage/backup
gives you a state PARTIAL
/_snapshot/storage/backup/_status
gives you a state SUCCESS

Why? The first one is blocking and refers to the backup status itself. The second one shouldn't be blocking and refers to the backup operation.

And yes. The backup operation state is SUCCESS, while the backup state might be PARTIAL (hence no full backup was made, there were errors).

So we have now an additional API that we query that then wraps the API of elasticsearch. With all these shiny scary workarounds like polling, since some APIs are blocking which might lead to a gateway timeout...

Gateway timeout? Yes. Since some operations can run a LONG (multiple hours) time and you don't want to have a ton of open connections hogging resources... You let the loadbalancer kill it. Most operations simply run in ES in the background, while the connection was killed.

So much joy and fun, isn't it?

Now add the latest SMR scandal and a few faulty (as in SMR instead of CMD) hdds in a hundred terabyte ZFS pool and you'll get my frustration level.

PS: The cluster has several dozen terabyte and a lot od nodes. If you have good advice, you're welcome - but please think carefully about this fact.

I might have accidentially vaporized people sending me links with solutions that don't work on large scale TM.

So I just found out I have to use amazon (instead of Heroku) for hosting a service I'm supposed to build. The environment just f**king refuses to bild and these are my logs. fml...

AWS Contractor

I've been putting a web application together that I'm looking to have published on AWS. Not having too much experience with AWS, I am looking to hire a contractor. I've had a number of quotes from different AWS admin's ranging from $40 an hour to $200 an hour, from 1-days worth of work to 2-months worth of work!

I'm not really sure what to make of it or to whom to trust. I believe they’re using my ignorance to overcharge me. I've listed my requirements below, could you guys use your professional experiences to let me know what you think is reasonable charge and where best I could find someone to help me.

My application is a US shopping website where people can set up an online shop and upload their products and maintain an inventory of the items.

This is what I’m looking for setup and configuration with the following two areas:

1) AWS SYSTEMS…

* AIM - Set up my server admin users.

* EC2 - Web Hosting.

* RDS - Fast DB.

* SES - To send emails.

* S3 Buckets - Uploaded image hosting.

Route 53 - I don’t know but someone said I should have this.

* Elastic Load Balancing - For, well, load balancing.

2) SCRIPTS…

* A script that would back up the database once a day and save it to a private S3 Bucket.

* A script that will run once a day that calls an internal API, and POST a query to it.

* A script that runs once every 90 days, to refresh the SSL using ZeroSSL.com

Is there anything that I've missed such as security systems, firewalls, auto scaling and CDNs?

The quotes that I've received arranged from $320 to $64,000. I know I am being abused because of my ignorance. I would never overcharge someone because the customer doesn't know the efforts of the work. I hope someone here can help to understand the efforts needed and can tell me the true cost.

Thank you

Sat here trying to decide and finalise my Dev process for Wordpress!

Roots.io clean, good code, deployment to staging and production through git

Vagrant then just push to live (which one?)

Docker then try and figure S*** out

Flywheel local!

Then decide where to deploy:

Digital Ocean or AWS Elastic with load balancers and S***!

Decisions!!

- Finish "Introduction to algorithms"
- Learn some genetic algorithms
- Get my hands dirty on reinforcement learning
- Learn more about data streaming application (My currently app is still using plain stupid REST to transport image). I don't know, maybe Kafka and RabbitMQ.
- Learn to implement some distributed system prototypes to get fitter at this topic. There must be more than REST for communicating between components.
- Implementing a searching module for my app with elastic search.
- Employ redis at sometime for background tasks.
- Get my handy dirty on some operating system concepts (Interprocess Communication, I am looking at you)
- Take a look at Assembly (I dont want to do much with Assembly, maybe just want to implement one or two programs to know how things work)
- Learn a bit of parallel computing with CUDA to know what the hell Tensorflow is doing with my graphic card.
- Maybe finishing my first research paper
- Pass my electrical engineering exam (I suck at EE)

To be a Java (or other business popular language) developer
* Java 6, 8 and features up to 14
* SQL + nosql
* Caching
* Logging eg log4j2,
* Searching eg elastic stack
* Reactive
* Framework (at least 1, but hey, knowing 1 is lame..)
* Networking or at least base http knowledge
* Tomcat, jboss or other shit
* Aws, heroku, GCE or other SAAS/paas
* Rest, RPC, soap
* Business Hello World example
* Hexagonal Architecture
* TDD
* Ddd
* Cqrs
* 12 app factor
* Solid
* Patterns
* docket
* Kubernetes
* Microservices
* Security, oauth2
* concurrency
* AMPQ
* Cloud
* Eureka or consul as service Discovery
* Config server
* Hazel cast
*
*
* Endless story ...

Then we can start hello word app

So i just learned aws elastic beanstalk (EBS, ECS, ALB, EC2, Amplify, S3, RDS, SQS)

Essentially i learned how to operate with aws to deploy a full stack web application with custom backend i built, with security and jwt token, certificate manager, ssl/tls to set up https and redirect from http, and react/angular/nextjs on frontend

All with custom CI/CD pipelines docker and other devops shit

But i still feel like im missing on A Lot of stuff regarding aws. I havent worked with Fargate for example and dont know how it works or when to use it, but i heard other devs use it

Can someone list me a number of things i as a dev should know more regarding aws?

The whole of the UK pretty much in standstill for a football match.

... and I'm here trying to get my fucking app to deploy into Glassfish 5 on AWS Elastic Beanstalk.

Fucking AWS Elastic Beanstalk took a week of work to get fucking mounts set up. They invented their own version of docker compose that is missing half the features so I had to work my way from hacking their scripts to include options they don't support, to restarting the whole fucking docker service on every fucking deployment and now the shit finally works. How can most StackOverflow answers just say restart docker, this shit is not ok! I fucking hate sysadmin work. I want to code :(

Manual EC2 instances + Elastic load balancer or Elastic beanstalk for a PHP 7 application? I might have some cron jobs to be run too...

What's a good book to learn ELK hands-on?

I have an instance set up and working but want to need some more advanced features? like mapping, index templates, querying remotely for aggregations.

The Elastic docs feel very high level and maybe assumes u can read their minds... A lot of snippets I just go "uhm.. where do I put this in the file, which file?"

The jruby jit compiler is trash. 25 minutes for logstash to show signs of life.

So... I got a simple task of choosing the best fitting NIDS/MIDS, as well as deploying it, configuring to fit a specific use case and monitor its outputs for one client at work today...

I'm a little... Anxious. At a first glance, setting up like... Snort... Doesn't seem all that difficult, but I have no idea where this takes me and if what I come up with will ultimately be useful or not... Until now I did simple service configuration changes like apache, nginx, php... And a bit of database management with things like mariadb, mysql, postgresql, mongo or elastic... I feel so... Out of my usual waters.

Do you guys thing a person without a title in network security (or... Any title for that matter) can even manage this?...

This invite to an ElasticSearch webinar is epic:

webinars/proven-architectural-patterns-for-mature-elastic-stack-deployments?ultron=reference-architecture-webinar&blade=invite&hulk=email

Is there some sort of Query Builder for ElasticSearch?

I have ELK setup and in Kibana can generate all the aggregation visualizations but now I want the data to be usable in a program so it can generate reports like who are our top users.

But the aggregation queries seem to be very verbose... not sure how anyone can generate or understand it by hand vs telling Kibana I want a chart with X and Y axes using these terms.

IDeally I'd like to have Kibana then tell me what's the actual JSON/Elastic query it used to generate that but can't seem to find something like that.

For six months, I have tried to put geodjango on AWS elastic beanstalk and failed. Anyone with a working solution?

What do you guys think about deploying elastic search on App Engine Custom Runtime?

(Basically, an empty folder with an elastic search Dockerfile.)

I think it's a good idea: you can now deploy your code and storage application (Elastic search, Redis, etc) as services on your cluster.

You can use GCP magic to auto scale those services, you have so many good stuff that come with it.

And it's inside the same network as your services running in the same AppEngine project.

Using AWS DocumentDB elastic cluster for a sharded managed MongoDB
It's ass is so heavy!! More than 15min for scaling 2 -> 3 shards!
Am I taking a bad decision?

used bootstrap, and the found a prety elastic side menu animation, turned out it can'twork with the best css fw.. fml

can you use elastic search as a search engine for your app ?

because i see several weak points in it.

the increased latency after every bulk uploading of docs, meaning u cant ensure fast response time for users

the inability to add synonyms without closing the index ? this is either downtime or ill have to replicate an index to update the original and then switch back to it !!

idk i feel i either must have wrong info or elastic is very inefficient. I might be wrong, not too experienced with it so if I am let me know of some good resources and workarounds that helped you

Where can find an all in one guide to set up/configure a ELK environment manually from the tars (not yum, apt-get, docker image)?

I am following the component docs from Elastic but not sure how each component integrates with each other or how to set the mongo connection (DB is not local)

Any Elasticsearch gurus here? I have a box with too many young gen GCs (one per 2 or 3 seconds), and irregular, very long old gen GCs (One per several hours, taking around a minute and freeing about 2/3's of the old gen space) -- I was thinking changing the new gen ratio from 2/3 to something like 3/4 or 4/5.

However, after reading an elastic article about settings to never touch... I'm no longer so sure...

Only other option I was considering is going from CMS to G1GC to cut back on the old gen GC time... A minute long downtime for Elastic is rather problematic.

Any thoughts? The box is rather old - running Elastic 5.6 with 20 GBs of heap, 207 shards and 306k docs.

Logstash can't find configuration files
:/

Django project in a docker container in an ec2 instance of my aws vpc... i did use elastic beanstalk though, but needed to let my media files be uploaded to S3... couldn't figure it out yet...

Hi fellow devRanters, I need some advice on how to detect web traffic coming from bad/malicious bots and block them.

I have ELK (Elastic) stack set up to capture the logs from the sites, I have already blocked the ones that are obviously bad (bad user-agent, IP addresses known for spamming etc). I know you can tell by looking at how fast/frequently they crawl the site but how would I know if I block the one that's causing the malicious and non-human traffic? I am not sure if I should block access from other countries because I think the bots are from local.

I am lost, I don't know what else I can do - I can't use rate limiting on the sites and I can't sign up for a paid service cause management wants everything with the price of peanuts.

Rant:

Someone asked why I can't just read through the logs (from several mid-large scale websites) and pick out the baddies.

*facepalm* Here's the gigabytes log files.

I started using WordPress in the 1990's - building all kinds of sites that looked OK until all the plugins and new themes came along. As the years have gone by I've become bored with all the tedious little errors and bugs. To the point that I abandoned my print website 10 years ago.
Just tried to edit it today and FUCK NO get me out of here. It's like painting a Rembrandt with a fucking elastic band.

SIEM: Security Information and Event Management system

Within a SIEM there is usually a reporting, alerting, and learning framework wherein you perform investigations and threat hunting. Our SIEM is connected to our data lake through a glorified elastic backend.

Today we were figuring out how to get dynamic data that we store in our SIEM to show up in the regular data lake presentation layer. All the solutions only half worked or had barriers to progress that seemed larger than the proposed solution.

So now we're going with the proposed solution: send static data back into the data lake in order to pull it out on the normal frontend with all the enriched info. We're basically turning this thing into a damn feedback loop.

I hate designing solutions within the confines of COTS products.

How the fuck do you use and make a fields.yml for dynamic filebeat indexes?

Aka what if i don't want all the fields?

Sophomore year starting soon so I'm looking for new project (s) to complete in parallel with the studies.
Some are more design-y and some more backend-y but I recently started getting better at designing so :)

1) Learn some fragment shader stuff. I've always been messing around with graphics and have a game on steam, so I think that's a good idea to be paired with signal processing.

2) Reactive web services. Preferably with spring-boot or vert.x but

3) I would also like to dive into golang (and make some reactive thing with it)

4) WebAssembly seems nice... But I got some concerns

5) exercise making wireframes -> CSS (with some js)

6) I've never really done any real backed work with nodejs, except serving and aot compiling js, or doing gulp tasks

7) Implementing a whole project, or a fraction of it as serverless on aws

* I'm definitely going to use a couple very simple services to make a docker swarm with load balancing, etc, just because I know how everything works but got no practical knowledge

8) Design an esports jersey for the university department I'm in (shouldn't take long)

So what do you guys think? Recommendations are welcome :)

P.S. last year in review:
> A webapp running on a raspberry pi powering a reflex testing game on gpio (java/spring-boot , codename: buttonmasher)
> small Elastic search cluster to monitor some random university servers through kibana dashboards
> laser tracking on wall of *any* colour and variable light conditions via a webcam (opencv) , controlling the mouse pointer, whether you run it against a projector or any wall
> jstrain.herokuapp.com => a small JavaScript powered tool with a DSL to help you train more efficiently without a coach
> Various random Photoshop stuff

Any good recommendations on how to gather user metrics/instrumentation and visualize data?

The program is a WPF application with not internet connection so logging to file and get file is probably the only solution.

I've played a little with Serilog to file and trying to import the log into elastic search and visualize data with Kibana.

so.. i'm in a call right now and one participant just dropped the info, that it seems to be the case, that someone just switched the elastic/elasticsearch-php repository on github to private - or worse, deleted it.

does anyone has any information about this?

Thoughts on the Elastic stack? I.e. if you have used it and regretted it, please share your horror stories. Or, if you feel that it's great, share why that's the case or how did it help your company/business/product/whatever.

Software is elastic and mouldable. Not software development process.