All the microservices are planning to communicate via kafka. And they are planning to put a REST API between kafka and the service. So all services hit the REST API with the events and the REST API send events to kafka.

I don't see any benefit of having a REST API between the service and kafka. What are the opinions in the wild?

sounds like an orchestrator service maybe?

Not a big fan personally

@Hazarth Seems just like a bridge to me. Between kafka and the event producer. I feel its unncessary and the services can send events to kafka directly.

What's the point of using Kafka then?

Use kafka directly.
In case you want a middleware wrapper - you can use a shared library for each microservice to use.
In case you want to encrypt what passes through kafka - use transparent sidecars.

@rantsauce Exactly my point. And the question. The REST server seems to only take JSON as input. I mean what's the point in using a schema registry then. FYI, It is also being used.

@magicMirror Shared lib is what I'm pushing for as well.

Btw, reg sidecars and encryption, that's new to me. I'm assuming the sidcars intercept events from the service and encrypt it before sending it to kafka. Correct me if I'm wrong. Could you point some ref to any implementation?

is it really just a bridge though? Is it not doing anything else, for example contacting multiple services to build up the Kafka event?

That's how we used something similar in the past since no single micro-service had the entire view of data, so the middle service was responsible for assembling whatever was necessary to push into stored or event pipes.

It kinda made sense on paper, but in reality it was hell

@Hazarth It is not contacting services to build up events. Services hit these APIs with the event payload. REST server job is simple here - get the payload send the payload to kafka.

I'm not sure if they have addressed producer errors from kafka in the design. AFAIK, they didn't.

@rantsauce The reason I got for the REST server's existence is "Today we use kafka and tomorrow we use something else. We need to build a wrapper around whatever that is". I disagree. If we go by the same, servers existed even before REST came into picture. I think they are overengineering this part of the design.

@kaipulla
Kafka can work over https/wss. That way there is no need to encrypt anything.

Using sidecars to handle the encryption is not really recommend. The idea is a reverse proxy that has a secure side, and an open side - it is similar to the middleware suggested in the original rant.

I was killing myself trying encrypt smth.

@magicMirror using a service mesh sidecar is the correct solution. You can get mTLS between the services.