We are using Envoy as an edge proxy and sidecar proxy for our service mesh data plane at Aimo. From the beginning we designed our architecture with service to service communication in mind, being careful not to add too much complexity.
Envoy at core is basically a network proxy, but its dynamic configuration and extensibility makes it a powerful building block for many other tasks related to microservice architecture, like service discovery, routing, and observability.
In our case the journey of each request starts by going from an Application Load Balancer (currently we manage TLS termination here) to the Envoy (as an edge proxy) cluster then it will forward the request to the corresponding service based on the url path. Envoy fetch periodically and dynamically information about services from a service discovery cluster to know their locations (ip and port).
At the edge proxy level we also use a filter for external authorization that allows us to check if each incoming request is authorized. If so original headers are modified to include relevant information for any service, like a user id or a request id for distributed tracing.
Each service has Envoy as a sidecar proxy so service communication (in and out) is only done through that proxy, each application container makes a request to the proxy (located in localhost:port) indicating with a header the name of the service it wants to communicate with, while also achieving client side load balancing in this way.
As I already mentioned, dynamic configuration is a powerful feature of Envoy, so you can configure the endpoints where it will get information about listeners and clusters. Envoy allows you set a level of dynamism but you can also have a simple static configuration.
Envoy is a powerful tool, it has many features that we expect to use in the future like GRPC for service to service communication or a full dynamic configuration to achieve a better integration with our control plane and deployment strategy.
If you have any experience with Envoy or you want to share some ideas, please send me a DM on Twitter @ArturoBermejo.