Internal Listener

Envoy supports user-space sockets that enable establishing TCP streams from an upstream cluster to a listener without using the system network API. A listener that accepts user space connections is called an internal listener. The internal listener name identifies the server for a client internal address.

To utilize internal listeners, several components need to be configured in conjunction. First, the bootstrap extension must be enabled. This extension registers a client connection factory that allows internal listeners to accept connections from within Envoy. Second, an internal listener must be defined:

name: internal_listener
internal_listener: {}
filter_chains:
- filters: []

Third, an upstream cluster must include an endpoint with an internal address referencing the internal listener by name:

name: cluster_0
load_assignment:
  cluster_name: cluster_0
  endpoints:
  - lb_endpoints:
    - endpoint:
        address:
          envoy_internal_address:
            server_listener_name: internal_listener

Internal upstream transport

Internal upstream transport extension enables exchange of the filter state from the downstream listener to the internal listener through a user space socket. This additional state can be in the form of the resource metadata obtained from the upstream host or the filter state objects. This is an optional extension that may be added to the upstream clusters with internal addresses.

This extension emits the following statistics:

Name	Type	Description
no_metadata	Counter	Metadata key is absent from the import location.

Endpoint disambiguation

In case there are multiple endpoints referencing the same internal listener in a single upstream cluster, use endpoint ID field to improve change tracking in the cluster pool. This field in combination with the internal listener name uniquely identify an endpoint in the pool. For example, if multiple endpoints use distinct upstream host metadata, the value of the field should be a hash or a distinct value from the host metadata. For tunneling internal listeners, the value should be the actual destination address propagated to the internal listener.

Examples

Simple chaining

A minimal example that chains two TCP proxies to forward connections from port 9999 to port 10000 via an internal listener can be found here

Encapsulate HTTP GET requests in a HTTP CONNECT request

Currently Envoy HTTP connection manager cannot proxy a GET request in an upstream HTTP CONNECT request. This requirement can be accomplished by setting up the upstream endpoint of HTTP connection manager to the internal listener address. Meanwhile, another internal listener binding to the above listener address includes a TCP proxy with tunneling config.

A sample config can be found here

Decapsulate the CONNECT requests

There are some complicated GET-in-CONNECT requests across services or edges. In order to proxy the GET request within Envoy, two layer of HTTP connection manager is demanded. The first HHTTP connection manager layer extract the TCP stream from a CONNECT request and redirect the TCP stream to the second HTTP connection manager layer to parse the common GET requests.

A sample config can be found here

The above two examples can be tested together as follows:

bazel-bin/source/exe/envoy-static --config-path configs/encapsulate_http_in_http2_connect.yaml --disable-hot-restart
bazel-bin/source/exe/envoy-static --config-path configs/terminate_http_in_http2_connect.yaml --disable-hot-restart.
curl 127.0.0.1:10000