Content microservice proxy error in a load-balanced environment

Question

We are getting proxy error while accessing content service URL . It tries to access it for 2-3 minutes and then show message:

Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /sdl/content.svc.

Reason: Error reading from remote server

When I tried to access load-balancer URL it again tries to access it for 2-3 minutes and then shows message

Available content services:
•None

In log file I can see these messages:

DEBUG ContentInfoResolverImpl - http://lbdomian.com:9081/sdl/client/v2/content.svc returned status code: 404
DEBUG ContentInfoResolverImpl - http://lbdomian.com:9081/sdl/client/v4/content.svc returned status code: 404

Here Infrastructure is setup to work as:

From log it seems that it is trying to access load balancer URL of V2 and V4 client which is not accessible from CD servers. I am wondering why it is trying to access load-balancer URL? Is there something I am missing here?

We can access V2 and V4 clients and from outside and this is able to fetch the content.

Details added based on questions asked:

Result of query https://externaldomian.com/discovery.svc/Environment/ContentServiceCapability:

<?xml version="1.0" encoding="UTF-8"?>
<entry xmlns:metadata="http://docs.oasis-open.org/odata/ns/metadata" xmlns:data="http://docs.oasis-open.org/odata/ns/data" xmlns="http://www.w3.org/2005/Atom" metadata:context="http://lbdomian.com:9082/sdl/discovery.svc/$metadata#Environment/ContentServiceCapability" xml:base="http://lbdomian.com:9082/sdl/discovery.svc">
    <id>http://lbdomian.com:9082/sdl/discovery.svc/ContentServiceCapabilities('DefaultContentService')</id>
    <title/>
    <summary/>
    <updated>2019-11-12T07:50:39.497Z</updated>
    <author>
        <name>SDL OData v4 framework</name>
    </author>
    <link rel="edit" title="ContentServiceCapability" href="ContentServiceCapabilities('DefaultContentService')"/>
    <link rel="http://docs.oasis-open.org/odata/ns/related/Environment" type="application/atom+xml;type=entry" title="Environment" href="ContentServiceCapabilities('DefaultContentService')/Environment"/>
    <link rel="http://docs.oasis-open.org/odata/ns/relatedlinks/Environment" type="application/xml" title="Environment" href="ContentServiceCapabilities('DefaultContentService')/Environment/$ref"/>
    <category scheme="http://docs.oasis-open.org/odata/ns/scheme" term="#Tridion.WebDelivery.Platform.ContentServiceCapability"/>
    <content type="application/xml">
        <metadata:properties>
            <data:id>DefaultContentService</data:id>
            <data:LastUpdateTime metadata:type="Int64">1573242354944</data:LastUpdateTime>
            <data:URI>https://externaldomian.com/sdl/content.svc</data:URI>
            <data:ExtensionProperties metadata:type="#Collection(Tridion.WebDelivery.Platform.ContentKeyValuePair)"/>
        </metadata:properties>
    </content>
</entry>

Answer 1

We had similar kind of setup... We created path forwarding rules for all the sdl micro services. We treated deployer and content service separately.

Synopsis of what we did:

• Configured all services on the deployer boxes using lb domain name... this you can do by replacing server name with lb name in the cd_storage file of the respective service and restart the services.
• Deployer: Only one deployer service will be up and running on one of the content deliver server other service is stopped on box 2.
• Content service: We added following configuration on the load balancer.

location ~* ^/qa-content-rule(.*) {
             rewrite ^ $request_uri;
             rewrite ^/qa-content-rule/(.*) $1 break;  
             return 400;
             proxy_redirect off;
             proxy_pass_header  Set-Cookie;
             proxy_cookie_domain $host <<domain>>;
             client_max_body_size 100M;
             proxy_set_header   X-Real-IP            $remote_addr;
             proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
             proxy_set_header   X-Forwarded-Proto $scheme;
             proxy_set_header   Host    <<servername>>:8081;
             proxy_set_header   X-NginX-Proxy    true;
             proxy_set_header   Connection "";
             proxy_http_version 1.1;
             proxy_pass         http://qa-content-rule/$1$is_args$args;
           }
upstream qa-content-rule {server <<DeployerServername>>:8081; server <<DeployerServername>>:8081; 
keepalive 64; }

Please note: we have added following notes to return the binaries as it was throwing 404 at LB

rewrite ^
   $request_uri;
             rewrite ^/qa-content-rule/(.*) $1 break;  
             return 400;

Hope this helps!