I’m trying to setup a HA node cluster behind NGINX with an SSL endpoint. I will be providing this on port 35997 and 35998. However, I’ve run into an issue that I cannot solve.
The FAILING configuration is below. I’m testing this on https://explorer.zenon.info. I’ve setup a default node on the explorer pointing to https://secure.deeznnodez.com:35997, the configuration file for that endpoint is below.
upstream node {
server 10.36.39.10:35997;
server 10.36.39.11:35997;
}
server {
root /var/www/secure.deeznnodez.com/html;
index index.html;
server_name secure.deeznnodez.com;
location / {
try_files $uri $uri/ =404;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/secure.deeznnodez.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/secure.deeznnodez.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = secure.deeznnodez.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name secure.deeznnodez.com;
return 404; # managed by Certbot
}
server {
listen 35997 ssl;
server_name secure.deeznnodez.com;
ssl_certificate /etc/letsencrypt/live/secure.deeznnodez.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/secure.deeznnodez.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
location / {
proxy_pass http://node;
}
}
This setup produces a CORS error below:
Access to XMLHttpRequest at 'https://secure.deeznnodez.com:35997/' from origin 'https://explorer.zenon.info' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
polyfills.1b391f26029e71638c65.js:1 GET https://secure.deeznnodez.com:35997/ net::ERR_FAILED 403
When I change the config file above and make the following change, the CORS error goes away, and the endpoint works exactly as expected. No errors and no issues.
location / {
proxy_pass http://10.36.39.10:35997; #Going direct to the endpoint rather than through Load Balancer
}
Here is the TCP dump from both setups
Load Balancer Successful Request
07:53:37.214184 IP REMOVE.20423 > deeznnutz-lb02.35997: Flags [P.], seq 1322034519:1322035136, ack 1355387046, win 1026, length 617
07:53:37.214348 IP deeznnutz-lb02.35997 > REMOVE.20423: Flags [.], ack 617, win 498, length 0
07:53:37.216028 IP deeznnutz-lb02.39304 > 10.36.39.10.35997: Flags [S], seq 861538721, win 64240, options [mss 1460,sackOK,TS val 3947167510 ecr 0,nop,wscale 7], length 0
07:53:37.216218 IP 10.36.39.10.35997 > deeznnutz-lb02.39304: Flags [S.], seq 2646200694, ack 861538722, win 65160, options [mss 1460,sackOK,TS val 750543776 ecr 3947167510,nop,wscale 7], length 0
07:53:37.216233 IP deeznnutz-lb02.39304 > 10.36.39.10.35997: Flags [.], ack 1, win 502, options [nop,nop,TS val 3947167510 ecr 750543776], length 0
07:53:37.216257 IP deeznnutz-lb02.39304 > 10.36.39.10.35997: Flags [P.], seq 1:581, ack 1, win 502, options [nop,nop,TS val 3947167510 ecr 750543776], length 580
07:53:37.216766 IP 10.36.39.10.35997 > deeznnutz-lb02.39304: Flags [.], ack 581, win 505, options [nop,nop,TS val 750543777 ecr 3947167510], length 0
07:53:37.217194 IP 10.36.39.10.35997 > deeznnutz-lb02.39304: Flags [P.], seq 1:170, ack 581, win 505, options [nop,nop,TS val 750543777 ecr 3947167510], length 169
07:53:37.429428 IP deeznnutz-lb02.39312 > 10.36.39.10.35997: Flags [.], ack 3908596493, win 1134, options [nop,nop,TS val 3947167723 ecr 750543989], length 0
07:53:37.430072 IP 10.36.39.10.35997 > deeznnutz-lb02.39312: Flags [P.], seq 1:4097, ack 0, win 504, options [nop,nop,TS val 750543990 ecr 3947167723], length 4096
07:53:37.431024 IP 10.36.39.10.35997 > deeznnutz-lb02.39312: Flags [P.], seq 16385:20481, ack 0, win 504, options [nop,nop,TS val 750543991 ecr 3947167724], length 4096
07:53:37.431035 IP deeznnutz-lb02.39312 > 10.36.39.10.35997: Flags [.], ack 20481, win 1204, options [nop,nop,TS val 3947167725 ecr 750543991], length 0
07:53:37.431060 IP 10.36.39.10.35997 > deeznnutz-lb02.39312: Flags [P.], seq 20481:24577, ack 0, win 504, options [nop,nop,TS val 750543991 ecr 3947167724], length 4096
07:53:37.431064 IP deeznnutz-lb02.39312 > 10.36.39.10.35997: Flags [.], ack 24577, win 1204, options [nop,nop,TS val 3947167725 ecr 750543991], length 0
07:53:37.431105 IP 10.36.39.10.35997 > deeznnutz-lb02.39312: Flags [P.], seq 24577:28673, ack 0, win 504, options [nop,nop,TS val 750543991 ecr 3947167725], length 4096
07:53:37.431109 IP deeznnutz-lb02.39312 > 10.36.39.10.35997: Flags [.], ack 28673, win 1204, options [nop,nop,TS val 3947167725 ecr 750543991], length 0
07:53:37.431677 IP 10.36.39.10.35997 > deeznnutz-lb02.39312: Flags [P.], seq 28673:32769, ack 0, win 504, options [nop,nop,TS val 750543991 ecr 3947167725], length 4096
07:53:37.431704 IP deeznnutz-lb02.39312 > 10.36.39.10.35997: Flags [.], ack 32769, win 1204, options [nop,nop,TS val 3947167726 ecr 750543991], length 0
07:53:37.431714 IP 10.36.39.10.35997 > deeznnutz-lb02.39312: Flags [P.], seq 32769:36865, ack 0, win 504, options [nop,nop,TS val 750543992 ecr 3947167725], length 4096
07:53:37.431768 IP 10.36.39.10.35997 > deeznnutz-lb02.39312: Flags [P.], seq 36865:40961, ack 0, win 504, options [nop,nop,TS val 750543992 ecr 3947167726], length 4096
Endpoint Successful Request
12:53:37.215409 IP 10.36.39.30.39304 > deeznnutz-node01.35997: Flags [S], seq 861538721, win 64240, options [mss 1460,sackOK,TS val 3947167510 ecr 0,nop,wscale 7], length 0
12:53:37.215499 IP deeznnutz-node01.35997 > 10.36.39.30.39304: Flags [S.], seq 2646200694, ack 861538722, win 65160, options [mss 1460,sackOK,TS val 750543776 ecr 3947167510,nop,wscale 7], length 0
12:53:37.215567 IP 10.36.39.30.39304 > deeznnutz-node01.35997: Flags [.], ack 1, win 502, options [nop,nop,TS val 3947167510 ecr 750543776], length 0
12:53:37.216018 IP 10.36.39.30.39304 > deeznnutz-node01.35997: Flags [P.], seq 1:581, ack 1, win 502, options [nop,nop,TS val 3947167510 ecr 750543776], length 580
12:53:37.216033 IP deeznnutz-node01.35997 > 10.36.39.30.39304: Flags [.], ack 581, win 505, options [nop,nop,TS val 750543777 ecr 3947167510], length 0
12:53:37.216479 IP deeznnutz-node01.35997 > 10.36.39.30.39304: Flags [P.], seq 1:170, ack 581, win 505, options [nop,nop,TS val 750543777 ecr 3947167510], length 169
12:53:37.216544 IP 10.36.39.30.39304 > deeznnutz-node01.35997: Flags [.], ack 170, win 501, options [nop,nop,TS val 3947167511 ecr 750543777], length 0
12:53:37.216561 IP deeznnutz-node01.35997 > 10.36.39.30.39304: Flags [F.], seq 170, ack 581, win 505, options [nop,nop,TS val 750543777 ecr 3947167511], length 0
12:53:37.306129 IP 10.36.39.30.39306 > deeznnutz-node01.35997: Flags [S], seq 3125397100, win 64240, options [mss 1460,sackOK,TS val 3947167601 ecr 0,nop,wscale 7], length 0
12:53:37.306157 IP deeznnutz-node01.35997 > 10.36.39.30.39306: Flags [S.], seq 3021095236, ack 3125397101, win 65160, options [mss 1460,sackOK,TS val 750543867 ecr 3947167601,nop,wscale 7], length 0
12:53:37.306243 IP 10.36.39.30.39306 > deeznnutz-node01.35997: Flags [.], ack 1, win 502, options [nop,nop,TS val 3947167601 ecr 750543867], length 0
12:53:37.306285 IP 10.36.39.30.39306 > deeznnutz-node01.35997: Flags [P.], seq 1:709, ack 1, win 502, options [nop,nop,TS val 3947167601 ecr 750543867], length 708
12:53:37.306288 IP deeznnutz-node01.35997 > 10.36.39.30.39306: Flags [.], ack 709, win 504, options [nop,nop,TS val 750543867 ecr 3947167601], length 0
12:53:37.307067 IP deeznnutz-node01.35997 > 10.36.39.30.39306: Flags [P.], seq 1:731, ack 709, win 504, options [nop,nop,TS val 750543868 ecr 3947167601], length 730
12:53:37.307150 IP 10.36.39.30.39306 > deeznnutz-node01.35997: Flags [.], ack 731, win 498, options [nop,nop,TS val 3947167602 ecr 750543868], length 0
12:53:37.307188 IP 10.36.39.30.39306 > deeznnutz-node01.35997: Flags [F.], seq 709, ack 731, win 501, options [nop,nop,TS val 3947167602 ecr 750543868], length 0
12:53:37.307270 IP deeznnutz-node01.35997 > 10.36.39.30.39306: Flags [F.], seq 731, ack 710, win 504, options [nop,nop,TS val 750543868 ecr 3947167602], length 0
12:53:37.307331 IP 10.36.39.30.39306 > deeznnutz-node01.35997: Flags [.], ack 732, win 501, options [nop,nop,TS val 3947167602 ecr 750543868], length 0
12:53:37.309724 IP 10.36.39.30.39308 > deeznnutz-node01.35997: Flags [S], seq 4018848022, win 64240, options [mss 1460,sackOK,TS val 3947167604 ecr 0,nop,wscale 7], length 0
12:53:37.309741 IP deeznnutz-node01.35997 > 10.36.39.30.39308: Flags [S.], seq 518442306, ack 4018848023, win 65160, options [mss 1460,sackOK,TS val 750543870 ecr 3947167604,nop,wscale 7], length 0
When I change back to
location / {
proxy_pass http://node; #Going THROUGH the NGINX load balancer now
}
Load Balancer Failed Request
07:58:13.680113 IP REMOVE.22118 > deeznnutz-lb02.35997: Flags [P.], seq 2091000900:2091001517, ack 4099951145, win 1022, length 617
07:58:13.680341 IP deeznnutz-lb02.39530 > 10.36.39.10.35997: Flags [S], seq 333245589, win 64240, options [mss 1460,sackOK,TS val 3947443974 ecr 0,nop,wscale 7], length 0
07:58:13.680624 IP 10.36.39.10.35997 > deeznnutz-lb02.39530: Flags [S.], seq 1268591503, ack 333245590, win 65160, options [mss 1460,sackOK,TS val 750820239 ecr 3947443974,nop,wscale 7], length 0
07:58:13.680651 IP deeznnutz-lb02.39530 > 10.36.39.10.35997: Flags [.], ack 1, win 502, options [nop,nop,TS val 3947443975 ecr 750820239], length 0
07:58:13.680676 IP deeznnutz-lb02.39530 > 10.36.39.10.35997: Flags [P.], seq 1:571, ack 1, win 502, options [nop,nop,TS val 3947443975 ecr 750820239], length 570
07:58:13.680734 IP 10.36.39.10.35997 > deeznnutz-lb02.39530: Flags [.], ack 571, win 505, options [nop,nop,TS val 750820239 ecr 3947443975], length 0
07:58:13.681074 IP 10.36.39.10.35997 > deeznnutz-lb02.39530: Flags [P.], seq 1:229, ack 571, win 505, options [nop,nop,TS val 750820239 ecr 3947443975], length 228
07:58:13.681079 IP deeznnutz-lb02.39530 > 10.36.39.10.35997: Flags [.], ack 229, win 501, options [nop,nop,TS val 3947443975 ecr 750820239], length 0
07:58:13.750491 IP REMOVE.22118 > deeznnutz-lb02.35997: Flags [.], ack 306, win 1021, length 0
Endpoint Failed Request
12:58:13.678232 IP 10.36.39.30.39530 > deeznnutz-node01.35997: Flags [S], seq 333245589, win 64240, options [mss 1460,sackOK,TS val 3947443974 ecr 0,nop,wscale 7], length 0
12:58:13.678394 IP deeznnutz-node01.35997 > 10.36.39.30.39530: Flags [S.], seq 1268591503, ack 333245590, win 65160, options [mss 1460,sackOK,TS val 750820239 ecr 3947443974,nop,wscale 7], length 0
12:58:13.678483 IP 10.36.39.30.39530 > deeznnutz-node01.35997: Flags [.], ack 1, win 502, options [nop,nop,TS val 3947443975 ecr 750820239], length 0
12:58:13.678514 IP 10.36.39.30.39530 > deeznnutz-node01.35997: Flags [P.], seq 1:571, ack 1, win 502, options [nop,nop,TS val 3947443975 ecr 750820239], length 570
12:58:13.678519 IP deeznnutz-node01.35997 > 10.36.39.30.39530: Flags [.], ack 571, win 505, options [nop,nop,TS val 750820239 ecr 3947443975], length 0
12:58:13.678838 IP deeznnutz-node01.35997 > 10.36.39.30.39530: Flags [P.], seq 1:229, ack 571, win 505, options [nop,nop,TS val 750820239 ecr 3947443975], length 228
12:58:13.678884 IP deeznnutz-node01.35997 > 10.36.39.30.39530: Flags [F.], seq 229, ack 571, win 505, options [nop,nop,TS val 750820239 ecr 3947443975], length 0
12:58:13.678913 IP 10.36.39.30.39530 > deeznnutz-node01.35997: Flags [.], ack 229, win 501, options [nop,nop,TS val 3947443975 ecr 750820239], length 0
When I use the load balancer in NGINX it seems to pass the first request directly to the endpoint and the connection fails. Here is what the headers look like in the failed request.
When I remove the load balancer in NGINX the Load Balancer seems to respond to the first request before sending it to the endpoint. Here is what a successful request looks like. The LB is stripping the “Remote Address” and removing Access-Control-Allow-Origin: * and Vary: Origin.
Can anyone see a problem with my Load Balancer Setup? I think I need the LB to respond BEFORE sending the request to the endpoint (like in the successful communications above).
I have generally followed all the guidelines out there. Specifically this one: