Mixing
Kubernetes nginx refresh ip address when upstream service IP changes
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
I am using nginx to proxy requests to multiple Headless services of StatefulSets in a kubernetes cluster. The problem I am having now is that whenever the service IP changes, the nginx does not resolve the service endpoint to updated IP address but still using the outdated cached IP address. I have tried to use the variable in proxy_pass in nginx configuration but it's to no avail at all. Both on my local cluster as well as deployed on AWS EKS. Here is a snippet of my nginx configuration:
upstream svc-foo {
server svc-foo:8080;
keepalive 1024;
}
server {
resolver 127.0.0.1 [::1]:5353 valid=10s;
set $foo http://svc-foo;
location /foo/ {
proxy_pass $foo;
proxy_http_version 1.1;
}
}
I expect no downtime when I update the service which causes the service IP to change. Any insight and advice is appreciated.
nginx dynamic service dns
kubernetes asked Jan 3 at 9:40
Kok How TehKok How Teh
158317
add a comment |
I am using nginx to proxy requests to multiple Headless services of StatefulSets in a kubernetes cluster. The problem I am having now is that whenever the service IP changes, the nginx does not resolve the service endpoint to updated IP address but still using the outdated cached IP address. I have tried to use the variable in proxy_pass in nginx configuration but it's to no avail at all. Both on my local cluster as well as deployed on AWS EKS. Here is a snippet of my nginx configuration:
upstream svc-foo {
server svc-foo:8080;
keepalive 1024;
}
server {
resolver 127.0.0.1 [::1]:5353 valid=10s;
set $foo http://svc-foo;
location /foo/ {
proxy_pass $foo;
proxy_http_version 1.1;
}
}
I expect no downtime when I update the service which causes the service IP to change. Any insight and advice is appreciated.
nginx dynamic service dns
kubernetes asked Jan 3 at 9:40
Kok How TehKok How Teh
158317
add a comment |
I am using nginx to proxy requests to multiple Headless services of StatefulSets in a kubernetes cluster. The problem I am having now is that whenever the service IP changes, the nginx does not resolve the service endpoint to updated IP address but still using the outdated cached IP address. I have tried to use the variable in proxy_pass in nginx configuration but it's to no avail at all. Both on my local cluster as well as deployed on AWS EKS. Here is a snippet of my nginx configuration:
upstream svc-foo {
server svc-foo:8080;
keepalive 1024;
}
server {
resolver 127.0.0.1 [::1]:5353 valid=10s;
set $foo http://svc-foo;
location /foo/ {
proxy_pass $foo;
proxy_http_version 1.1;
}
}
I expect no downtime when I update the service which causes the service IP to change. Any insight and advice is appreciated.
nginx dynamic service dns
kubernetes asked Jan 3 at 9:40
Kok How TehKok How Teh
158317
I am using nginx to proxy requests to multiple Headless services of StatefulSets in a kubernetes cluster. The problem I am having now is that whenever the service IP changes, the nginx does not resolve the service endpoint to updated IP address but still using the outdated cached IP address. I have tried to use the variable in proxy_pass in nginx configuration but it's to no avail at all. Both on my local cluster as well as deployed on AWS EKS. Here is a snippet of my nginx configuration:
upstream svc-foo {
server svc-foo:8080;
keepalive 1024;
}
server {
resolver 127.0.0.1 [::1]:5353 valid=10s;
set $foo http://svc-foo;
location /foo/ {
proxy_pass $foo;
proxy_http_version 1.1;
}
}
I expect no downtime when I update the service which causes the service IP to change. Any insight and advice is appreciated.
nginx dynamic service dns
kubernetes nginx dynamic service dns
kubernetes asked Jan 3 at 9:40
Kok How TehKok How Teh
158317
asked Jan 3 at 9:40
Kok How TehKok How Teh
158317
asked Jan 3 at 9:40
Kok How TehKok How Teh
158317
asked Jan 3 at 9:40
Kok How TehKok How Teh
158317
asked Jan 3 at 9:40
Kok How TehKok How Teh
158317
158317
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
Best way is to use an DNS sidecar on your nginx pod as below:
apiVersion: v1
kind: ConfigMap
metadata:
namespace: issue-795
name: nginx-config
data:
nginx.conf: |-
user nginx;
worker_processes 1;
events {
worker_connections 4096; ## Default: 1024
}
http {
server { # php/fastcgi
listen 80;
resolver 127.0.0.1:53 ipv6=off valid=10s;
set $upstream http://backend:8080;
location / {
proxy_pass $upstream;
proxy_http_version 1.1;
}
}
}
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
namespace: issue-795
name: proxy
spec:
replicas: 1
template:
metadata:
labels:
app: proxy
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
volumeMounts:
- name: nginx-config
mountPath: /etc/nginx/nginx.conf
subPath: nginx.conf
- name: dnsmasq
image: "janeczku/go-dnsmasq:release-1.0.7"
args:
- --listen
- "127.0.0.1:53"
- --default-resolver
- --append-search-domains
volumes:
- name: nginx-config
configMap:
name: nginx-config
---
apiVersion: v1
kind: Service
metadata:
namespace: issue-795
name: backend
spec:
ports:
- port: 80
targetPort: 8080
clusterIP: None
selector:
app: backend
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: backend
namespace: issue-795
spec:
serviceName: "backend"
replicas: 2
selector:
matchLabels:
app: backend
template:
metadata:
labels:
app: backend
spec:
containers:
- name: echoserver
image: gcr.io/google_containers/echoserver:1.4
imagePullPolicy: Always
ports:
- containerPort: 8080
answered Jan 3 at 22:40
M00nF1shM00nF1sh
361
add a comment |
I would recommand the use of Ingress resource on Kubernetes with the Nginx Ingress Controller.
Its whole purpose is to have a proxy inside your Kubernetes cluster that redirects the traffic to ClusterIP Services.
So you only have one external ELB that redirects all the traffic into your Kubernetes cluster. The Ingress Controller then redirects the traffic to different services.
For more advanced ingress controllers, you can look at Kong Ingress Controller.
answered Jan 3 at 9:51
Quentin RevelQuentin Revel
75539
I tried but it only creates a classic ELB. I need ALB and therefore I use aws-alb-ingress-controller > Ingress > Nginx (NodePort) > svc-<foo> and the outdated IP problem happens in the Nginx NodePort :(
– Kok How Teh
Jan 3 at 13:07
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54019648%2fkubernetes-nginx-refresh-ip-address-when-upstream-service-ip-changes%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
Best way is to use an DNS sidecar on your nginx pod as below:
apiVersion: v1
kind: ConfigMap
metadata:
namespace: issue-795
name: nginx-config
data:
nginx.conf: |-
user nginx;
worker_processes 1;
events {
worker_connections 4096; ## Default: 1024
}
http {
server { # php/fastcgi
listen 80;
resolver 127.0.0.1:53 ipv6=off valid=10s;
set $upstream http://backend:8080;
location / {
proxy_pass $upstream;
proxy_http_version 1.1;
}
}
}
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
namespace: issue-795
name: proxy
spec:
replicas: 1
template:
metadata:
labels:
app: proxy
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
volumeMounts:
- name: nginx-config
mountPath: /etc/nginx/nginx.conf
subPath: nginx.conf
- name: dnsmasq
image: "janeczku/go-dnsmasq:release-1.0.7"
args:
- --listen
- "127.0.0.1:53"
- --default-resolver
- --append-search-domains
volumes:
- name: nginx-config
configMap:
name: nginx-config
---
apiVersion: v1
kind: Service
metadata:
namespace: issue-795
name: backend
spec:
ports:
- port: 80
targetPort: 8080
clusterIP: None
selector:
app: backend
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: backend
namespace: issue-795
spec:
serviceName: "backend"
replicas: 2
selector:
matchLabels:
app: backend
template:
metadata:
labels:
app: backend
spec:
containers:
- name: echoserver
image: gcr.io/google_containers/echoserver:1.4
imagePullPolicy: Always
ports:
- containerPort: 8080
answered Jan 3 at 22:40
M00nF1shM00nF1sh
361
add a comment |
Best way is to use an DNS sidecar on your nginx pod as below:
apiVersion: v1
kind: ConfigMap
metadata:
namespace: issue-795
name: nginx-config
data:
nginx.conf: |-
user nginx;
worker_processes 1;
events {
worker_connections 4096; ## Default: 1024
}
http {
server { # php/fastcgi
listen 80;
resolver 127.0.0.1:53 ipv6=off valid=10s;
set $upstream http://backend:8080;
location / {
proxy_pass $upstream;
proxy_http_version 1.1;
}
}
}
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
namespace: issue-795
name: proxy
spec:
replicas: 1
template:
metadata:
labels:
app: proxy
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
volumeMounts:
- name: nginx-config
mountPath: /etc/nginx/nginx.conf
subPath: nginx.conf
- name: dnsmasq
image: "janeczku/go-dnsmasq:release-1.0.7"
args:
- --listen
- "127.0.0.1:53"
- --default-resolver
- --append-search-domains
volumes:
- name: nginx-config
configMap:
name: nginx-config
---
apiVersion: v1
kind: Service
metadata:
namespace: issue-795
name: backend
spec:
ports:
- port: 80
targetPort: 8080
clusterIP: None
selector:
app: backend
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: backend
namespace: issue-795
spec:
serviceName: "backend"
replicas: 2
selector:
matchLabels:
app: backend
template:
metadata:
labels:
app: backend
spec:
containers:
- name: echoserver
image: gcr.io/google_containers/echoserver:1.4
imagePullPolicy: Always
ports:
- containerPort: 8080
answered Jan 3 at 22:40
M00nF1shM00nF1sh
361
add a comment |
Best way is to use an DNS sidecar on your nginx pod as below:
apiVersion: v1
kind: ConfigMap
metadata:
namespace: issue-795
name: nginx-config
data:
nginx.conf: |-
user nginx;
worker_processes 1;
events {
worker_connections 4096; ## Default: 1024
}
http {
server { # php/fastcgi
listen 80;
resolver 127.0.0.1:53 ipv6=off valid=10s;
set $upstream http://backend:8080;
location / {
proxy_pass $upstream;
proxy_http_version 1.1;
}
}
}
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
namespace: issue-795
name: proxy
spec:
replicas: 1
template:
metadata:
labels:
app: proxy
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
volumeMounts:
- name: nginx-config
mountPath: /etc/nginx/nginx.conf
subPath: nginx.conf
- name: dnsmasq
image: "janeczku/go-dnsmasq:release-1.0.7"
args:
- --listen
- "127.0.0.1:53"
- --default-resolver
- --append-search-domains
volumes:
- name: nginx-config
configMap:
name: nginx-config
---
apiVersion: v1
kind: Service
metadata:
namespace: issue-795
name: backend
spec:
ports:
- port: 80
targetPort: 8080
clusterIP: None
selector:
app: backend
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: backend
namespace: issue-795
spec:
serviceName: "backend"
replicas: 2
selector:
matchLabels:
app: backend
template:
metadata:
labels:
app: backend
spec:
containers:
- name: echoserver
image: gcr.io/google_containers/echoserver:1.4
imagePullPolicy: Always
ports:
- containerPort: 8080
answered Jan 3 at 22:40
M00nF1shM00nF1sh
361
Best way is to use an DNS sidecar on your nginx pod as below:
apiVersion: v1
kind: ConfigMap
metadata:
namespace: issue-795
name: nginx-config
data:
nginx.conf: |-
user nginx;
worker_processes 1;
events {
worker_connections 4096; ## Default: 1024
}
http {
server { # php/fastcgi
listen 80;
resolver 127.0.0.1:53 ipv6=off valid=10s;
set $upstream http://backend:8080;
location / {
proxy_pass $upstream;
proxy_http_version 1.1;
}
}
}
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
namespace: issue-795
name: proxy
spec:
replicas: 1
template:
metadata:
labels:
app: proxy
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
volumeMounts:
- name: nginx-config
mountPath: /etc/nginx/nginx.conf
subPath: nginx.conf
- name: dnsmasq
image: "janeczku/go-dnsmasq:release-1.0.7"
args:
- --listen
- "127.0.0.1:53"
- --default-resolver
- --append-search-domains
volumes:
- name: nginx-config
configMap:
name: nginx-config
---
apiVersion: v1
kind: Service
metadata:
namespace: issue-795
name: backend
spec:
ports:
- port: 80
targetPort: 8080
clusterIP: None
selector:
app: backend
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: backend
namespace: issue-795
spec:
serviceName: "backend"
replicas: 2
selector:
matchLabels:
app: backend
template:
metadata:
labels:
app: backend
spec:
containers:
- name: echoserver
image: gcr.io/google_containers/echoserver:1.4
imagePullPolicy: Always
ports:
- containerPort: 8080
answered Jan 3 at 22:40
M00nF1shM00nF1sh
361
answered Jan 3 at 22:40
M00nF1shM00nF1sh
361
answered Jan 3 at 22:40
M00nF1shM00nF1sh
361
answered Jan 3 at 22:40
M00nF1shM00nF1sh
361
361
add a comment |
add a comment |
I would recommand the use of Ingress resource on Kubernetes with the Nginx Ingress Controller.
Its whole purpose is to have a proxy inside your Kubernetes cluster that redirects the traffic to ClusterIP Services.
So you only have one external ELB that redirects all the traffic into your Kubernetes cluster. The Ingress Controller then redirects the traffic to different services.
For more advanced ingress controllers, you can look at Kong Ingress Controller.
answered Jan 3 at 9:51
Quentin RevelQuentin Revel
75539
I tried but it only creates a classic ELB. I need ALB and therefore I use aws-alb-ingress-controller > Ingress > Nginx (NodePort) > svc-<foo> and the outdated IP problem happens in the Nginx NodePort :(
– Kok How Teh
Jan 3 at 13:07
add a comment |
I would recommand the use of Ingress resource on Kubernetes with the Nginx Ingress Controller.
Its whole purpose is to have a proxy inside your Kubernetes cluster that redirects the traffic to ClusterIP Services.
So you only have one external ELB that redirects all the traffic into your Kubernetes cluster. The Ingress Controller then redirects the traffic to different services.
For more advanced ingress controllers, you can look at Kong Ingress Controller.
answered Jan 3 at 9:51
Quentin RevelQuentin Revel
75539
I tried but it only creates a classic ELB. I need ALB and therefore I use aws-alb-ingress-controller > Ingress > Nginx (NodePort) > svc-<foo> and the outdated IP problem happens in the Nginx NodePort :(
– Kok How Teh
Jan 3 at 13:07
add a comment |
I would recommand the use of Ingress resource on Kubernetes with the Nginx Ingress Controller.
Its whole purpose is to have a proxy inside your Kubernetes cluster that redirects the traffic to ClusterIP Services.
So you only have one external ELB that redirects all the traffic into your Kubernetes cluster. The Ingress Controller then redirects the traffic to different services.
For more advanced ingress controllers, you can look at Kong Ingress Controller.
answered Jan 3 at 9:51
Quentin RevelQuentin Revel
75539
I would recommand the use of Ingress resource on Kubernetes with the Nginx Ingress Controller.
Its whole purpose is to have a proxy inside your Kubernetes cluster that redirects the traffic to ClusterIP Services.
So you only have one external ELB that redirects all the traffic into your Kubernetes cluster. The Ingress Controller then redirects the traffic to different services.
For more advanced ingress controllers, you can look at Kong Ingress Controller.
answered Jan 3 at 9:51
Quentin RevelQuentin Revel
75539
answered Jan 3 at 9:51
Quentin RevelQuentin Revel
75539
answered Jan 3 at 9:51
Quentin RevelQuentin Revel
75539
answered Jan 3 at 9:51
Quentin RevelQuentin Revel
75539
75539
I tried but it only creates a classic ELB. I need ALB and therefore I use aws-alb-ingress-controller > Ingress > Nginx (NodePort) > svc-<foo> and the outdated IP problem happens in the Nginx NodePort :(
– Kok How Teh
Jan 3 at 13:07
add a comment |
I tried but it only creates a classic ELB. I need ALB and therefore I use aws-alb-ingress-controller > Ingress > Nginx (NodePort) > svc-<foo> and the outdated IP problem happens in the Nginx NodePort :(
– Kok How Teh
Jan 3 at 13:07
I tried but it only creates a classic ELB. I need ALB and therefore I use aws-alb-ingress-controller > Ingress > Nginx (NodePort) > svc-<foo> and the outdated IP problem happens in the Nginx NodePort :(
– Kok How Teh
Jan 3 at 13:07
I tried but it only creates a classic ELB. I need ALB and therefore I use aws-alb-ingress-controller > Ingress > Nginx (NodePort) > svc-<foo> and the outdated IP problem happens in the Nginx NodePort :(
– Kok How Teh
Jan 3 at 13:07
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54019648%2fkubernetes-nginx-refresh-ip-address-when-upstream-service-ip-changes%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
