Mixing
kubectl timeout inside kube-addon-manager
0
I was debugging a issue from my cluster, seems kubectl commands timeout inside the kube-addon-manager pod, while the equivalent curl command works fine.
bash-4.3# kubectl get node --v 10
I1119 16:35:55.506867 54 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.10.5 (linux/amd64) kubernetes/32ac1c9" http://localhost:8080/api
I1119 16:36:25.507550 54 round_trippers.go:405] GET http://localhost:8080/api in 30000 milliseconds
I1119 16:36:25.507959 54 round_trippers.go:411] Response Headers:
I1119 16:36:25.508122 54 cached_discovery.go:124] skipped caching discovery info due to Get http://localhost:8080/api: dial tcp: i/o timeout
Equivalent curl command output
bash-4.3# curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.10.5 (linux/amd64) kubernetes/32ac1c9" http://localhost:8080/api
Note: Unnecessary use of -X or --request, GET is already inferred.
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /api HTTP/1.1
> Host: localhost:8080
> Accept: application/json, */*
> User-Agent: kubectl/v1.10.5 (linux/amd64) kubernetes/32ac1c9
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Mon, 19 Nov 2018 16:43:00 GMT
< Content-Length: 134
<
{"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.16.1.13:6443"}]}
* Connection #0 to host localhost left intact
Also tried to run a docker container with host network mode, kubectl command still timeout.
kube-addon-manager.yaml
apiVersion: v1
kind: Pod
metadata:
name: kube-addon-manager
namespace: kube-system
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
labels:
component: kube-addon-manager
spec:
hostNetwork: true
containers:
- name: kube-addon-manager
image: gcr.io/google-containers/kube-addon-manager:v8.6
imagePullPolicy: IfNotPresent
command:
- /bin/bash
- -c
- /opt/kube-addons.sh
resources:
requests:
cpu: 5m
memory: 50Mi
volumeMounts:
- mountPath: /etc/kubernetes/
name: addons
readOnly: true
volumes:
- name: addons
hostPath:
path: /etc/kubernetes/
docker
kubernetes kubectl client-go asked Nov 19 '18 at 16:49
flynn
1316
add a comment |
0
I was debugging a issue from my cluster, seems kubectl commands timeout inside the kube-addon-manager pod, while the equivalent curl command works fine.
bash-4.3# kubectl get node --v 10
I1119 16:35:55.506867 54 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.10.5 (linux/amd64) kubernetes/32ac1c9" http://localhost:8080/api
I1119 16:36:25.507550 54 round_trippers.go:405] GET http://localhost:8080/api in 30000 milliseconds
I1119 16:36:25.507959 54 round_trippers.go:411] Response Headers:
I1119 16:36:25.508122 54 cached_discovery.go:124] skipped caching discovery info due to Get http://localhost:8080/api: dial tcp: i/o timeout
Equivalent curl command output
bash-4.3# curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.10.5 (linux/amd64) kubernetes/32ac1c9" http://localhost:8080/api
Note: Unnecessary use of -X or --request, GET is already inferred.
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /api HTTP/1.1
> Host: localhost:8080
> Accept: application/json, */*
> User-Agent: kubectl/v1.10.5 (linux/amd64) kubernetes/32ac1c9
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Mon, 19 Nov 2018 16:43:00 GMT
< Content-Length: 134
<
{"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.16.1.13:6443"}]}
* Connection #0 to host localhost left intact
Also tried to run a docker container with host network mode, kubectl command still timeout.
kube-addon-manager.yaml
apiVersion: v1
kind: Pod
metadata:
name: kube-addon-manager
namespace: kube-system
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
labels:
component: kube-addon-manager
spec:
hostNetwork: true
containers:
- name: kube-addon-manager
image: gcr.io/google-containers/kube-addon-manager:v8.6
imagePullPolicy: IfNotPresent
command:
- /bin/bash
- -c
- /opt/kube-addons.sh
resources:
requests:
cpu: 5m
memory: 50Mi
volumeMounts:
- mountPath: /etc/kubernetes/
name: addons
readOnly: true
volumes:
- name: addons
hostPath:
path: /etc/kubernetes/
docker
kubernetes kubectl client-go asked Nov 19 '18 at 16:49
flynn
1316
add a comment |
0
0
0
I was debugging a issue from my cluster, seems kubectl commands timeout inside the kube-addon-manager pod, while the equivalent curl command works fine.
bash-4.3# kubectl get node --v 10
I1119 16:35:55.506867 54 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.10.5 (linux/amd64) kubernetes/32ac1c9" http://localhost:8080/api
I1119 16:36:25.507550 54 round_trippers.go:405] GET http://localhost:8080/api in 30000 milliseconds
I1119 16:36:25.507959 54 round_trippers.go:411] Response Headers:
I1119 16:36:25.508122 54 cached_discovery.go:124] skipped caching discovery info due to Get http://localhost:8080/api: dial tcp: i/o timeout
Equivalent curl command output
bash-4.3# curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.10.5 (linux/amd64) kubernetes/32ac1c9" http://localhost:8080/api
Note: Unnecessary use of -X or --request, GET is already inferred.
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /api HTTP/1.1
> Host: localhost:8080
> Accept: application/json, */*
> User-Agent: kubectl/v1.10.5 (linux/amd64) kubernetes/32ac1c9
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Mon, 19 Nov 2018 16:43:00 GMT
< Content-Length: 134
<
{"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.16.1.13:6443"}]}
* Connection #0 to host localhost left intact
Also tried to run a docker container with host network mode, kubectl command still timeout.
kube-addon-manager.yaml
apiVersion: v1
kind: Pod
metadata:
name: kube-addon-manager
namespace: kube-system
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
labels:
component: kube-addon-manager
spec:
hostNetwork: true
containers:
- name: kube-addon-manager
image: gcr.io/google-containers/kube-addon-manager:v8.6
imagePullPolicy: IfNotPresent
command:
- /bin/bash
- -c
- /opt/kube-addons.sh
resources:
requests:
cpu: 5m
memory: 50Mi
volumeMounts:
- mountPath: /etc/kubernetes/
name: addons
readOnly: true
volumes:
- name: addons
hostPath:
path: /etc/kubernetes/
docker
kubernetes kubectl client-go asked Nov 19 '18 at 16:49
flynn
1316
I was debugging a issue from my cluster, seems kubectl commands timeout inside the kube-addon-manager pod, while the equivalent curl command works fine.
bash-4.3# kubectl get node --v 10
I1119 16:35:55.506867 54 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.10.5 (linux/amd64) kubernetes/32ac1c9" http://localhost:8080/api
I1119 16:36:25.507550 54 round_trippers.go:405] GET http://localhost:8080/api in 30000 milliseconds
I1119 16:36:25.507959 54 round_trippers.go:411] Response Headers:
I1119 16:36:25.508122 54 cached_discovery.go:124] skipped caching discovery info due to Get http://localhost:8080/api: dial tcp: i/o timeout
Equivalent curl command output
bash-4.3# curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.10.5 (linux/amd64) kubernetes/32ac1c9" http://localhost:8080/api
Note: Unnecessary use of -X or --request, GET is already inferred.
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /api HTTP/1.1
> Host: localhost:8080
> Accept: application/json, */*
> User-Agent: kubectl/v1.10.5 (linux/amd64) kubernetes/32ac1c9
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Mon, 19 Nov 2018 16:43:00 GMT
< Content-Length: 134
<
{"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.16.1.13:6443"}]}
* Connection #0 to host localhost left intact
Also tried to run a docker container with host network mode, kubectl command still timeout.
kube-addon-manager.yaml
apiVersion: v1
kind: Pod
metadata:
name: kube-addon-manager
namespace: kube-system
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
labels:
component: kube-addon-manager
spec:
hostNetwork: true
containers:
- name: kube-addon-manager
image: gcr.io/google-containers/kube-addon-manager:v8.6
imagePullPolicy: IfNotPresent
command:
- /bin/bash
- -c
- /opt/kube-addons.sh
resources:
requests:
cpu: 5m
memory: 50Mi
volumeMounts:
- mountPath: /etc/kubernetes/
name: addons
readOnly: true
volumes:
- name: addons
hostPath:
path: /etc/kubernetes/
docker
kubernetes kubectl client-go docker
kubernetes kubectl client-go asked Nov 19 '18 at 16:49
flynn
1316
asked Nov 19 '18 at 16:49
flynn
1316
edited Nov 19 '18 at 16:55
asked Nov 19 '18 at 16:49
flynn
1316
asked Nov 19 '18 at 16:49
flynn
1316
asked Nov 19 '18 at 16:49
flynn
1316
1316
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
1
Seems like in your config you are trying to talk to port 8080 which is the insecure port in the kube-apiserver.
You can try starting your kube-apiserver with this option:
--insecure-port
The default for the insecure port is 8080. Note that this option might be deprecated in the future.
Also, keep in mind the the kube-addon-manager is part of the legacy add-ons.
answered Nov 20 '18 at 2:20
Rico
26.5k94864
No, I am usingkubectlinsidekube-addon-managerpod, not trying to connect tokube-addon-manager. That's whatkube-addon-manageris doing, it useskubectlto check addons status, reconcile them if needed. But now,kubectlcommand always timeout, makeskube-addon-managerunable to do reconcile thing.
– flynn
Nov 20 '18 at 2:46
I see what did you change in your cluster?
– Rico
Nov 20 '18 at 5:40
Actually, I have two clusters with exactly the same configuration, the other one works fine, this one doesn't.kubectlinsidekube-addon-managerpod connects to apiserver insecure port8080, but if I use a kubeconfig, which will connect to port 6443 instead, everything looks good. I know there is always one thing I miss, just don't know which one.
– flynn
Nov 20 '18 at 6:43
Try--insecure-portin your kube-apiserver, updated the answer
– Rico
Nov 20 '18 at 6:59
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53379246%2fkubectl-timeout-inside-kube-addon-manager%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
1
Seems like in your config you are trying to talk to port 8080 which is the insecure port in the kube-apiserver.
You can try starting your kube-apiserver with this option:
--insecure-port
The default for the insecure port is 8080. Note that this option might be deprecated in the future.
Also, keep in mind the the kube-addon-manager is part of the legacy add-ons.
answered Nov 20 '18 at 2:20
Rico
26.5k94864
No, I am usingkubectlinsidekube-addon-managerpod, not trying to connect tokube-addon-manager. That's whatkube-addon-manageris doing, it useskubectlto check addons status, reconcile them if needed. But now,kubectlcommand always timeout, makeskube-addon-managerunable to do reconcile thing.
– flynn
Nov 20 '18 at 2:46
I see what did you change in your cluster?
– Rico
Nov 20 '18 at 5:40
Actually, I have two clusters with exactly the same configuration, the other one works fine, this one doesn't.kubectlinsidekube-addon-managerpod connects to apiserver insecure port8080, but if I use a kubeconfig, which will connect to port 6443 instead, everything looks good. I know there is always one thing I miss, just don't know which one.
– flynn
Nov 20 '18 at 6:43
Try--insecure-portin your kube-apiserver, updated the answer
– Rico
Nov 20 '18 at 6:59
add a comment |
1
Seems like in your config you are trying to talk to port 8080 which is the insecure port in the kube-apiserver.
You can try starting your kube-apiserver with this option:
--insecure-port
The default for the insecure port is 8080. Note that this option might be deprecated in the future.
Also, keep in mind the the kube-addon-manager is part of the legacy add-ons.
answered Nov 20 '18 at 2:20
Rico
26.5k94864
No, I am usingkubectlinsidekube-addon-managerpod, not trying to connect tokube-addon-manager. That's whatkube-addon-manageris doing, it useskubectlto check addons status, reconcile them if needed. But now,kubectlcommand always timeout, makeskube-addon-managerunable to do reconcile thing.
– flynn
Nov 20 '18 at 2:46
I see what did you change in your cluster?
– Rico
Nov 20 '18 at 5:40
Actually, I have two clusters with exactly the same configuration, the other one works fine, this one doesn't.kubectlinsidekube-addon-managerpod connects to apiserver insecure port8080, but if I use a kubeconfig, which will connect to port 6443 instead, everything looks good. I know there is always one thing I miss, just don't know which one.
– flynn
Nov 20 '18 at 6:43
Try--insecure-portin your kube-apiserver, updated the answer
– Rico
Nov 20 '18 at 6:59
add a comment |
1
1
1
Seems like in your config you are trying to talk to port 8080 which is the insecure port in the kube-apiserver.
You can try starting your kube-apiserver with this option:
--insecure-port
The default for the insecure port is 8080. Note that this option might be deprecated in the future.
Also, keep in mind the the kube-addon-manager is part of the legacy add-ons.
answered Nov 20 '18 at 2:20
Rico
26.5k94864
Seems like in your config you are trying to talk to port 8080 which is the insecure port in the kube-apiserver.
You can try starting your kube-apiserver with this option:
--insecure-port
The default for the insecure port is 8080. Note that this option might be deprecated in the future.
Also, keep in mind the the kube-addon-manager is part of the legacy add-ons.
answered Nov 20 '18 at 2:20
Rico
26.5k94864
edited Nov 20 '18 at 6:59
answered Nov 20 '18 at 2:20
Rico
26.5k94864
answered Nov 20 '18 at 2:20
Rico
26.5k94864
answered Nov 20 '18 at 2:20
Rico
26.5k94864
26.5k94864
No, I am usingkubectlinsidekube-addon-managerpod, not trying to connect tokube-addon-manager. That's whatkube-addon-manageris doing, it useskubectlto check addons status, reconcile them if needed. But now,kubectlcommand always timeout, makeskube-addon-managerunable to do reconcile thing.
– flynn
Nov 20 '18 at 2:46
I see what did you change in your cluster?
– Rico
Nov 20 '18 at 5:40
Actually, I have two clusters with exactly the same configuration, the other one works fine, this one doesn't.kubectlinsidekube-addon-managerpod connects to apiserver insecure port8080, but if I use a kubeconfig, which will connect to port 6443 instead, everything looks good. I know there is always one thing I miss, just don't know which one.
– flynn
Nov 20 '18 at 6:43
Try--insecure-portin your kube-apiserver, updated the answer
– Rico
Nov 20 '18 at 6:59
add a comment |
No, I am usingkubectlinsidekube-addon-managerpod, not trying to connect tokube-addon-manager. That's whatkube-addon-manageris doing, it useskubectlto check addons status, reconcile them if needed. But now,kubectlcommand always timeout, makeskube-addon-managerunable to do reconcile thing.
– flynn
Nov 20 '18 at 2:46
I see what did you change in your cluster?
– Rico
Nov 20 '18 at 5:40
Actually, I have two clusters with exactly the same configuration, the other one works fine, this one doesn't.kubectlinsidekube-addon-managerpod connects to apiserver insecure port8080, but if I use a kubeconfig, which will connect to port 6443 instead, everything looks good. I know there is always one thing I miss, just don't know which one.
– flynn
Nov 20 '18 at 6:43
Try--insecure-portin your kube-apiserver, updated the answer
– Rico
Nov 20 '18 at 6:59
No, I am using
kubectl inside kube-addon-manager pod, not trying to connect to kube-addon-manager. That's what kube-addon-manager is doing, it uses kubectl to check addons status, reconcile them if needed. But now, kubectl command always timeout, makes kube-addon-manager unable to do reconcile thing.– flynn
Nov 20 '18 at 2:46
No, I am using
kubectl inside kube-addon-manager pod, not trying to connect to kube-addon-manager. That's what kube-addon-manager is doing, it uses kubectl to check addons status, reconcile them if needed. But now, kubectl command always timeout, makes kube-addon-manager unable to do reconcile thing.– flynn
Nov 20 '18 at 2:46
I see what did you change in your cluster?
– Rico
Nov 20 '18 at 5:40
I see what did you change in your cluster?
– Rico
Nov 20 '18 at 5:40
Actually, I have two clusters with exactly the same configuration, the other one works fine, this one doesn't.
kubectl inside kube-addon-manager pod connects to apiserver insecure port 8080, but if I use a kubeconfig, which will connect to port 6443 instead, everything looks good. I know there is always one thing I miss, just don't know which one.– flynn
Nov 20 '18 at 6:43
Actually, I have two clusters with exactly the same configuration, the other one works fine, this one doesn't.
kubectl inside kube-addon-manager pod connects to apiserver insecure port 8080, but if I use a kubeconfig, which will connect to port 6443 instead, everything looks good. I know there is always one thing I miss, just don't know which one.– flynn
Nov 20 '18 at 6:43
Try
--insecure-port in your kube-apiserver, updated the answer– Rico
Nov 20 '18 at 6:59
Try
--insecure-port in your kube-apiserver, updated the answer– Rico
Nov 20 '18 at 6:59
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53379246%2fkubectl-timeout-inside-kube-addon-manager%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
