Mixing
Docker Network not Found
In our team, we are currently transitioning to Docker to deploy everything on our server.
We are using Docker Swarm and multiple (10+) compose files defining plenty (20+) of services. Everything works beautifully so far, except when we take down our stack using docker stack rm <name> (and redeploy using docker stack deploy <options> <name>): about every second time, we get the following error:
Failed to remove network <id>: Error response from daemon: network <id> not foundFailed to remove some resources from stack: <name>
When using docker network ls, the network is indeed not removed, however, docker network rm <id> always results in the following:
Error response from daemon: network <id> not found
What makes that even more strange is the fact that docker network inspect <id> returns a normal output. The networks are always overlay networks that are created with the compose files used to deploy our stack. Currently, we only have a single node in our Swarm.
Our current "workaround" is to restart Docker (which resolves the issue), but that is not a viable solution in a production environment. Leaving the swarm and joining it again does not resolve the issue either.
At first, we thought that this issue is related to Docker for Mac only (as we first encountered the issue on local machines), however, the same issue arises on Debian Stretch. In both cases, we use the latest Docker distribution available.
I would really appreciate any help!
docker docker-swarm docker-networking
asked Nov 17 '18 at 3:36
borcheroborchero
2,32022952
add a comment |
In our team, we are currently transitioning to Docker to deploy everything on our server.
We are using Docker Swarm and multiple (10+) compose files defining plenty (20+) of services. Everything works beautifully so far, except when we take down our stack using docker stack rm <name> (and redeploy using docker stack deploy <options> <name>): about every second time, we get the following error:
Failed to remove network <id>: Error response from daemon: network <id> not foundFailed to remove some resources from stack: <name>
When using docker network ls, the network is indeed not removed, however, docker network rm <id> always results in the following:
Error response from daemon: network <id> not found
What makes that even more strange is the fact that docker network inspect <id> returns a normal output. The networks are always overlay networks that are created with the compose files used to deploy our stack. Currently, we only have a single node in our Swarm.
Our current "workaround" is to restart Docker (which resolves the issue), but that is not a viable solution in a production environment. Leaving the swarm and joining it again does not resolve the issue either.
At first, we thought that this issue is related to Docker for Mac only (as we first encountered the issue on local machines), however, the same issue arises on Debian Stretch. In both cases, we use the latest Docker distribution available.
I would really appreciate any help!
docker docker-swarm docker-networking
asked Nov 17 '18 at 3:36
borcheroborchero
2,32022952
Did you check to see if all the containers using this network are not running? the stack rm issues the command to delete all the services immediately but the containers themselves takes a bit of time. Also, are the networks has any configurations? IPAM, specific drivers?
– nadavbrkt
Nov 24 '18 at 16:57
add a comment |
In our team, we are currently transitioning to Docker to deploy everything on our server.
We are using Docker Swarm and multiple (10+) compose files defining plenty (20+) of services. Everything works beautifully so far, except when we take down our stack using docker stack rm <name> (and redeploy using docker stack deploy <options> <name>): about every second time, we get the following error:
Failed to remove network <id>: Error response from daemon: network <id> not foundFailed to remove some resources from stack: <name>
When using docker network ls, the network is indeed not removed, however, docker network rm <id> always results in the following:
Error response from daemon: network <id> not found
What makes that even more strange is the fact that docker network inspect <id> returns a normal output. The networks are always overlay networks that are created with the compose files used to deploy our stack. Currently, we only have a single node in our Swarm.
Our current "workaround" is to restart Docker (which resolves the issue), but that is not a viable solution in a production environment. Leaving the swarm and joining it again does not resolve the issue either.
At first, we thought that this issue is related to Docker for Mac only (as we first encountered the issue on local machines), however, the same issue arises on Debian Stretch. In both cases, we use the latest Docker distribution available.
I would really appreciate any help!
docker docker-swarm docker-networking
asked Nov 17 '18 at 3:36
borcheroborchero
2,32022952
In our team, we are currently transitioning to Docker to deploy everything on our server.
We are using Docker Swarm and multiple (10+) compose files defining plenty (20+) of services. Everything works beautifully so far, except when we take down our stack using docker stack rm <name> (and redeploy using docker stack deploy <options> <name>): about every second time, we get the following error:
Failed to remove network <id>: Error response from daemon: network <id> not foundFailed to remove some resources from stack: <name>
When using docker network ls, the network is indeed not removed, however, docker network rm <id> always results in the following:
Error response from daemon: network <id> not found
What makes that even more strange is the fact that docker network inspect <id> returns a normal output. The networks are always overlay networks that are created with the compose files used to deploy our stack. Currently, we only have a single node in our Swarm.
Our current "workaround" is to restart Docker (which resolves the issue), but that is not a viable solution in a production environment. Leaving the swarm and joining it again does not resolve the issue either.
At first, we thought that this issue is related to Docker for Mac only (as we first encountered the issue on local machines), however, the same issue arises on Debian Stretch. In both cases, we use the latest Docker distribution available.
I would really appreciate any help!
docker docker-swarm docker-networking
docker docker-swarm docker-networking
asked Nov 17 '18 at 3:36
borcheroborchero
2,32022952
asked Nov 17 '18 at 3:36
borcheroborchero
2,32022952
asked Nov 17 '18 at 3:36
borcheroborchero
2,32022952
asked Nov 17 '18 at 3:36
borcheroborchero
2,32022952
asked Nov 17 '18 at 3:36
borcheroborchero
2,32022952
2,32022952
Did you check to see if all the containers using this network are not running? the stack rm issues the command to delete all the services immediately but the containers themselves takes a bit of time. Also, are the networks has any configurations? IPAM, specific drivers?
– nadavbrkt
Nov 24 '18 at 16:57
add a comment |
Did you check to see if all the containers using this network are not running? the stack rm issues the command to delete all the services immediately but the containers themselves takes a bit of time. Also, are the networks has any configurations? IPAM, specific drivers?
– nadavbrkt
Nov 24 '18 at 16:57
Did you check to see if all the containers using this network are not running? the stack rm issues the command to delete all the services immediately but the containers themselves takes a bit of time. Also, are the networks has any configurations? IPAM, specific drivers?
– nadavbrkt
Nov 24 '18 at 16:57
Did you check to see if all the containers using this network are not running? the stack rm issues the command to delete all the services immediately but the containers themselves takes a bit of time. Also, are the networks has any configurations? IPAM, specific drivers?
– nadavbrkt
Nov 24 '18 at 16:57
add a comment |
2 Answers
2
active
oldest
votes
That sounds exactly like this issue.
Stack rm followed "too fast" by stack deploy would race for the creation/removal of networks, possibly other stack resources.
The issue is still open as of today (docker/cli), but you could try the workaround suggested:
until [ -z "$(docker service ls --filter label=com.docker.stack.namespace=$COMPOSE_PROJECT_NAME -q)" ] || [ "$limit" -lt 0 ]; do
sleep 1;
done
until [ -z "$(docker network ls --filter label=com.docker.stack.namespace=$COMPOSE_PROJECT_NAME -q)" ] || [ "$limit" -lt 0 ]; do
sleep 1;
done
answered Nov 21 '18 at 17:59
Julio Daniel ReyesJulio Daniel Reyes
2,082815
Unfortunately, this does not solve the problem .. when I use this code in a script, it is stuck in an infinite loop as (at least) one network is never removed.
– borchero
Nov 21 '18 at 20:08
Well... here is the same issue again, still open. They also suggested the restart of the daemon, did you do that before testing the script?
– Julio Daniel Reyes
Nov 21 '18 at 20:37
Also it says it takes a while to actually delete the network, can you try running it for a couple of minutes?
– Julio Daniel Reyes
Nov 21 '18 at 20:39
Yeah, as indicated in my question, restarting the daemon is my "hardfix", but it really seems kind of strange.
– borchero
Nov 21 '18 at 20:40
1
It ran for about 10 minutes and did not complete. When I rundocker network ls, one of the networks is still visible, over 30 minutes after removing the stack. But I guess there seems to be no solution to the problem at the moment (based on the issues that you linked).
– borchero
Nov 21 '18 at 20:41
add a comment |
You can always use docker system prune -a to get rid of the old network. This will not delete your volumes.
It will take longer to docker-compose up --build -d the next time, but it will get you past your current problem.
edited Nov 27 '18 at 10:46
JoSSte
98021531
answered Nov 27 '18 at 1:58
fcnormanfcnorman
584317
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53347951%2fdocker-network-not-found%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
That sounds exactly like this issue.
Stack rm followed "too fast" by stack deploy would race for the creation/removal of networks, possibly other stack resources.
The issue is still open as of today (docker/cli), but you could try the workaround suggested:
until [ -z "$(docker service ls --filter label=com.docker.stack.namespace=$COMPOSE_PROJECT_NAME -q)" ] || [ "$limit" -lt 0 ]; do
sleep 1;
done
until [ -z "$(docker network ls --filter label=com.docker.stack.namespace=$COMPOSE_PROJECT_NAME -q)" ] || [ "$limit" -lt 0 ]; do
sleep 1;
done
answered Nov 21 '18 at 17:59
Julio Daniel ReyesJulio Daniel Reyes
2,082815
Unfortunately, this does not solve the problem .. when I use this code in a script, it is stuck in an infinite loop as (at least) one network is never removed.
– borchero
Nov 21 '18 at 20:08
Well... here is the same issue again, still open. They also suggested the restart of the daemon, did you do that before testing the script?
– Julio Daniel Reyes
Nov 21 '18 at 20:37
Also it says it takes a while to actually delete the network, can you try running it for a couple of minutes?
– Julio Daniel Reyes
Nov 21 '18 at 20:39
Yeah, as indicated in my question, restarting the daemon is my "hardfix", but it really seems kind of strange.
– borchero
Nov 21 '18 at 20:40
1
It ran for about 10 minutes and did not complete. When I rundocker network ls, one of the networks is still visible, over 30 minutes after removing the stack. But I guess there seems to be no solution to the problem at the moment (based on the issues that you linked).
– borchero
Nov 21 '18 at 20:41
add a comment |
That sounds exactly like this issue.
Stack rm followed "too fast" by stack deploy would race for the creation/removal of networks, possibly other stack resources.
The issue is still open as of today (docker/cli), but you could try the workaround suggested:
until [ -z "$(docker service ls --filter label=com.docker.stack.namespace=$COMPOSE_PROJECT_NAME -q)" ] || [ "$limit" -lt 0 ]; do
sleep 1;
done
until [ -z "$(docker network ls --filter label=com.docker.stack.namespace=$COMPOSE_PROJECT_NAME -q)" ] || [ "$limit" -lt 0 ]; do
sleep 1;
done
answered Nov 21 '18 at 17:59
Julio Daniel ReyesJulio Daniel Reyes
2,082815
Unfortunately, this does not solve the problem .. when I use this code in a script, it is stuck in an infinite loop as (at least) one network is never removed.
– borchero
Nov 21 '18 at 20:08
Well... here is the same issue again, still open. They also suggested the restart of the daemon, did you do that before testing the script?
– Julio Daniel Reyes
Nov 21 '18 at 20:37
Also it says it takes a while to actually delete the network, can you try running it for a couple of minutes?
– Julio Daniel Reyes
Nov 21 '18 at 20:39
Yeah, as indicated in my question, restarting the daemon is my "hardfix", but it really seems kind of strange.
– borchero
Nov 21 '18 at 20:40
1
It ran for about 10 minutes and did not complete. When I rundocker network ls, one of the networks is still visible, over 30 minutes after removing the stack. But I guess there seems to be no solution to the problem at the moment (based on the issues that you linked).
– borchero
Nov 21 '18 at 20:41
add a comment |
That sounds exactly like this issue.
Stack rm followed "too fast" by stack deploy would race for the creation/removal of networks, possibly other stack resources.
The issue is still open as of today (docker/cli), but you could try the workaround suggested:
until [ -z "$(docker service ls --filter label=com.docker.stack.namespace=$COMPOSE_PROJECT_NAME -q)" ] || [ "$limit" -lt 0 ]; do
sleep 1;
done
until [ -z "$(docker network ls --filter label=com.docker.stack.namespace=$COMPOSE_PROJECT_NAME -q)" ] || [ "$limit" -lt 0 ]; do
sleep 1;
done
answered Nov 21 '18 at 17:59
Julio Daniel ReyesJulio Daniel Reyes
2,082815
That sounds exactly like this issue.
Stack rm followed "too fast" by stack deploy would race for the creation/removal of networks, possibly other stack resources.
The issue is still open as of today (docker/cli), but you could try the workaround suggested:
until [ -z "$(docker service ls --filter label=com.docker.stack.namespace=$COMPOSE_PROJECT_NAME -q)" ] || [ "$limit" -lt 0 ]; do
sleep 1;
done
until [ -z "$(docker network ls --filter label=com.docker.stack.namespace=$COMPOSE_PROJECT_NAME -q)" ] || [ "$limit" -lt 0 ]; do
sleep 1;
done
answered Nov 21 '18 at 17:59
Julio Daniel ReyesJulio Daniel Reyes
2,082815
answered Nov 21 '18 at 17:59
Julio Daniel ReyesJulio Daniel Reyes
2,082815
answered Nov 21 '18 at 17:59
Julio Daniel ReyesJulio Daniel Reyes
2,082815
answered Nov 21 '18 at 17:59
Julio Daniel ReyesJulio Daniel Reyes
2,082815
2,082815
Unfortunately, this does not solve the problem .. when I use this code in a script, it is stuck in an infinite loop as (at least) one network is never removed.
– borchero
Nov 21 '18 at 20:08
Well... here is the same issue again, still open. They also suggested the restart of the daemon, did you do that before testing the script?
– Julio Daniel Reyes
Nov 21 '18 at 20:37
Also it says it takes a while to actually delete the network, can you try running it for a couple of minutes?
– Julio Daniel Reyes
Nov 21 '18 at 20:39
Yeah, as indicated in my question, restarting the daemon is my "hardfix", but it really seems kind of strange.
– borchero
Nov 21 '18 at 20:40
1
It ran for about 10 minutes and did not complete. When I rundocker network ls, one of the networks is still visible, over 30 minutes after removing the stack. But I guess there seems to be no solution to the problem at the moment (based on the issues that you linked).
– borchero
Nov 21 '18 at 20:41
add a comment |
Unfortunately, this does not solve the problem .. when I use this code in a script, it is stuck in an infinite loop as (at least) one network is never removed.
– borchero
Nov 21 '18 at 20:08
Well... here is the same issue again, still open. They also suggested the restart of the daemon, did you do that before testing the script?
– Julio Daniel Reyes
Nov 21 '18 at 20:37
Also it says it takes a while to actually delete the network, can you try running it for a couple of minutes?
– Julio Daniel Reyes
Nov 21 '18 at 20:39
Yeah, as indicated in my question, restarting the daemon is my "hardfix", but it really seems kind of strange.
– borchero
Nov 21 '18 at 20:40
1
It ran for about 10 minutes and did not complete. When I rundocker network ls, one of the networks is still visible, over 30 minutes after removing the stack. But I guess there seems to be no solution to the problem at the moment (based on the issues that you linked).
– borchero
Nov 21 '18 at 20:41
Unfortunately, this does not solve the problem .. when I use this code in a script, it is stuck in an infinite loop as (at least) one network is never removed.
– borchero
Nov 21 '18 at 20:08
Unfortunately, this does not solve the problem .. when I use this code in a script, it is stuck in an infinite loop as (at least) one network is never removed.
– borchero
Nov 21 '18 at 20:08
Well... here is the same issue again, still open. They also suggested the restart of the daemon, did you do that before testing the script?
– Julio Daniel Reyes
Nov 21 '18 at 20:37
Well... here is the same issue again, still open. They also suggested the restart of the daemon, did you do that before testing the script?
– Julio Daniel Reyes
Nov 21 '18 at 20:37
Also it says it takes a while to actually delete the network, can you try running it for a couple of minutes?
– Julio Daniel Reyes
Nov 21 '18 at 20:39
Also it says it takes a while to actually delete the network, can you try running it for a couple of minutes?
– Julio Daniel Reyes
Nov 21 '18 at 20:39
Yeah, as indicated in my question, restarting the daemon is my "hardfix", but it really seems kind of strange.
– borchero
Nov 21 '18 at 20:40
Yeah, as indicated in my question, restarting the daemon is my "hardfix", but it really seems kind of strange.
– borchero
Nov 21 '18 at 20:40
1
1
It ran for about 10 minutes and did not complete. When I run
docker network ls, one of the networks is still visible, over 30 minutes after removing the stack. But I guess there seems to be no solution to the problem at the moment (based on the issues that you linked).– borchero
Nov 21 '18 at 20:41
It ran for about 10 minutes and did not complete. When I run
docker network ls, one of the networks is still visible, over 30 minutes after removing the stack. But I guess there seems to be no solution to the problem at the moment (based on the issues that you linked).– borchero
Nov 21 '18 at 20:41
add a comment |
You can always use docker system prune -a to get rid of the old network. This will not delete your volumes.
It will take longer to docker-compose up --build -d the next time, but it will get you past your current problem.
edited Nov 27 '18 at 10:46
JoSSte
98021531
answered Nov 27 '18 at 1:58
fcnormanfcnorman
584317
add a comment |
You can always use docker system prune -a to get rid of the old network. This will not delete your volumes.
It will take longer to docker-compose up --build -d the next time, but it will get you past your current problem.
edited Nov 27 '18 at 10:46
JoSSte
98021531
answered Nov 27 '18 at 1:58
fcnormanfcnorman
584317
add a comment |
You can always use docker system prune -a to get rid of the old network. This will not delete your volumes.
It will take longer to docker-compose up --build -d the next time, but it will get you past your current problem.
edited Nov 27 '18 at 10:46
JoSSte
98021531
answered Nov 27 '18 at 1:58
fcnormanfcnorman
584317
You can always use docker system prune -a to get rid of the old network. This will not delete your volumes.
It will take longer to docker-compose up --build -d the next time, but it will get you past your current problem.
edited Nov 27 '18 at 10:46
JoSSte
98021531
answered Nov 27 '18 at 1:58
fcnormanfcnorman
584317
edited Nov 27 '18 at 10:46
JoSSte
98021531
edited Nov 27 '18 at 10:46
JoSSte
98021531
edited Nov 27 '18 at 10:46
JoSSte
98021531
98021531
answered Nov 27 '18 at 1:58
fcnormanfcnorman
584317
answered Nov 27 '18 at 1:58
fcnormanfcnorman
584317
answered Nov 27 '18 at 1:58
fcnormanfcnorman
584317
584317
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53347951%2fdocker-network-not-found%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Did you check to see if all the containers using this network are not running? the stack rm issues the command to delete all the services immediately but the containers themselves takes a bit of time. Also, are the networks has any configurations? IPAM, specific drivers?
– nadavbrkt
Nov 24 '18 at 16:57