Mixing
Slow mounting of Docker volumes with large number of files on Linux
We are experiencing some very strange behaviour when mounting volumes with large amount of data (e.g a million files )
The current setup:
- Docker host: Container Linux by CoreOS 1465.7.0 (Ladybug)
- Docker version client: 18.06.1-ce
- Docker version host: 17.09.0-ce
I have tried different versions of docker and CoreOs, both newer and older releases without any differences.
The data-directory on the docker-host is probably mapped to some storage tech, im not sure about the setup here, but I can fill out with details if necessary, but from my point of view it looks like a normal folder.
The initial error happened when switching from an anonymous volume mounted through a dummy-container (docker-compose v1) to a named volume (docker-compose v3). After creating a named volume, i shut down the docker-service and does a manual copy of the files from the old volume to the new volume. This has been tested with small data amounts, and that works so it doesnt seem to be an issue with the actual moving data in the internal /var/lib/docker-domain. I am also able to recreate this issue with a new installation where I copy a decently large amount of data.
Building container with compose works fine:
myservice:
build: myservice
restart: always
ports:
- "8080:8080"
volumes:
- type: volume
source: repo
target: /home/repo
volumes:
repo:
The repo-volume beeing the volume with a lot of data. Now, when trying to up the services, I get a timeout on the up, and the service gets stuck in "Created":
ERROR: for my-servce-new_exp_1 HTTPSConnectionPool(host='xxxx.xx.xxx.xxx', port=2376): Read timed out. (read timeout=60)
ERROR: for exp HTTPSConnectionPool(host='xxx.xx.xxx.xxx', port=2376): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
I have tried to increase the timeout, but something I get another timeout after a while.
Now, if I RESTART the docker-service or host now, the service is getting up and running (but doing it this way causes issues with internal dns-mappings between services)
If i up the service with an empty / small volume, it works as expected.
As a curiosity, I found something possibly releated when trying to mount the same volume to a docker-container:
docker run -it --rm --name rmytest --volume my-service-new_repo:/data ubuntu:latest
This will time out after e.g 30 minutes or so.
If I, on the other hand, adds any option to the consistency-parameter of the volume-mapping, it runs within a couple of seconds:
docker run -it --rm --name rmytest --volume my-service-new_repo:/data:consistent ubuntu:latest
I have had no success adding the same options to the compose files either, e.g
volumes:
- type: volume
source: repo
target: /home/repo
consistency: delegated
Yields the same results; timeout and not working. Any help and pointers in the right direction would be much appreciated.
docker docker-compose coreos
asked Jan 2 at 22:03
runarMrunarM
1,11511119
add a comment |
We are experiencing some very strange behaviour when mounting volumes with large amount of data (e.g a million files )
The current setup:
- Docker host: Container Linux by CoreOS 1465.7.0 (Ladybug)
- Docker version client: 18.06.1-ce
- Docker version host: 17.09.0-ce
I have tried different versions of docker and CoreOs, both newer and older releases without any differences.
The data-directory on the docker-host is probably mapped to some storage tech, im not sure about the setup here, but I can fill out with details if necessary, but from my point of view it looks like a normal folder.
The initial error happened when switching from an anonymous volume mounted through a dummy-container (docker-compose v1) to a named volume (docker-compose v3). After creating a named volume, i shut down the docker-service and does a manual copy of the files from the old volume to the new volume. This has been tested with small data amounts, and that works so it doesnt seem to be an issue with the actual moving data in the internal /var/lib/docker-domain. I am also able to recreate this issue with a new installation where I copy a decently large amount of data.
Building container with compose works fine:
myservice:
build: myservice
restart: always
ports:
- "8080:8080"
volumes:
- type: volume
source: repo
target: /home/repo
volumes:
repo:
The repo-volume beeing the volume with a lot of data. Now, when trying to up the services, I get a timeout on the up, and the service gets stuck in "Created":
ERROR: for my-servce-new_exp_1 HTTPSConnectionPool(host='xxxx.xx.xxx.xxx', port=2376): Read timed out. (read timeout=60)
ERROR: for exp HTTPSConnectionPool(host='xxx.xx.xxx.xxx', port=2376): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
I have tried to increase the timeout, but something I get another timeout after a while.
Now, if I RESTART the docker-service or host now, the service is getting up and running (but doing it this way causes issues with internal dns-mappings between services)
If i up the service with an empty / small volume, it works as expected.
As a curiosity, I found something possibly releated when trying to mount the same volume to a docker-container:
docker run -it --rm --name rmytest --volume my-service-new_repo:/data ubuntu:latest
This will time out after e.g 30 minutes or so.
If I, on the other hand, adds any option to the consistency-parameter of the volume-mapping, it runs within a couple of seconds:
docker run -it --rm --name rmytest --volume my-service-new_repo:/data:consistent ubuntu:latest
I have had no success adding the same options to the compose files either, e.g
volumes:
- type: volume
source: repo
target: /home/repo
consistency: delegated
Yields the same results; timeout and not working. Any help and pointers in the right direction would be much appreciated.
docker docker-compose coreos
asked Jan 2 at 22:03
runarMrunarM
1,11511119
After a lot of research, this seems to be related to SeLinux and labelling of data when mounting. The conclusion is still not final, I will post the answer when its ready.
– runarM
Jan 9 at 9:05
add a comment |
We are experiencing some very strange behaviour when mounting volumes with large amount of data (e.g a million files )
The current setup:
- Docker host: Container Linux by CoreOS 1465.7.0 (Ladybug)
- Docker version client: 18.06.1-ce
- Docker version host: 17.09.0-ce
I have tried different versions of docker and CoreOs, both newer and older releases without any differences.
The data-directory on the docker-host is probably mapped to some storage tech, im not sure about the setup here, but I can fill out with details if necessary, but from my point of view it looks like a normal folder.
The initial error happened when switching from an anonymous volume mounted through a dummy-container (docker-compose v1) to a named volume (docker-compose v3). After creating a named volume, i shut down the docker-service and does a manual copy of the files from the old volume to the new volume. This has been tested with small data amounts, and that works so it doesnt seem to be an issue with the actual moving data in the internal /var/lib/docker-domain. I am also able to recreate this issue with a new installation where I copy a decently large amount of data.
Building container with compose works fine:
myservice:
build: myservice
restart: always
ports:
- "8080:8080"
volumes:
- type: volume
source: repo
target: /home/repo
volumes:
repo:
The repo-volume beeing the volume with a lot of data. Now, when trying to up the services, I get a timeout on the up, and the service gets stuck in "Created":
ERROR: for my-servce-new_exp_1 HTTPSConnectionPool(host='xxxx.xx.xxx.xxx', port=2376): Read timed out. (read timeout=60)
ERROR: for exp HTTPSConnectionPool(host='xxx.xx.xxx.xxx', port=2376): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
I have tried to increase the timeout, but something I get another timeout after a while.
Now, if I RESTART the docker-service or host now, the service is getting up and running (but doing it this way causes issues with internal dns-mappings between services)
If i up the service with an empty / small volume, it works as expected.
As a curiosity, I found something possibly releated when trying to mount the same volume to a docker-container:
docker run -it --rm --name rmytest --volume my-service-new_repo:/data ubuntu:latest
This will time out after e.g 30 minutes or so.
If I, on the other hand, adds any option to the consistency-parameter of the volume-mapping, it runs within a couple of seconds:
docker run -it --rm --name rmytest --volume my-service-new_repo:/data:consistent ubuntu:latest
I have had no success adding the same options to the compose files either, e.g
volumes:
- type: volume
source: repo
target: /home/repo
consistency: delegated
Yields the same results; timeout and not working. Any help and pointers in the right direction would be much appreciated.
docker docker-compose coreos
asked Jan 2 at 22:03
runarMrunarM
1,11511119
We are experiencing some very strange behaviour when mounting volumes with large amount of data (e.g a million files )
The current setup:
- Docker host: Container Linux by CoreOS 1465.7.0 (Ladybug)
- Docker version client: 18.06.1-ce
- Docker version host: 17.09.0-ce
I have tried different versions of docker and CoreOs, both newer and older releases without any differences.
The data-directory on the docker-host is probably mapped to some storage tech, im not sure about the setup here, but I can fill out with details if necessary, but from my point of view it looks like a normal folder.
The initial error happened when switching from an anonymous volume mounted through a dummy-container (docker-compose v1) to a named volume (docker-compose v3). After creating a named volume, i shut down the docker-service and does a manual copy of the files from the old volume to the new volume. This has been tested with small data amounts, and that works so it doesnt seem to be an issue with the actual moving data in the internal /var/lib/docker-domain. I am also able to recreate this issue with a new installation where I copy a decently large amount of data.
Building container with compose works fine:
myservice:
build: myservice
restart: always
ports:
- "8080:8080"
volumes:
- type: volume
source: repo
target: /home/repo
volumes:
repo:
The repo-volume beeing the volume with a lot of data. Now, when trying to up the services, I get a timeout on the up, and the service gets stuck in "Created":
ERROR: for my-servce-new_exp_1 HTTPSConnectionPool(host='xxxx.xx.xxx.xxx', port=2376): Read timed out. (read timeout=60)
ERROR: for exp HTTPSConnectionPool(host='xxx.xx.xxx.xxx', port=2376): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
I have tried to increase the timeout, but something I get another timeout after a while.
Now, if I RESTART the docker-service or host now, the service is getting up and running (but doing it this way causes issues with internal dns-mappings between services)
If i up the service with an empty / small volume, it works as expected.
As a curiosity, I found something possibly releated when trying to mount the same volume to a docker-container:
docker run -it --rm --name rmytest --volume my-service-new_repo:/data ubuntu:latest
This will time out after e.g 30 minutes or so.
If I, on the other hand, adds any option to the consistency-parameter of the volume-mapping, it runs within a couple of seconds:
docker run -it --rm --name rmytest --volume my-service-new_repo:/data:consistent ubuntu:latest
I have had no success adding the same options to the compose files either, e.g
volumes:
- type: volume
source: repo
target: /home/repo
consistency: delegated
Yields the same results; timeout and not working. Any help and pointers in the right direction would be much appreciated.
docker docker-compose coreos
docker docker-compose coreos
asked Jan 2 at 22:03
runarMrunarM
1,11511119
asked Jan 2 at 22:03
runarMrunarM
1,11511119
edited Jan 9 at 9:02
runarM
asked Jan 2 at 22:03
runarMrunarM
1,11511119
asked Jan 2 at 22:03
runarMrunarM
1,11511119
asked Jan 2 at 22:03
runarMrunarM
1,11511119
1,11511119
After a lot of research, this seems to be related to SeLinux and labelling of data when mounting. The conclusion is still not final, I will post the answer when its ready.
– runarM
Jan 9 at 9:05
add a comment |
After a lot of research, this seems to be related to SeLinux and labelling of data when mounting. The conclusion is still not final, I will post the answer when its ready.
– runarM
Jan 9 at 9:05
After a lot of research, this seems to be related to SeLinux and labelling of data when mounting. The conclusion is still not final, I will post the answer when its ready.
– runarM
Jan 9 at 9:05
After a lot of research, this seems to be related to SeLinux and labelling of data when mounting. The conclusion is still not final, I will post the answer when its ready.
– runarM
Jan 9 at 9:05
add a comment |
0
active
oldest
votes
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54013774%2fslow-mounting-of-docker-volumes-with-large-number-of-files-on-linux%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54013774%2fslow-mounting-of-docker-volumes-with-large-number-of-files-on-linux%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
After a lot of research, this seems to be related to SeLinux and labelling of data when mounting. The conclusion is still not final, I will post the answer when its ready.
– runarM
Jan 9 at 9:05