Mixing
Cassandra Process Killed by OS
I'm using apache cassandra server. After random amount of time my cassandra service stops. when I try to check its status using 'service cassandra status' using centOS7 it shows me the following log
[centos@ip-172-31-24-101 routes]$ service cassandra status
cassandra.service - LSB: distributed storage system for structured data
Loaded: loaded (/etc/rc.d/init.d/cassandra; bad; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2018-12-31 10:26:13 UTC; 34min ago
Docs: man:systemd-sysv-generator(8)
Main PID: 2078 (code=killed, signal=KILL)
Dec 31 05:12:46 ip-172-31-24-101.ap-south-1.compute.internal su[781]: (to cassandra) root on none
Dec 31 05:12:49 ip-172-31-24-101.ap-south-1.compute.internal cassandra[761]: Starting Cassandra: OK
Dec 31 05:12:49 ip-172-31-24-101.ap-south-1.compute.internal systemd[1]: Started LSB: distributed storage system for structured data.
Dec 31 10:25:46 ip-172-31-24-101.ap-south-1.compute.internal systemd[1]: cassandra.service: main process exited, code=killed, s...KILL
Dec 31 10:25:47 ip-172-31-24-101.ap-south-1.compute.internal su[15760]: (to cassandra) root on none
Dec 31 10:25:47 ip-172-31-24-101.ap-south-1.compute.internal cassandra[15746]: Shutdown Cassandra: bash: line 0: kill: (2078) - ...ess
Dec 31 10:26:13 ip-172-31-24-101.ap-south-1.compute.internal cassandra[15746]: ERROR: could not stop cassandra
Dec 31 10:26:13 ip-172-31-24-101.ap-south-1.compute.internal systemd[1]: cassandra.service: control process exited, code=exited...us=1
Dec 31 10:26:13 ip-172-31-24-101.ap-south-1.compute.internal systemd[1]: Unit cassandra.service entered failed state.
Dec 31 10:26:13 ip-172-31-24-101.ap-south-1.compute.internal systemd[1]: cassandra.service failed."
How can I figure out what wrong with the Cassandra? why its crashing?
cassandra cassandra-3.0
edited Jan 2 at 15:58
Mehul Gupta
312317
asked Dec 31 '18 at 11:37
Suleman TanveerSuleman Tanveer
227
add a comment |
I'm using apache cassandra server. After random amount of time my cassandra service stops. when I try to check its status using 'service cassandra status' using centOS7 it shows me the following log
[centos@ip-172-31-24-101 routes]$ service cassandra status
cassandra.service - LSB: distributed storage system for structured data
Loaded: loaded (/etc/rc.d/init.d/cassandra; bad; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2018-12-31 10:26:13 UTC; 34min ago
Docs: man:systemd-sysv-generator(8)
Main PID: 2078 (code=killed, signal=KILL)
Dec 31 05:12:46 ip-172-31-24-101.ap-south-1.compute.internal su[781]: (to cassandra) root on none
Dec 31 05:12:49 ip-172-31-24-101.ap-south-1.compute.internal cassandra[761]: Starting Cassandra: OK
Dec 31 05:12:49 ip-172-31-24-101.ap-south-1.compute.internal systemd[1]: Started LSB: distributed storage system for structured data.
Dec 31 10:25:46 ip-172-31-24-101.ap-south-1.compute.internal systemd[1]: cassandra.service: main process exited, code=killed, s...KILL
Dec 31 10:25:47 ip-172-31-24-101.ap-south-1.compute.internal su[15760]: (to cassandra) root on none
Dec 31 10:25:47 ip-172-31-24-101.ap-south-1.compute.internal cassandra[15746]: Shutdown Cassandra: bash: line 0: kill: (2078) - ...ess
Dec 31 10:26:13 ip-172-31-24-101.ap-south-1.compute.internal cassandra[15746]: ERROR: could not stop cassandra
Dec 31 10:26:13 ip-172-31-24-101.ap-south-1.compute.internal systemd[1]: cassandra.service: control process exited, code=exited...us=1
Dec 31 10:26:13 ip-172-31-24-101.ap-south-1.compute.internal systemd[1]: Unit cassandra.service entered failed state.
Dec 31 10:26:13 ip-172-31-24-101.ap-south-1.compute.internal systemd[1]: cassandra.service failed."
How can I figure out what wrong with the Cassandra? why its crashing?
cassandra cassandra-3.0
edited Jan 2 at 15:58
Mehul Gupta
312317
asked Dec 31 '18 at 11:37
Suleman TanveerSuleman Tanveer
227
add a comment |
I'm using apache cassandra server. After random amount of time my cassandra service stops. when I try to check its status using 'service cassandra status' using centOS7 it shows me the following log
[centos@ip-172-31-24-101 routes]$ service cassandra status
cassandra.service - LSB: distributed storage system for structured data
Loaded: loaded (/etc/rc.d/init.d/cassandra; bad; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2018-12-31 10:26:13 UTC; 34min ago
Docs: man:systemd-sysv-generator(8)
Main PID: 2078 (code=killed, signal=KILL)
Dec 31 05:12:46 ip-172-31-24-101.ap-south-1.compute.internal su[781]: (to cassandra) root on none
Dec 31 05:12:49 ip-172-31-24-101.ap-south-1.compute.internal cassandra[761]: Starting Cassandra: OK
Dec 31 05:12:49 ip-172-31-24-101.ap-south-1.compute.internal systemd[1]: Started LSB: distributed storage system for structured data.
Dec 31 10:25:46 ip-172-31-24-101.ap-south-1.compute.internal systemd[1]: cassandra.service: main process exited, code=killed, s...KILL
Dec 31 10:25:47 ip-172-31-24-101.ap-south-1.compute.internal su[15760]: (to cassandra) root on none
Dec 31 10:25:47 ip-172-31-24-101.ap-south-1.compute.internal cassandra[15746]: Shutdown Cassandra: bash: line 0: kill: (2078) - ...ess
Dec 31 10:26:13 ip-172-31-24-101.ap-south-1.compute.internal cassandra[15746]: ERROR: could not stop cassandra
Dec 31 10:26:13 ip-172-31-24-101.ap-south-1.compute.internal systemd[1]: cassandra.service: control process exited, code=exited...us=1
Dec 31 10:26:13 ip-172-31-24-101.ap-south-1.compute.internal systemd[1]: Unit cassandra.service entered failed state.
Dec 31 10:26:13 ip-172-31-24-101.ap-south-1.compute.internal systemd[1]: cassandra.service failed."
How can I figure out what wrong with the Cassandra? why its crashing?
cassandra cassandra-3.0
edited Jan 2 at 15:58
Mehul Gupta
312317
asked Dec 31 '18 at 11:37
Suleman TanveerSuleman Tanveer
227
I'm using apache cassandra server. After random amount of time my cassandra service stops. when I try to check its status using 'service cassandra status' using centOS7 it shows me the following log
[centos@ip-172-31-24-101 routes]$ service cassandra status
cassandra.service - LSB: distributed storage system for structured data
Loaded: loaded (/etc/rc.d/init.d/cassandra; bad; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2018-12-31 10:26:13 UTC; 34min ago
Docs: man:systemd-sysv-generator(8)
Main PID: 2078 (code=killed, signal=KILL)
Dec 31 05:12:46 ip-172-31-24-101.ap-south-1.compute.internal su[781]: (to cassandra) root on none
Dec 31 05:12:49 ip-172-31-24-101.ap-south-1.compute.internal cassandra[761]: Starting Cassandra: OK
Dec 31 05:12:49 ip-172-31-24-101.ap-south-1.compute.internal systemd[1]: Started LSB: distributed storage system for structured data.
Dec 31 10:25:46 ip-172-31-24-101.ap-south-1.compute.internal systemd[1]: cassandra.service: main process exited, code=killed, s...KILL
Dec 31 10:25:47 ip-172-31-24-101.ap-south-1.compute.internal su[15760]: (to cassandra) root on none
Dec 31 10:25:47 ip-172-31-24-101.ap-south-1.compute.internal cassandra[15746]: Shutdown Cassandra: bash: line 0: kill: (2078) - ...ess
Dec 31 10:26:13 ip-172-31-24-101.ap-south-1.compute.internal cassandra[15746]: ERROR: could not stop cassandra
Dec 31 10:26:13 ip-172-31-24-101.ap-south-1.compute.internal systemd[1]: cassandra.service: control process exited, code=exited...us=1
Dec 31 10:26:13 ip-172-31-24-101.ap-south-1.compute.internal systemd[1]: Unit cassandra.service entered failed state.
Dec 31 10:26:13 ip-172-31-24-101.ap-south-1.compute.internal systemd[1]: cassandra.service failed."
How can I figure out what wrong with the Cassandra? why its crashing?
cassandra cassandra-3.0
cassandra cassandra-3.0
edited Jan 2 at 15:58
Mehul Gupta
312317
asked Dec 31 '18 at 11:37
Suleman TanveerSuleman Tanveer
227
edited Jan 2 at 15:58
Mehul Gupta
312317
asked Dec 31 '18 at 11:37
Suleman TanveerSuleman Tanveer
227
edited Jan 2 at 15:58
Mehul Gupta
312317
edited Jan 2 at 15:58
Mehul Gupta
312317
edited Jan 2 at 15:58
Mehul Gupta
312317
312317
asked Dec 31 '18 at 11:37
Suleman TanveerSuleman Tanveer
227
asked Dec 31 '18 at 11:37
Suleman TanveerSuleman Tanveer
227
asked Dec 31 '18 at 11:37
Suleman TanveerSuleman Tanveer
227
227
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
A process being killed by SIGKILL is often the result of Linux's "OOM Killer" - which kills processes when running out of memory (see, for example,
https://unix.stackexchange.com/questions/136291/will-linux-start-killing-my-processes-without-asking-me-if-memory-gets-short/136294 for some more details on the OOM Killer).
This may indicate that you are giving too much memory to Cassandra (for both heap and off-heap), don't have enough swap space, or both. If it's the OOM killer which killed you r Cassandra, you should be able to find log messages in the usual places (dmesg, /var/log/messages, journalctl, depending on your distribution). Messages which look something like:
[ 54.125380] Out of memory: Kill process 8320 (cassandra) score 324 or sacrifice child
[ 54.125382] Killed process 8320 (cassandra) total-vm:1309660kB, anon-rss:1287796kB, file-rss:76kB
answered Jan 2 at 10:19
Nadav Har'ElNadav Har'El
1,342413
add a comment |
I would suggest taking a look at the system.log for the Cassandra process as it should point you to where the problem is. Depending on if you installed C* from a package or a tarball will depend where it is located. The default for a package install is /var/log/cassandra and for a tarball I think it is installation_directory/log/cassandra (not positive on this).
answered Jan 1 at 21:47
bechbdbechbd
3,74332146
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53987017%2fcassandra-process-killed-by-os%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
A process being killed by SIGKILL is often the result of Linux's "OOM Killer" - which kills processes when running out of memory (see, for example,
https://unix.stackexchange.com/questions/136291/will-linux-start-killing-my-processes-without-asking-me-if-memory-gets-short/136294 for some more details on the OOM Killer).
This may indicate that you are giving too much memory to Cassandra (for both heap and off-heap), don't have enough swap space, or both. If it's the OOM killer which killed you r Cassandra, you should be able to find log messages in the usual places (dmesg, /var/log/messages, journalctl, depending on your distribution). Messages which look something like:
[ 54.125380] Out of memory: Kill process 8320 (cassandra) score 324 or sacrifice child
[ 54.125382] Killed process 8320 (cassandra) total-vm:1309660kB, anon-rss:1287796kB, file-rss:76kB
answered Jan 2 at 10:19
Nadav Har'ElNadav Har'El
1,342413
add a comment |
A process being killed by SIGKILL is often the result of Linux's "OOM Killer" - which kills processes when running out of memory (see, for example,
https://unix.stackexchange.com/questions/136291/will-linux-start-killing-my-processes-without-asking-me-if-memory-gets-short/136294 for some more details on the OOM Killer).
This may indicate that you are giving too much memory to Cassandra (for both heap and off-heap), don't have enough swap space, or both. If it's the OOM killer which killed you r Cassandra, you should be able to find log messages in the usual places (dmesg, /var/log/messages, journalctl, depending on your distribution). Messages which look something like:
[ 54.125380] Out of memory: Kill process 8320 (cassandra) score 324 or sacrifice child
[ 54.125382] Killed process 8320 (cassandra) total-vm:1309660kB, anon-rss:1287796kB, file-rss:76kB
answered Jan 2 at 10:19
Nadav Har'ElNadav Har'El
1,342413
add a comment |
A process being killed by SIGKILL is often the result of Linux's "OOM Killer" - which kills processes when running out of memory (see, for example,
https://unix.stackexchange.com/questions/136291/will-linux-start-killing-my-processes-without-asking-me-if-memory-gets-short/136294 for some more details on the OOM Killer).
This may indicate that you are giving too much memory to Cassandra (for both heap and off-heap), don't have enough swap space, or both. If it's the OOM killer which killed you r Cassandra, you should be able to find log messages in the usual places (dmesg, /var/log/messages, journalctl, depending on your distribution). Messages which look something like:
[ 54.125380] Out of memory: Kill process 8320 (cassandra) score 324 or sacrifice child
[ 54.125382] Killed process 8320 (cassandra) total-vm:1309660kB, anon-rss:1287796kB, file-rss:76kB
answered Jan 2 at 10:19
Nadav Har'ElNadav Har'El
1,342413
A process being killed by SIGKILL is often the result of Linux's "OOM Killer" - which kills processes when running out of memory (see, for example,
https://unix.stackexchange.com/questions/136291/will-linux-start-killing-my-processes-without-asking-me-if-memory-gets-short/136294 for some more details on the OOM Killer).
This may indicate that you are giving too much memory to Cassandra (for both heap and off-heap), don't have enough swap space, or both. If it's the OOM killer which killed you r Cassandra, you should be able to find log messages in the usual places (dmesg, /var/log/messages, journalctl, depending on your distribution). Messages which look something like:
[ 54.125380] Out of memory: Kill process 8320 (cassandra) score 324 or sacrifice child
[ 54.125382] Killed process 8320 (cassandra) total-vm:1309660kB, anon-rss:1287796kB, file-rss:76kB
answered Jan 2 at 10:19
Nadav Har'ElNadav Har'El
1,342413
answered Jan 2 at 10:19
Nadav Har'ElNadav Har'El
1,342413
answered Jan 2 at 10:19
Nadav Har'ElNadav Har'El
1,342413
answered Jan 2 at 10:19
Nadav Har'ElNadav Har'El
1,342413
1,342413
add a comment |
add a comment |
I would suggest taking a look at the system.log for the Cassandra process as it should point you to where the problem is. Depending on if you installed C* from a package or a tarball will depend where it is located. The default for a package install is /var/log/cassandra and for a tarball I think it is installation_directory/log/cassandra (not positive on this).
answered Jan 1 at 21:47
bechbdbechbd
3,74332146
add a comment |
I would suggest taking a look at the system.log for the Cassandra process as it should point you to where the problem is. Depending on if you installed C* from a package or a tarball will depend where it is located. The default for a package install is /var/log/cassandra and for a tarball I think it is installation_directory/log/cassandra (not positive on this).
answered Jan 1 at 21:47
bechbdbechbd
3,74332146
add a comment |
I would suggest taking a look at the system.log for the Cassandra process as it should point you to where the problem is. Depending on if you installed C* from a package or a tarball will depend where it is located. The default for a package install is /var/log/cassandra and for a tarball I think it is installation_directory/log/cassandra (not positive on this).
answered Jan 1 at 21:47
bechbdbechbd
3,74332146
I would suggest taking a look at the system.log for the Cassandra process as it should point you to where the problem is. Depending on if you installed C* from a package or a tarball will depend where it is located. The default for a package install is /var/log/cassandra and for a tarball I think it is installation_directory/log/cassandra (not positive on this).
answered Jan 1 at 21:47
bechbdbechbd
3,74332146
answered Jan 1 at 21:47
bechbdbechbd
3,74332146
answered Jan 1 at 21:47
bechbdbechbd
3,74332146
answered Jan 1 at 21:47
bechbdbechbd
3,74332146
3,74332146
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53987017%2fcassandra-process-killed-by-os%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
