Mixing
Perf stack trace is missing __libc_start_main for some processes
1
I have a C++ program compiled with -fno-omit-frame-pointer. I run the same program in two processes and use perf record -p 24509,24470 -F 977 --call-graph fp -- sleep 120 then perf script to get stack traces.
What I found is that all stack traces from 24509 have main | __libc_start_main at the bottom but stack traces from 24470 only have main.
My question is why perf stack trace doesn't have __libc_start_main for one process but have it for the other process even though they are running the same program.
My perf version is 3.10.0-514.36.5.el7.x86_64.debug.
linux x86-64 glibc perf backtrace
edited Nov 20 '18 at 21:26
Peter Cordes
124k18189316
asked Nov 20 '18 at 21:19
jjyaojjyao
100415
|
show 2 more comments
1
I have a C++ program compiled with -fno-omit-frame-pointer. I run the same program in two processes and use perf record -p 24509,24470 -F 977 --call-graph fp -- sleep 120 then perf script to get stack traces.
What I found is that all stack traces from 24509 have main | __libc_start_main at the bottom but stack traces from 24470 only have main.
My question is why perf stack trace doesn't have __libc_start_main for one process but have it for the other process even though they are running the same program.
My perf version is 3.10.0-514.36.5.el7.x86_64.debug.
linux x86-64 glibc perf backtrace
edited Nov 20 '18 at 21:26
Peter Cordes
124k18189316
asked Nov 20 '18 at 21:19
jjyaojjyao
100415
Are the two processes both started from the exact same executable, or different builds of it? Are they started with the same options and environment variables? Is this repeatable if you restart the processes and runperf recordagain? Or even if you runperf recordagain without restarting them?
– Peter Cordes
Nov 20 '18 at 21:27
1
Hi @PeterCordes, those two processes are started from the exact same executable. The only difference is that they log to different log files. Also I restarted the process that's missing__libc_start_mainand__libc_start_mainappears after restart. If I don't restart, the process is consistently missing__libc_start_main.
– jjyao
Nov 21 '18 at 20:18
If you try repeated restarts, is it sometimes missing__libc_start_main? Or was that a one-off that you can't repeat at all? Or maybe after running for a long time, something gets overwritten on the stack and the info is lost? And BTW, command line args and the environment are on the stack on process entry, but they're above the stack frames that lead to__libc_start_mainso should have no impact.
– Peter Cordes
Nov 21 '18 at 20:21
Have you upgraded your systemlibc.so.6after starting the original process? Maybeperfis using stack-unwind metadata from the file of that name in the filesystem, but the libc the process still has mapped has different symbol addresses so its.eh_frameaddresses don't match.
– Peter Cordes
Nov 21 '18 at 20:23
Hi @PeterCordes, you are right. I realized that after running the process for a long time,__libc_start_mainmay disappear. That's why I can always see__libc_start_mainif I immediately runperf recordafter restarting the process. So my question is why the stack gets overwritten if the process runs for a long time.
– jjyao
Nov 21 '18 at 23:25
|
show 2 more comments
1
1
1
1
I have a C++ program compiled with -fno-omit-frame-pointer. I run the same program in two processes and use perf record -p 24509,24470 -F 977 --call-graph fp -- sleep 120 then perf script to get stack traces.
What I found is that all stack traces from 24509 have main | __libc_start_main at the bottom but stack traces from 24470 only have main.
My question is why perf stack trace doesn't have __libc_start_main for one process but have it for the other process even though they are running the same program.
My perf version is 3.10.0-514.36.5.el7.x86_64.debug.
linux x86-64 glibc perf backtrace
edited Nov 20 '18 at 21:26
Peter Cordes
124k18189316
asked Nov 20 '18 at 21:19
jjyaojjyao
100415
I have a C++ program compiled with -fno-omit-frame-pointer. I run the same program in two processes and use perf record -p 24509,24470 -F 977 --call-graph fp -- sleep 120 then perf script to get stack traces.
What I found is that all stack traces from 24509 have main | __libc_start_main at the bottom but stack traces from 24470 only have main.
My question is why perf stack trace doesn't have __libc_start_main for one process but have it for the other process even though they are running the same program.
My perf version is 3.10.0-514.36.5.el7.x86_64.debug.
linux x86-64 glibc perf backtrace
linux x86-64 glibc perf backtrace
edited Nov 20 '18 at 21:26
Peter Cordes
124k18189316
asked Nov 20 '18 at 21:19
jjyaojjyao
100415
edited Nov 20 '18 at 21:26
Peter Cordes
124k18189316
asked Nov 20 '18 at 21:19
jjyaojjyao
100415
edited Nov 20 '18 at 21:26
Peter Cordes
124k18189316
edited Nov 20 '18 at 21:26
Peter Cordes
124k18189316
edited Nov 20 '18 at 21:26
Peter Cordes
124k18189316
124k18189316
asked Nov 20 '18 at 21:19
jjyaojjyao
100415
asked Nov 20 '18 at 21:19
jjyaojjyao
100415
asked Nov 20 '18 at 21:19
jjyaojjyao
100415
100415
Are the two processes both started from the exact same executable, or different builds of it? Are they started with the same options and environment variables? Is this repeatable if you restart the processes and runperf recordagain? Or even if you runperf recordagain without restarting them?
– Peter Cordes
Nov 20 '18 at 21:27
1
Hi @PeterCordes, those two processes are started from the exact same executable. The only difference is that they log to different log files. Also I restarted the process that's missing__libc_start_mainand__libc_start_mainappears after restart. If I don't restart, the process is consistently missing__libc_start_main.
– jjyao
Nov 21 '18 at 20:18
If you try repeated restarts, is it sometimes missing__libc_start_main? Or was that a one-off that you can't repeat at all? Or maybe after running for a long time, something gets overwritten on the stack and the info is lost? And BTW, command line args and the environment are on the stack on process entry, but they're above the stack frames that lead to__libc_start_mainso should have no impact.
– Peter Cordes
Nov 21 '18 at 20:21
Have you upgraded your systemlibc.so.6after starting the original process? Maybeperfis using stack-unwind metadata from the file of that name in the filesystem, but the libc the process still has mapped has different symbol addresses so its.eh_frameaddresses don't match.
– Peter Cordes
Nov 21 '18 at 20:23
Hi @PeterCordes, you are right. I realized that after running the process for a long time,__libc_start_mainmay disappear. That's why I can always see__libc_start_mainif I immediately runperf recordafter restarting the process. So my question is why the stack gets overwritten if the process runs for a long time.
– jjyao
Nov 21 '18 at 23:25
|
show 2 more comments
Are the two processes both started from the exact same executable, or different builds of it? Are they started with the same options and environment variables? Is this repeatable if you restart the processes and runperf recordagain? Or even if you runperf recordagain without restarting them?
– Peter Cordes
Nov 20 '18 at 21:27
1
Hi @PeterCordes, those two processes are started from the exact same executable. The only difference is that they log to different log files. Also I restarted the process that's missing__libc_start_mainand__libc_start_mainappears after restart. If I don't restart, the process is consistently missing__libc_start_main.
– jjyao
Nov 21 '18 at 20:18
If you try repeated restarts, is it sometimes missing__libc_start_main? Or was that a one-off that you can't repeat at all? Or maybe after running for a long time, something gets overwritten on the stack and the info is lost? And BTW, command line args and the environment are on the stack on process entry, but they're above the stack frames that lead to__libc_start_mainso should have no impact.
– Peter Cordes
Nov 21 '18 at 20:21
Have you upgraded your systemlibc.so.6after starting the original process? Maybeperfis using stack-unwind metadata from the file of that name in the filesystem, but the libc the process still has mapped has different symbol addresses so its.eh_frameaddresses don't match.
– Peter Cordes
Nov 21 '18 at 20:23
Hi @PeterCordes, you are right. I realized that after running the process for a long time,__libc_start_mainmay disappear. That's why I can always see__libc_start_mainif I immediately runperf recordafter restarting the process. So my question is why the stack gets overwritten if the process runs for a long time.
– jjyao
Nov 21 '18 at 23:25
Are the two processes both started from the exact same executable, or different builds of it? Are they started with the same options and environment variables? Is this repeatable if you restart the processes and run
perf record again? Or even if you run perf record again without restarting them?– Peter Cordes
Nov 20 '18 at 21:27
Are the two processes both started from the exact same executable, or different builds of it? Are they started with the same options and environment variables? Is this repeatable if you restart the processes and run
perf record again? Or even if you run perf record again without restarting them?– Peter Cordes
Nov 20 '18 at 21:27
1
1
Hi @PeterCordes, those two processes are started from the exact same executable. The only difference is that they log to different log files. Also I restarted the process that's missing
__libc_start_main and __libc_start_main appears after restart. If I don't restart, the process is consistently missing __libc_start_main.– jjyao
Nov 21 '18 at 20:18
Hi @PeterCordes, those two processes are started from the exact same executable. The only difference is that they log to different log files. Also I restarted the process that's missing
__libc_start_main and __libc_start_main appears after restart. If I don't restart, the process is consistently missing __libc_start_main.– jjyao
Nov 21 '18 at 20:18
If you try repeated restarts, is it sometimes missing
__libc_start_main? Or was that a one-off that you can't repeat at all? Or maybe after running for a long time, something gets overwritten on the stack and the info is lost? And BTW, command line args and the environment are on the stack on process entry, but they're above the stack frames that lead to __libc_start_main so should have no impact.– Peter Cordes
Nov 21 '18 at 20:21
If you try repeated restarts, is it sometimes missing
__libc_start_main? Or was that a one-off that you can't repeat at all? Or maybe after running for a long time, something gets overwritten on the stack and the info is lost? And BTW, command line args and the environment are on the stack on process entry, but they're above the stack frames that lead to __libc_start_main so should have no impact.– Peter Cordes
Nov 21 '18 at 20:21
Have you upgraded your system
libc.so.6 after starting the original process? Maybe perf is using stack-unwind metadata from the file of that name in the filesystem, but the libc the process still has mapped has different symbol addresses so its .eh_frame addresses don't match.– Peter Cordes
Nov 21 '18 at 20:23
Have you upgraded your system
libc.so.6 after starting the original process? Maybe perf is using stack-unwind metadata from the file of that name in the filesystem, but the libc the process still has mapped has different symbol addresses so its .eh_frame addresses don't match.– Peter Cordes
Nov 21 '18 at 20:23
Hi @PeterCordes, you are right. I realized that after running the process for a long time,
__libc_start_main may disappear. That's why I can always see __libc_start_main if I immediately run perf record after restarting the process. So my question is why the stack gets overwritten if the process runs for a long time.– jjyao
Nov 21 '18 at 23:25
Hi @PeterCordes, you are right. I realized that after running the process for a long time,
__libc_start_main may disappear. That's why I can always see __libc_start_main if I immediately run perf record after restarting the process. So my question is why the stack gets overwritten if the process runs for a long time.– jjyao
Nov 21 '18 at 23:25
|
show 2 more comments
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53401691%2fperf-stack-trace-is-missing-libc-start-main-for-some-processes%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53401691%2fperf-stack-trace-is-missing-libc-start-main-for-some-processes%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Are the two processes both started from the exact same executable, or different builds of it? Are they started with the same options and environment variables? Is this repeatable if you restart the processes and run
perf recordagain? Or even if you runperf recordagain without restarting them?– Peter Cordes
Nov 20 '18 at 21:27
1
Hi @PeterCordes, those two processes are started from the exact same executable. The only difference is that they log to different log files. Also I restarted the process that's missing
__libc_start_mainand__libc_start_mainappears after restart. If I don't restart, the process is consistently missing__libc_start_main.– jjyao
Nov 21 '18 at 20:18
If you try repeated restarts, is it sometimes missing
__libc_start_main? Or was that a one-off that you can't repeat at all? Or maybe after running for a long time, something gets overwritten on the stack and the info is lost? And BTW, command line args and the environment are on the stack on process entry, but they're above the stack frames that lead to__libc_start_mainso should have no impact.– Peter Cordes
Nov 21 '18 at 20:21
Have you upgraded your system
libc.so.6after starting the original process? Maybeperfis using stack-unwind metadata from the file of that name in the filesystem, but the libc the process still has mapped has different symbol addresses so its.eh_frameaddresses don't match.– Peter Cordes
Nov 21 '18 at 20:23
Hi @PeterCordes, you are right. I realized that after running the process for a long time,
__libc_start_mainmay disappear. That's why I can always see__libc_start_mainif I immediately runperf recordafter restarting the process. So my question is why the stack gets overwritten if the process runs for a long time.– jjyao
Nov 21 '18 at 23:25