unexpected behaviour in shell command substitution












6















Given the shell script test.sh



for var in "$@"
do
echo "$var"
done


and the invocation sh test.sh $(echo '"hello " "hi and bye"')



I would expect the output



hello
hi and bye


but instead get:



"hello
"
"hi
and
bye"


If I change to sh test.sh $(echo "hello " "hi and bye")



then I get



hello
hi
and
bye


Neither of these behaviours are desired, how can I get the desired behaviour?



What I want to understand is why sh test.sh $(echo "hello " "hi and bye") and sh test.sh "hello " "hi and bye" are not equivalent?










share|improve this question





























    6















    Given the shell script test.sh



    for var in "$@"
    do
    echo "$var"
    done


    and the invocation sh test.sh $(echo '"hello " "hi and bye"')



    I would expect the output



    hello
    hi and bye


    but instead get:



    "hello
    "
    "hi
    and
    bye"


    If I change to sh test.sh $(echo "hello " "hi and bye")



    then I get



    hello
    hi
    and
    bye


    Neither of these behaviours are desired, how can I get the desired behaviour?



    What I want to understand is why sh test.sh $(echo "hello " "hi and bye") and sh test.sh "hello " "hi and bye" are not equivalent?










    share|improve this question



























      6












      6








      6








      Given the shell script test.sh



      for var in "$@"
      do
      echo "$var"
      done


      and the invocation sh test.sh $(echo '"hello " "hi and bye"')



      I would expect the output



      hello
      hi and bye


      but instead get:



      "hello
      "
      "hi
      and
      bye"


      If I change to sh test.sh $(echo "hello " "hi and bye")



      then I get



      hello
      hi
      and
      bye


      Neither of these behaviours are desired, how can I get the desired behaviour?



      What I want to understand is why sh test.sh $(echo "hello " "hi and bye") and sh test.sh "hello " "hi and bye" are not equivalent?










      share|improve this question
















      Given the shell script test.sh



      for var in "$@"
      do
      echo "$var"
      done


      and the invocation sh test.sh $(echo '"hello " "hi and bye"')



      I would expect the output



      hello
      hi and bye


      but instead get:



      "hello
      "
      "hi
      and
      bye"


      If I change to sh test.sh $(echo "hello " "hi and bye")



      then I get



      hello
      hi
      and
      bye


      Neither of these behaviours are desired, how can I get the desired behaviour?



      What I want to understand is why sh test.sh $(echo "hello " "hi and bye") and sh test.sh "hello " "hi and bye" are not equivalent?







      shell






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Jan 21 at 11:10







      user1207217

















      asked Jan 21 at 10:59









      user1207217user1207217

      60849




      60849






















          4 Answers
          4






          active

          oldest

          votes


















          8














          Your command substitution will generate a string.



          In the case of



          $(echo '"hello " "hi and bye"')


          this string will be "hello " "hi and bye".



          The string is then undergoing word splitting (and filename globbing, but it doesn't affect this example). The word splitting happen on every character that is the same as one of the characters in $IFS (by default, spaces, tabs and newlines).



          The words generated by the default value of IFS would be "hello, ", "hi, and, and bye".



          These are then given as separate arguments to your script.



          In your second command, the command substitution is



          $(echo "hello " "hi and bye")


          This generates the string hello hi and bye and the word splitting would result in the four words hello, hi, and, and bye.



          In your last example, you use the two arguments hello and hi and bye directly with your script. These won't undergo word splitting because they are quoted.






          share|improve this answer


























          • Thanks. Is the short answer to this then that you cannot (reliably) use command substitution to generate arguments to another command?

            – user1207217
            Jan 21 at 11:26











          • @user1207217 Correct, especially if the arguments contain characters from $IFS or filename globbing characters.

            – Kusalananda
            Jan 21 at 11:29











          • @Kusalananda thanks, although thats a huge pain ....

            – user1207217
            Jan 21 at 11:30






          • 1





            @user1207217 You may want to ask a separate question about what it actually is that you are intending or trying to do.

            – Kusalananda
            Jan 21 at 11:31











          • Good idea, thanks. unix.stackexchange.com/questions/495769/…

            – user1207217
            Jan 21 at 11:40



















          3














          TL;DR: the result of echo is a victim of word splitting on $() result, and sh test.sh "hello " "hi and bye" uses a different rule.



          Let's take a peak as what's actually happening, when you add set -x for debugging:



          > set -x
          > ./main.sh $(echo '"hello " "hi and bye"')
          ++ echo '"hello " "hi and bye"'
          + ./main.sh '"hello' '"' '"hi' and 'bye"'
          "hello
          "
          "hi
          and
          bye"


          echo receives "hello " "hi and bye" as a single argument (yep, including all spaces and single quote) due to single-quotes in the command itself. But command substitution turns that into "hello " "hi and bye". To quote bash manual:




          If the substitution appears within double quotes, word splitting and pathname expansion are not performed on the results.




          However, in your case, unquoted command substitution allows exactly for word splitting to occur based on blank character (which is one of the 3 characters set in the $IFS variable, which is what is used for word splitting). Thus, if you break the resulting string at each space, what do you have?





          1. "hello, then space, after which we have next token


          2. " - that one character by itself, isolated by spaces from both sides.


          3. "hi, again with space on left and right

          4. and

          5. bye"


          all in all, you get 5 tokens broken at whitespaces. It's important to realize that word splitting, in this case, occurs exactly a result of command substitution. Later, all of these are treated as individual tokens that have been already parsed. As you see in set -x output, the shell script is called with exactly those tokens.



          As for why sh test.sh "hello " "hi and bye" behaves differently, that's because the different rule applies. Your command-line is parsed after you enter it, and quoted strings are treated as a single unit, i.e. hello and hi and bye






          share|improve this answer

































            2














            You could change IFS to something other than space (since you want to protect the space in hi and bye), and use that something to separate the two strings:



            $ IFS=:
            $ sh test.sh $(echo "hello ":"hi and bye")
            hello
            hi and bye


            The order of operations (those relevant in this example, anyway) is somewhat like this:




            1. substitutions

            2. word splitting

            3. quote removal


            Note that quotes created as a result of substitutions aren't special in word splitting or quote removal, so a quote in the output of (1) passes through (2) and (3) like any other character. So:




            1. In echo "hello " "hi and bye", the shell removes these quotes, so echo gets hello and hi and bye, so outputs hello hi and bye, joining the strings with a space.

            2. In echo '"hello " "hi and bye"', the shell removes the outer ', the echo gets "hello " "hi and bye" and outputs that.


            3. In sh test.sh $(echo '"hello " "hi and bye"'), the command substitution is replaced with "hello " "hi and bye", but these quotes are a result of a substitution, and so aren't involved in word splitting or quote removal.



              So the shell splits those into "hello, ", "hi, and, bye", hence the output you get.



            4. In sh test.sh $(echo "hello " "hi and bye"), the command substitution is replaced with hello hi and bye, which gets split into hello, hi and bye.






            share|improve this answer





















            • 1





              Thanks. I guess what I am not understanding is why the usual shell substitution is not occurring, i.e. why is the behaviour not the same as if I do sh test.sh "hello " "hi and bye"?

              – user1207217
              Jan 21 at 11:09











            • @user1207217 does the edit help?

              – Olorin
              Jan 21 at 11:27











            • In $(echo "hello ":"hi and bye"), you might as well write $(echo "hello :hi and bye"). The colon works the same quoted or not.

              – ilkkachu
              Jan 21 at 11:31



















            0














            How about



            sh test.sh "hello " "hi and bye"


            ?






            share|improve this answer
























            • I'm aware I can do this, the shell script is a MWE of an unexpected behaviour -- I'm trying to use a command substitution to generate a long list of args (some of which contain spaces) but it doesn't work correctly because even with quoting in the command output the above is occurring, i.e. the shell is not doing its usual parsing and treating everything inside a single or double quote pair as a single arg

              – user1207217
              Jan 21 at 11:15











            Your Answer








            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "106"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: false,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: null,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














            draft saved

            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f495755%2funexpected-behaviour-in-shell-command-substitution%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown

























            4 Answers
            4






            active

            oldest

            votes








            4 Answers
            4






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            8














            Your command substitution will generate a string.



            In the case of



            $(echo '"hello " "hi and bye"')


            this string will be "hello " "hi and bye".



            The string is then undergoing word splitting (and filename globbing, but it doesn't affect this example). The word splitting happen on every character that is the same as one of the characters in $IFS (by default, spaces, tabs and newlines).



            The words generated by the default value of IFS would be "hello, ", "hi, and, and bye".



            These are then given as separate arguments to your script.



            In your second command, the command substitution is



            $(echo "hello " "hi and bye")


            This generates the string hello hi and bye and the word splitting would result in the four words hello, hi, and, and bye.



            In your last example, you use the two arguments hello and hi and bye directly with your script. These won't undergo word splitting because they are quoted.






            share|improve this answer


























            • Thanks. Is the short answer to this then that you cannot (reliably) use command substitution to generate arguments to another command?

              – user1207217
              Jan 21 at 11:26











            • @user1207217 Correct, especially if the arguments contain characters from $IFS or filename globbing characters.

              – Kusalananda
              Jan 21 at 11:29











            • @Kusalananda thanks, although thats a huge pain ....

              – user1207217
              Jan 21 at 11:30






            • 1





              @user1207217 You may want to ask a separate question about what it actually is that you are intending or trying to do.

              – Kusalananda
              Jan 21 at 11:31











            • Good idea, thanks. unix.stackexchange.com/questions/495769/…

              – user1207217
              Jan 21 at 11:40
















            8














            Your command substitution will generate a string.



            In the case of



            $(echo '"hello " "hi and bye"')


            this string will be "hello " "hi and bye".



            The string is then undergoing word splitting (and filename globbing, but it doesn't affect this example). The word splitting happen on every character that is the same as one of the characters in $IFS (by default, spaces, tabs and newlines).



            The words generated by the default value of IFS would be "hello, ", "hi, and, and bye".



            These are then given as separate arguments to your script.



            In your second command, the command substitution is



            $(echo "hello " "hi and bye")


            This generates the string hello hi and bye and the word splitting would result in the four words hello, hi, and, and bye.



            In your last example, you use the two arguments hello and hi and bye directly with your script. These won't undergo word splitting because they are quoted.






            share|improve this answer


























            • Thanks. Is the short answer to this then that you cannot (reliably) use command substitution to generate arguments to another command?

              – user1207217
              Jan 21 at 11:26











            • @user1207217 Correct, especially if the arguments contain characters from $IFS or filename globbing characters.

              – Kusalananda
              Jan 21 at 11:29











            • @Kusalananda thanks, although thats a huge pain ....

              – user1207217
              Jan 21 at 11:30






            • 1





              @user1207217 You may want to ask a separate question about what it actually is that you are intending or trying to do.

              – Kusalananda
              Jan 21 at 11:31











            • Good idea, thanks. unix.stackexchange.com/questions/495769/…

              – user1207217
              Jan 21 at 11:40














            8












            8








            8







            Your command substitution will generate a string.



            In the case of



            $(echo '"hello " "hi and bye"')


            this string will be "hello " "hi and bye".



            The string is then undergoing word splitting (and filename globbing, but it doesn't affect this example). The word splitting happen on every character that is the same as one of the characters in $IFS (by default, spaces, tabs and newlines).



            The words generated by the default value of IFS would be "hello, ", "hi, and, and bye".



            These are then given as separate arguments to your script.



            In your second command, the command substitution is



            $(echo "hello " "hi and bye")


            This generates the string hello hi and bye and the word splitting would result in the four words hello, hi, and, and bye.



            In your last example, you use the two arguments hello and hi and bye directly with your script. These won't undergo word splitting because they are quoted.






            share|improve this answer















            Your command substitution will generate a string.



            In the case of



            $(echo '"hello " "hi and bye"')


            this string will be "hello " "hi and bye".



            The string is then undergoing word splitting (and filename globbing, but it doesn't affect this example). The word splitting happen on every character that is the same as one of the characters in $IFS (by default, spaces, tabs and newlines).



            The words generated by the default value of IFS would be "hello, ", "hi, and, and bye".



            These are then given as separate arguments to your script.



            In your second command, the command substitution is



            $(echo "hello " "hi and bye")


            This generates the string hello hi and bye and the word splitting would result in the four words hello, hi, and, and bye.



            In your last example, you use the two arguments hello and hi and bye directly with your script. These won't undergo word splitting because they are quoted.







            share|improve this answer














            share|improve this answer



            share|improve this answer








            edited Jan 21 at 11:30

























            answered Jan 21 at 11:20









            KusalanandaKusalananda

            134k17255418




            134k17255418













            • Thanks. Is the short answer to this then that you cannot (reliably) use command substitution to generate arguments to another command?

              – user1207217
              Jan 21 at 11:26











            • @user1207217 Correct, especially if the arguments contain characters from $IFS or filename globbing characters.

              – Kusalananda
              Jan 21 at 11:29











            • @Kusalananda thanks, although thats a huge pain ....

              – user1207217
              Jan 21 at 11:30






            • 1





              @user1207217 You may want to ask a separate question about what it actually is that you are intending or trying to do.

              – Kusalananda
              Jan 21 at 11:31











            • Good idea, thanks. unix.stackexchange.com/questions/495769/…

              – user1207217
              Jan 21 at 11:40



















            • Thanks. Is the short answer to this then that you cannot (reliably) use command substitution to generate arguments to another command?

              – user1207217
              Jan 21 at 11:26











            • @user1207217 Correct, especially if the arguments contain characters from $IFS or filename globbing characters.

              – Kusalananda
              Jan 21 at 11:29











            • @Kusalananda thanks, although thats a huge pain ....

              – user1207217
              Jan 21 at 11:30






            • 1





              @user1207217 You may want to ask a separate question about what it actually is that you are intending or trying to do.

              – Kusalananda
              Jan 21 at 11:31











            • Good idea, thanks. unix.stackexchange.com/questions/495769/…

              – user1207217
              Jan 21 at 11:40

















            Thanks. Is the short answer to this then that you cannot (reliably) use command substitution to generate arguments to another command?

            – user1207217
            Jan 21 at 11:26





            Thanks. Is the short answer to this then that you cannot (reliably) use command substitution to generate arguments to another command?

            – user1207217
            Jan 21 at 11:26













            @user1207217 Correct, especially if the arguments contain characters from $IFS or filename globbing characters.

            – Kusalananda
            Jan 21 at 11:29





            @user1207217 Correct, especially if the arguments contain characters from $IFS or filename globbing characters.

            – Kusalananda
            Jan 21 at 11:29













            @Kusalananda thanks, although thats a huge pain ....

            – user1207217
            Jan 21 at 11:30





            @Kusalananda thanks, although thats a huge pain ....

            – user1207217
            Jan 21 at 11:30




            1




            1





            @user1207217 You may want to ask a separate question about what it actually is that you are intending or trying to do.

            – Kusalananda
            Jan 21 at 11:31





            @user1207217 You may want to ask a separate question about what it actually is that you are intending or trying to do.

            – Kusalananda
            Jan 21 at 11:31













            Good idea, thanks. unix.stackexchange.com/questions/495769/…

            – user1207217
            Jan 21 at 11:40





            Good idea, thanks. unix.stackexchange.com/questions/495769/…

            – user1207217
            Jan 21 at 11:40













            3














            TL;DR: the result of echo is a victim of word splitting on $() result, and sh test.sh "hello " "hi and bye" uses a different rule.



            Let's take a peak as what's actually happening, when you add set -x for debugging:



            > set -x
            > ./main.sh $(echo '"hello " "hi and bye"')
            ++ echo '"hello " "hi and bye"'
            + ./main.sh '"hello' '"' '"hi' and 'bye"'
            "hello
            "
            "hi
            and
            bye"


            echo receives "hello " "hi and bye" as a single argument (yep, including all spaces and single quote) due to single-quotes in the command itself. But command substitution turns that into "hello " "hi and bye". To quote bash manual:




            If the substitution appears within double quotes, word splitting and pathname expansion are not performed on the results.




            However, in your case, unquoted command substitution allows exactly for word splitting to occur based on blank character (which is one of the 3 characters set in the $IFS variable, which is what is used for word splitting). Thus, if you break the resulting string at each space, what do you have?





            1. "hello, then space, after which we have next token


            2. " - that one character by itself, isolated by spaces from both sides.


            3. "hi, again with space on left and right

            4. and

            5. bye"


            all in all, you get 5 tokens broken at whitespaces. It's important to realize that word splitting, in this case, occurs exactly a result of command substitution. Later, all of these are treated as individual tokens that have been already parsed. As you see in set -x output, the shell script is called with exactly those tokens.



            As for why sh test.sh "hello " "hi and bye" behaves differently, that's because the different rule applies. Your command-line is parsed after you enter it, and quoted strings are treated as a single unit, i.e. hello and hi and bye






            share|improve this answer






























              3














              TL;DR: the result of echo is a victim of word splitting on $() result, and sh test.sh "hello " "hi and bye" uses a different rule.



              Let's take a peak as what's actually happening, when you add set -x for debugging:



              > set -x
              > ./main.sh $(echo '"hello " "hi and bye"')
              ++ echo '"hello " "hi and bye"'
              + ./main.sh '"hello' '"' '"hi' and 'bye"'
              "hello
              "
              "hi
              and
              bye"


              echo receives "hello " "hi and bye" as a single argument (yep, including all spaces and single quote) due to single-quotes in the command itself. But command substitution turns that into "hello " "hi and bye". To quote bash manual:




              If the substitution appears within double quotes, word splitting and pathname expansion are not performed on the results.




              However, in your case, unquoted command substitution allows exactly for word splitting to occur based on blank character (which is one of the 3 characters set in the $IFS variable, which is what is used for word splitting). Thus, if you break the resulting string at each space, what do you have?





              1. "hello, then space, after which we have next token


              2. " - that one character by itself, isolated by spaces from both sides.


              3. "hi, again with space on left and right

              4. and

              5. bye"


              all in all, you get 5 tokens broken at whitespaces. It's important to realize that word splitting, in this case, occurs exactly a result of command substitution. Later, all of these are treated as individual tokens that have been already parsed. As you see in set -x output, the shell script is called with exactly those tokens.



              As for why sh test.sh "hello " "hi and bye" behaves differently, that's because the different rule applies. Your command-line is parsed after you enter it, and quoted strings are treated as a single unit, i.e. hello and hi and bye






              share|improve this answer




























                3












                3








                3







                TL;DR: the result of echo is a victim of word splitting on $() result, and sh test.sh "hello " "hi and bye" uses a different rule.



                Let's take a peak as what's actually happening, when you add set -x for debugging:



                > set -x
                > ./main.sh $(echo '"hello " "hi and bye"')
                ++ echo '"hello " "hi and bye"'
                + ./main.sh '"hello' '"' '"hi' and 'bye"'
                "hello
                "
                "hi
                and
                bye"


                echo receives "hello " "hi and bye" as a single argument (yep, including all spaces and single quote) due to single-quotes in the command itself. But command substitution turns that into "hello " "hi and bye". To quote bash manual:




                If the substitution appears within double quotes, word splitting and pathname expansion are not performed on the results.




                However, in your case, unquoted command substitution allows exactly for word splitting to occur based on blank character (which is one of the 3 characters set in the $IFS variable, which is what is used for word splitting). Thus, if you break the resulting string at each space, what do you have?





                1. "hello, then space, after which we have next token


                2. " - that one character by itself, isolated by spaces from both sides.


                3. "hi, again with space on left and right

                4. and

                5. bye"


                all in all, you get 5 tokens broken at whitespaces. It's important to realize that word splitting, in this case, occurs exactly a result of command substitution. Later, all of these are treated as individual tokens that have been already parsed. As you see in set -x output, the shell script is called with exactly those tokens.



                As for why sh test.sh "hello " "hi and bye" behaves differently, that's because the different rule applies. Your command-line is parsed after you enter it, and quoted strings are treated as a single unit, i.e. hello and hi and bye






                share|improve this answer















                TL;DR: the result of echo is a victim of word splitting on $() result, and sh test.sh "hello " "hi and bye" uses a different rule.



                Let's take a peak as what's actually happening, when you add set -x for debugging:



                > set -x
                > ./main.sh $(echo '"hello " "hi and bye"')
                ++ echo '"hello " "hi and bye"'
                + ./main.sh '"hello' '"' '"hi' and 'bye"'
                "hello
                "
                "hi
                and
                bye"


                echo receives "hello " "hi and bye" as a single argument (yep, including all spaces and single quote) due to single-quotes in the command itself. But command substitution turns that into "hello " "hi and bye". To quote bash manual:




                If the substitution appears within double quotes, word splitting and pathname expansion are not performed on the results.




                However, in your case, unquoted command substitution allows exactly for word splitting to occur based on blank character (which is one of the 3 characters set in the $IFS variable, which is what is used for word splitting). Thus, if you break the resulting string at each space, what do you have?





                1. "hello, then space, after which we have next token


                2. " - that one character by itself, isolated by spaces from both sides.


                3. "hi, again with space on left and right

                4. and

                5. bye"


                all in all, you get 5 tokens broken at whitespaces. It's important to realize that word splitting, in this case, occurs exactly a result of command substitution. Later, all of these are treated as individual tokens that have been already parsed. As you see in set -x output, the shell script is called with exactly those tokens.



                As for why sh test.sh "hello " "hi and bye" behaves differently, that's because the different rule applies. Your command-line is parsed after you enter it, and quoted strings are treated as a single unit, i.e. hello and hi and bye







                share|improve this answer














                share|improve this answer



                share|improve this answer








                edited Feb 4 at 15:01









                msp9011

                4,38044066




                4,38044066










                answered Jan 21 at 11:19









                Sergiy KolodyazhnyySergiy Kolodyazhnyy

                10.5k42663




                10.5k42663























                    2














                    You could change IFS to something other than space (since you want to protect the space in hi and bye), and use that something to separate the two strings:



                    $ IFS=:
                    $ sh test.sh $(echo "hello ":"hi and bye")
                    hello
                    hi and bye


                    The order of operations (those relevant in this example, anyway) is somewhat like this:




                    1. substitutions

                    2. word splitting

                    3. quote removal


                    Note that quotes created as a result of substitutions aren't special in word splitting or quote removal, so a quote in the output of (1) passes through (2) and (3) like any other character. So:




                    1. In echo "hello " "hi and bye", the shell removes these quotes, so echo gets hello and hi and bye, so outputs hello hi and bye, joining the strings with a space.

                    2. In echo '"hello " "hi and bye"', the shell removes the outer ', the echo gets "hello " "hi and bye" and outputs that.


                    3. In sh test.sh $(echo '"hello " "hi and bye"'), the command substitution is replaced with "hello " "hi and bye", but these quotes are a result of a substitution, and so aren't involved in word splitting or quote removal.



                      So the shell splits those into "hello, ", "hi, and, bye", hence the output you get.



                    4. In sh test.sh $(echo "hello " "hi and bye"), the command substitution is replaced with hello hi and bye, which gets split into hello, hi and bye.






                    share|improve this answer





















                    • 1





                      Thanks. I guess what I am not understanding is why the usual shell substitution is not occurring, i.e. why is the behaviour not the same as if I do sh test.sh "hello " "hi and bye"?

                      – user1207217
                      Jan 21 at 11:09











                    • @user1207217 does the edit help?

                      – Olorin
                      Jan 21 at 11:27











                    • In $(echo "hello ":"hi and bye"), you might as well write $(echo "hello :hi and bye"). The colon works the same quoted or not.

                      – ilkkachu
                      Jan 21 at 11:31
















                    2














                    You could change IFS to something other than space (since you want to protect the space in hi and bye), and use that something to separate the two strings:



                    $ IFS=:
                    $ sh test.sh $(echo "hello ":"hi and bye")
                    hello
                    hi and bye


                    The order of operations (those relevant in this example, anyway) is somewhat like this:




                    1. substitutions

                    2. word splitting

                    3. quote removal


                    Note that quotes created as a result of substitutions aren't special in word splitting or quote removal, so a quote in the output of (1) passes through (2) and (3) like any other character. So:




                    1. In echo "hello " "hi and bye", the shell removes these quotes, so echo gets hello and hi and bye, so outputs hello hi and bye, joining the strings with a space.

                    2. In echo '"hello " "hi and bye"', the shell removes the outer ', the echo gets "hello " "hi and bye" and outputs that.


                    3. In sh test.sh $(echo '"hello " "hi and bye"'), the command substitution is replaced with "hello " "hi and bye", but these quotes are a result of a substitution, and so aren't involved in word splitting or quote removal.



                      So the shell splits those into "hello, ", "hi, and, bye", hence the output you get.



                    4. In sh test.sh $(echo "hello " "hi and bye"), the command substitution is replaced with hello hi and bye, which gets split into hello, hi and bye.






                    share|improve this answer





















                    • 1





                      Thanks. I guess what I am not understanding is why the usual shell substitution is not occurring, i.e. why is the behaviour not the same as if I do sh test.sh "hello " "hi and bye"?

                      – user1207217
                      Jan 21 at 11:09











                    • @user1207217 does the edit help?

                      – Olorin
                      Jan 21 at 11:27











                    • In $(echo "hello ":"hi and bye"), you might as well write $(echo "hello :hi and bye"). The colon works the same quoted or not.

                      – ilkkachu
                      Jan 21 at 11:31














                    2












                    2








                    2







                    You could change IFS to something other than space (since you want to protect the space in hi and bye), and use that something to separate the two strings:



                    $ IFS=:
                    $ sh test.sh $(echo "hello ":"hi and bye")
                    hello
                    hi and bye


                    The order of operations (those relevant in this example, anyway) is somewhat like this:




                    1. substitutions

                    2. word splitting

                    3. quote removal


                    Note that quotes created as a result of substitutions aren't special in word splitting or quote removal, so a quote in the output of (1) passes through (2) and (3) like any other character. So:




                    1. In echo "hello " "hi and bye", the shell removes these quotes, so echo gets hello and hi and bye, so outputs hello hi and bye, joining the strings with a space.

                    2. In echo '"hello " "hi and bye"', the shell removes the outer ', the echo gets "hello " "hi and bye" and outputs that.


                    3. In sh test.sh $(echo '"hello " "hi and bye"'), the command substitution is replaced with "hello " "hi and bye", but these quotes are a result of a substitution, and so aren't involved in word splitting or quote removal.



                      So the shell splits those into "hello, ", "hi, and, bye", hence the output you get.



                    4. In sh test.sh $(echo "hello " "hi and bye"), the command substitution is replaced with hello hi and bye, which gets split into hello, hi and bye.






                    share|improve this answer















                    You could change IFS to something other than space (since you want to protect the space in hi and bye), and use that something to separate the two strings:



                    $ IFS=:
                    $ sh test.sh $(echo "hello ":"hi and bye")
                    hello
                    hi and bye


                    The order of operations (those relevant in this example, anyway) is somewhat like this:




                    1. substitutions

                    2. word splitting

                    3. quote removal


                    Note that quotes created as a result of substitutions aren't special in word splitting or quote removal, so a quote in the output of (1) passes through (2) and (3) like any other character. So:




                    1. In echo "hello " "hi and bye", the shell removes these quotes, so echo gets hello and hi and bye, so outputs hello hi and bye, joining the strings with a space.

                    2. In echo '"hello " "hi and bye"', the shell removes the outer ', the echo gets "hello " "hi and bye" and outputs that.


                    3. In sh test.sh $(echo '"hello " "hi and bye"'), the command substitution is replaced with "hello " "hi and bye", but these quotes are a result of a substitution, and so aren't involved in word splitting or quote removal.



                      So the shell splits those into "hello, ", "hi, and, bye", hence the output you get.



                    4. In sh test.sh $(echo "hello " "hi and bye"), the command substitution is replaced with hello hi and bye, which gets split into hello, hi and bye.







                    share|improve this answer














                    share|improve this answer



                    share|improve this answer








                    edited Jan 21 at 11:27

























                    answered Jan 21 at 11:07









                    OlorinOlorin

                    3,8581721




                    3,8581721








                    • 1





                      Thanks. I guess what I am not understanding is why the usual shell substitution is not occurring, i.e. why is the behaviour not the same as if I do sh test.sh "hello " "hi and bye"?

                      – user1207217
                      Jan 21 at 11:09











                    • @user1207217 does the edit help?

                      – Olorin
                      Jan 21 at 11:27











                    • In $(echo "hello ":"hi and bye"), you might as well write $(echo "hello :hi and bye"). The colon works the same quoted or not.

                      – ilkkachu
                      Jan 21 at 11:31














                    • 1





                      Thanks. I guess what I am not understanding is why the usual shell substitution is not occurring, i.e. why is the behaviour not the same as if I do sh test.sh "hello " "hi and bye"?

                      – user1207217
                      Jan 21 at 11:09











                    • @user1207217 does the edit help?

                      – Olorin
                      Jan 21 at 11:27











                    • In $(echo "hello ":"hi and bye"), you might as well write $(echo "hello :hi and bye"). The colon works the same quoted or not.

                      – ilkkachu
                      Jan 21 at 11:31








                    1




                    1





                    Thanks. I guess what I am not understanding is why the usual shell substitution is not occurring, i.e. why is the behaviour not the same as if I do sh test.sh "hello " "hi and bye"?

                    – user1207217
                    Jan 21 at 11:09





                    Thanks. I guess what I am not understanding is why the usual shell substitution is not occurring, i.e. why is the behaviour not the same as if I do sh test.sh "hello " "hi and bye"?

                    – user1207217
                    Jan 21 at 11:09













                    @user1207217 does the edit help?

                    – Olorin
                    Jan 21 at 11:27





                    @user1207217 does the edit help?

                    – Olorin
                    Jan 21 at 11:27













                    In $(echo "hello ":"hi and bye"), you might as well write $(echo "hello :hi and bye"). The colon works the same quoted or not.

                    – ilkkachu
                    Jan 21 at 11:31





                    In $(echo "hello ":"hi and bye"), you might as well write $(echo "hello :hi and bye"). The colon works the same quoted or not.

                    – ilkkachu
                    Jan 21 at 11:31











                    0














                    How about



                    sh test.sh "hello " "hi and bye"


                    ?






                    share|improve this answer
























                    • I'm aware I can do this, the shell script is a MWE of an unexpected behaviour -- I'm trying to use a command substitution to generate a long list of args (some of which contain spaces) but it doesn't work correctly because even with quoting in the command output the above is occurring, i.e. the shell is not doing its usual parsing and treating everything inside a single or double quote pair as a single arg

                      – user1207217
                      Jan 21 at 11:15
















                    0














                    How about



                    sh test.sh "hello " "hi and bye"


                    ?






                    share|improve this answer
























                    • I'm aware I can do this, the shell script is a MWE of an unexpected behaviour -- I'm trying to use a command substitution to generate a long list of args (some of which contain spaces) but it doesn't work correctly because even with quoting in the command output the above is occurring, i.e. the shell is not doing its usual parsing and treating everything inside a single or double quote pair as a single arg

                      – user1207217
                      Jan 21 at 11:15














                    0












                    0








                    0







                    How about



                    sh test.sh "hello " "hi and bye"


                    ?






                    share|improve this answer













                    How about



                    sh test.sh "hello " "hi and bye"


                    ?







                    share|improve this answer












                    share|improve this answer



                    share|improve this answer










                    answered Jan 21 at 11:13









                    gerhard d.gerhard d.

                    1,256311




                    1,256311













                    • I'm aware I can do this, the shell script is a MWE of an unexpected behaviour -- I'm trying to use a command substitution to generate a long list of args (some of which contain spaces) but it doesn't work correctly because even with quoting in the command output the above is occurring, i.e. the shell is not doing its usual parsing and treating everything inside a single or double quote pair as a single arg

                      – user1207217
                      Jan 21 at 11:15



















                    • I'm aware I can do this, the shell script is a MWE of an unexpected behaviour -- I'm trying to use a command substitution to generate a long list of args (some of which contain spaces) but it doesn't work correctly because even with quoting in the command output the above is occurring, i.e. the shell is not doing its usual parsing and treating everything inside a single or double quote pair as a single arg

                      – user1207217
                      Jan 21 at 11:15

















                    I'm aware I can do this, the shell script is a MWE of an unexpected behaviour -- I'm trying to use a command substitution to generate a long list of args (some of which contain spaces) but it doesn't work correctly because even with quoting in the command output the above is occurring, i.e. the shell is not doing its usual parsing and treating everything inside a single or double quote pair as a single arg

                    – user1207217
                    Jan 21 at 11:15





                    I'm aware I can do this, the shell script is a MWE of an unexpected behaviour -- I'm trying to use a command substitution to generate a long list of args (some of which contain spaces) but it doesn't work correctly because even with quoting in the command output the above is occurring, i.e. the shell is not doing its usual parsing and treating everything inside a single or double quote pair as a single arg

                    – user1207217
                    Jan 21 at 11:15


















                    draft saved

                    draft discarded




















































                    Thanks for contributing an answer to Unix & Linux Stack Exchange!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f495755%2funexpected-behaviour-in-shell-command-substitution%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    Popular posts from this blog

                    MongoDB - Not Authorized To Execute Command

                    in spring boot 2.1 many test slices are not allowed anymore due to multiple @BootstrapWith

                    How to fix TextFormField cause rebuild widget in Flutter