Referencing RETURN values in sequential INSERT statements, within a transaction (in Postgres)












0















I am attempting to insert several records across multiple tables, all in the one transaction. In this instance, its to generate some mock data. Smaller sub-sets of this insert sequence would be used in 'production'.



The records to be inserted are:




  1. an 'account'

  2. a 'gist'

  3. a 'version' of the gist

  4. several 'files', representing the actual contents of the gist version


Schema available here for reference:
https://github.com/thomasgwatson/thegistofit/tree/master/sql



My current iteration looks like this



BEGIN;

INSERT INTO account (email, name, password_hash)
VALUES
(text 'HAH@YAY.COM', text 'HAH', text 'HASHHASH')
RETURNING account_id AS new_account_id;

INSERT INTO gist (account_id)
SELECT account_id from new_account_id
RETURNING gist_id AS new_gist_id;

INSERT INTO version (gist_id, title)
SELECT gist_id, 'my gist' from new_gist_id
RETURNING version_id AS new_version_id;

INSERT INTO file (langauge, file_name, content)
VALUES
(text 'python', text 'first.py', text 'asfoasdfj'),
(text 'python', text 'second.py', text 'ipsum asdfasdf'),
(text 'python', text 'third.py', text 'okaoskdaoskd')
RETURNING file_id AS new_file_ids;

INSERT INTO version_file (version_id, file_id)
-- need to do the cartesian joint here
SELECT version_id, file_id from new_version_id, new_file_ids;


COMMIT;


I've tried a variety of ways to do this within PostgresQL, to no avail. Looking up Common Table Expressions (WITH and AS clauses) has been one main thread. However, one line from the docs might be what is sinking me:




A fine point of the above example is that the WITH clause is attached
to the INSERT, not the sub-SELECT within the INSERT. This is necessary
because data-modifying statements are only allowed in WITH clauses
that are attached to the top-level statement. However, normal WITH
visibility rules apply, so it is possible to refer to the WITH
statement's output from the sub-SELECT.




from https://www.postgresql.org/docs/current/queries-with.html



Meta commentary




  1. Using DataGrip as something of a linter but it's not saving me.

  2. SQL error messages don't seem to help much, likely that they have a slightly different mental model to error messages I'm used to

  3. I'd be more than comfortable using Knex/Node or Psycopg2/Python to abstract the transactions, INSERT statements, and to handle the data munging there. Trying to make it work with pure SQL










share|improve this question





























    0















    I am attempting to insert several records across multiple tables, all in the one transaction. In this instance, its to generate some mock data. Smaller sub-sets of this insert sequence would be used in 'production'.



    The records to be inserted are:




    1. an 'account'

    2. a 'gist'

    3. a 'version' of the gist

    4. several 'files', representing the actual contents of the gist version


    Schema available here for reference:
    https://github.com/thomasgwatson/thegistofit/tree/master/sql



    My current iteration looks like this



    BEGIN;

    INSERT INTO account (email, name, password_hash)
    VALUES
    (text 'HAH@YAY.COM', text 'HAH', text 'HASHHASH')
    RETURNING account_id AS new_account_id;

    INSERT INTO gist (account_id)
    SELECT account_id from new_account_id
    RETURNING gist_id AS new_gist_id;

    INSERT INTO version (gist_id, title)
    SELECT gist_id, 'my gist' from new_gist_id
    RETURNING version_id AS new_version_id;

    INSERT INTO file (langauge, file_name, content)
    VALUES
    (text 'python', text 'first.py', text 'asfoasdfj'),
    (text 'python', text 'second.py', text 'ipsum asdfasdf'),
    (text 'python', text 'third.py', text 'okaoskdaoskd')
    RETURNING file_id AS new_file_ids;

    INSERT INTO version_file (version_id, file_id)
    -- need to do the cartesian joint here
    SELECT version_id, file_id from new_version_id, new_file_ids;


    COMMIT;


    I've tried a variety of ways to do this within PostgresQL, to no avail. Looking up Common Table Expressions (WITH and AS clauses) has been one main thread. However, one line from the docs might be what is sinking me:




    A fine point of the above example is that the WITH clause is attached
    to the INSERT, not the sub-SELECT within the INSERT. This is necessary
    because data-modifying statements are only allowed in WITH clauses
    that are attached to the top-level statement. However, normal WITH
    visibility rules apply, so it is possible to refer to the WITH
    statement's output from the sub-SELECT.




    from https://www.postgresql.org/docs/current/queries-with.html



    Meta commentary




    1. Using DataGrip as something of a linter but it's not saving me.

    2. SQL error messages don't seem to help much, likely that they have a slightly different mental model to error messages I'm used to

    3. I'd be more than comfortable using Knex/Node or Psycopg2/Python to abstract the transactions, INSERT statements, and to handle the data munging there. Trying to make it work with pure SQL










    share|improve this question



























      0












      0








      0








      I am attempting to insert several records across multiple tables, all in the one transaction. In this instance, its to generate some mock data. Smaller sub-sets of this insert sequence would be used in 'production'.



      The records to be inserted are:




      1. an 'account'

      2. a 'gist'

      3. a 'version' of the gist

      4. several 'files', representing the actual contents of the gist version


      Schema available here for reference:
      https://github.com/thomasgwatson/thegistofit/tree/master/sql



      My current iteration looks like this



      BEGIN;

      INSERT INTO account (email, name, password_hash)
      VALUES
      (text 'HAH@YAY.COM', text 'HAH', text 'HASHHASH')
      RETURNING account_id AS new_account_id;

      INSERT INTO gist (account_id)
      SELECT account_id from new_account_id
      RETURNING gist_id AS new_gist_id;

      INSERT INTO version (gist_id, title)
      SELECT gist_id, 'my gist' from new_gist_id
      RETURNING version_id AS new_version_id;

      INSERT INTO file (langauge, file_name, content)
      VALUES
      (text 'python', text 'first.py', text 'asfoasdfj'),
      (text 'python', text 'second.py', text 'ipsum asdfasdf'),
      (text 'python', text 'third.py', text 'okaoskdaoskd')
      RETURNING file_id AS new_file_ids;

      INSERT INTO version_file (version_id, file_id)
      -- need to do the cartesian joint here
      SELECT version_id, file_id from new_version_id, new_file_ids;


      COMMIT;


      I've tried a variety of ways to do this within PostgresQL, to no avail. Looking up Common Table Expressions (WITH and AS clauses) has been one main thread. However, one line from the docs might be what is sinking me:




      A fine point of the above example is that the WITH clause is attached
      to the INSERT, not the sub-SELECT within the INSERT. This is necessary
      because data-modifying statements are only allowed in WITH clauses
      that are attached to the top-level statement. However, normal WITH
      visibility rules apply, so it is possible to refer to the WITH
      statement's output from the sub-SELECT.




      from https://www.postgresql.org/docs/current/queries-with.html



      Meta commentary




      1. Using DataGrip as something of a linter but it's not saving me.

      2. SQL error messages don't seem to help much, likely that they have a slightly different mental model to error messages I'm used to

      3. I'd be more than comfortable using Knex/Node or Psycopg2/Python to abstract the transactions, INSERT statements, and to handle the data munging there. Trying to make it work with pure SQL










      share|improve this question
















      I am attempting to insert several records across multiple tables, all in the one transaction. In this instance, its to generate some mock data. Smaller sub-sets of this insert sequence would be used in 'production'.



      The records to be inserted are:




      1. an 'account'

      2. a 'gist'

      3. a 'version' of the gist

      4. several 'files', representing the actual contents of the gist version


      Schema available here for reference:
      https://github.com/thomasgwatson/thegistofit/tree/master/sql



      My current iteration looks like this



      BEGIN;

      INSERT INTO account (email, name, password_hash)
      VALUES
      (text 'HAH@YAY.COM', text 'HAH', text 'HASHHASH')
      RETURNING account_id AS new_account_id;

      INSERT INTO gist (account_id)
      SELECT account_id from new_account_id
      RETURNING gist_id AS new_gist_id;

      INSERT INTO version (gist_id, title)
      SELECT gist_id, 'my gist' from new_gist_id
      RETURNING version_id AS new_version_id;

      INSERT INTO file (langauge, file_name, content)
      VALUES
      (text 'python', text 'first.py', text 'asfoasdfj'),
      (text 'python', text 'second.py', text 'ipsum asdfasdf'),
      (text 'python', text 'third.py', text 'okaoskdaoskd')
      RETURNING file_id AS new_file_ids;

      INSERT INTO version_file (version_id, file_id)
      -- need to do the cartesian joint here
      SELECT version_id, file_id from new_version_id, new_file_ids;


      COMMIT;


      I've tried a variety of ways to do this within PostgresQL, to no avail. Looking up Common Table Expressions (WITH and AS clauses) has been one main thread. However, one line from the docs might be what is sinking me:




      A fine point of the above example is that the WITH clause is attached
      to the INSERT, not the sub-SELECT within the INSERT. This is necessary
      because data-modifying statements are only allowed in WITH clauses
      that are attached to the top-level statement. However, normal WITH
      visibility rules apply, so it is possible to refer to the WITH
      statement's output from the sub-SELECT.




      from https://www.postgresql.org/docs/current/queries-with.html



      Meta commentary




      1. Using DataGrip as something of a linter but it's not saving me.

      2. SQL error messages don't seem to help much, likely that they have a slightly different mental model to error messages I'm used to

      3. I'd be more than comfortable using Knex/Node or Psycopg2/Python to abstract the transactions, INSERT statements, and to handle the data munging there. Trying to make it work with pure SQL







      sql database postgresql






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Nov 20 '18 at 1:43







      Underwater_developer

















      asked Nov 20 '18 at 1:31









      Underwater_developerUnderwater_developer

      176110




      176110
























          1 Answer
          1






          active

          oldest

          votes


















          0














          Fortunately, I came up with a SQL statement that runs!



          BEGIN;

          with new_account as (
          INSERT INTO account (email, name, password_hash)
          VALUES
          (text 'HAH@YAY.COM', text 'HAH', text 'HASHHASH')
          RETURNING account_id
          ), new_gist as (
          INSERT INTO gist (account_id)
          SELECT account_id from new_account
          RETURNING gist_id
          ), new_version as (
          INSERT INTO version (gist_id, title)
          SELECT gist_id, 'my gist' from new_gist
          RETURNING version_id
          ), new_files as (
          INSERT INTO file (langauge, file_name, content)
          VALUES
          (text 'python', text 'first.py', text 'ipsum hhb'),
          (text 'python', text 'second.py', text 'ipsum iug'),
          (text 'python', text 'third.py', text 'ipsum thd')
          RETURNING file_id
          )

          INSERT INTO version_file (version_id, file_id)
          -- yes, doing a cartesian joint here
          SELECT version_id, file_id from new_version, new_files;

          COMMIT;





          share|improve this answer





















          • 1





            I'm not precisely sure what I changed ... then why are you offering this as an answer to the community?

            – Tim Biegeleisen
            Nov 20 '18 at 1:32











          • Honestly, I was documenting my process in the question and as I was about to submit, I used the (slightly simplified) query I was about to put into the post, into DataGrip to confirm the error message and... it worked. At a broader level, I have looked multiple times across the internet, via Google, docs, Duck Duck Go, various Stack Exchanges and could not find an example of what I was trying to create. By posting it here, I hope I can save someone the hours I spent in frustration to get to this point

            – Underwater_developer
            Nov 20 '18 at 1:39








          • 2





            The move to CTEs is the solution. You need to consume the values returned by the returning clause somehow. With plain SQL you can only store them in a temporary table or pass them on by chaining CTEs

            – a_horse_with_no_name
            Nov 20 '18 at 6:46











          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53384992%2freferencing-return-values-in-sequential-insert-statements-within-a-transaction%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          1 Answer
          1






          active

          oldest

          votes








          1 Answer
          1






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          0














          Fortunately, I came up with a SQL statement that runs!



          BEGIN;

          with new_account as (
          INSERT INTO account (email, name, password_hash)
          VALUES
          (text 'HAH@YAY.COM', text 'HAH', text 'HASHHASH')
          RETURNING account_id
          ), new_gist as (
          INSERT INTO gist (account_id)
          SELECT account_id from new_account
          RETURNING gist_id
          ), new_version as (
          INSERT INTO version (gist_id, title)
          SELECT gist_id, 'my gist' from new_gist
          RETURNING version_id
          ), new_files as (
          INSERT INTO file (langauge, file_name, content)
          VALUES
          (text 'python', text 'first.py', text 'ipsum hhb'),
          (text 'python', text 'second.py', text 'ipsum iug'),
          (text 'python', text 'third.py', text 'ipsum thd')
          RETURNING file_id
          )

          INSERT INTO version_file (version_id, file_id)
          -- yes, doing a cartesian joint here
          SELECT version_id, file_id from new_version, new_files;

          COMMIT;





          share|improve this answer





















          • 1





            I'm not precisely sure what I changed ... then why are you offering this as an answer to the community?

            – Tim Biegeleisen
            Nov 20 '18 at 1:32











          • Honestly, I was documenting my process in the question and as I was about to submit, I used the (slightly simplified) query I was about to put into the post, into DataGrip to confirm the error message and... it worked. At a broader level, I have looked multiple times across the internet, via Google, docs, Duck Duck Go, various Stack Exchanges and could not find an example of what I was trying to create. By posting it here, I hope I can save someone the hours I spent in frustration to get to this point

            – Underwater_developer
            Nov 20 '18 at 1:39








          • 2





            The move to CTEs is the solution. You need to consume the values returned by the returning clause somehow. With plain SQL you can only store them in a temporary table or pass them on by chaining CTEs

            – a_horse_with_no_name
            Nov 20 '18 at 6:46
















          0














          Fortunately, I came up with a SQL statement that runs!



          BEGIN;

          with new_account as (
          INSERT INTO account (email, name, password_hash)
          VALUES
          (text 'HAH@YAY.COM', text 'HAH', text 'HASHHASH')
          RETURNING account_id
          ), new_gist as (
          INSERT INTO gist (account_id)
          SELECT account_id from new_account
          RETURNING gist_id
          ), new_version as (
          INSERT INTO version (gist_id, title)
          SELECT gist_id, 'my gist' from new_gist
          RETURNING version_id
          ), new_files as (
          INSERT INTO file (langauge, file_name, content)
          VALUES
          (text 'python', text 'first.py', text 'ipsum hhb'),
          (text 'python', text 'second.py', text 'ipsum iug'),
          (text 'python', text 'third.py', text 'ipsum thd')
          RETURNING file_id
          )

          INSERT INTO version_file (version_id, file_id)
          -- yes, doing a cartesian joint here
          SELECT version_id, file_id from new_version, new_files;

          COMMIT;





          share|improve this answer





















          • 1





            I'm not precisely sure what I changed ... then why are you offering this as an answer to the community?

            – Tim Biegeleisen
            Nov 20 '18 at 1:32











          • Honestly, I was documenting my process in the question and as I was about to submit, I used the (slightly simplified) query I was about to put into the post, into DataGrip to confirm the error message and... it worked. At a broader level, I have looked multiple times across the internet, via Google, docs, Duck Duck Go, various Stack Exchanges and could not find an example of what I was trying to create. By posting it here, I hope I can save someone the hours I spent in frustration to get to this point

            – Underwater_developer
            Nov 20 '18 at 1:39








          • 2





            The move to CTEs is the solution. You need to consume the values returned by the returning clause somehow. With plain SQL you can only store them in a temporary table or pass them on by chaining CTEs

            – a_horse_with_no_name
            Nov 20 '18 at 6:46














          0












          0








          0







          Fortunately, I came up with a SQL statement that runs!



          BEGIN;

          with new_account as (
          INSERT INTO account (email, name, password_hash)
          VALUES
          (text 'HAH@YAY.COM', text 'HAH', text 'HASHHASH')
          RETURNING account_id
          ), new_gist as (
          INSERT INTO gist (account_id)
          SELECT account_id from new_account
          RETURNING gist_id
          ), new_version as (
          INSERT INTO version (gist_id, title)
          SELECT gist_id, 'my gist' from new_gist
          RETURNING version_id
          ), new_files as (
          INSERT INTO file (langauge, file_name, content)
          VALUES
          (text 'python', text 'first.py', text 'ipsum hhb'),
          (text 'python', text 'second.py', text 'ipsum iug'),
          (text 'python', text 'third.py', text 'ipsum thd')
          RETURNING file_id
          )

          INSERT INTO version_file (version_id, file_id)
          -- yes, doing a cartesian joint here
          SELECT version_id, file_id from new_version, new_files;

          COMMIT;





          share|improve this answer















          Fortunately, I came up with a SQL statement that runs!



          BEGIN;

          with new_account as (
          INSERT INTO account (email, name, password_hash)
          VALUES
          (text 'HAH@YAY.COM', text 'HAH', text 'HASHHASH')
          RETURNING account_id
          ), new_gist as (
          INSERT INTO gist (account_id)
          SELECT account_id from new_account
          RETURNING gist_id
          ), new_version as (
          INSERT INTO version (gist_id, title)
          SELECT gist_id, 'my gist' from new_gist
          RETURNING version_id
          ), new_files as (
          INSERT INTO file (langauge, file_name, content)
          VALUES
          (text 'python', text 'first.py', text 'ipsum hhb'),
          (text 'python', text 'second.py', text 'ipsum iug'),
          (text 'python', text 'third.py', text 'ipsum thd')
          RETURNING file_id
          )

          INSERT INTO version_file (version_id, file_id)
          -- yes, doing a cartesian joint here
          SELECT version_id, file_id from new_version, new_files;

          COMMIT;






          share|improve this answer














          share|improve this answer



          share|improve this answer








          edited Nov 20 '18 at 23:33

























          answered Nov 20 '18 at 1:31









          Underwater_developerUnderwater_developer

          176110




          176110








          • 1





            I'm not precisely sure what I changed ... then why are you offering this as an answer to the community?

            – Tim Biegeleisen
            Nov 20 '18 at 1:32











          • Honestly, I was documenting my process in the question and as I was about to submit, I used the (slightly simplified) query I was about to put into the post, into DataGrip to confirm the error message and... it worked. At a broader level, I have looked multiple times across the internet, via Google, docs, Duck Duck Go, various Stack Exchanges and could not find an example of what I was trying to create. By posting it here, I hope I can save someone the hours I spent in frustration to get to this point

            – Underwater_developer
            Nov 20 '18 at 1:39








          • 2





            The move to CTEs is the solution. You need to consume the values returned by the returning clause somehow. With plain SQL you can only store them in a temporary table or pass them on by chaining CTEs

            – a_horse_with_no_name
            Nov 20 '18 at 6:46














          • 1





            I'm not precisely sure what I changed ... then why are you offering this as an answer to the community?

            – Tim Biegeleisen
            Nov 20 '18 at 1:32











          • Honestly, I was documenting my process in the question and as I was about to submit, I used the (slightly simplified) query I was about to put into the post, into DataGrip to confirm the error message and... it worked. At a broader level, I have looked multiple times across the internet, via Google, docs, Duck Duck Go, various Stack Exchanges and could not find an example of what I was trying to create. By posting it here, I hope I can save someone the hours I spent in frustration to get to this point

            – Underwater_developer
            Nov 20 '18 at 1:39








          • 2





            The move to CTEs is the solution. You need to consume the values returned by the returning clause somehow. With plain SQL you can only store them in a temporary table or pass them on by chaining CTEs

            – a_horse_with_no_name
            Nov 20 '18 at 6:46








          1




          1





          I'm not precisely sure what I changed ... then why are you offering this as an answer to the community?

          – Tim Biegeleisen
          Nov 20 '18 at 1:32





          I'm not precisely sure what I changed ... then why are you offering this as an answer to the community?

          – Tim Biegeleisen
          Nov 20 '18 at 1:32













          Honestly, I was documenting my process in the question and as I was about to submit, I used the (slightly simplified) query I was about to put into the post, into DataGrip to confirm the error message and... it worked. At a broader level, I have looked multiple times across the internet, via Google, docs, Duck Duck Go, various Stack Exchanges and could not find an example of what I was trying to create. By posting it here, I hope I can save someone the hours I spent in frustration to get to this point

          – Underwater_developer
          Nov 20 '18 at 1:39







          Honestly, I was documenting my process in the question and as I was about to submit, I used the (slightly simplified) query I was about to put into the post, into DataGrip to confirm the error message and... it worked. At a broader level, I have looked multiple times across the internet, via Google, docs, Duck Duck Go, various Stack Exchanges and could not find an example of what I was trying to create. By posting it here, I hope I can save someone the hours I spent in frustration to get to this point

          – Underwater_developer
          Nov 20 '18 at 1:39






          2




          2





          The move to CTEs is the solution. You need to consume the values returned by the returning clause somehow. With plain SQL you can only store them in a temporary table or pass them on by chaining CTEs

          – a_horse_with_no_name
          Nov 20 '18 at 6:46





          The move to CTEs is the solution. You need to consume the values returned by the returning clause somehow. With plain SQL you can only store them in a temporary table or pass them on by chaining CTEs

          – a_horse_with_no_name
          Nov 20 '18 at 6:46


















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53384992%2freferencing-return-values-in-sequential-insert-statements-within-a-transaction%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          MongoDB - Not Authorized To Execute Command

          in spring boot 2.1 many test slices are not allowed anymore due to multiple @BootstrapWith

          Npm cannot find a required file even through it is in the searched directory