spring batch : Read Twice one after other from database












0















I need to know what is best approach to read the data from one database in chunk(100) and on the basis of that data we read the data from other database server .
example : taking id from one database server and on the basis of that id we take data from other database server.



I have searched on google but have'nt got solution to read twice and write once in batch.



One approach is read in chunk and inside process we take id and hit the database. But process will take single data at a time which is most time consuming.



Second approach is make two different step but in this we can't able share list of id to other step because we can share only small amount of data to other step.



Need to know what is best approach to read twice one after other.










share|improve this question



























    0















    I need to know what is best approach to read the data from one database in chunk(100) and on the basis of that data we read the data from other database server .
    example : taking id from one database server and on the basis of that id we take data from other database server.



    I have searched on google but have'nt got solution to read twice and write once in batch.



    One approach is read in chunk and inside process we take id and hit the database. But process will take single data at a time which is most time consuming.



    Second approach is make two different step but in this we can't able share list of id to other step because we can share only small amount of data to other step.



    Need to know what is best approach to read twice one after other.










    share|improve this question

























      0












      0








      0








      I need to know what is best approach to read the data from one database in chunk(100) and on the basis of that data we read the data from other database server .
      example : taking id from one database server and on the basis of that id we take data from other database server.



      I have searched on google but have'nt got solution to read twice and write once in batch.



      One approach is read in chunk and inside process we take id and hit the database. But process will take single data at a time which is most time consuming.



      Second approach is make two different step but in this we can't able share list of id to other step because we can share only small amount of data to other step.



      Need to know what is best approach to read twice one after other.










      share|improve this question














      I need to know what is best approach to read the data from one database in chunk(100) and on the basis of that data we read the data from other database server .
      example : taking id from one database server and on the basis of that id we take data from other database server.



      I have searched on google but have'nt got solution to read twice and write once in batch.



      One approach is read in chunk and inside process we take id and hit the database. But process will take single data at a time which is most time consuming.



      Second approach is make two different step but in this we can't able share list of id to other step because we can share only small amount of data to other step.



      Need to know what is best approach to read twice one after other.







      spring-boot spring-batch






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Nov 21 '18 at 7:09









      dj Bravodj Bravo

      77311




      77311
























          1 Answer
          1






          active

          oldest

          votes


















          0














          There is no best approach as it depends on the use case.




          One approach is read in chunk and inside process we take id and hit the database. But process will take single data at a time which is most time consuming.




          This approach is a common pattern called the "Driving Query Pattern" explained in detail in the Common Batch Patterns section of the reference documentation. The idea is that the reader reads only IDs, and the processor enriches the item by querying the second server with additional data for that item. Of course this will generate a query for each item, but this what you want anyway, unless you want your second query to send the list of all IDs in the chunk. In this case, you can do it in org.springframework.batch.core.ItemWriteListener#beforeWrite where you get the list of all items to be written.




          Second approach is make two different step but in this we can't able share list of id to other step because we can share only small amount of data to other step.




          Yes, sharing a lot of data via the execution context is not recommended as this execution context will be persisted between steps. So I think this is not a good option for you.



          Hope this helps.






          share|improve this answer























            Your Answer






            StackExchange.ifUsing("editor", function () {
            StackExchange.using("externalEditor", function () {
            StackExchange.using("snippets", function () {
            StackExchange.snippets.init();
            });
            });
            }, "code-snippets");

            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "1"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: true,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: 10,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














            draft saved

            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53406906%2fspring-batch-read-twice-one-after-other-from-database%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown

























            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            0














            There is no best approach as it depends on the use case.




            One approach is read in chunk and inside process we take id and hit the database. But process will take single data at a time which is most time consuming.




            This approach is a common pattern called the "Driving Query Pattern" explained in detail in the Common Batch Patterns section of the reference documentation. The idea is that the reader reads only IDs, and the processor enriches the item by querying the second server with additional data for that item. Of course this will generate a query for each item, but this what you want anyway, unless you want your second query to send the list of all IDs in the chunk. In this case, you can do it in org.springframework.batch.core.ItemWriteListener#beforeWrite where you get the list of all items to be written.




            Second approach is make two different step but in this we can't able share list of id to other step because we can share only small amount of data to other step.




            Yes, sharing a lot of data via the execution context is not recommended as this execution context will be persisted between steps. So I think this is not a good option for you.



            Hope this helps.






            share|improve this answer




























              0














              There is no best approach as it depends on the use case.




              One approach is read in chunk and inside process we take id and hit the database. But process will take single data at a time which is most time consuming.




              This approach is a common pattern called the "Driving Query Pattern" explained in detail in the Common Batch Patterns section of the reference documentation. The idea is that the reader reads only IDs, and the processor enriches the item by querying the second server with additional data for that item. Of course this will generate a query for each item, but this what you want anyway, unless you want your second query to send the list of all IDs in the chunk. In this case, you can do it in org.springframework.batch.core.ItemWriteListener#beforeWrite where you get the list of all items to be written.




              Second approach is make two different step but in this we can't able share list of id to other step because we can share only small amount of data to other step.




              Yes, sharing a lot of data via the execution context is not recommended as this execution context will be persisted between steps. So I think this is not a good option for you.



              Hope this helps.






              share|improve this answer


























                0












                0








                0







                There is no best approach as it depends on the use case.




                One approach is read in chunk and inside process we take id and hit the database. But process will take single data at a time which is most time consuming.




                This approach is a common pattern called the "Driving Query Pattern" explained in detail in the Common Batch Patterns section of the reference documentation. The idea is that the reader reads only IDs, and the processor enriches the item by querying the second server with additional data for that item. Of course this will generate a query for each item, but this what you want anyway, unless you want your second query to send the list of all IDs in the chunk. In this case, you can do it in org.springframework.batch.core.ItemWriteListener#beforeWrite where you get the list of all items to be written.




                Second approach is make two different step but in this we can't able share list of id to other step because we can share only small amount of data to other step.




                Yes, sharing a lot of data via the execution context is not recommended as this execution context will be persisted between steps. So I think this is not a good option for you.



                Hope this helps.






                share|improve this answer













                There is no best approach as it depends on the use case.




                One approach is read in chunk and inside process we take id and hit the database. But process will take single data at a time which is most time consuming.




                This approach is a common pattern called the "Driving Query Pattern" explained in detail in the Common Batch Patterns section of the reference documentation. The idea is that the reader reads only IDs, and the processor enriches the item by querying the second server with additional data for that item. Of course this will generate a query for each item, but this what you want anyway, unless you want your second query to send the list of all IDs in the chunk. In this case, you can do it in org.springframework.batch.core.ItemWriteListener#beforeWrite where you get the list of all items to be written.




                Second approach is make two different step but in this we can't able share list of id to other step because we can share only small amount of data to other step.




                Yes, sharing a lot of data via the execution context is not recommended as this execution context will be persisted between steps. So I think this is not a good option for you.



                Hope this helps.







                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered Nov 21 '18 at 8:34









                Mahmoud Ben HassineMahmoud Ben Hassine

                4,3101714




                4,3101714






























                    draft saved

                    draft discarded




















































                    Thanks for contributing an answer to Stack Overflow!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53406906%2fspring-batch-read-twice-one-after-other-from-database%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    Popular posts from this blog

                    android studio warns about leanback feature tag usage required on manifest while using Unity exported app?

                    SQL update select statement

                    'app-layout' is not a known element: how to share Component with different Modules