Forum Discussion

Parex122334's avatar
Parex122334
New Contributor III
2 years ago
Solved

Unable to insert 50000 records

Hi all,

I want to insert whole 50000 records from CSV File  to Sql Server,but its inserting only 2000 records only in a table. How to resolve this issue or how to insert whole data from CSV to  Sql Server Database

 



  • Try executing the pipeline instead of just validating it. Then use the Pipeline Execution Statistics to see how many records were processed.

  • I would recommend going ahead with Parent-Child Pipeline and define a pool size so that there are no too many calls on the server.
    50k hitting at the same time is a lot and may break the downstream. I've had more than ~3m rows going from snapLogic and I would say for sure that you wouldn't get any errors and agreed with ptaylor - please execute the pipeline instead.

    You may want to change your settings from 2000 to 50, sharing some screenshots for your references...

     

    Thank you.

     

    Regards,

    Darsh

4 Replies

  • Try executing the pipeline instead of just validating it. Then use the Pipeline Execution Statistics to see how many records were processed.

  • darshthakkar's avatar
    darshthakkar
    Valued Contributor

    I would recommend going ahead with Parent-Child Pipeline and define a pool size so that there are no too many calls on the server.
    50k hitting at the same time is a lot and may break the downstream. I've had more than ~3m rows going from snapLogic and I would say for sure that you wouldn't get any errors and agreed with ptaylor - please execute the pipeline instead.

    You may want to change your settings from 2000 to 50, sharing some screenshots for your references...

     

    Thank you.

     

    Regards,

    Darsh