How to consolidate multiple similar SSIS packages that export data to txt file
I am in the middle of migrating jobs and packages from an old server and I need your help guys on how to consolidate packages. I have these very old DTS packages that extracts data to a text file using some version of a function for each state. The way it is implemented now is there are 50 packages for all states and one package that executes the other packages. For example for the state of Virginia there is one text file required and the following query is used to extract the data and this would be exported to c:\\va.txt `(SELECT * FROM dbo.FN_all_states() WHERE SUBSTRING(PRIMARY_CHAP_PRODUCT,1,3) = 'VA'` and for the state of Maryland I have which would be exported to c:\\md.txt `(SELECT * FROM dbo.FN_all_states() WHERE SUBSTRING(PRIMARY_CHAP_PRODUCT,1,3) = 'MD'` and the same for the other states too. What I have in mind now is to build a single package [something like this] for all states but it doesnt seem the best solution. I got a feeling that there should be a neat and easy way like using Foreach Loop Container instead of building a package with 50 dataflows but I cant wrap my head around it now and I need help. :
Hey DaniSQL, Here is a solution that should help you out, I did this in SQL 2005, 1. Create 3 variables all with scope of the package; a) Name: allStates, Data Type: Object, Value: System.Object b) Name: state, Data Type: String, Value: VA (arbitrary value, the value is changed by the package) c) Name: textFile, Data Type: String. For this variable, in the properties window change the EvaluateAsExpression value to True, click on the ellipses for the expression property and enter this as the expression "C:\\\" + @[User::state] +".txt" (NOTE there should be a double backslash in the expression, the editor is taking it out) click on evaluate expression and it should equal C:\AB.txt. 2. Add a data flow to the control flow, and add a OLEDB source and a recordset Destination to the dataflow. 3. configure you OLEDB source to your connection and use SQL command (I created a simple table for this test, but you can change to your function) but you get the idea, it is getting a distinct list of the states in your table or you can create another table with a list of the states if you like, the choice is up to you (The editor is doing funny stuff with the code, there are underscores in the query) SELECT DISTINCT(SUBSTRING(fas.primary_chap_product,1,2)) FROM dbo.FN_all_states 4. Connect the OLEDB source to the recordset destination, open the recordset and change the VariableName to User::allStates. Select the Input Columns tab and select states, click OK. This creates an object which we will loop through to get the states. 5. Go back to the control Flow and add a Foreach Loop Container click on Collection, and change the Enumerator to Foreach ADO Enumerator. In the Enumerator configuration window, select User::allStates as the Variable. Select Rows in all the tables (
ADO.NET dataset only) from the Enumeration Mode window. Select the Variable Mappings option on the left. select User::state for the variable and the index is 0 (gets the first column from the recordset) click OK. 6. Add a data flow container to the Foreach loop container. in the data flow add a OLEDB source and a Flat File Destination. Open the OLEDB source and select the connection, and choose SQL command. Enter your query, and change the 'VA' to a ? SELECT * FROM dbo.FN_all_states fas WHERE SUBSTRING(fas.primary_chap_product,1,2)=? Click on Parameters, and select User::state for Parameter0 7. Connect the OLEDB source to the Flat File Destination (FFD). Open the FFD and click New connection. Select your format option. Click on Browse and Enter an arbitrary name like c:\AB.txt. click on Columns to make sure the inputs are OK, then click on OK. Select the mappings and it should default to the mappings. click OK. 8. Now we have to change the flat file connection manager to the variable textFile. click on the Flat File conenction manager and click on the Ellipses for the expressions Select ConnectionString from the Property window, and click on the ellipses. Drag the User::textFile down from the variables into the Expression area, and Click on Evaluate Expression. It should say c:\AB.txt. Click ok twice. and go back to the control flow. 9. You are now ready to run the package, however it will be a good idea to right click on the For each loop container select edit Breakpoints, and select the Break at the beginning of every iteration, just so you can see each file created at a time, click on the green arrow to continue each iteration. That should be it! Let me know if you need any further info.