question

Carly B avatar image
Carly B asked

How to import bulk .log files and add a column to the table

I want to import some log files from the past years to a single table. There are 100+ files per directory with the date of the log included in the name, for example: app_07122009.log I would like to import every single items in all the files both inside and outside the directories with the name as year into a single table.

On top of this I have to add a column which will show the log date of the item, i.e. truncate the date from the log file name and add a column to all the rows for every logs.

What can I do with this? Thanks a lot!

sql-server-2005data-import
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Kristen avatar image
Kristen answered

I would use BPC, but I can't offhand thing of a way to get the Filename into an additional column in the imported data.

Perhaps:

BCP the data into a staging table Run a SQL command - e.g. using OSQL - to transfer the data from Staging table to Real table, and pass that the name of the original file.

BATCH file should be able to do this with

FOR %%f IN (*.log) DO BCP ... etc

10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Jason Cumberland avatar image
Jason Cumberland answered

If you have the SQL 2005 or above client tools then you could do this in an Integration Services Package. Assuming that all of your text files have the same schema, fire up BIDS and do the following:

  • Add a column to the table to hold the file name
  • add a flat file and oledb data source
  • create a variable to hold the file name, scoped at the package level
  • create a data flow
  • add a fat file source and oledb destination
  • use the derived column to add the file name variable (or use expression to change it) to the data flow pipeline
  • map the columns in the data flow
  • create a for each loop in the package, set it to iterate through your directory and assign the file name to your variable
  • add the data flow to the for each container
  • use an expression on the connection string property of the flat file data source so the connection string is changed every time the file name variable changes

Run the package and it should loop through the file in the directory and enter them into the table with the file name in your new column.

10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.