Raster Catalog: import

Showing posts with label import. Show all posts

Friday, March 30, 2012

reading a txt file

I import data by omporting a txt file that is tab delimited.
On of the fileds in the txt file has a date 08/06/2007 (for example)
I need to read that into my field in my table that is set as datetime.
Is there somethign special that need to get done? When I try, i get teh
following error:
Conversion invalid for datatypes on column pair 1 (source column
'DoRecd'(dbstype_str), destination column 'doRecd'(DBTYPE_DBtimestamp)On Aug 6, 11:14 pm, "Johnfli" <j...@.ivhs.us> wrote:
> I import data by omporting a txt file that is tab delimited.
> On of the fileds in the txt file has a date 08/06/2007 (for example)
> I need to read that into my field in my table that is set as datetime.
> Is there somethign special that need to get done? When I try, i get teh
> following error:
> Conversion invalid for datatypes on column pair 1 (source column
> 'DoRecd'(dbstype_str), destination column 'doRecd'(DBTYPE_DBtimestamp)
You should change the source data type to datetime or use text for
both.|||it's a text file.
Before I export it to a text file, I have it formated as datetime in excel.
"SB" <othellomy@.yahoo.com> wrote in message
news:1186485944.181307.110770@.w3g2000hsg.googlegroups.com...
> On Aug 6, 11:14 pm, "Johnfli" <j...@.ivhs.us> wrote:
>> I import data by omporting a txt file that is tab delimited.
>> On of the fileds in the txt file has a date 08/06/2007 (for example)
>> I need to read that into my field in my table that is set as datetime.
>> Is there somethign special that need to get done? When I try, i get teh
>> following error:
>> Conversion invalid for datatypes on column pair 1 (source column
>> 'DoRecd'(dbstype_str), destination column 'doRecd'(DBTYPE_DBtimestamp)
> You should change the source data type to datetime or use text for
> both.
>|||On Aug 8, 3:51 am, "Johnfli" <j...@.ivhs.us> wrote:
> it's a text file.
> Before I export it to a text file, I have it formated as datetime in excel.
> "SB" <othell...@.yahoo.com> wrote in message
> news:1186485944.181307.110770@.w3g2000hsg.googlegroups.com...
>
> > On Aug 6, 11:14 pm, "Johnfli" <j...@.ivhs.us> wrote:
> >> I import data by omporting a txt file that is tab delimited.
> >> On of the fileds in the txt file has a date 08/06/2007 (for example)
> >> I need to read that into my field in my table that is set as datetime.
> >> Is there somethign special that need to get done? When I try, i get teh
> >> following error:
> >> Conversion invalid for datatypes on column pair 1 (source column
> >> 'DoRecd'(dbstype_str), destination column 'doRecd'(DBTYPE_DBtimestamp)
> > You should change the source data type to datetime or use text for
> > both.- Hide quoted text -
> - Show quoted text -
Hi,
I think it be easier if you import the text file AS IS that is all
columns as text (including the column with datetime) you can call it
Excel_to_text_TEMP. Once you have this table imported in the server
you can populate your main table from this temp table and use a
conversion function for datetime etc. HTH.|||I do intailly load it into a tmp table, as after its in teh temps table, I
have it go and only copy new original records into the main table, I also
have it update exsisting records. so with that said...
How do I convert it to datetime as it goes from the tmp table to teh main
table?
I am not an sql master or anythign, heck, i'm not ever a sql novice lol
so please have your example as clear as possible..
THANK YOU
"SB" <othellomy@.yahoo.com> wrote in message
news:1186545477.061884.9860@.19g2000hsx.googlegroups.com...
> On Aug 8, 3:51 am, "Johnfli" <j...@.ivhs.us> wrote:
>> it's a text file.
>> Before I export it to a text file, I have it formated as datetime in
>> excel.
>> "SB" <othell...@.yahoo.com> wrote in message
>> news:1186485944.181307.110770@.w3g2000hsg.googlegroups.com...
>>
>> > On Aug 6, 11:14 pm, "Johnfli" <j...@.ivhs.us> wrote:
>> >> I import data by omporting a txt file that is tab delimited.
>> >> On of the fileds in the txt file has a date 08/06/2007 (for
>> >> example)
>> >> I need to read that into my field in my table that is set as datetime.
>> >> Is there somethign special that need to get done? When I try, i get
>> >> teh
>> >> following error:
>> >> Conversion invalid for datatypes on column pair 1 (source column
>> >> 'DoRecd'(dbstype_str), destination column 'doRecd'(DBTYPE_DBtimestamp)
>> > You should change the source data type to datetime or use text for
>> > both.- Hide quoted text -
>> - Show quoted text -
> Hi,
> I think it be easier if you import the text file AS IS that is all
> columns as text (including the column with datetime) you can call it
> Excel_to_text_TEMP. Once you have this table imported in the server
> you can populate your main table from this temp table and use a
> conversion function for datetime etc. HTH.
>|||On Aug 8, 10:06 pm, "Johnfli" <j...@.ivhs.us> wrote:
> I do intailly load it into a tmp table, as after its in teh temps table, I
> have it go and only copy new original records into the main table, I also
> have it update exsisting records. so with that said...
> How do I convert it to datetime as it goes from the tmp table to teh main
> table?
> I am not an sql master or anythign, heck, i'm not ever a sql novice lol
> so please have your example as clear as possible..
> THANK YOU
> "SB" <othell...@.yahoo.com> wrote in message
> news:1186545477.061884.9860@.19g2000hsx.googlegroups.com...
>
> > On Aug 8, 3:51 am, "Johnfli" <j...@.ivhs.us> wrote:
> >> it's a text file.
> >> Before I export it to a text file, I have it formated as datetime in
> >> excel.
> >> "SB" <othell...@.yahoo.com> wrote in message
> >>news:1186485944.181307.110770@.w3g2000hsg.googlegroups.com...
> >> > On Aug 6, 11:14 pm, "Johnfli" <j...@.ivhs.us> wrote:
> >> >> I import data by omporting a txt file that is tab delimited.
> >> >> On of the fileds in the txt file has a date 08/06/2007 (for
> >> >> example)
> >> >> I need to read that into my field in my table that is set as datetime.
> >> >> Is there somethign special that need to get done? When I try, i get
> >> >> teh
> >> >> following error:
> >> >> Conversion invalid for datatypes on column pair 1 (source column
> >> >> 'DoRecd'(dbstype_str), destination column 'doRecd'(DBTYPE_DBtimestamp)
> >> > You should change the source data type to datetime or use text for
> >> > both.- Hide quoted text -
> >> - Show quoted text -
> > Hi,
> > I think it be easier if you import the text file AS IS that is all
> > columns as text (including the column with datetime) you can call it
> > Excel_to_text_TEMP. Once you have this table imported in the server
> > you can populate your main table from this temp table and use a
> > conversion function for datetime etc. HTH.- Hide quoted text -
> - Show quoted text -
Hi,
Then you should drop the TEMP table and re-create it where all columns
should be varchar(255) or varchar(1000) etc. Then once the temp table
is loaded you should run a query such as (preferably in a sored
procedure, see Books online for syntax):
insert Excel_To_Text (
column1,
column2,
etc...,
date_Column1,
etc...)
select column1,
column2,
etc...,
convert(datetime,Date_Column1),
etc...
from Excel_To_Text_TEMP
Let me know if you have questions. Regards,
SB|||SB <othellomy@.yahoo.com> wrote in
news:1186631817.654696.162750@.19g2000hsx.googlegroups.com:
> On Aug 8, 10:06 pm, "Johnfli" <j...@.ivhs.us> wrote:
>> I do intailly load it into a tmp table, as after its in teh temps
>> table, I have it go and only copy new original records into the main
>> table, I also have it update exsisting records. so with that
>> said...
>> How do I convert it to datetime as it goes from the tmp table to teh
>> main table?
>> I am not an sql master or anythign, heck, i'm not ever a sql novice
>> lol
>> so please have your example as clear as possible..
>> THANK YOU
>> "SB" <othell...@.yahoo.com> wrote in message
>> news:1186545477.061884.9860@.19g2000hsx.googlegroups.com...
>>
>> > On Aug 8, 3:51 am, "Johnfli" <j...@.ivhs.us> wrote:
>> >> it's a text file.
>> >> Before I export it to a text file, I have it formated as datetime
>> >> in excel.
>> >> "SB" <othell...@.yahoo.com> wrote in message
>> >>news:1186485944.181307.110770@.w3g2000hsg.googlegroups.com...
>> >> > On Aug 6, 11:14 pm, "Johnfli" <j...@.ivhs.us> wrote:
>> >> >> I import data by omporting a txt file that is tab delimited.
>> >> >> On of the fileds in the txt file has a date 08/06/2007 (for
>> >> >> example)
>> >> >> I need to read that into my field in my table that is set as
>> >> >> datetime.
>> >> >> Is there somethign special that need to get done? When I try,
>> >> >> i get teh
>> >> >> following error:
>> >> >> Conversion invalid for datatypes on column pair 1 (source
>> >> >> column 'DoRecd'(dbstype_str), destination column
>> >> >> 'doRecd'(DBTYPE_DBtimestamp)
>> >> > You should change the source data type to datetime or use text
>> >> > for both.- Hide quoted text -
>> >> - Show quoted text -
>> > Hi,
>> > I think it be easier if you import the text file AS IS that is all
>> > columns as text (including the column with datetime) you can call
>> > it Excel_to_text_TEMP. Once you have this table imported in the
>> > server you can populate your main table from this temp table and
>> > use a conversion function for datetime etc. HTH.- Hide quoted text
>> > -
>> - Show quoted text -
> Hi,
> Then you should drop the TEMP table and re-create it where all columns
> should be varchar(255) or varchar(1000) etc. Then once the temp table
> is loaded you should run a query such as (preferably in a sored
> procedure, see Books online for syntax):
> insert Excel_To_Text (
> column1,
> column2,
> etc...,
> date_Column1,
> etc...)
> select column1,
> column2,
> etc...,
> convert(datetime,Date_Column1),
> etc...
> from Excel_To_Text_TEMP
> Let me know if you have questions. Regards,
> SB
You might eliminate the text file and instead:
SELECT F1 AS [Col1], ... CAST(Fn AS datetime) AS [DateCol], ...
INTO [DestinationTable]
FROM OPENROWSET('Microsoft.Jet.OLEDB.4.0',
'Excel 8.0;HDR=YES;Database=full_path_to\filename.xls',
'SELECT * FROM [SheetName$]')
Use the Microsoft.ACE.OLEDB.12.0 provider if you don't have Jet or if the
Excel version is 2007 (and, for Excel 2007, change Excel 8.0 to Excel
12.0).
You will probably find this a little tricky to get working (I surely
did!) so build up from a simple import, e.g. of a single integer column.sql

Monday, March 26, 2012

Read SQL Profiler Trace output

Hi Folks,

I want to Read the SQL Profiler Trace Output using VB or VB.NET

I read some forums. They advice to import SMO in VB.NET..

But how can i Import this....

This is very Urgent for me....

Please Reply me immediately...You don't need SMO to read trace output. select from ::fn_get_trace into a table, then use normal ADO.Net to access the trace data from the table.|||Thanks for Reply...

Please can u give me a sample code for this....

Please this is very urgent for me...|||

First thing you need to do is set up your trace to go into a file. (Sending the trace data directly into SQL Server is very bad for performance.) Once you've got the trace file you can do something like this (which is right out of Books Online - see that document for more details.)

USE AdventureWorks;

SELECT * INTO temp_trc

FROM fn_trace_gettable('c:\temp\my_trace.trc', default);

GO|||Thanks for help me...

Read perfmon log into SQL table

Is there a way i can import the .blg perfmon counter log into a table for
reporting purposesUsing Performance (System) Monitor in Windows 2000 and 2003, you can save
the log to the text file. Then you can use Bulk Insert to copy the data to a
table. I don't remember for Win 2k, but in Win 2003 you can save it even
directly to a SQL table.
--
Dejan Sarka, SQL Server MVP
FAQ from Neil & others at: http://www.sqlserverfaq.com
Please reply only to the newsgroups.
PASS - the definitive, global community
for SQL Server professionals - http://www.sqlpass.org
"Hassan" <fatima_ja@.hotmail.com> wrote in message
news:#XqypYBfDHA.1888@.TK2MSFTNGP12.phx.gbl...
> Is there a way i can import the .blg perfmon counter log into a table for
> reporting purposes
>|||Since you have the perfmon file in binary format you can pull out the actual
counters you want as csv using win2k resource kit program relog.exe. Then
create a table with the desired structure and bcp it in, or do a DTS import.
If you are running on xp or 2003 you can have perfmon log straight to a sql
server database.
"Hassan" <fatima_ja@.hotmail.com> wrote in message
news:#XqypYBfDHA.1888@.TK2MSFTNGP12.phx.gbl...
> Is there a way i can import the .blg perfmon counter log into a table for
> reporting purposes
>

Friday, March 23, 2012

READ EXCEL DATA CELL BY CELL FROM SP

HI,

I HAVE AN EXCEL SHEET WITH SOME DATA, I WANT TO IMPORT THAT DATA (CELL
BY CELL WITH MANIPULATION) INTO THE SQL SERVER TABLES BY USING STORED
PROCEDURE(IF POSSIBLE).

IF ANYBODY HAVE DONE SIMILER TYPE OF JOB OR KNOWING ABOUT IT, PLS. LET
ME KNOW.

THANKS IN ADV.

T.S.NEGIWrap sql server with an object tag and create an instance for every
spreadsheet you have. Seriously, save your spreadsheet as a CSV and
use Bulk Insert. It's straight forward and can be found on the sql
Books On Line Help. -- Louis|||louisducnguyen@.hotmail.com (louis nguyen) wrote in message news:<b0e9d53.0311140737.280f0fda@.posting.google.com>...
> Wrap sql server with an object tag and create an instance for every
> spreadsheet you have. Seriously, save your spreadsheet as a CSV and
> use Bulk Insert. It's straight forward and can be found on the sql
> Books On Line Help. -- Louis

Louis thanks for your reply.

But this does not seems to solve my problem.

Bulk insert will not be right in this situation, coz. I have to
manipulate data also.

for example. if some record (id) already exist in data, in that case
only updation may require.

If I could read cell by cell data from excel sheet, then only this
level of manipulation is possible.

T.S.Negi|||tilak.negi@.mind-infotech.com (T.S.Negi) wrote in message news:<a1930058.0311240332.35ae29ef@.posting.google.com>...
> louisducnguyen@.hotmail.com (louis nguyen) wrote in message news:<b0e9d53.0311140737.280f0fda@.posting.google.com>...
> > Wrap sql server with an object tag and create an instance for every
> > spreadsheet you have. Seriously, save your spreadsheet as a CSV and
> > use Bulk Insert. It's straight forward and can be found on the sql
> > Books On Line Help. -- Louis
>
> Louis thanks for your reply.
> But this does not seems to solve my problem.
> Bulk insert will not be right in this situation, coz. I have to
> manipulate data also.
> for example. if some record (id) already exist in data, in that case
> only updation may require.
>
> If I could read cell by cell data from excel sheet, then only this
> level of manipulation is possible.
>
>
> T.S.Negi

Probably the easiest approach is to BULK INSERT or DTS the spreadsheet
into a staging table, then do an INSERT into the target table. That
way you can check and clean up the data while it's in the staging
table, and before it goes to the 'real' table.

Simon|||> If I could read cell by cell data from excel sheet, then only this
> level of manipulation is possible.

If you're more comfortable doing arrays/loops, I don't think T-SQL has
it. I would suggest bulk inserting into a temp table. Update
(manipulate it). Identify new records and insert. Identify existing
records and update.

Wednesday, March 21, 2012

Read CSV file - Save Columns into rows

I want to import CSV file and convert columns into rows depending on Customer count(2nd record in each row of CSV file) and save to SQL table

--CSV file format
State, Customer_Count, Name_1, Total_1,Name_2, Total_2,Name_3, Total_3..can go upto 350

GA,2,'John Doe',14.00,'Roger Smith',15.00
FL,3,'John Doe',14.00,'Roger Smith',15.00,'Sally Cox',16.00
SC,5,'John Doe',14.00,'Roger Smith',15.00,'Sally Cox',16.00,'James Brown',17.00,'Rick Davis',18.00

Data in SQL table from csv file should look like this

State,Name,Total
GA,John Doe,14.00
GA,Roger Smith,15.00
FL,John Doe,14.00,
FL,Roger Smith,15.00
FL,Sally Cox,16.00

I have multiple CSV files with millions of records. How can i achieve this using Integration Services or Bulk Data Import.

Amar:

I think for what you are describing I would use SSIS; if you are running SQL Server 2000, that would be DTS instead of SSIS. Other alternatives include the use of OPENROWSET, BULK INSERT, or BCP. Read about these alternatives in books online and choose the alternative that you think best fits.

Dave

|||

Hi Amar,

refer http://blogs.msdn.com/euanga/archive/2006/07/20/672272.aspx also.

Hemantgiri S. Goswami

sql

Read CSV file - Save Columns into Rows

I want to import CSV file and convert columns into rows depending on
Customer count(2nd record in each row of CSV file) and save to SQL
table
--CSV file format
State, Customer_Count, Name_1, Total_1,Name_2, Total_2,Name_3,
Total_3..can go upto 350
GA,2,'John Doe',14.00,'Roger Smith',15.00
FL,3,'John Doe',14.00,'Roger Smith',15.00,'Sally Cox',16.00
SC,5,'John Doe',14.00,'Roger Smith',15.00,'Sally Cox',16.00,'James
Brown',17.00,'Rick Davis',18.00
Data in SQL table from csv file should look like this
State,Name,Total
GA,John Doe,14.00
GA,Roger Smith,15.00
FL,John Doe,14.00,
FL,Roger Smith,15.00
FL,Sally Cox,16.00
I have multiple CSV files with millions of records.What is the best and
fastest way to achieve this?
Throw me anything
Hi
"pintoo" wrote:

> I want to import CSV file and convert columns into rows depending on
> Customer count(2nd record in each row of CSV file) and save to SQL
> table
> --CSV file format
> State, Customer_Count, Name_1, Total_1,Name_2, Total_2,Name_3,
> Total_3..can go upto 350
> GA,2,'John Doe',14.00,'Roger Smith',15.00
> FL,3,'John Doe',14.00,'Roger Smith',15.00,'Sally Cox',16.00
> SC,5,'John Doe',14.00,'Roger Smith',15.00,'Sally Cox',16.00,'James
> Brown',17.00,'Rick Davis',18.00
> Data in SQL table from csv file should look like this
> State,Name,Total
> GA,John Doe,14.00
> GA,Roger Smith,15.00
> FL,John Doe,14.00,
> FL,Roger Smith,15.00
> FL,Sally Cox,16.00
>
If you are moving these into rows the data is not normalized! You will only
be able to have a finite number of rows that can be converted, therefore you
could self join if each row is allocated a sequence number or the methid
described by Erland in
http://groups-beta.google.com/group/comp.databases.ms-sqlserver/msg/250f0c68596ce22e?&hl=en

> I have multiple CSV files with millions of records.What is the best and
> fastest way to achieve this?
> Throw me anything
>
If you are using SQL 2005 you could use the PIVOT command.
John

Read CSV file - Save Columns into Rows

I want to import CSV file and convert columns into rows depending on Customer count(2nd record in each row of CSV file) and save to SQL table

--CSV file format
State, Customer_Count, Name_1, Total_1,Name_2, Total_2,Name_3, Total_3..can go upto 350

Data in SQL table from csv file should look like this

State,Name,Total
GA,John Doe,14.00
GA,Roger Smith,15.00
FL,John Doe,14.00,
FL,Roger Smith,15.00
FL,Sally Cox,16.00

I have multiple CSV files with millions of records. How can i achieve this using Integration Services or Bulk Data Import.

Perhaps you should look at creating a VB/C# application to handle this situation.|||

I am trying do something very similar to what you are asking...I found import wizard by right clicking on the database itself. Under the "Tasks" option. I just got this administrators handbook which is somewhat helpful. First, you will have to import so it seems the column names first there is a check box on the screen you need to select to get the column names to appear this helps out with the data import too. Then go back from the top and do the same thing to get the data. You need to arrive at the copy or query screen to handle this. And choose the option to write a query. For, me however when I go back for the second pass through. It doesn't seem to let me get to the screen again, I saw it on the initial pass. Let me know if this worked for you?

|||

Once you have imported the rows to a temp. table then you might need to run a self-join to extract the required columns like:

insert into TableB(column1,column2,column3)
select A.column1 , B.column1, C.column1
from
(select column1 from TableA where column1 like 'A%') A
cross join (select column1 from TableA where column1 like 'B%') B
cross join (select column1 from TableA where column1 like 'C%') C
order by A.column1 , B.column1, C.column1

Read CSV file - Save Columns into Rows

I want to import CSV file and convert columns into rows depending on Customer count(2nd record in each row of CSV file) and save to SQL table

--CSV file format
State, Customer_Count, Name_1, Total_1,Name_2, Total_2,Name_3, Total_3..can go upto 350

Data in SQL table from csv file should look like this

State,Name,Total
GA,John Doe,14.00
GA,Roger Smith,15.00
FL,John Doe,14.00,
FL,Roger Smith,15.00
FL,Sally Cox,16.00

I have multiple CSV files with millions of records. How can i achieve this using Integration Services or Bulk Data Import.

Perhaps you should look at creating a VB/C# application to handle this situation.|||

|||

Once you have imported the rows to a temp. table then you might need to run a self-join to extract the required columns like:

Read CSV file - Save Columns into Rows

I want to import CSV file and convert columns into rows depending on Customer count(2nd record in each row of CSV file) and save to SQL table

--CSV file format
State, Customer_Count, Name_1, Total_1,Name_2, Total_2,Name_3, Total_3..can go upto 600

Data in SQL table from csv file should look like this

State,Name,Total
GA,John Doe,14.00
GA,Roger Smith,15.00
FL,John Doe,14.00,
FL,Roger Smith,15.00
FL,Sally Cox,16.00

I have multiple CSV files with millions of records. How can i achieve this using Integration Services or Bulk Data Import.

You can use something like this..

Create Table #DataIntoRows
(
State Varchar(10),
Name Varchar(100),
Total Int
)
SELECT * INTO #Data FROM OPENROWSET('MSDASQL','Driver={Microsoft Text Driver (*.txt; *.csv)};DefaultDir={PATH}','SELECT * FROM Data.csv')

Declare @.I as Int;
Declare @.Count as Int;
Select
@.I = 1,
@.Count=Count(Colid)
From
tempdb..Syscolumns
Where
id = Object_Id('tempdb..#Data') and Colid > 2;

While @.I < @.Count
begin
Declare @.SQL as Varchar(300);
Select @.SQL = 'Insert Into #DataIntoRows Select State,'
Select @.SQL = @.SQL + name From tempdb..Syscolumns Where id = Object_Id('tempdb..#Data') and colId= @.I + 2
Select @.SQL = @.SQL + ',' + name + ' From #Data Where ' + name + ' Is NOT NULL' From tempdb..Syscolumns Where id = Object_Id('tempdb..#Data') and colId= @.I + 3
Exec(@.SQL)
Select @.I = @.I + 2;
End

Read CSV file - Save Columns into Rows

I want to import CSV file and convert columns into rows depending on Customer count(2nd record in each row of CSV file) and save to SQL table

--CSV file format
State, Customer_Count, Name_1, Total_1,Name_2, Total_2,Name_3, Total_3..can go upto 600

Data in SQL table from csv file should look like this

State,Name,Total
GA,John Doe,14.00
GA,Roger Smith,15.00
FL,John Doe,14.00,
FL,Roger Smith,15.00
FL,Sally Cox,16.00

I have multiple CSV files with millions of records. How can i achieve this using Integration Services or Bulk Data Import.

See if my post in this thread helps get you started. I know it's not quite the same, but it may be a good foundation for you.

http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=880588&SiteID=1

Phil|||

Thanks Phil for your reply

I am new to SSID can you please provide some more detail

|||You want to use the unpivot transform to accomplish this. It does a fantastic job about flipping over columns to rows and purging nulls for you. Let me know if you need more info but there are some good blog posts out there on it.|||

Brian Knight wrote:
You want to use the unpivot transform to accomplish this. It does a fantastic job about flipping over columns to rows and purging nulls for you. Let me know if you need more info but there are some good blog posts out there on it.

I don't believe the unpivot will work because there can be a variable number of columns to unpivot. If it was a set number of columns, then this would work, however from row to row the number of columns changes.|||

Phil Brammer wrote:

Brian Knight wrote:
You want to use the unpivot transform to accomplish this. It does a fantastic job about flipping over columns to rows and purging nulls for you. Let me know if you need more info but there are some good blog posts out there on it.
I don't believe the unpivot will work because there can be a variable number of columns to unpivot. If it was a set number of columns, then this would work, however from row to row the number of columns changes.

I think it should be OK. If the original poster populates those columns wich don't have a value with NULL,then they won't get unpivoted.

If unpivot doesn't do the job let us know, there are alternatives using SORT and UNION ALL.

-Jamie

|||

Jamie Thomson wrote:

I think it should be OK. If the original poster populates those columns wich don't have a value with NULL,then they won't get unpivoted.
If unpivot doesn't do the job let us know, there are alternatives using SORT and UNION ALL.
-Jamie

My guess is that the OP can't control the file. That seems to usually be the case. The OP also states that the number of columns can go up to 600. Wow.

So for his data:
GA,2,'John Doe',14.00,'Roger Smith',15.00
FL,3,'John Doe',14.00,'Roger Smith',15.00,'Sally Cox',16.00
SC,5,'John Doe',14.00,'Roger Smith',15.00,'Sally Cox',16.00,'James Brown',17.00,'Rick Davis',18.00

Row 1 has 6 columns. Row 2 has 8 columns. Row 3 has 12 columns.

Will the SSIS flat file connector work to correctly set the number of columns if the file is 10,000 rows long with the longest record (say 50 columns) is at the end when all of the other records were under 40 columns?|||

Phil Brammer wrote:

Jamie Thomson wrote:

I think it should be OK. If the original poster populates those columns wich don't have a value with NULL,then they won't get unpivoted.
If unpivot doesn't do the job let us know, there are alternatives using SORT and UNION ALL.
-Jamie
My guess is that the OP can't control the file. That seems to usually be the case. The OP also states that the number of columns can go up to 600. Wow.
So for his data:
GA,2,'John Doe',14.00,'Roger Smith',15.00
FL,3,'John Doe',14.00,'Roger Smith',15.00,'Sally Cox',16.00
SC,5,'John Doe',14.00,'Roger Smith',15.00,'Sally Cox',16.00,'James Brown',17.00,'Rick Davis',18.00
Row 1 has 6 columns. Row 2 has 8 columns. Row 3 has 12 columns.
Will the SSIS flat file connector work to correctly set the number of columns if the file is 10,000 rows long with the longest record (say 50 columns) is at the end when all of the other records were under 40 columns?

No it won't But then the complexity here is in the source file - there's no getting around that. The original poster will just have to accept that he has a ridiculously complex source file and bite the bullet. There are still lots of ways to achieve this.

-Jamie

|||

Can you guys please point me to any examples or blogs.

|||

Amar Khaira wrote:
Can you please point me any examples or blogs

One way is to search the forums for examples. On the main page of this forum is a search box on the left-hand side of the screen. Type in unpivot and you'll get a few examples/topics that discuss that transformation.|||

Here's a great blog post: http://sqljunkies.com/WebLog/ashvinis/archive/2005/03.aspx

Send me an email and I'll send you an example code and I'll eventually do a video on JumpstartTV.com. The variable amount of columns is not a problem though. bknight<at>jumpstarttv.com is my email addr if needed.

|||

Brian Knight wrote:
Here's a great blog post: http://sqljunkies.com/WebLog/ashvinis/archive/2005/03.aspx
Send me an email and I'll send you an example code and I'll eventually do a video on JumpstartTV.com. The variable amount of columns is not a problem though. bknight<at>jumpstarttv.com is my email addr if needed.

How can the variable columns not be a problem? Because the flat file source won't account for the variability, you'll have to read in each record as one column, and then split it with a script transformation. In doing that, you might be able to set a fixed number of columns and fill them with NULLs if no data exists. I'm trying to understand how you think you can expose all of the columns to the unpivot transformation when the metadata can't be pre-populated with certainty. (That is, based on my previous example, there's a certain row limit that the flat file connector uses to assess how many columns there are in the file.)|||

I am getting following error while trying Unpivot Transform:

PivotKeyValue is not valid. In an UnPivot transform with more than one unpivoted DestinationColumn, the set of PivotKeyValues per destination must match exactly.

Read CSV file - Save Columns into Rows

I want to import CSV file and convert columns into rows depending on Customer count(2nd record in each row of CSV file) and save to SQL table

--CSV file format
State, Customer_Count, Name_1, Total_1,Name_2, Total_2,Name_3, Total_3..can go upto 600

Data in SQL table from csv file should look like this

State,Name,Total
GA,John Doe,14.00
GA,Roger Smith,15.00
FL,John Doe,14.00,
FL,Roger Smith,15.00
FL,Sally Cox,16.00

I have multiple CSV files with millions of records. How can i achieve this using Integration Services or Bulk Data Import.

See if my post in this thread helps get you started. I know it's not quite the same, but it may be a good foundation for you.

http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=880588&SiteID=1

Phil|||

Thanks Phil for your reply

I am new to SSID can you please provide some more detail

Brian Knight wrote:
You want to use the unpivot transform to accomplish this. It does a fantastic job about flipping over columns to rows and purging nulls for you. Let me know if you need more info but there are some good blog posts out there on it.

Phil Brammer wrote:

Brian Knight wrote:
You want to use the unpivot transform to accomplish this. It does a fantastic job about flipping over columns to rows and purging nulls for you. Let me know if you need more info but there are some good blog posts out there on it.
I don't believe the unpivot will work because there can be a variable number of columns to unpivot. If it was a set number of columns, then this would work, however from row to row the number of columns changes.

I think it should be OK. If the original poster populates those columns wich don't have a value with NULL,then they won't get unpivoted.

If unpivot doesn't do the job let us know, there are alternatives using SORT and UNION ALL.

-Jamie

|||

Jamie Thomson wrote:

I think it should be OK. If the original poster populates those columns wich don't have a value with NULL,then they won't get unpivoted.
If unpivot doesn't do the job let us know, there are alternatives using SORT and UNION ALL.
-Jamie

My guess is that the OP can't control the file. That seems to usually be the case. The OP also states that the number of columns can go up to 600. Wow.

Row 1 has 6 columns. Row 2 has 8 columns. Row 3 has 12 columns.

Phil Brammer wrote:

Jamie Thomson wrote:

I think it should be OK. If the original poster populates those columns wich don't have a value with NULL,then they won't get unpivoted.
If unpivot doesn't do the job let us know, there are alternatives using SORT and UNION ALL.
-Jamie
My guess is that the OP can't control the file. That seems to usually be the case. The OP also states that the number of columns can go up to 600. Wow.
So for his data:
GA,2,'John Doe',14.00,'Roger Smith',15.00
FL,3,'John Doe',14.00,'Roger Smith',15.00,'Sally Cox',16.00
SC,5,'John Doe',14.00,'Roger Smith',15.00,'Sally Cox',16.00,'James Brown',17.00,'Rick Davis',18.00
Row 1 has 6 columns. Row 2 has 8 columns. Row 3 has 12 columns.
Will the SSIS flat file connector work to correctly set the number of columns if the file is 10,000 rows long with the longest record (say 50 columns) is at the end when all of the other records were under 40 columns?

-Jamie

|||

Can you guys please point me to any examples or blogs.

|||

Amar Khaira wrote:
Can you please point me any examples or blogs

Here's a great blog post: http://sqljunkies.com/WebLog/ashvinis/archive/2005/03.aspx

|||

Brian Knight wrote:
Here's a great blog post: http://sqljunkies.com/WebLog/ashvinis/archive/2005/03.aspx
Send me an email and I'll send you an example code and I'll eventually do a video on JumpstartTV.com. The variable amount of columns is not a problem though. bknight<at>jumpstarttv.com is my email addr if needed.

I am getting following error while trying Unpivot Transform:

PivotKeyValue is not valid. In an UnPivot transform with more than one unpivoted DestinationColumn, the set of PivotKeyValues per destination must match exactly.

Read CSV file - Save Columns into rows

I want to import CSV file and convert columns into rows depending on Customer count(2nd record in each row of CSV file) and save to SQL table

--CSV file format
State, Customer_Count, Name_1, Total_1,Name_2, Total_2,Name_3, Total_3..can go upto 350

Data in SQL table from csv file should look like this

State,Name,Total
GA,John Doe,14.00
GA,Roger Smith,15.00
FL,John Doe,14.00,
FL,Roger Smith,15.00
FL,Sally Cox,16.00

I have multiple CSV files with millions of records. How can i achieve this using Integration Services or Bulk Data Import.

Amar:

Dave

|||

Hi Amar,

refer http://blogs.msdn.com/euanga/archive/2006/07/20/672272.aspx also.

Hemantgiri S. Goswami

Read CSV file - Save Columns into Rows

I want to import CSV file and convert columns into rows depending on
Customer count(2nd record in each row of CSV file) and save to SQL
table
--CSV file format
State, Customer_Count, Name_1, Total_1,Name_2, Total_2,Name_3,
Total_3..can go upto 350
GA,2,'John Doe',14.00,'Roger Smith',15.00
FL,3,'John Doe',14.00,'Roger Smith',15.00,'Sally Cox',16.00
SC,5,'John Doe',14.00,'Roger Smith',15.00,'Sally Cox',16.00,'James
Brown',17.00,'Rick Davis',18.00
Data in SQL table from csv file should look like this
State,Name,Total
GA,John Doe,14.00
GA,Roger Smith,15.00
FL,John Doe,14.00,
FL,Roger Smith,15.00
FL,Sally Cox,16.00
I have multiple CSV files with millions of records.What is the best and
fastest way to achieve this?
Throw me anythingHi
"pintoo" wrote:

> I want to import CSV file and convert columns into rows depending on
> Customer count(2nd record in each row of CSV file) and save to SQL
> table
> --CSV file format
> State, Customer_Count, Name_1, Total_1,Name_2, Total_2,Name_3,
> Total_3..can go upto 350
> GA,2,'John Doe',14.00,'Roger Smith',15.00
> FL,3,'John Doe',14.00,'Roger Smith',15.00,'Sally Cox',16.00
> SC,5,'John Doe',14.00,'Roger Smith',15.00,'Sally Cox',16.00,'James
> Brown',17.00,'Rick Davis',18.00
> Data in SQL table from csv file should look like this
> State,Name,Total
> GA,John Doe,14.00
> GA,Roger Smith,15.00
> FL,John Doe,14.00,
> FL,Roger Smith,15.00
> FL,Sally Cox,16.00
>
If you are moving these into rows the data is not normalized! You will only
be able to have a finite number of rows that can be converted, therefore you
could self join if each row is allocated a sequence number or the methid
described by Erland in
http://groups-beta.google.com/group...r />
2e?&hl=en

> I have multiple CSV files with millions of records.What is the best and
> fastest way to achieve this?
> Throw me anything
>
If you are using SQL 2005 you could use the PIVOT command.
John

Friday, March 9, 2012

RDF XML

I know SQL Server 2005 can import/export xml files. Can it be specific for RDF XML files?

There is nothing specific in SQL Server for RDF-XML processing.

Friday, March 30, 2012

reading a txt file

reading a txt file

reading a txt file

Monday, March 26, 2012

Read SQL Profiler Trace output

Read perfmon log into SQL table

Friday, March 23, 2012

READ EXCEL DATA CELL BY CELL FROM SP

Wednesday, March 21, 2012

Read CSV file - Save Columns into rows

Read CSV file - Save Columns into Rows

Read CSV file - Save Columns into Rows

Read CSV file - Save Columns into Rows

Read CSV file - Save Columns into Rows

Read CSV file - Save Columns into Rows

Read CSV file - Save Columns into Rows

Read CSV file - Save Columns into rows

Read CSV file - Save Columns into Rows

Friday, March 9, 2012

RDF XML

Raster Catalog

Blog Archive

About Me