Showing posts with label believe. Show all posts
Showing posts with label believe. Show all posts

Monday, March 12, 2012

Lookup / Merge Join / Script - Howto look up values by comparing to a range of values?

Hello all,

I am trying to think my way through a solution which I believe others have probably come across... I am trying to implement a matching routine wherein I need to match an address against a high value and a low value (or, for that matter an input date vs. a start and end date) to return the desired row ... i.e. if I were to use a straight vb program I would just use the following lookup:

"SELECT DISTINCT fire_id, police_ID, fire_opt_in_out, police_opt_in_out FROM ipt_tbl " & _

" WHERE zip_code = @.zip_code AND addr_prim_lo <= @.street_number AND addr_prim_hi >= @.street_number " & _

" AND addr_prim_oe = @.addr_prim_oe AND street_pre = @.street_pre AND street_name = @.street_name " & _

" AND street_suff = @.street_suff AND street_post = @.street_post " & _

" AND (expiry_date = '' OR expiry_date = '00000000' OR expiry_date > @.expiry_date)" & _

" GROUP BY fire_ID, police_ID, fire_opt_in_out, police_opt_in_out"

My question, then, is how would you perform this type of query using a lookup / merge join or script? I have not found a way to implement a way to set the input columns? I can set the straight matches without a problem, i.e. lookup zip code = input zip code, but can't think of the correct way to set comparisons, i.e. lookup value 1 <= input value AND lookup value 2 >= input value

Any suggestions?

thanks for your time...

After a bit of research, I have found a few different solutions to this problem. The first uses the lookup by altering the sql statement and parameter set under the advanced tab (enable memory restriction / modify the sql statement) as discussed here http://www.julian-kuiters.id.au/article.php/ssis-lookup-with-range. The second (a bit less likeable) option was to use a script component and found here http://sqlblogcasts.com/blogs/simons/archive/2005/10/04/628.aspx

I am a bit disappointed that the range functionality was not prebuilt in to the components (other than through the memory restriction)... Does anyone else have any ideas or suggestions?

Thanks for your time.

|||Although the logic in your query is pretty clear, the context and output is not - at least not to me. What exactly do you want to have happen in your data flow as the output of this process?|||

Reading back I can see how you would be confused...

I am trying to look up information based on an address range (i.e. if you are on the 23rd block (2300 - 2400) there will be one set of police and fire municipality codes, if you are on the 24th block 2401 - 2500 there will be seperate municipality code outputs) We are aggregatting these facts based on addresses supplied from our customer base to a table provided by the state which maps out the various address blocks for which we will need to break down our earned premiums such that we can pay taxes to the firefighter and police pension plans.

I think that the main point was that there is not an intuitive way of looking up information based on a fact tables value being within the dimension tables value range. I have also had this problem looking up values based on date ranges...

|||

I don't know how much anyone is interested, but here is a topic on the feedback site which you can vote on to bring this to the ssis teams attention...

https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=263594

Lookup / Merge Join / Script - How to?

Hello all,

I am trying to think my way through a solution which I believe others have probably come across... I am trying to implement a matching routine wherein I need to match an address against a high value and a low value (or, for that matter an input date vs. a start and end date) to return the desired row ... i.e. if I were to use a straight vb program I would just use the following lookup:

"SELECT DISTINCT fire_id, police_ID, fire_opt_in_out, police_opt_in_out FROM ipt_tbl " & _

" WHERE zip_code = @.zip_code AND addr_prim_lo <= @.street_number AND addr_prim_hi >= @.street_number " & _

" AND addr_prim_oe = @.addr_prim_oe AND street_pre = @.street_pre AND street_name = @.street_name " & _

" AND street_suff = @.street_suff AND street_post = @.street_post " & _

" AND (expiry_date = '' OR expiry_date = '00000000' OR expiry_date > @.expiry_date)" & _

" GROUP BY fire_ID, police_ID, fire_opt_in_out, police_opt_in_out"

My question, then, is how would you perform this type of query using a lookup / merge join or script? I have not found a way to implement a way to set the input columns? I can set the straight matches without a problem, i.e. lookup zip code = input zip code, but can't think of the correct way to set comparisons, i.e. lookup value 1 <= input value AND lookup value 2 >= input value

Any suggestions?

thanks for your time...

After a bit of research, I have found a few different solutions to this problem. The first uses the lookup by altering the sql statement and parameter set under the advanced tab (enable memory restriction / modify the sql statement) as discussed here http://www.julian-kuiters.id.au/article.php/ssis-lookup-with-range. The second (a bit less likeable) option was to use a script component and found here http://sqlblogcasts.com/blogs/simons/archive/2005/10/04/628.aspx

I am a bit disappointed that the range functionality was not prebuilt in to the components (other than through the memory restriction)... Does anyone else have any ideas or suggestions?

Thanks for your time.

|||Although the logic in your query is pretty clear, the context and output is not - at least not to me. What exactly do you want to have happen in your data flow as the output of this process?|||

Reading back I can see how you would be confused...

I am trying to look up information based on an address range (i.e. if you are on the 23rd block (2300 - 2400) there will be one set of police and fire municipality codes, if you are on the 24th block 2401 - 2500 there will be seperate municipality code outputs) We are aggregatting these facts based on addresses supplied from our customer base to a table provided by the state which maps out the various address blocks for which we will need to break down our earned premiums such that we can pay taxes to the firefighter and police pension plans.

I think that the main point was that there is not an intuitive way of looking up information based on a fact tables value being within the dimension tables value range. I have also had this problem looking up values based on date ranges...

|||

I don't know how much anyone is interested, but here is a topic on the feedback site which you can vote on to bring this to the ssis teams attention...

https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=263594

Friday, March 9, 2012

Lookup / Merge Join / Script - How to?

Hello all,

I am trying to think my way through a solution which I believe others have probably come across... I am trying to implement a matching routine wherein I need to match an address against a high value and a low value (or, for that matter an input date vs. a start and end date) to return the desired row ... i.e. if I were to use a straight vb program I would just use the following lookup:

"SELECT DISTINCT fire_id, police_ID, fire_opt_in_out, police_opt_in_out FROM ipt_tbl " & _

" WHERE zip_code = @.zip_code AND addr_prim_lo <= @.street_number AND addr_prim_hi >= @.street_number " & _

" AND addr_prim_oe = @.addr_prim_oe AND street_pre = @.street_pre AND street_name = @.street_name " & _

" AND street_suff = @.street_suff AND street_post = @.street_post " & _

" AND (expiry_date = '' OR expiry_date = '00000000' OR expiry_date > @.expiry_date)" & _

" GROUP BY fire_ID, police_ID, fire_opt_in_out, police_opt_in_out"

My question, then, is how would you perform this type of query using a lookup / merge join or script? I have not found a way to implement a way to set the input columns? I can set the straight matches without a problem, i.e. lookup zip code = input zip code, but can't think of the correct way to set comparisons, i.e. lookup value 1 <= input value AND lookup value 2 >= input value

Any suggestions?

thanks for your time...

After a bit of research, I have found a few different solutions to this problem. The first uses the lookup by altering the sql statement and parameter set under the advanced tab (enable memory restriction / modify the sql statement) as discussed here http://www.julian-kuiters.id.au/article.php/ssis-lookup-with-range. The second (a bit less likeable) option was to use a script component and found here http://sqlblogcasts.com/blogs/simons/archive/2005/10/04/628.aspx

I am a bit disappointed that the range functionality was not prebuilt in to the components (other than through the memory restriction)... Does anyone else have any ideas or suggestions?

Thanks for your time.

|||Although the logic in your query is pretty clear, the context and output is not - at least not to me. What exactly do you want to have happen in your data flow as the output of this process?|||

Reading back I can see how you would be confused...

I am trying to look up information based on an address range (i.e. if you are on the 23rd block (2300 - 2400) there will be one set of police and fire municipality codes, if you are on the 24th block 2401 - 2500 there will be seperate municipality code outputs) We are aggregatting these facts based on addresses supplied from our customer base to a table provided by the state which maps out the various address blocks for which we will need to break down our earned premiums such that we can pay taxes to the firefighter and police pension plans.

I think that the main point was that there is not an intuitive way of looking up information based on a fact tables value being within the dimension tables value range. I have also had this problem looking up values based on date ranges...

|||

I don't know how much anyone is interested, but here is a topic on the feedback site which you can vote on to bring this to the ssis teams attention...

https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=263594

Wednesday, March 7, 2012

Looking for some wisdom

My company has a sql database with a 100 gig database. There are many
performance issues that I believe are linked to database design and
programming. I have a couple of questions that I hope can be
answered.
The database only has 26 gig of real data the rest are indexes.
Is this normal? I know the extra indexes cause performance problems
with inserts,updates and deletes.
The databse has huge stored procedures many pages long. Is it the
right thing to do putting all the work onto the sql server itself?
Shouldn't these long procedures be handled in the middle tier using vb
or c?
Triggers using inserted and deleted tables. These triggers are used
on tansactions for inserts udates and deletes on the database. From
what I have seen monitoring the server these triggers run twice as
long as the update delete or insert and since the trigger is fired
during a transaction I would guess that the transaction is not
commited until the trigger is done. Would I be correct in assuming
this?
Thats all I have for right now any help would be great. If you had
any documention to back this up would help alot. I seem to be in a
battle with the programming group on this whole performance issue. By
the way the server hardware is dual 2 gig xeons 4 gig memory 165 gig
hd space on raid 5.
Jim
jmaddox@.oaktreesys.comi have frequently seen databases where there was as much
space used for indexes as for data
i think the highest index to data size ratio i saw was
~2X, and i felt that one had unnecessary indexes. good
table design is also part of index minimization.
btw, each index adds between 15-40% overhead to the base
cost of modifying a row, depending on a few factors (SQL
Server Connections conference, Oct 2003, SDB417)
i like to maintain a script that executes each sp once
(plus consideration for multiple code paths)
then i can drop indexes one by one to look for table scans.
a big problem with very long sp's is recompiles, an insert
into a temp table or other factor could trigger a
recompile of the entire sp (fixed in Yukon), so if can't
fix the cause of the recompile, breaking a big proc into
smaller procs can be helpful
i prefer using sprocs and not triggers. triggers are good
if you are using sql statements, so you need only one
network roundtrip to handle the complete transaction.
i believe triggers to be less efficient in multi-row
operations, where the trigger may fire once per row,
>--Original Message--
>My company has a sql database with a 100 gig database.
There are many
>performance issues that I believe are linked to database
design and
>programming. I have a couple of questions that I hope
can be
>answered.
>The database only has 26 gig of real data the rest are
indexes.
>Is this normal? I know the extra indexes cause
performance problems
>with inserts,updates and deletes.
>The databse has huge stored procedures many pages long.
Is it the
>right thing to do putting all the work onto the sql
server itself?
>Shouldn't these long procedures be handled in the middle
tier using vb
>or c?
>Triggers using inserted and deleted tables. These
triggers are used
>on tansactions for inserts udates and deletes on the
database. From
>what I have seen monitoring the server these triggers run
twice as
>long as the update delete or insert and since the trigger
is fired
>during a transaction I would guess that the transaction
is not
>commited until the trigger is done. Would I be correct
in assuming
>this?
>Thats all I have for right now any help would be great.
If you had
>any documention to back this up would help alot. I seem
to be in a
>battle with the programming group on this whole
performance issue. By
>the way the server hardware is dual 2 gig xeons 4 gig
memory 165 gig
>hd space on raid 5.
>Jim
>jmaddox@.oaktreesys.com
>.
>|||I don't think triggers fire once per row in SQL Server since there is not
ROW level triggers like in Oracle.
They are set based only. But beware of triggers since they make it harder to
follow the flow of what is happening.
If you are running big inserts, deletes or updates (multiple rows per
command) and use the inserted, deleted table in joins in the trigger I'm not
sure the performance will be incredible. You should probably try to see if
sp's with all the logic of what the triggers are doing could be created and
called instead of relying on the trigger processing.
Triggers are part of your transaction. So if these commands are long lasting
and touch lots of data you can get into blocking problems. Which obviously
doesn't help performance. In a sp you could control the transactions
explicitly and commit (or rollback) at more than one point.
As for having sp's or a middle tier in vb or other :
I prefer having SQL code located on SQL Server. This way it's easy to
isolate and change SQL code that's not optimal. You can see that is what MS
thinks also in Yukon by having SQL Server host .NET so we can create more
complex procs.
The middle tier can perhaps generate the commands used to access the DB but
it should put it in a proc and use that next time around. This way the
middle tier can call one proc to return multiple datasets instead of
executing each command separately occurring a round trip each time. Also
your middle tier can perhaps cache some amount of data as to not always hit
the DB.
As for your index check this site out :
http://www.sql-server-performance.com/optimizing_indexes.asp
The above site holds a great deal of info you should probably browse it and
you will surely find a load of answers to your questions.
Chris.
"joe chang" <anonymous@.discussions.microsoft.com> wrote in message
news:0c7201c3be83$258e93c0$a001280a@.phx.gbl...
> i have frequently seen databases where there was as much
> space used for indexes as for data
> i think the highest index to data size ratio i saw was
> ~2X, and i felt that one had unnecessary indexes. good
> table design is also part of index minimization.
> btw, each index adds between 15-40% overhead to the base
> cost of modifying a row, depending on a few factors (SQL
> Server Connections conference, Oct 2003, SDB417)
> i like to maintain a script that executes each sp once
> (plus consideration for multiple code paths)
> then i can drop indexes one by one to look for table scans.
> a big problem with very long sp's is recompiles, an insert
> into a temp table or other factor could trigger a
> recompile of the entire sp (fixed in Yukon), so if can't
> fix the cause of the recompile, breaking a big proc into
> smaller procs can be helpful
> i prefer using sprocs and not triggers. triggers are good
> if you are using sql statements, so you need only one
> network roundtrip to handle the complete transaction.
> i believe triggers to be less efficient in multi-row
> operations, where the trigger may fire once per row,
> >--Original Message--
> >My company has a sql database with a 100 gig database.
> There are many
> >performance issues that I believe are linked to database
> design and
> >programming. I have a couple of questions that I hope
> can be
> >answered.
> >
> >The database only has 26 gig of real data the rest are
> indexes.
> >Is this normal? I know the extra indexes cause
> performance problems
> >with inserts,updates and deletes.
> >
> >The databse has huge stored procedures many pages long.
> Is it the
> >right thing to do putting all the work onto the sql
> server itself?
> >Shouldn't these long procedures be handled in the middle
> tier using vb
> >or c?
> >
> >Triggers using inserted and deleted tables. These
> triggers are used
> >on tansactions for inserts udates and deletes on the
> database. From
> >what I have seen monitoring the server these triggers run
> twice as
> >long as the update delete or insert and since the trigger
> is fired
> >during a transaction I would guess that the transaction
> is not
> >commited until the trigger is done. Would I be correct
> in assuming
> >this?
> >
> >Thats all I have for right now any help would be great.
> If you had
> >any documention to back this up would help alot. I seem
> to be in a
> >battle with the programming group on this whole
> performance issue. By
> >the way the server hardware is dual 2 gig xeons 4 gig
> memory 165 gig
> >hd space on raid 5.
> >
> >Jim
> >jmaddox@.oaktreesys.com
> >.
> >