The ‘FIFO Stock Inventory’ SQL Problem

Phil Factor SQL Speed Phreak Competition: No 2

alt text

This competition is now over, but the winner, Dave, got an Amazon Voucher for $60, and the privilege of being able to display the 'Phil Factor SQL Speed Phreak' award on their own site

It was quite a struggle with some close competition from many of those who participated in this competition. However, Dave came up with a clever solution that produced an FIFO calculation from a million rows in a just a few seconds. The third Phil Factor Speed Phreak competition will soon be held on here. Watch out!

(here is the original preamble.)

This competition is to calculate current items in stock and current stock value in FIFO order.
I'll tell you the business rules for the algorithm and provide a sample cursor-based routine that generates the correct result, but slowly (about 40 minutes).

I have seen many different algorithms to do this and most of them involve a cursor. Can it be done more quickly without, or even with, a cursor? In other words, what is the fastest way in SQL Server (any version) to provide this stock inventory report?
It is a reasonable request. We have a stock transaction list (1,000,001 records, 17.6MB zipped) with 15,002 unique articles and we need to do a stock inventory report that gives the current breakdown of the ArticleID, the number of items in stock and the current stock value according to FIFO rules. The list should be in ArticleID order. The sample stock transaction list will include two running totals so you can check the results of your routine. They are NOT part of the competition, and you are not allowed to use them in your calculation.

Beware that the sample file includes the two extra columns; CurrentItems (INT) and CurrentValue (MONEY) !
If you want a smaller subset to test on, import all records and remove all articles but two or three. These are the samples for which all suggestions will be measured with.

The table is in this form (simplified from the way we'd do it in a real system of course).
In this made-up but realistic problem, you have an existing table and database design which you can’t do anything about. (we can all relate to that!) You have an existing cursor-based solution that is taking several minutes to run. (Yes, we all relate to that). The database is performing badly, and you need to take effective steps to remedy this. The only thing you can do is to come up with a better-performing routine. Redesigning the database isn’t an option very often, in the real world, because this requires team sign-in. This is a competition based on the real, sometimes imperfect, world: not an exposition of database design. The point is that we are faced with designs like this and we have to heal them. The competition is all about making stuff go faster.

CREATE TABLE    dbo.Stock
                    StockID INT IDENTITY(1, 1) NOT NULL,
                    ArticleID SMALLINT NOT NULL,
                    TranDate DATETIME NOT NULL,
                    TranCode VARCHAR(3) NOT NULL,
                    Items INT NOT NULL,
                    Price MONEY NULL,
                    CONSTRAINT [PK_Stock] PRIMARY KEY CLUSTERED 
                        StockID ASC

CREATE NONCLUSTERED INDEX IX_Input ON dbo.Stock (TranCode, ArticleID)
--INCLUDE (TranDate, Items, Price) -- Remove comment for SQL Server 2005 and later
--WHERE TranCode IN ('IN', 'RET')   -- Remove comment for SQL Server 2008

CREATE NONCLUSTERED INDEX IX_Output ON dbo.Stock (TranCode, ArticleID)
-- INCLUDE (TranDate, Items) -- Remove comment for SQL Server 2005 and later
--WHERE TranCode = 'OUT'  -- Remove comment for SQL Server 2008

You are welcome to change the two nonclustered indexes to suit your solution. You can download the complete sample data here. I have an idea of my own of the way to do this but I don’t know if it is the fastest.

Explanation of FIFO rules (example, abbreviated)

StockID ArticleID TranDate TranCode Items  Price CurrentItems CurrentValue
   4567     10000 10:45:07 IN         738 245.94          738   181,503.72
  21628     10000 12:05:25 OUT        600                 138    33,939.72
  22571     10000 14:39:27 IN          62 199.95          200    46,336.62
  30263     10000 16:14:13 OUT        165                  35     6,998.25
  42090     10000 18:18:58 RET          5                  40     7,998.00
  58143     10000 20:18:54 IN         500 135.91          540    75,953.00

a) First IN add 738 items (each $245.94) to the stock, for a total of $181,503.72
b) Then we take out 600 items (each 245.94) from the stock, leaving a total of $33,939.72
c) Then we insert 62 items (each 199.95) to the stock, for a total of $46,336.62
d) Then we take out 165 items (138 each 245.94 and 27 each 199.95), leaving a total of $6,998.25
e) Then we return 5 items. We can’t track at which price we took them out; so all returns are priced at the price of the latest ones inserted before the return. Even if there should be items left for the price of 245.94, the returned items are valued for 199.95. After the return, the current stock value is $7,998.00
f) The final insert adds $67,995.00 to the stock value, for a total of $75,953.00

As mentioned before, the CurrentItems and CurrentValue columns in the sample data are only included for you to validate your routines.

Here are some guidelines for your entries:

1) Include a header in your suggestion. Make sure your name and date is present.
2) Include an edition number. First edition is 1. If you later improve your current suggestion post it again as version 2. Example: “Peso 1” and if improved, “Peso 1b”, “Peso 1c” etc.
3) If you are trying a new algorithm, change the edition to “Peso 2”. If you improve this algorithm, change the version to “Peso 2b”, “Peso 2c” etc. This will save Phil hours of work in the test harness!
4) If you create a temp table, make sure you delete it in the script.
5) Keep the order of columns in output as ArticleID, CurrentItems, CurrentValue

I will allow you to use an existing tally number table (make sure it starts with 0). You can use any kind of object for this competition, except SQLCLR. If you are using a fixed tally number table, it has to be named dbo.TallyNumbers and the column named Number.
The time taken for their creation will not be included in the timings. The time measured is the “Main” script/procedure. If you want to call sub-procedures, go ahead.

The winner will be amongst the tied fastest entrants (generally there is a group of these) and it will be the one with the highest number of votes. We'll announce the winner in three week's time on 16th November.

For a starter, here is a common cursor based solution that you will find in production in many places.

                    RowID INT IDENTITY(1, 1) PRIMARY KEY CLUSTERED,
                    Price MONEY

DECLARE @ArticleID INT = 1,
        @PrevID INT = 0,
        @TranCode VARCHAR(3),
        @Items INT,
        @Price MONEY,
        @Loop INT = 0,
        @StockID INT,
        @LatestPrice MONEY,
        @Total INT = (SELECT COUNT(*) FROM dbo.Stock)

                SELECT      ArticleID,
                FROM        dbo.Stock
                ORDER BY    ArticleID,

OPEN    curYak

FROM    curYak
INTO    @ArticleID,

        IF @ArticleID > @PrevID
                TRUNCATE TABLE  #Work

                SET @LatestPrice = NULL

        IF @TranCode = 'IN'
                INSERT  #Work
                SELECT  @Price
                FROM    dbo.TallyNumbers
                WHERE   Number < @Items

                SET     @LatestPrice = @Price

        IF @TranCode = 'RET'
            INSERT  #Work
            SELECT  @LatestPrice
            FROM    dbo.TallyNumbers
            WHERE   Number < @Items

        IF @TranCode = 'OUT'
            DELETE  w
            FROM    (
                        SELECT      TOP(@Items)
                        FROM        #Work
                        ORDER BY    RowID
                    ) AS w

        UPDATE      s
        SET         s.CurrentItems = w.CurrentItems,
                    s.CurrentValue = COALESCE(w.CurrentValue, 0)
        FROM        dbo.Stock AS s
        INNER JOIN  (
                        SELECT  COUNT(*) AS CurrentItems,
                                SUM(Price) AS CurrentValue
                        FROM    #Work
                    ) AS w ON s.StockID = @StockID

        SELECT  @PrevID = @ArticleID,
                @Loop += 1

        IF @Loop % 1000 = 0
            RAISERROR('Now updating record %d of %d.', 10, 1, @Loop, @Total) WITH NOWAIT

        FETCH   NEXT
        FROM    curYak
        INTO    @ArticleID,


CLOSE       curYak

The above is already done. Here is the code to produce the final resultset.

SELECT      ArticleID,
FROM        (
                SELECT  ArticleID,
                        ROW_NUMBER() OVER (PARTITION BY ArticleID ORDER BY TranDate DESC) AS recID
                FROM    dbo.Stock
            ) AS d
WHERE       recID = 1
ORDER BY    ArticleID

Good luck to you all!

Peter Larsson

more ▼

asked Oct 23, 2009 at 03:56 PM in Default

Peso gravatar image

1.6k 5 6 8

Was there any particular reason you disallowed CLR functionality?

On an initial look, this would be the ideal scenario in which to use a CLR type...
Oct 23, 2009 at 04:34 PM Peso
Two things had us made this decision: 1) We have tried CLR before and there was no performance gain. 2) To test and verify, poster need to disclode the source code for the suggestion. But if you have a CLR routine that runs on 15 seconds or less, I think Phil will be interested in testing it.
Oct 23, 2009 at 05:08 PM Peso
Questions: 1) Is the Stock Transaction List referred to the one that will be used for Final Evaluation? If not, can you describe the differences? 2) What is the configuration that entries will be (finally) evaluated on?
Oct 23, 2009 at 05:09 PM RBarryYoung
Yes, the downloadable file is the data against all suggestions will be measured. Phil will drop or rename the two last columns so that no suggestion can use them. I guess Phil will be using same test harness as last time with 'Subscription List' competition.
Oct 23, 2009 at 05:16 PM Peso
Those filtered indexes will only work on SQL Server 2008, right?
Oct 23, 2009 at 05:46 PM RBarryYoung
(comments are locked)
10|1200 characters needed characters left

28 answers: sort oldest

Dave Ballantyne - Phil Factor Challenge Entry 3.d

Peso had adding some index hints and shuffled a calculation to my initial 3.a. So this is 3.a(+) with comments

with 3.d doing the TotalStock sum in one operation shaves some more time off

“A problem well stated is a problem half solved” - Charles F. Kettering

Let us think about what has been asked within the challenge.

We have a warehouse which starts empty , the data contains the stock movements in (including returns) and out. The stock movements ALWAYS happen in a first in first out basis. With this we know that if the present stock level (sum(in's) - sum(out's) is 50 that will be the last 50 that have entered the warehouse. The key to this query is to efficiently find the cost of those 50 items.

        [ArticleID] ASC,
        [TranDate] ASC
INCLUDE ( [Items])
WHERE ([TranCode] IN ('IN', 'RET'))


        [ArticleID] ASC,
        [TranDate] ASC
INCLUDE ( [Price])
WHERE ([TranCode]='IN')


/* Sum up the ins and outs to calculate the remaining stock level */ ;WITH cteStockSum AS (

  SELECT  ArticleID,
        SUM(CASE  WHEN TranCode = 'OUT' THEN 0-Items
                  ELSE Items END) AS TotalStock
        FROM  dbo.Stock
  GROUP BY  ArticleID

), /* Perform a rolling balance ( in reverse order ) through the stock movements in / cteReverseInSum AS ( SELECT s.ArticleID, s.TranDate, ( SELECT SUM(i.Items) FROM dbo.Stock AS i WITH (INDEX (IX_Dave_Items)) WHERE i.ArticleID = s.ArticleID AND i.TranCode IN ('IN', 'RET') AND i.TranDate >= s.TranDate / SELECT SUM(i.Items) FROM dbo.vwStockIn as i with (index (idxStockInArtTrans)) WHERE i.ArticleID = s.ArticleID AND i.TranDate >= s.TranDate*/

       ) AS RollingStock,
       s.Items AS ThisStock
 FROM  dbo.Stock AS s
WHERE  s.TranCode IN ('IN', 'RET')

), /* Using the rolling balance above find the first stock movement in that meets (or exceeds) our required stock level / / and calculate how much stock is required from the earliest stock in */ cteWithLastTranDate AS ( SELECT w.ArticleID, w.TotalStock, LastPartialStock.TranDate, LastPartialStock.StockToUse, LastPartialStock.RunningTotal, w.TotalStock - LastPartialStock.RunningTotal + LastPartialStock.StockToUse AS UseThisStock

  FROM cteStockSum AS w
                             z.ThisStock AS StockToUse,
                             z.RollingStock AS RunningTotal
                        FROM cteReverseInSum AS z
                       WHERE z.ArticleID = w.ArticleID
                         AND z.RollingStock &gt;= w.TotalStock
                    ORDER BY z.TranDate DESC ) AS LastPartialStock
) /* Sum up the cost of 100% of the stock movements in after the returned stockid and for that stockid we need 'UseThisStock' items' / SELECT y.ArticleID, y.TotalStock as CurrentItems , SUM(CASE WHEN e.TranDate = y.TranDate THEN y.UseThisStock ELSE e.Items END * Price.Price) AS CurrentValue FROM cteWithLastTranDate AS y INNER JOIN dbo.Stock AS e WITH (INDEX (IX_Dave_Items)) ON e.ArticleID = y.ArticleID AND e.TranDate >= y.TranDate AND e.TranCode IN ('IN', 'RET') CROSS APPLY ( / Find the Price of the item in */ SELECT TOP(1)
p.Price FROM dbo.Stock AS p WITH (INDEX (IX_Dave_Price)) WHERE p.ArticleID = e.ArticleID AND p.TranDate <= e.TranDate AND p.TranCode = 'IN' ORDER BY p.TranDate DESC ) AS Price GROUP BY y.ArticleID, y.TotalStock ORDER BY y.ArticleID go
more ▼

answered Nov 05, 2009 at 06:03 AM

dave ballantyne gravatar image

dave ballantyne
928 1 1 4

(comments are locked)
10|1200 characters needed characters left

What do you think?
Am I allowed to compete, since all rules are known?

Please comment here what you think.

more ▼

answered Oct 23, 2009 at 05:32 PM

Peso gravatar image

1.6k 5 6 8

Wow, have I been there, Peso. It's probably Phil's call, but my feeling is NO. I say this knowing that I may be in your position too in the near future (hopefully :-)). The problem is that no matter how fair you are, sooner or later someone is going to feel cheated because you were both a competitor and the Question Asker (or worse, Tester/Judge). What I would recommend is just throwing you solution in at the end, in the final measurements, as an example non-competing entry. I.e., non-eligible to win.
Oct 23, 2009 at 05:41 PM RBarryYoung
I respect that. If this is the consensus, I will post my set-based solution very last before deadline so that no-one can steal my thunder :-)
Oct 23, 2009 at 05:51 PM Peso
Oh boy! Someone already voted this as an unhelpful post grin
Oct 23, 2009 at 05:59 PM Peso
I think the series would be more fun if we disallow the current winner from entering, since it gives other people a chance! Actually, I think we should be allowed to steal Peso's thunder too at some point!
Oct 24, 2009 at 05:23 AM Phil Factor
Well, I'm upvoting it, so they should cancel out, ;-)
Oct 24, 2009 at 03:24 PM RBarryYoung
(comments are locked)
10|1200 characters needed characters left

Here will the preliminary timings be presented. I write preliminary since Phil is (will be) away for a few days, so I will do the preliminary timings on a similar machine as Phil has.

Current standings

 1 - Dave 3d 1.3 seconds  2 - Dave 3c 1.5 seconds  - Matt 1a 1.8 seconds (correct items, wrong value)  3 - Dave 3b 2.1 seconds  4 - Gianluca 1a 3 seconds  5 - Steve 1a 5 seconds  6 - Gianluca 2a 5 seconds  7 - Scot 1 6 seconds  8 - Barry 2d 7 seconds  - Andriy 1a 7 seconds (correct items, wrong value)  9 - Herman 1d 9 seconds 10 - Herman 1c 12 seconds  - Peso "Cursor" 12 seconds (not in competition) 11 - Barry 2c 18 seconds 12 - Andriy 1c 22 seconds 13 - Dave 2b 26 seconds  - Matt CLR 44 seconds (not in competition) 14 - Barry 2 50 seconds
15 - Dave 1b 96 seconds 16 - Barry 1b 97 seconds
17 - Herman 1a 172 seconds
- Article code ~2,400 seconds (not in competition)
more ▼

answered Oct 24, 2009 at 04:08 PM

Peso gravatar image

1.6k 5 6 8

Barry's timings on his solution varies due to a lot of tempdb space. But it is still a great suggestion.
Oct 26, 2009 at 08:51 AM Peso
What configuration are you testing on, Peso? I would think that my solution #2 would do a lot better than that on a multi-core environment. (Or is MAXDOP set down?)
Oct 26, 2009 at 08:58 PM RBarryYoung
It's probably my tempDB going through the roof. I'll try the suggestions on a better tempdb configuration. Your queries generates a lot of activity on the tempdbs.
Oct 27, 2009 at 09:44 AM Peso
(comments are locked)
10|1200 characters needed characters left
more ▼

answered Oct 24, 2009 at 05:22 PM

Matt Whitfield gravatar image

Matt Whitfield ♦♦
29.5k 61 65 87

It's a great entry! However, have you disabled parallelism manually?
Oct 24, 2009 at 05:48 PM Peso
I did the MAXDOP 1 because it was a typical quirky update safety addition - I did remove it at one point but didn't find it make any difference to the performance overall...
Oct 24, 2009 at 05:56 PM Matt Whitfield ♦♦
It's a great learning experience for me! Thank you very much. Until now, I have only dealt with aggregate SQLCLR. Take a look here http://www.developerworkshop.net/software.html
Oct 24, 2009 at 06:17 PM Peso
(comments are locked)
10|1200 characters needed characters left

RBarryYoung_1b: Naive, Pure Set-Based Solution

OK, just to get things rolling, here is a straight-forward Set-based solution, with no T-SQL tricks (though there is a very important algorithmic trick in cteOverlapValue).

    Naive pure Set-based solution to SSC FIFO inventory challenge.

    - RBarryYoung, Oct-24, 2009
-- the base CTE, (just an easy way to Alias to other version of the test table)
 cteStock AS ( SELECT * FROM Stock )
-- Just the 'IN' records:
, cteStockIN AS ( 
    SELECT * 
    FROM cteStock
    WHERE TranCode = 'IN' 
-- Fill in the Last IN records date for lookups later:
, cteLastINDate AS (
    SELECT * 
    , CASE TranCode WHEN 'IN' THEN TranDate
        ELSE (
            SELECT MAX(s3.TranDate)
            FROM cteStockIN s3
            WHERE s.ArticleID = s3.ArticleID
            AND  s.TranDate > s3.TranDate
            ) END AS LastINDate
    FROM cteStock s
-- Carry-forward the last price for RET records:
, cteLastPriceDate AS (
    SELECT * 
    , COALESCE(Price, (
        SELECT Price 
        FROM cteStockIN s2
        WHERE s.ArticleID = s2.ArticleID
        AND  s.LastINDate =  s2.TranDate
        )) AS LastPrice
    FROM cteLastINDate s
-- Build running sums of InItems and OutItems (same column, but separate counts)
, cteRunningItems AS (
    SELECT *
    , CASE WHEN TranCode IN('IN','RET') THEN (
            SELECT SUM(Items) 
            FROM cteLastPriceDate p2
            WHERE p2.TranCode IN('IN','RET')
            AND   p2.ArticleID = p.ArticleID
            AND   p2.TranDate <= p.TranDate
        WHEN TranCode = 'OUT' THEN (
            SELECT SUM(Items) 
            FROM cteLastPriceDate p2
            WHERE p2.TranCode = 'OUT'
            AND   p2.ArticleID = p.ArticleID
            AND   p2.TranDate <= p.TranDate
        END AS InOutItems
    , CASE WHEN TranCode IN('IN','RET') THEN (
            SELECT SUM(p2.Items*p2.LastPrice) 
            FROM cteLastPriceDate p2
            WHERE p2.TranCode IN('IN','RET')
            AND   p2.ArticleID = p.ArticleID
            AND   p2.TranDate <= p.TranDate
            ) END AS TotalValueIN
    FROM cteLastPriceDate p
-- Map each OUT record to the (IN,RET) records that it will be consuming:
, cteOutMapIn AS (
    SELECT t1.*
    , t2.Items AS InItems
    , t2.LastPrice AS InPrice
    , t2.InOutItems AS TotalIn
    FROM cteRunningItems t1
    LEFT JOIN cteRunningItems t2
        ON  t1.ArticleID = t2.ArticleID
        AND t1.TranCode = 'OUT'
        AND t2.TranCode IN('IN','RET')
        AND t1.InOutItems-t1.Items <= t2.InOutItems
        AND t1.InOutItems >= t2.InOutItems-t2.Items
-- Calculate the OUT consumed values, 
--based on how the OUT items overlap the (IN,RET) items:
, cteOverlapValue AS (
    SELECT *
    , CASE WHEN TranCode = 'OUT' THEN
            WHEN InOutItems >= TotalIn  AND (InOutItems-Items) <= (TotalIn-InItems)
                THEN InItems            -- OUT consumes all the INs
            WHEN InOutItems >= TotalIn  -- OUT consumes only the upper part of the INs
                THEN (TotalIn - (InOutItems-Items)) 
            WHEN (InOutItems-Items) <= (TotalIn-InItems)    -- OUT consumes the lower INs
                THEN (InOutItems - (TotalIn-InItems)) 
            ELSE Items  END             -- All OUTs consume an interior slice of IN
      END AS ItemsConsumed
    , CASE WHEN TranCode = 'OUT' THEN
            WHEN InOutItems >= TotalIn  AND (InOutItems-Items) <= (TotalIn-InItems)
                THEN InItems * InPrice  -- OUT consumes all the INs
            WHEN InOutItems >= TotalIn  -- OUT consumes only the upper part of the INs
                THEN (TotalIn - (InOutItems-Items)) * InPrice
            WHEN (InOutItems-Items) <= (TotalIn-InItems)    -- OUT consumes the lower INs
                THEN (InOutItems - (TotalIn-InItems)) * InPrice
            ELSE Items * InPrice END    -- All OUTs consume an interior slice of IN
      END AS ConsumedValue
    FROM cteOutMapIn 
-- Total everything up for each ArticleID:
, cteArticleTotals AS (
    SELECT t1.ArticleID
    , SUM( CASE WHEN TranCode='OUT' THEN -ItemsConsumed ELSE Items END ) AS CurrentItems
    , MAX( TotalValueIN ) - COALESCE(SUM( ConsumedValue ),0) AS CurrentValue
    FROM cteOverlapValue t1
    GROUP BY t1.ArticleID
-- Output the results:
FROM cteArticleTotals t1
ORDER BY t1.ArticleID

Performance is not good, 9 seconds on the 5000 row (75 Articles) test set, and about 18 minutes on the full set.

Oops! I got those numbers completely wrong,...

In reality, performance is OK, but not great. it takes 9 seconds on 100,000 rows (about 1500 Articles) and 90 seconds on the full set. Not bad for virtually no optimizations.

1b: OK, I have made the corrections Requested by Peso (ORDER BY and Fixed NULLS on Article 25001). My current timings for it are 9sec on 100k rows and 95sec on 1M rows.

more ▼

answered Oct 24, 2009 at 11:49 PM

RBarryYoung gravatar image

782 5 5 8

Excellent suggestion Barry! Great to see a pure set-based solution. However, there are two things I miss. 1) Articles need to be sorted by ArticleID. 2) Last ArticleID 25001 reports correct Items but wrong Value.
Oct 25, 2009 at 04:43 AM Peso
OK, thanks Peso. Obviously I can fix (1) right now ... (:-)), (2) is going to take some investigation though.
Oct 25, 2009 at 09:58 AM RBarryYoung
The procedure header says 1a, but the code (I think) is 1b?
Oct 26, 2009 at 08:42 AM Peso
Right. I'm not very good with directions (that's why I need computers :-)). Fixed now.
Oct 26, 2009 at 11:26 AM RBarryYoung
Ok, sorry for the really slow comment, but this was the one showing incorrect values for me - first article, value shows as 0.0, should be 15446.39. Sorry for the delay!
Oct 30, 2009 at 06:23 PM Matt Whitfield ♦♦
(comments are locked)
10|1200 characters needed characters left
Your answer
toggle preview:

Up to 2 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

New code box

There's a new way to format code on the site - the red speech bubble logo will automatically format T-SQL for you. The original code box is still there for XML, etc. More details here.

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here



Answers and Comments

SQL Server Central

Need long-form SQL discussion? SQLserverCentral.com is the place.



asked: Oct 23, 2009 at 03:56 PM

Seen: 29543 times

Last Updated: Nov 14, 2009 at 01:55 PM