Does your temporary table have the same indexes as your "physical" table? Are you really making fair comparisons - are you clearing down the buffers between running on "physical" vs running on temp table?
As @ThomasRushton mentioned are the tables really comparable? What about the file group configuration on which the physical table resides. You mentioned 4 discs RAID 10 for tempdb, but not mentioned configuration of the RAID level and discs for file group on which the physical table is placed.