The handling part was supposedly due to the orange DeCarbon shocks installed. Other than that even the suspension bits were rpetty much the same unless you opted for the SLP stuff. But like some LS6 blocks ending up in Fbodies this wasn't always the case my Z for instance recieved the DeCarbon shocks from the factory.
The 16" rims were standard on the Z's as mentioned, freer flowing muffler (debatable on whether or not it was). The Ram air hood with matching air box base along with the rims spoiler and other looks package. Another difference was gearing. Z's got 2.73's with 3.23's optional (automatic transmissions) wheras the SS's got 3.23's standard for the same trans.
The faster cars were probably the 1SC "strippers" (not sayin that because mine is one) like Brandon said being lighter. Had a coworker with an auto SS. Mine was faster even when we switched drivers and despite me having 2.73's vs his 3.23's.
2000 Red 1SC Z28 A4 - dead for now
2005 Black GTO M6 6.0 - Split daily driver
1987 Silver Fiero GT - Split daily driver