Every object in the game has to be stored in memory and processed every "round" (for want of a better term). The more there are, the more it has to do, so the longer it takes = the slower things are.... And if you DO have shared memory as many older (crappier.... ) cards, then it will reduce the available system ram and possibly result in your system "thrashing" (constantly swapping memory to/from hard disk as more memory has been requested by all your running apps (including Sins) then is in your machine...

Also, make sure you haven't upped the graphics too much - played a 5v5 game online with someone who had graphics on high with a card that couldn't handle it and it took us 5 minutes to get going... We lost 2 people while we waited and he eventually dropped the settings - his CPU was at 100 on the latency screen!!!!