Hi — I know this is an old thread, though the idea is very current.
I've been looking around for a multi-shop/multi-vendor cart system for a client. I almost certain that I've found what I want in IXXO Multi-Vendor. It's fairly expensive at $1495 for a downloadable install.
It requires all vendors to have PayPal accounts if they want to receive payments from the system. In my case, we'd be using Authorize.net so we'll have to generate checks for vendors, but the cart will track all sales, site owner commissions, and the payments owed (and made) to vendors for sales.
It is an extension for Joomla. Which also fits our needs. I've also recently heard from Dioscouri.com — a Joomla extension developer — that they are going to be releasing an Enterprise version of their Tienda cart system which will handle multi-vendor stores.
Hope this helps someone...