I had the same problem, I may have found a solution...
It turns out for some reason paypal isn't accepting the invoice numbers, which are being created as 1, 2, 3.... etc... however if you go to includes/modules/payments/paypayl_ipn.php and change the line (I think 326) that says...
$parameters['invoice'] = substr($cart_PayPal_IPN_ID, strpos($cart_PayPal_IPN_ID, '-')+1);
to
$parameters['invoice'] = $cart_PayPal_IPN_ID;
Then the transaction should work and you will be returned to your site. However on paypay the Invoice number will be cartNo-InvNo where as in your osCommerce records it will just be InvNo. I'm still working on getting around that issue...
Hope this helps