I used a contrib IPN. When I use the standard one, nothing is subtracted at all.
Now, I also have QT Pro installed, and when I use QT Pro and have Products Attributes used (Sizes, etc.), the correct number(s) are subtracted. Only when this is not used is something fouling up. And yes, only through PayPal.