You can add additional decimals for a currency through a work around method.
Set up a currency let's say instead of USD call it US$ ( OY03 ) and define the number of decimal places ( OY04 ) to be 3 or more depending on your requirement.
Maintain the exchange rate for between US$ and USD to be 1 to 1 ( OBBS and OB08 ).
Create pricing condition records for those customers requiring 3 decimal places using Current US$ instead of USD.
That will give you 3 decimal places for your prices. However, one thing you will have to watch out for is rounding.
You can try transaction OB90, define rounding rule for currency. Here you define the rounding rule for your customer's currency