You don't mention whether you're using backtrader to do real trading and want to round per exchange / currency rules, or just have more accurate simulations.
I actually wasn't able to get this to work in backtrader either, but honestly didn't try a whole lot on the simulation side. Rounding to 0.1 vs 1 if you're using simulated cash anyway isn't super meaningful. As Paska mentions, shifting up and down orders of magnitude could work but difficult (in my case, I simulated this by just increasing my starting cash by an order of magnitude instead -- e.g. if I'm trading LTC in increments of 0.1, I multiply my starting cash by 10 and divide the size of each trade by 10).
In my case, I am using backtrader to generate signals to automatically trade. At the time I generate the orders for the exchanges, I perform exchange / currency-appropriate rounding of the approximated orders generated from backtrader. For me, I'm allocating a percentage of my capital in the account for each trade, and then sizing the order based on current bid/ask prices. Since a lot of this is dynamic based on what is actually on the order book, spending a lot of time getting backtrader to exactly size the order seemed not really to be worth the effort in my case.
Not sure if this helps but thought I'd offer a suggestion.