Sadly, that's true. I made a mistake with my jolly geologist, but I don't care. I'll live with a couple of tomes a little out of place rather than sacrificing 10 books. I get it if I want to start completely over, but just to change one thing should not carry that much of a penalty whereby you need to learn everything else over again.
Yes the penalty is a bit high when you only need to move a couple of books.
More of the same "new" content with a different bow? I will give more of the "same" towards my gem accumulation.