Sadly, that's true. I made a mistake with my jolly geologist, but I don't care. I'll live with a couple of tomes a little out of place rather than sacrificing 10 books. I get it if I want to start completely over, but just to change one thing should not carry that much of a penalty whereby you need to learn everything else over again.
Yes the penalty is a bit high when you only need to move a couple of books.
