I just got a Quartermaster General today. When I set it down on my island it brought my building permits to -1. When I sent it to an adventure it gave me the building permit back. It appears that the supply garrison when built on the island is counting towards your permit count.