When accepting invitations to join an adventure you must make sure to have atleast 1 general on your island. (i.e. In case you were in an adventure and moved them all there.) If you don't have any general's present on your island and accept an invitation to join an adventure it will give the the initial popup saying you 'Joined adventure' but will never give the adventure button on the right hand side to allow you to send general's there and whoever sent you the invitation will show you as part of the adventure and will not be able to reinvite you or anyone else.
The adventure therefore must then be completed by whoever sent the invite solo and you will not receive any rewards since you weren't able to help out. It happened to me twice.. First time I was suspiscious that no general was the cause then a friend of mine accepted an invite and she did not have general's present and same thing happened to her.
<Edit>
Also it does not appear in the quest menu "!" for you to check it so it's not possible to get it to appear that way. Even logging in and out will not fix it.
************************
2nd Invitation Bug - Sort of what Sirroy was talking about down below.. Except I was on my island and I clicked accept invite.. but I didn't wait for the invitation confirmation popup before I switched to another island to check buffs.. when I returned I noticed I didn't have the adventure button again.
So also make sure to stay put after you accept the invite till you actually receive it and get the adventure icon on the right.