If there's more than one person attending an adventure, doesn't matter if everyone else is offline, there's 4, 5 or more flags worth of LAG!
Alone, fine. Multiplayer, LAAAAG
It's not just me I've heard the same from several people. Time of day doesn't seem to make a difference either