Fellow house sitters Charli and Jeff swung by in San Diego on Christmas Day