Yes. It takes them a while to find you, and they may or may not make it to you without aggro-ing on something else.

If you worm-holed to an island, then the hounds can't follow you, but will ambush you the next time you're on a piece of land they can reach.

If you go into a cave, they will wait for you outside the cave.

They may spawn while you're in a cave anyway, so it's not a terrible idea to set up some spike traps around the cave entrance.