I used to get those exact tube-looking things inside the windows of the house when I lived in NC. Best thing to do is spray them. Also, it's best after spraying a nest of a wasp to leave it up. New wasps won't move in for some reason, and it removes a place for them to build a fresh and lethal dive-bombing hangar. What I usually did (and this won't work for your situation) is let them fly away, and just dribble some of that spray all over the outside of the tube and soak it real good. Evidently that Raid spray has some pretty wicked residual effects, because every once in a while I'd see some dead/dying in between the windows.
You could probably spray some Raid in a glass and soak a cotton ball in it. Get the bottom of the tube REALLY good. Those things DO creep the hell out of you, but I never got stung by 'em. Dunno if that's cuz I Hong Kong Fooey-ed 'em, or if they were afraid of how toxic I was.