Oh yeah, most definitely need to plug it if you're actually using that top half of the carb.
Originally I bought a second carb (you can see in the pics that there are two halves there) and I was going to drill into it anyways and put an NPT plug in its place. Then I realized it was a welch plug covering the hole.
A set screw would probably be a decent idea if you don't have a replacement welch plug, make sure you loctite it or something, though.