Definitely get it checked out, Im not trying to scare you, but if it is a loose connection, it is generating heat.
Many many houses have burned because of loose connections.
A short circuit will most often trip a breaker, but a loose connection most often just generates heat, but it does give off warning signs, lights dimming, TV's turning off, computers re-booting.
Loose connections can also damage motor devices, (refrigerators, AC's, etc), and TV's and computers, because they dont like the voltage drop that a loose connection causes.
If your lights momentarily go brighter when your refrigerator kicks on, that is a sure sign of a loose neutral in your breaker panel, or between your breaker panel and the light pole.
This causes an imbalance between the two 120 volt "legs", which can cause dangerously high voltages on 120 volt circuits. Voltages can swing from 120 to 240 volts when the neutral "breaks".
Call an electrician!