Well, I suppose it depends on the stick.
If you are using a mechanical rotary (one that clicks for each rotation), the stick must be connected to a rotary interface to infer what the rotation means. The output from that card will be a button press for each rotation direction. There is no special setting in MAME required to make this work other than mapping that click to the corresponding rotation.
If you are using an optical rotary stick (smooth rotation in either direction), then the stick connects via an optical interface. This interface is essentially a single axis of a mouse to the computer. In this case, you map the rotation in MAME via the analog settings. If you are connected this way, you must also inform MAME that you are using a mouse with the -mouse switch or setting mouse=1 in the mame.ini file.