Thinking about this some more:
I am not sure producing an API for this is really appropriate because the approach I have taken is very different. I am not concerning the user with video modelines as users dont need to know anything about pixel clock, horizontal totals etc etc. All the user needs to know about, and change, is the picture size, position and refresh rates. So this is what the utility does. Then it stores the settings and invokes using a command line.
Not sure if adding the ability to change all the video mode timings add any benefit for the user.
I'll have to look at this more and play around with the the utility. I'm sure there's a way switchres can use it to improve how it acts with an AVGA card. I like the idea of the API, just because from my testing I've seen that modelines can do a lot of different things for timings that different monitors are picky about. Sure a person can fiddle with it themselves, but you can predetermine it all from the testing Calamity and I have been doing. So for the most part, the advantage would be to be able to tell the card what we want after calculating it ourselves from the Mame XML information. I am not sure how the rsi format is setup, I'm really curious since it sounds like I might for the most part be able to setup .rsi files of what I want the resolution to be and run them with your utility, hence avoiding manually creating them with the utility but use it to pass things to the card we calculate.
I guess the main thing is that not all modeline calculators are equal, and the one we are using is developed by Calamity and an amazing one from what I have seen. So a neat feature would be to see these modelines calculated to the desired specs pushed onto the AVGA card, so far they seem to have done pretty well. Also the way I've done things, my program basically avoids having all those little .ini files statically setup, they are dynamically produced, so that combined with feeding the resolution the .ini file would have had but dynamically calculated, plus using your utility and feeding it more of the timing information we feel the monitor likes, that would be really a cool combination.