Yes, I would patch that tearout. Although the button bezels will cover the majority of it, it's pretty easy to do and is worthwhile.
You can use wood filler, bondo, wood glue mixed with sawdust, fibreglass resin or any of a number of different types of fill. If the chips from the tearout are still in your work area, you can even superglue them back in. Whatever type of filler is used, just sand the top flat again to blend it in and once the vinyl is applied you'll never know any of it is there.