SUBTARGET="mess" gives you MAME with just the MESS parts, and MAME's default UI (the MEWUI thing)
SUBTARGET="mess" OSD="newui" gives you the same, but you can enable the white menubar.
SUBTARGET="mess" OSD="messui" gives you the MESSUI frontend plus the newui menubar.
If you are trying to build 0.174, you need to download MAME source from MAMEDEV, then apply the MESS source changes from the .rar file.
Or if you just want whatever is the latest GIT, just download a github zip package and compile that.
The program runs here. I've heard there might be trouble on Windows 10. I don't have it (win10), therefore cannot support it.