Basically, how to install x264 on a Mac. A beginner's guide would be helpful. :) I have experimented with Compressor and Media Encoder, but some areas of film (grain etc) sometimes cause problems. The encodes need to be compliant with DoStudio.
You don't need to install it. It's a binary. Run it in the terminal. I don't know anything about DoStudio, but if it's any good, it'll accept BD-compliant files, so use the guide to which I linked in my last post.
DoStudio is discontinued and, above all, it doesn't made all the checks for bd compliancy. It is possible to mux a 23.98 SD video which is not compliant, and mux SD subtitles on an HD source... You need to be sure to use the right settings of x264 because there's high chance it will accept the file even if it's not compliant.