Another update but with some bad news:
Since I'm impatient, I installed a nightly build (non-released versions that contain some unofficial fixes) of VLC and am now able to convert an ISO to MP4 (H264 video and AAC audio) on Win7-64bit without crashing. Yay!
My plan was to fire off the conversion of the selected ISO file via my web app (by selecting a video to play), which would start converting the ISO to a "temporary" MP4 file on my server. Then, have the web player pause for ~10 sec to give the server enough time to partially convert the MP4 file before I attempted to play it. Since my server can convert video at faster than real-time frame-rates (approx 3x faster actually), I wouldn't have to worry about the player "catching up" to the point where the file was currently converted.
Excited, I quickly ported the command line to do this in some ASP code and fired the conversion from within my web app (using ASP). However, I discovered I cannot play a partially-converted file. And even if I choose a format that
does allow playback of a partial file (like WMV), I can only play the video up to the point where it was initiated (would stop after the ~10 sec mark).
Looks like I will have to devise a way to convert the video and stream in real-time after all. This is a bit complex since there is no "file" to play; the web player will be playing a direct stream. So back to the drawing board.