One of the writers for Atomic mag spent some time upscalling his dvd collection to 720p. This is the commandline he used in the end to get a great clear result.
commandline ;
mencoder -sws 9 -vf scale=1280:720,hqdn3d=6:5:8,unsharp=17×5:1,pp=ha:128:7/va/dr/al – ovc x264 -x264encopts subq=9:b_pyramid:weight_b:8x8dct:me=umh:cabac:partitions=all:trellis=2:mixed_refs:direct_pred=auto:psy-rd=1.0,0.2:qp=16:threads=2:nodct_decimate:nofast_pskip:bframes=6:frameref=6 -oac copy dvd://1 -o output.avi
Of course it may change depending on what you are encoding, but its a good place to start. Check the atomic fourms (au) for more detail etc…