I was also looking into this as I hoped to set it up as the external merge tool for SourceTree. You can use Process Explorer to view all the strings in a running program and usually that will show you the command line options as well. When I did that for RSLCompare.exe, the very first string it shows is "!This program cannot be run in DOS mode.".
However, it also has hidden in the strings this information:
"Command-line input should be of the form:
RSLCompare.exe BaseLogixProject[.L5K|.ACD] CompareLogixProject[.L5K|.ACD]
RSLCompare.exe BaseLogixProject[.L5K|.ACD] CompareLogixProject[.L5K|.ACD] CompareToolProject.compare
RSLCompare.exe CompareToolProject.compare CompareToolReport.pdf
RSLCompare.exe [/d]
Where /d creates an XML dump of the project at ProjectName.xml"
Sure enough, running RSLCompare from the command line with the /d switch brings up a prompt about how to use RSLCompare from the command line.
EDIT: I've tried a few tests and I can't for the life of me get the /d switch to do anything.