My question to everyone is.... Can you come up with some tests to put the different software packages through that will produce objectively measurable results?
I think it could be doable, if resources were easily obtained such as licensing and hardware environments needed.
I think one of the biggest problems is that everyone tends to lump HMI/SCADA into the same bucket, and you simply can't do that. You've mentioned the apples/oranges when it comes to HMI and SCADA software. To be objective, SCADA and HMI packages would have to differentiated, and while SCADA can be an HMI, an HMI package simply cannot be SCADA.
From a SCADA standpoint, much like PLCs, once one becomes fluent in a package, jumping ship to another is very hard due to the amount of effort originally put in. So people will always have the bias. I do think a number of things of a SCADA package are measurable:
1 - Platform features: Visualization, SQL interactions/transactions, Email connectivity, Alarming, Reporting, Voice notifications, Historian, MES, OEE, Track & Trace, Recipe Management, Batching, Redundancy,Deployment...etc
2 - Licensing/Support costs, compare 1 server, 5 clients. Redundant severs 10 clients, but tag based licensing really makes this a mess to compare.
3 - Software installation, configuration to get up and running
4 - Device connectivity, opc servers, PLC support
5 - Development: Tag generation/management features (UDTs, attributes)
6 - Development: Objects/templates/global objects
7 - Development: Redundancy configuration/fail-over
8 - Development: Historian logging/SQL transactions
9 - Development: Visualizations, Navigation, included objects
10 - Scripting: Library, language
11 - Security/Role-management, integration w/Active Directory
12 - Training services
13 - "Headache factor", how many support calls do you have to make, how many times do you have to start-over, bang your head against the wall to make something work?
I think all of these are measurable to an extent. However the amount of work and resources required to generate a measurable review is quite high.