Software Testing

Most of us have used software that sometimes got released a little too soon, and maybe has a few glitches in it.

Should the manufacturer release it as soon as it is "ready", or should they continue to test it. Should we be able to test it for them? The opportunity to field test and relate back how things are in the real world, not in their ideal lab conditions.

So, the AA PLC Company has new programming software, do you want to test it for them? Who should beta test it, the biggest customers? The best known? Those who want to? Or maybe some picked at random?

So, what are your thoughts?

Maybe you could be testing RSLogik601 for the new ML2100 plc?

I voted "Turn it loose". to me..I meant that both the manufacturer(s) and the customers do the tests, and there will be feedback from both sides (or both parties), and I believe this will make better progress for the product.
just my point of view.

regards
Sherine T.

ps. I really enjoy these polls.

I voted for biggest customers, you can't please everyone, so try to please the people who buy the most.

Commissioning new machinery is hectic enough as it is without having to chase a supplier due to "undocumented behaviour" in their software. I would much prefer to see well tested software from the PLC makers rather than being a guinea pig. I think a "Windows Update" service for PLC firmware is a long way off and not particularly desirable. My customer's expect software that works before they hand over money, I expect the same from my suppliers.

This testing should only be done by users who are fully aware of the state of the software and who agree to take part.
My first experience of Control Logix and Logix 5000 was on new installation consisting of 12 PLC's on ControlNet. The current version of L5K at the start of the project was 2.27. By the end of the job we were using V6. The experience was horrendous and left a very bad taste in the mouths of all concerned. We are now using V13 and consider both Control Logix and Logix 5000 to be excellent products but the damage done in the early days has never been fully repaired. This would not have happened if more care had been taken releasing these products into the field.

The biggest customer - but with a good mix of industry. Say, 25% manufacturing customers, 25% process, 25% OEM and 25% integrators. Getting integrators involved in something like this would be critical, since I think they tend to "break" stuff more often because they are pushing the envelope in may instances, across the full spectrum of industry. And I think the very large manufacturing-type customers already rely on provider services (like Rockwell GMS), so if they hit a snag, they would be more likely to call in the cavalry rather than trying to find the root of the problem, and some important lessons about that product might not be learned. And there is nothing wrong with that approach, but not everyone can afford it, which is why I think getting large integrators involved with "Beta-testing" is critical.

The biggest customers should be offered the oportunity to test the new software, but not forced into it. The big customers typically already have well developed relationships with their suppliers, everyone is already on friendly terms, etc... This goes a long way when trying to figure out why XYZ is happening when it shouldn't.

Just my two cents though

Only those who want to, but anyone who wants to, not just the biggest customers. As a compensation for finishing the manufacturer's job for them those who use pre-release or initial release software should be given a discount on the final release, and those who first identify a bug or recommend an incorporated improvement should get a bigger discount.

Beta testing doesn't mean what it used to. Once upon a time if a product was in beta testing it meant that the manufacturer truly believed it was ready for real world use, but understood that end user input was needed to verify this was the case.

Nowadays beta testing means that the manufacturer's internal staff are going through marketing's checklist of features to verify they are more or less kind of almost just about there, and they can't find anything that likely will bankrupt the company with lawsuits.

Initial release nowadays puts the adventurous end user in the position of having to find the kind of bugs, glitches, and anamolies that internal programmers knowledgable in real world applications used to find. Version 3 means that the software has gotten to the point where you can use it for actual work and not expect to spend more time re-loading and re-booting than actually generating useful logic or configurations.

It's getting worse. I don't think it's just that the manufacturers are getting more irresponsible or that the market is more competitive. I think they just don't have guys on staff that have spent days and days in a factory looking over their shoulders while production managers polish their guns and foremen tie nooses.

Although it's not PLC software, DVT does something interesting with new releases of their vision software: they pay grad students from Georgia Tech to come in at night and try to break the software. They have a station set up for each language version of the new release and a student fluent in that language puts it thru its paces for a given period of time. Then they fix the identified bugs, test some more and release it.

Not a bad system...

Tom, I think you hit the nail on the head, companies don't have the same man-power they used to (wether its the guy looking over the programmers sholder or the total number of programmers)... I think the trend that is really causing the software bugs is the do more with less and do it faster mentality.

I still think (for PLC software) that using the bigger (or maybe the right word is better) customers is a good idea. Given the pace at which things must now work life is much easier if the communication channels are already open before trying to debug (again, just my two cents though).

You didn't have an option for "Put out an APB asking for volunteers. Then, approach the customers that are known to have good test cases." So, I chose "Only those who want to."

It's interesting how the "control world" works. Adopting Revision 0 of a given software package can be a real chore. The reason, I think, is that it's easier to get software installed new and fix it "in the field" than it is to get a retrofit job. There's no money in sitting on a product, and you're missing opportunities by waiting.

Since the end customer probably doesn't have to use the software until the integrator has already worked out the kinks, all the cussing and swearing goes unnoticed. The end customer gets a working system (hopefully), and he's happy... On the next job, the customer specifies the same software package because he's familiar with it and we start all over again.

AK

I selected "Turn It Loose - Every Tests It " - I want has many people has possible to test for me cause I sure ain't gonna use this version. No way!

I will get the next one. Maybee!

But darn. Why did I have to get SP2 in my laptop.

Now I have to reformat and re-install XP all over.

Sometimes I should listen to myself.

I voted 'Biggest and best known', although another option may have been "Most trusted customers".
Beta release software is essentially used for customer feedback, and as an overall 'smoke test' against potential problems. It should never be assumed that Beta versions are suitable for use in a production environment.
I'm sure that there are plenty of less than reputable controls people out there (although obviously none of them are PLCs.Net members ) who would love the chance to get their hands on what would essentially be free (even if not fully proven) software, and given that PLC software is so safety critical, the consequences of handing it out to anyone who 'fancies having a go' could be disasterous.
Just my 0.0383871 Euros worth!