Reading text file with Asian fonts in Codesys results in garbage

[AlfredoQuintero]

Hello. I have a problem which may be common in this part of the world. My CoDeSys program reads data from a text file that I generate from the program itself with a system call for Windows text console in the form of "dir <PATH> /B > <PATH>dir_list.txt", which generates a text file with the filenames in that directory, separated by line feed characters. If the filenames are pure ASCII everything is fine. But if the filenames contain Asian fonts (in this case Japanese fonts), then what I read from the program is garbage display, as I show below.
Before I tell my customer that he can only use filenames in ASCII so I can continue with the development (and make him get quite grumpy), I though I would fire a question to the forum, to see if someone would be familiar wit this problem and could suggest a fix, in which case I will be as ever very grateful.
Thanks.

 

[JeremyM]

It’s Unicode. Characters are represented by two bytes rather than one in ASCII.

 

[AlfredoQuintero]

Yes, thanks I though this is where the problem comes from. What I do not know is how to solve this problem on the CoDeSys program. Codesys has a WSTRING type Unicode. But the file functions only support STRING. Unless there are file functions that I am unaware of that will work with WSTRING.

 

[JesperMP]

Is this in the HMI program ?
Is it a script language ?

My own experience with VBS in Siemens WinCC is that you need to tell the file operations that it must read unicode.

i.e.:

Const Unicode = -1
Const forreading = 1
Set ts = fso.OpenTextFile(str_Path_and_File, forreading , False , Unicode)
SmartTags("mytextstring") = ts.readline

 

[BryanG]

Don't ask me questions on this, I don't use Codesys, but I am interested in understanding how to use Unicode, so I did some searching:

https://forge.codesys.com/forge/talk/Engineering/thread/3608e375a7/

https://faq.codesys.com/display/CDSFAQ/SysFile:+Saving+Files+in+UTF-8+Format

https://forge.codesys.com/forge/talk/Visualization/thread/aac699818f/
Particularly this bit:
Interesting. When the first two characters of a text file are ÿþ, that is a notice to the text editor that it is a UTF-16 LE encoded file (is unicode). More details on Wikipedia.

 

[JesperMP]

Alternatively, do the WSTRING operations in the PLC program instead of the HMI program.

 

[AlfredoQuintero]

Is this in the HMI program ?
Is it a script language ?

My own experience with VBS in Siemens WinCC is that you need to tell the file operations that it must read unicode.

i.e.:

Const Unicode = -1
Const forreading = 1
Set ts = fso.OpenTextFile(str_Path_and_File, forreading , False , Unicode)
SmartTags("mytextstring") = ts.readline

I am using CoDeSys in Windows so the PLC and the HMI are both contained in the PLC application.

 

[AlfredoQuintero]

Don't ask me questions on this, I don't use Codesys, but I am interested in understanding how to use Unicode, so I did some searching:

https://forge.codesys.com/forge/talk/Engineering/thread/3608e375a7/

https://faq.codesys.com/display/CDSFAQ/SysFile:+Saving+Files+in+UTF-8+Format

https://forge.codesys.com/forge/talk/Visualization/thread/aac699818f/
Particularly this bit:
Interesting. When the first two characters of a text file are ÿþ, that is a notice to the text editor that it is a UTF-16 LE encoded file (is unicode). More details on Wikipedia.

BryanG, thanks for these articles. I think I know what I need todo. The way I am generating the text file is the problem. I will generate the text file as indicated in one of the articles. Too late here in Japan. I hope I can get this fixed tomorrow.

 

[JesperMP]

The MID function you use only work on STRINGs, not on WSTRINGs.
Take a look at the suggestions posted here:
https://forge.codesys.com/forge/talk/Engineering/thread/4c1aa23dd3/

 

[AlfredoQuintero]

The MID function you use only work on STRINGs, not on WSTRINGs.
Take a look at the suggestions posted here:
https://forge.codesys.com/forge/talk/Engineering/thread/4c1aa23dd3/

Thanks a lot Jesper: With the info in this article and the ones pointed to by BryanG I think I can come-up with a solution.
It is late here, so I will work on this tomorrow.

 

[AlfredoQuintero]

Gentlemen, In think I have seen the light and it costs only EUR 50.

This morning I was discussing with a potential customer the possibility of using CoDeSys for a project and to my astonishment, while I was showing them the CoDeSys website, I discovered a "software package “IIoT Libraries SL” contains libraries for “IIoT communication” and libraries for reading and writing of data structures." I have just purchased this package. It has libraries for reading and writing CSV files an INI files. I am interested in the CSV file funtion. Just by reading the datasheet of the product I immediately realized that I would need weeks to code these functions myself. Will give them a try if anyone is interested, I can post some feedback.

https://store.codesys.com/en/iiot-libraries-sl.html