[Solved]Using special characters in RMXP scripts

[Moonpearl 32]

Hi guys,

 

Here's a quite technical question, but I'm having trouble using special characters in RMXP scripts. What I want to do is a script to allow different languages for a project. Since I'm planning to use it on my own project to translate it to French, namely, I'd need characters such as accented vowels (é, è, à...). I know RMXP can handle them because it is possible to input some via the Show Message event command, and they display properly in-game.

 

The thing is, no matter what I try, I can't get such characters to display properly with the source string encoded inside a script. The best I get is a blank character, like the string was empty.

 

What I tried:

• Character as is between quotes: issues a syntax error
  

"è"
'è'

​

• Escaped character: returns an empty string with double quotes and only a backslash with simple quotes
  

"\è"
'\è'

​

• Character hex code: returns an empty string with double quotes and the code itself with simple quotes
  

"\xC3"
"\303"

• Conversion from an integer: returns an empty string
  

0xC3.chr
195.chr

• Packing array into a string: returns an empty string
  

[0xC3].pack('C')
[195].pack('C')

• Any of the above using the print, puts, and inspect methods (in case some miracle would happen): returns the exact same contents
  
• Using a Win32API which reads characters directly from keyboard input: pressing the 'è' key returns a blank string (whereas all non-special characters work perfectly fine, of course)
  

So I think I've done it all. I really wonder how I can't get these characters to show, especially since when I ask a Show Message command how its 'è' is encoded, it returns "\xC3" (one of the encodings I tried with no luck).

 

So, if any of you has got some sort of solution or even a clue, I'm all ears.

Edited July 5, 2012 by Moonpearl

[ForeverZer0 44]

Does the font the game is using support those characters?

[kellessdee 48]

In this case, I'm not sure if the font in question is an issue (it could be, but let me explain)

 

The first clue:

Character as is between quotes: issues a syntax error

"è"
'è'

​

After trying to add those characters to an rpg maker project, it was successful (character was displayed, no syntax errors).

 

Now, as far as I know, the ruby interpreter (using C's standard I/O facilities) reads a script byte for byte, and really only understands ASCII character encoding (or, more accurately, any encoding that is a direct superset of ASCII; such as UTF-8). So, looking at what you have here:

0xC3.chr

C3 or 195 does not exist in any ASCII character set (nor UTF-8). "è", in UTF-8 is C3 A8 or U+00E8 (2-byte char) (hence why the attempts to use exact hex values displayed blank characters)

 

However, this (in my opinion) still doesn't really explain what's going on. I can't be certain (I don't really know HOW RPG Maker saves scripts/event data exactly), but I have a theory.

 

I'm assuming scripts are encoded as UTF-8. I know you are from France, and I am going to assume your window's locale settings match this fact...correct? Windows likes to hijack certain aspects of applications sometimes or sometimes applications purposely give control to Windows, and potentially this is messing with the input character set in RPG Maker's script editor. If ruby tries to read a character it doesn't understand, it's going to cry (and throw a syntax error).

 

I still find it strange that the "Show Text" event command has no issues with the character...my theory here - assuming my previous theory was correct - is that, since event data is directly written to a marshal format file (scripts are compressed then written to marshal format), I guess RPG Maker does some work for you, using a readable character instead... It still doesn't make much sense in my head, but I don't know, I don't have RPG Maker's source code.

 

Here's what you can try:

1. Try copying+pasting this --> "è" into the script editor (rather than directly inputting said character) and see if it works.

2. Try changing your windows' localization settings to US, and input the character (and try copying+pasting if that doesn't work)

 

If none of that works, my theories are probably incorrect and you might have to resort to storing the string data in an external file (encoded as "UTF-8") and read the text out of the file.

 

Other than that, I'm as stumped as you are.

 

Hope this helps;

[ForeverZer0 44]

RMXP sets the KCODE to UTF-8, so it shouldn't be that it is reading the string in the wrong encoding.

Later versions of Ruby are more "automatic", and has the String#force_encoding method for when you need something else.

 

If you want to double-check, simply put "p $KCODE" somewhere in the editor and run it. If its not "UTF8", then set to be so. Simply place $KCODE = "UTF8" in the first line of the editor.

[Moonpearl 32]

What you said made me think of something, Kellessdee. Actually I don't use RMXP's script editor anymore since I do everything using the Script Manager system. I tried to input strings containing special characters in the script editor and it works fine. So, actually, the issue is that those characters aren't recognized when loaded from a file, certainly because it is read byte-to-byte as you suggested. I guess it is a matter of file encoding because scripts from RMXP's script editor are compressed using Zlib::deflate before being written ro Scripts.rxdata. So my issue is half-solved, even though it kinda pains me to be forced to resort to the script editor (I would like to use the script in both projects with multiple languages and a language editor, and the Script Manager stuff sounds just perfect for this). I'll manage with that for now: however, if you have further insight regarding how to use special characters in external Ruby scripts, I'm still interested.

[ForeverZer0 44]

Just make sure that all files are using UTF8 encoding for reading and writing. I ran into this problem when developing Gemini.

[Moonpearl 32]

You got it, ForeverZer0, thank you so much.