Text encoding

JavaScript strings represent Unicode characters (internally encoded in UTF-16). Care must be taken when converting a Unicode string to or from a sequence of 8-bite bytes, such as a file on disk (File class) or a byte array in memory (ByteArray class).

The Codecs supported by scripting API can be classified as follows:

Codec

A codec (short-hand for "code-decode") specifies the text encoding used to represent a Unicode string as a sequence of 8-bit bytes. Various codecs are in use throughout the industry.

The utility module refers to supported codecs through predefined names. To specify a codec in a function argument, use one of the following strings:

• Apple Roman

• Big5

• Big5-HKSCS

• CP949

• EUC-JP

• EUC-KR

• GB18030-0

• IBM 850

• IBM 866

• IBM 874

• ISO 2022-JP

• ISO 8859-1 to 10

• ISO 8859-13 to 16

• Iscii-Bng, Dev, Gjr, Knd, Mlm, Ori, Pnj, Tlg, and Tml

• JIS X 0201

• JIS X 0208

• KOI8-R

• KOI8-U

• MuleLao-1

• ROMAN8

• SHIFT-JIS

• TIS-620

• TSCII

• UTF-8

• UTF-8BOM (see byte order marker below, does NOT work for ByteArrays)

• UTF-16

• UTF-16BE

• UTF-16LE

• UTF-32

• UTF-32BE

• UTF-32LE

• Windows-1250 to 1258

• WINSAMI2

In addition the utility module supports the following special codecs (See Hex and Base64 below for more information):

• Hex (corresponds to xsd:hexBinary)

• Base64 (corresponds to xsd:base64Binary and SOAP-ENC:base64)

• toHex, toBase64

• fromHex, fromBase64

Conversion errors

When converting from a byte sequence to a text string, byte sequences that do not match the requirements for the codec are replaced by the Unicode representation of a question mark ("?").

When converting from a text string to a byte sequence, Unicode code points that cannot be represented as a byte sequence using the codec are replaced by the ASCII representation of a question mark ("?").

Byte order marker (BOM): Input (from a byte sequence to a text string)

Regardless of the specified codec, the utility module automatically recognizes a UTF-16 BOM and when found, the codec is automatically adjusted to the appropriate UTF-16 variant. For example, if the codec argument is set to "UTF-8", the utility module will correctly read UTF-8, UTF-16BE and UTF-16LE.

var ba = new ByteArray("un garçon", "UTF-16");  
var bastr = "";
for(i = 0; i < ba.length; i++){   
    bastr += ba.getAt(i) + " ";  
}  
job.log(1, bastr); //starts with 255 254 (UTF-16LE) or 254 255 (UTF-16BE)  
var str = ba.toString("UTF-16");  
job.log(1, str); //"un garçon" 

Byte order marker (BOM): Output (from a text string to a byte sequence)

When the codec is set to one of the UTF-16 variants, the utility module always outputs an appropriate 2-byte UTF-16 BOM at the beginning of the data.

When the codec is set to UTF-8, the utility module does not output a BOM. This is the standard behavior expected by most applications.

var ba = new ByteArray("sample string", "UTF-8"); 
var bastr = "";
for(i = 0; i < ba.length; i++){
    bastr += ba.getAt(i) + " ";
}  
job.log(1, bastr); //starts with 115 (73h:'s') 97 (61h:'a') 109 (6dh:'m')
var str = ba.toString("UTF-8");
job.log(1, str); //"sample string" 

When the codec is set to UTF-8BOM, the utility module outputs a 3-byte UTF-8 BOM at the beginning of the data (this is just a marker to indicate UTF-8 since there are no byte-order issues in this encoding). Be careful when using this codec, because many applications are not able to correctly interpret a UTF-8 BOM. This encoding is not supported for ByteArrays.

//UTF-8BOM use case
//create a sample file
var path = job.createPathWithName("file.txt");
var f = new File(path, "UTF-8BOM");
f.open(File.WriteOnly);
f.writeLine("This is a line.");
f.close();
job.sendToSingle(path);//sends file to single outgoing connection
//result in hex starts with ef bb bf (BOM) 54 68 ...

Hex and Base64

Converting between a String and a ByteArray with any of the codecs (except Hex and Base64) is always unambiguous: the String contains the Unicode code points, the ByteArray contains the 16-bit or 8-bit representation.

With a Hex or Base64 codec, the situation is problematic because:

• Both decoded and encoded sides can be represented in 8 bits

• Following the logic that "the encoded side is in the ByteArray" (as with the other codecs) contradicts the frequent use case where "the encoded side is in a String" (for example, part of an XML stream).

To resolve this issue, the 10 release introduces the following directional codecs:

For compatibility reasons, the unidirectional codecs Hex and Base64 can still be used:

• Hex/Base64 will operate as toHex/toBase64 when the target is a ByteArray

  var ba = new ByteArray("sample string", 'Hex'); //toHex
  var bastr = "";
  for(i = 0; i < ba.length; i++){
     bastr += ba.getAt(i) + " ";  
  } 
  job.log(1, bastr); //starts with 55 (7 ASCII) 51 (3 ASCII): 73 (115dec:'s')

• Hex/Base64 will operate as fromHex/fromBase64 when target is a String.

  //declaration of ba see above
  var str = ba.toString('Hex'); //fromHex  
  job.log(1, str); //"sample string"