Upland OL User community

Datamapper script Charset UTF-8 encoding problem with AJAX Request

Hello there


Datamapper createHTTPRequest does not encode POST request properly to UTF8.

I did stumble over the following problem (code below):

In my datamapper i send some data to an external server with json data.
When my json contains special characters (like ö,ä,ü – I’m Swiss, so thats not that special after all) the server receives the data, but the encoding is wrong.

Wrong result

php json_decode results in:

JSON_ERROR_UTF8 Malformed UTF-8 characters, possibly incorrectly encoded

My guess, whats happening here:

When javascript is used on proper webpage, the encoding is defined by the page and internally javascript uses UTF16 (I think). And its transparently handled by the engine itself.
But here i think that the request just sends the UTF16 string to the server as-is and does not properly convert it to UTF8 encoding. Xmlhttprequest has to use UTF8 encoding acc. to the docs.

A (bad?) fix

I did find a solution when i use utf8.js lib [2] to encode the extracted strings “again” the json seems to be valid. But i dont know if i can trust this library at all its very old.

Version of PPC

I do use the quite old Version 2019.2. Maybe there was a fix in the newer versions?

My code datamapper extraction script field (reduced to the core part):

var payload = {
  "uuid": "75a567ef-7796-4a6f-a0d2-133df059ae7d",
  "Field": record.fields['FieldWithSpecialChars'], // This doesn't work but ': "ä"' does not work either

var myJSON = JSON.stringify(payload);

var request = createHTTPRequest();
var my_url = "https://webhook.site/xxxxx";  // To check the request

request.setRequestHeader("Content-type", "application/json");


[1] Datamapper createHTTPRequest() - Bad Request: Received a GET request with a text/plain; charset=UTF-8 Content-Type
[2] https://mths.be/utf8js


Do you think you could open a case for this with our Support team? I’d be interested in taking a closer look at it but the forum here isn’t quite appropriate because we would probably need the resources to recreate the problem, as it may have an environmental or data specific component to it. Please mention my name (Yannick Fortin) saying that I sent you. :wink: