PMG and cyrillic at WebUI

[Kizu]

Good day.

Got some troubles with PMG WebUI and Cyrillic characters.
Now tested at our production pmg/6.3-4(with subscription) and fresh installed 6.3-4(without subscription).
This problem was in the past, but we decided report it now.

Spoiler: Local Users and Cyrillic

root@mx01:~# locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
root@mx01:~# cat /etc/pmg/user.conf
#Unix Superuser
root:1:0::root:::::

root@mx01:~# cat /etc/pmg/user.conf
#Unix Superuser
root:1:0::root:::::
#Коммтест
t.test:1:0:$5$6R9KWT3F$9VtVsgEqGdusORJoR/nz/KJIecTm4SmTUQrywDDbih0:admin:t.testov@foobar.local:Тест:Тестов::

But in UI incorrect symbols

Spoiler: Mail Filter Action Objectes and Cyrillic

Double click at this object after that

Browser debugger gave me that

Spoiler: LDAP Users and Cyrillic

Created with pmg gui
# cat /etc/pmg/ldap.conf
ldap: ActiveDirectory
    comment AD domain
    server1 10.205.255.123
    accountattr sAMAccountName
    basedn DC=domain,DC=ru
    binddn ldap_user@domain.ru
    bindpw *******
    mailattr mail,proxyAddresses
    mode ldap
    server2 10.205.255.223
    verify 1

Spoiler: Checking receiver in LDAP DB and Cyrillic

When we got letter with wrong receiver (first letter "e" not latin - cyrillic "e.dedede@***") process pmg-smtp-filter crashed and letter stored in queue.

And in log got that

дек 01 13:48:11 mx02 pmg-smtp-filter[1035]: Wide character in subroutine entry at /usr/share/perl5/PMG/LDAPCache.pm line 638.
дек 01 13:48:11 mx02 pmg-smtp-filter[1035]: WARNING: Wide character in print at /usr/share/perl5/PMG/SMTP.pm line 48.
дек 01 13:48:11 mx02 pmg-smtp-filter[1035]: fast exit because of errors (free 310657024 bytes)
дек 01 13:48:11 mx02 postfix/lmtp[31158]: D445F60A42: to=<е.deli************>, relay=127.0.0.1[127.0.0.1]:10024, delay=348223, delays=348223/0/0.01/0.09, dsn=4.4.0, status=deferred (host 127.0.0.1[127.0.0.1] said: 451 4.4.0 detected undelivered mail to <е.de*** (in reply to end of DATA command))

As I understant it can not check receiver using imported ldap db.
/usr/share/perl5/PMG/LDAPCache.pm

sub mail_exists {
    my ($self, $mail) = @_;

    my $dbh = $self->{dbstat}->{mails}->{dbh};
    return 0 if !$dbh;

    $mail = lc($mail);

    my $res;
    $dbh->get($mail, $res);   <== 638 line
    return $res;
}

 

[Stoiko Ivanov]

We already have a bugzilla entry for a related problem: https://bugzilla.proxmox.com/show_bug.cgi?id=3005
You can subscribe to the issue there to get notified as soon as there is some progress.
Thanks for reporting !

 

[Kizu]

Thanks, but this problem, as I can see, not only with ldap. With user-management(first spoiler), with mail filter(second spoiler) for example.
Sometimes /api2/json/* returns not utf encoded characters

https://10.205.255.2:8006/api2/json/access/users

{"data":[{"username":"root",
          "expire":0,
          "enable":1,
          "comment":"Unix Superuser",
          "userid":"root@pam",
          "realm":"pam",
          "role":"root"},
         {"comment":"ÐоммÑеÑÑ",
          "enable":"1",
          "expire":0,
          "username":"t.test",
          "realm":"pmg",
          "email":"t.testov@foobar.local",
          "role":"admin",
          "lastname":"ТеÑÑов",
          "firstname":"ТеÑÑ",
          "userid":"t.test@pmg"}]}

https://10.205.255.2:8006/api2/extjs/config/ruledb/action/disclaimer/27_40?_dc=1606821326789

{"success":1,"data":{"otype_text":"Disclaimer",
                     "editable":1,
                     "info":"Тест002",
                      "id":"27_40",
                      "ogroup":27,
                      "receivertest":0,
                      "descr":"disclaimer",
                      "otype":4009,
                      "name":"Тест001",
                      "disclaimer":"ТеÑÑ003"}}

Is that already known?



UPD:
Found bug with Deferred mail in queue "Modified: 2019-11-20"
https://bugzilla.proxmox.com/show_bug.cgi?id=2465

 

[Stoiko Ivanov]

UPD:
Found bug with Deferred mail in queue "Modified: 2019-11-20"
https://bugzilla.proxmox.com/show_bug.cgi?id=2465

exactly - we have adapting PMG to being able to handle non-ascii (or rather non-perl-native) encodings on our roadmap, and plan on tackling that.

Without guarantee this could be addressed before PMG 7.0

 

[Kizu]

Thanks. Will wait.

 

[Kizu]

Good afternoon! Any news on non-ascii characters and the above issues?
https://bugzilla.proxmox.com/show_bug.cgi?id=2465
https://bugzilla.proxmox.com/show_bug.cgi?id=3005

We are using pmg7, and issue is still exists.