Clever Geek Handbook

Converting Unicode characters to Perl?

I want to convert text (Hindi) to Unicode in Perl. I have searched on CPAN. But I couldn't find the exact module / way I am looking for. Basically, I'm looking for something like this .

My input:

इस परीक्षण के लिए है

My expected output:

\u0907\u0938\u0020\u092a\u0930\u0940\u0915\u094d\u0937\u0923\u0020\u0915\u0947\u0020\u0932\u093f\u090f\u0020\u0939\u0948

How do you achieve this in Perl?

Give me some tips.

+3


source to share


4 answers


try it



use utf8;

my $str = 'इस परीक्षण के लिए है';

for my $c (split //, $str) {
    printf("\\u%04x", ord($c));
}
print "\n";
+7


source


You don't really need any module. ord

to extract the char code and printf

to format it as 4 numbers a zero filled hex is more than enough:



use utf8;
my $str = 'इस परीक्षण के लिए है';
(my $u_encoded = $str) =~ s/(.)/sprintf "\\u%04x", ord($1)/sge;
# \u0907\u0938\u0020\u092a\u0930\u0940\u0915\u094d\u0937\u0923\u0020\u0915\u0947\u0020\u0932\u093f\u090f\u0020\u0939\u0948
+5


source


If you only want a simple converter, you can use the following filter

perl -CSDA -nle 'printf "\\u%*v04x\n", "\\u",$_'
#or
perl -CSDA -nlE 'printf "\\u%04x",$_ for unpack "U*"'

as:

echo "इस परीक्षण के लिए है" | perl -CSDA -ne 'printf "\\u%*v04x\n", "\\u",$_'
#or
perl -CSDA -ne 'printf "\\u%*v04x\n", "\\u",$_' <<<  "इस परीक्षण के लिए है"

prints:

\u0907\u0938\u0020\u092a\u0930\u0940\u0915\u094d\u0937\u0923\u0020\u0915\u0947\u0020\u0932\u093f\u090f\u0020\u0939\u0948\u000a

Unicode with surrogate pairs.

use strict;
use warnings;
use utf8;
use open qw(:std :utf8);

my $str = "if( \N{U+1F42A}+\N{U+1F410} == \N{U+1F41B} ){ \N{U+1F602} = \N{U+1F52B} } # ορισμός ";

print "$str\n";
for my $ch (unpack "U*", $str) {
        if( $ch > 0xffff ) {
                my $h = ($ch - 0x10000) / 0x400 + 0xD800;
                my $l = ($ch - 0x10000) % 0x400 + 0xDC00;
                printf "\\u%04x\\u%04x", $h, $l;
        }
        else {
                printf "\\u%04x", $ch;
        }
}
print "\n";

prints

if( 🐪+🐐 == 🐛 ){ 😂 = 🔫 } # ορισμός 
\u0069\u0066\u0028\u0020\ud83d\udc2a\u002b\ud83d\udc10\u0020\u003d\u003d\u0020\ud83d\udc1b\u0020\u0029\u007b\u0020\ud83d\ude02\u0020\u003d\u0020\ud83d\udd2b\u0020\u007d\u0020\u0023\u0020\u03bf\u03c1\u03b9\u03c3\u03bc\u03cc\u03c2\u0020
+3


source


Since I left a few comments about how other answers might not meet the expectations of various tools, I would like to share a solution that encodes characters outside of the Basic Multilingual Plane as two-screen pairs: "😃"

will \ud83d\ude03

.

This is done with:

  • Encode the string as UTF-16 unsigned byte order. We are clearly choosing entiance. Here we arbitrarily use the big end shape. This creates a string of octets ("bytes") where two octets form one UTF-16 code block, and two or four octets form a Unicode code point.

    This is for convenience and performance; we could also pinpoint the numerical values ​​of the UTF-16 code units.

  • unpack

    inputting a binary string into 16-bit integers that represent each block of UTF-16 code. We have to respect the correct endianess, so we use the pattern n*

    for unpack

    (i.e. 16 bit unsigned integer).

  • Formatting each block of code as escape \uxxxx

    .

As a Perl subroutine, it would look like

use strict;
use warnings;
use Encode ();

sub unicode_escape {
    my ($str) = @_;
    my $UTF_16BE_octets = Encode::encode("UTF-16BE", $str);
    my @code_units = unpack "n*", $UTF_16BE_octets;
    return join '', map { sprintf "\\u%04x", $_ } @code_units;
}

Test cases:

use Test::More tests => 3;
use utf8;

is unicode_escpape(''), '',
    'empty string is empty string';

is unicode_escape("\N{SMILING FACE WITH OPEN MOUTH}"), '\ud83d\ude03',
    'non-BMP code points are escaped as surrogate halves';

my $input = 'इस परीक्षण के लिए है';
my $output = '\u0907\u0938\u0020\u092a\u0930\u0940\u0915\u094d\u0937\u0923\u0020\u0915\u0947\u0020\u0932\u093f\u090f\u0020\u0939\u0948';
is unicode_escape($input), $output,
    'ordinary BMP code points each have a single escape';
+3


source






More articles:


All Articles