On 2020-07-19 03:46, Ajith R wrote:
I seem to recall that he puts Perl at the top of the heap, and notes that Perl compatible regular expressions (PCRE) are available via libraries in other programming languages.Thanks for confirming that I didn't make a wrong choice. Programs that claim to use PCRE don't support everything that PERL does.
My use of regular expressions is primarily via grep/ egrep and Perl. I am curious to see what Erlang offers.
I wanted to clean many documents (Wikipedia dump) to analyse the Malayalam content. As I was not comfortable with scripting, I was looking for some prorgam that could remove the foreign language text from the files. As, I could find none that could do the job, I had to use a Perl script with the line below (among others)
s/[^\p{Block: Malayalam}\p{Block: Basic_Latin}\p{Block: General_Punctuation}\s]//g; # remove characters outside the specified unicode blocks.
As of now, the simple substitute command of perl is sufficient for my requirements. Even that one command appears powerful compared to others.
That sounds like you need lexing and parsing, followed by your desired processing. In simple cases, Perl regular expressions can accomplish two or three of those tasks. But as complexity grows, you will need more and more code. Writing a lexer/ parser in any language is a non-trivial task. I have used the Perl 'LWP' library to parse HTML 4 pages, but I have not tried to parse HTML 5 and/or Wikipedia pages. I would look for a library:
https://metacpan.org/search?q=parse https://metacpan.org/search?q=wikipediaI spent a little time with Raku (formerly Perl 6). AIUI improving regular expressions was a design goal of Perl 6, and features were added specifically for parsing. There is a book dedicated to the subject. I have a friend who put some time into this area, and he seemed impressed:
https://www.apress.com/us/book/9781484232279 David