From: Jonathan Fine on
Wes James wrote:
> On Wed, Feb 24, 2010 at 5:03 AM, Jonathan Fine <J.Fine(a)open.ac.uk> wrote:
>> Hi
>>
>> Does anyone know of a collection of regular expressions that will break a
>> TeX/LaTeX document into tokens? Assume that there is no verbatim or other
>> category code changes.
>
> I'm not sure how this does it, but it might help:
>
> http://plastex.sourceforge.net/plastex/sect0025.html

Thanks, Wes. I'm already using PlasTeX

It handles changes of category codes, which makes it over the top for
what I want to do. In addition it is a fairly large complex
application, and sadly it's not at all easy to use just a part of the
code base.

There's been more discussion of this thread on comp.text.tex (which is
where I set the follow-up to).

--
Jonathan