The core algorithms are pretty creaky as well. Knuth did a brilliant job coming up with efficient algorithms that could do a fairly good job at breaking lines/paragraphs/pages and could handle book-length documents even on 70s-era hardware, but they've had minimal improvement since then. Figure placement in particular is pretty bad, especially for two-column, relatively short conference papers, which is currently a frequent TeX use case. And all of those things are really hard to exercise any control over, with black-magic parameters that often don't have obvious effects ("goddammit, why won't it put the figure at the top of page 5?!").

LuaTeX does seem to be experimenting with some improvements in these areas though.

All true. Also, the way indexes, table of contents, bibliographies and cross references work in LaTeX are unholy, fragile hacks. Foot- and endnotes could use some TLC. And on 2011-era hardware we really should have some approximation of globally optimal page breaking.

Ugh, and fonts and character sets. Do you still have to put in magic incantations like

    \usepackage[T1]{fontenc}
    \usepackage[latin1]{inputenc}

to get working hyphenation with wacky characters such as æ, ø, and å?

LuaTeX does seem to be experimenting with some improvements in these areas though.

About when I stopped paying attention to TeX and LaTeX many years ago (2002-ish?), there was some effort under way to rewrite all of TeX in Java. Did that go anywhere?

> About when I stopped paying attention to TeX and LaTeX many years ago (2002-ish?), there was some effort under way to rewrite all of TeX in Java. Did that go anywhere?

I'd say no. See http://extex.org/

I don't have any insight at that level of abstraction. Figure placement like a bunch of other things has required me to hack around to get my desired result. Is there any newer research on these algorithms that could make it better given much better hardware?

As for LuaTex I wasn't able to find any examples of actual markup written in it. I would expect that for the actual markup you'd want a language with a bigger and more flexible grammar so you could build DSLs. Ruby seems particularly good at that kind of stuff, but I'm biased there.

LuaTeX has the same macro language as the other TeX engines (pdftex and it's ilk). The true power of LuaTeX comes into play when you use Lua scripting. I am not sure if ConTeXt has more advanced figure placement algorithms due to LuaTeX. LaTeX for sure has the same algorithms as before.

But then is the Lua scripting at the user level? Can it be used in the building of documents? Or is it a scripting language for the underlying engine, invisible to the end-user?

Incidentally I went looking for what pure TeX actually looks like as most people say TeX when they really mean LaTeX. Here's Knuth's TAOCP Volume I errata:

http://www-cs-faculty.stanford.edu/~uno/err1.textxt

Talk about Perl being line noise... I wonder if Knuth has ended up being the only person to actually write pure TeX.

I wrote a LaTeX macro package for my university's PhD thesis format, and like a lot of such packages, the bulk of the work was in TeX rather than LaTeX. It's...well, I'm struggling to find the appropriate word that means "awful, but sort of comfortably awful."

Like a lot of things, if you're handed a finished product, it looks like random noise. If you could watch it being written, it pretty much makes perfect sense. It's a hideous syntax, and sometimes I still wake up in a cold sweat with long strings of '#' signs running taunting me from the corner of my eyes. But I can't say it was all that daunting to write, although I suspect my TeX code was highly non-idiomatic.

You can do Lua scripting a the user level (\directlua{...}). But you have to know the Lua interface. What purpose is having Lua inside your document? Do some calculations? Yes, that's easy. Do complex algorithms for placing floats? Then you need to replace a lot of TeX/LaTeX code anyway and then I wouldn't speak of user level. What you can do easily from inside Lua: shuffle nodes (nodelists) around. Everything you see on a page is represented by a node. So you can reposition anything. That requires a lot of knowledge of the internal representation of nodes and the LuaTeX API, but once you grok it, it is much fun to do these things.

And no, Knuth is definitely not the only one using plain TeX. Yes, it is a niche.

I did a few times. It's actually easier to make something look exactly the way you want it to look in plain TeX than fight with some poorly documented LaTeX package.

I'm sure its possible and useful for specialty cases but would you want to write TAOCP in it?

My experience of using LaTeX compared to WYSIWYG is that it's easier to focus on the text in the editor instead of fighting with the UI interface but it then requires some chasing around package bugs and compile errors. In total it's clearly a net positive for me.

Writing pure TeX would seem to push the tradeoff quite a bit, having so much distracting stuff around the text that requires a totally different mindset than the text being written. It was probably designed for a world where typesetting was done as a final step before publication, potentially by someone else.

The idea is that you put together the few macros that you need and then the book is mostly tag-free. IDK you can find the source to the TeXbook on CTAN, see if it seems too noisy to you (I don't remember).

>I would disagree that the base is wonderful and it's the editors that are bad

OK. (As I said elsewhere...) When I said the language was fine, I meant that the language as a means for the user expressing an idea was fine, not that there aren't problems at a deeper level with the language. I'm a physicists, not a programmer. Nevertheless, the limitations of the given language could be massively ameliorated with a powerful editor.

>At least in LaTeX...

Sorry, I wasn't clear. I wasn't distinguishing between TeX and LaTeX. Everyone I know in Academia used LaTeX. I was just using TeX as a catch all to distinguish from other typesetting options.

> The features you mention would be needed to put it in front of less geeky users...

There's no reason I should have to go on the internet to find out how to use columns. Or sub figures. Or appendices. It should just work.

I'm not a less-geeky user who needs to be coddled (unless by "less geeky" you mean 99.99% of users). I just want to write up my research, and I don't want to have to think about typesetting theory.

>There's no reason I should have to go on the internet to find out how to use columns. Or sub figures. Or appendices. It should just work. >I'm not a less-geeky user who needs to be coddled (unless by "less geeky" you mean 99.99% of users). I just want to write up my research, and I don't want to have to think about typesetting theory.

I agree that those things should be easy to find. What I meant is that for the current user base those are nice to have but not essential. I have long since accumulated enough examples of the type of documents I write that I can quickly do the basic things just by copying them from somewhere else. I assume that people that write scientific articles already have working examples of two columns, sub figures and appendices that they just copy when they do a new article.

If you wanted to make LaTeX much more widely use then you'd need all the goodies you mentioned. But to even make those possible the environment under them needs to be much more consistent and robust than it is now. Something like LyX could work well if manual intervention in the typesetting wasn't so needed.

I guess we're disagreeing about what's essential. At this point, I prize robustness more than power. I'd be pretty happy if LaTeX would just work smoothly. Additional capabilities would be gravy.

I'm confused. You're the one that wanted extra features in the environment. All I hoped for would be for the typesetting engine to be more robust and modules to work well together. All I want are bugfixes.