Could LaTeX and ASCII Math be speech navigation options?

[joanmarie]

An Orca user wants to navigate in MathML content rendered in LaTeX and/or ASCII Math.

I realize that I can obtain the LaTeX/ASCII Math from MathCAT. Orca already supports those options for braille and for copying. The reasons why I'm requesting this as a MathCAT navigation option are:

• Orca's MathCAT implementation turns navigation in MathML expressions completely over to MathCAT, treating those expressions as black boxes. Being able to do so fills me with joy. 😄 I don't want to move away from that. But even if I were ok with it:
• Unlike other screen readers, Orca does not have a virtual buffer or any notion of virtual objects. The user is always in the actual document. The user's location is always a valid/real AT-SPI accessible object with a valid character offset. If I need to do this on my end, I'll need to create some sort of virtual object and take navigation over from MathCAT.

So... Any possibility of this happening on your end? (Please and thank you in advance.)

[RastislavKish]

@joanmarie thank you for working on this!

My original idea was that Orca could treat Math objects like span elements, so the caret could seamlessly navigate through the page as well as the content of math elements, with the actual content being a textual representation of the formulas, either verbal-one or a LaTeX / AsciiMath / etc.

However, if this is not technically feasible, as you mention, I think it might be easier to solve this on the browser side. I could write a script turning all math elements into spans and make them receive key presses, so when I press space on one it triggers the formula exploration.

Do you think it could be feasible to expose "Open MathML in a math browser window" in the Orca plugin API?

I think this would be a better and cleaner solution, as needing to open the explorer would sort of beat the point of having everything in text. And text selection would work properly too.

[joanmarie]

@RastislavKish: Orca feature requests should probably be filed and discussed in Orca's issue tracker. Having said that, either Orca would implement this to-be-specified "math browser window" command or someone could create a user extension. If someone wants to create a user extension, then I would need the specification of what that extension needs.

[RastislavKish]

@joanmarie

Orca feature requests should probably be filed and discussed in Orca's issue tracker.

Sure, that's why I wonder whether you'd see this path feasible, we can sort out details afterwards on a proper place. :)

someone could create a user extension.

I will, that's not an issue thanks to the Orca plugin system. My only question is, whether Orca could provide a function for the plugins, so an user extension could give MathML code to Orca and the screenreader would display the same dialog as it does currently when a shortcut is pressed. I believe this could be the simplest solution for everyone involved, and is also functionally the most effective-one.

[joanmarie]

@RastislavKish: I'm afraid I'm still confused. To be clear, right now when you press Orca+Alt+M to activate MathCAT, you're not in a dialog. Does that in any way change what you're suggesting?

[RastislavKish]

@joanmarie yes 😃

Sorry for the confusion on my side, usually screenreaders just pop up a dialog when an expression is activated and the dialog takes care of everything math related. And I thought Orca was doing the same.

If the math browser is just a keyboard override, then a hypothetical browser extension would need to:

• Swap the focused span element for a MathML element, so Orca would have something to navigate in
• Contact an external plugin in Orca, that the user requested exploring a Math Expression X in detail
• Now what. The plugin could focus the math expression and activate the command for invoking the math browser
• Then, the Orca plugin would wait while the math browser is active. When it's not anymore, it would let the browser extension know the user has finished browsing the formula
• And finally, the browser extension would swap the math element back for a span object

This is all assuming orca needs a math element on the page to switch into math browsing mode, correct? that sure complicates things a bit, because the browser extension can no longer just send the MathML code and mind its own business, but I think it is something that could be solved.

I'm not yet sure on the specific API the Orca plugin system would need to provide in order for this to work, it would ideally be something that:

• Is easy for you to implement and requires no architectural changes to Orca's codebase,
• Is not specific for a single plugin, but is rather a general API for accessing the screen content / monitoring the math browsing etc. e.g. a generally useful thing.

If you can imagine something like this working, we can sort out the implementation in a separate discussion. If there are considerable structural barriers for implementing this, I will need to think of a different approach.

[joanmarie]

What you describe sounds more complicated to me than MathCAT giving us the LaTeX/ASCII Math. But maybe I'm still missing something.

When you enter MathCAT mode, Orca:

• suspends its caret navigation and structural navigation commands
• un-suspends its command bindings for MathCAT: https://daisy.github.io/MathCAT/nav-commands.html

That's pretty much it. It's just like caret navigation, structural navigation, and table navigation. No dialogs, you're still in the web browser. The only difference is that in those (caret, structural, table) navigation modes, Orca has to do the work to figure out where to move and what to say. With MathCAT, Orca just calls libmathcat_py.DoNavigateCommand() with the named command. The return value is the thing to speak. It's really quite awesome.

As for being able to navigate in the expressions like they were a span, you can already do that: In MathCAT mode, you can use Shift+Up/Down to change the navigation mode to character. Then you can use left/right to navigate by character. MathCAT keeps track of the current location and gives Orca the thing to speak.

My understanding of your original request was that you wanted to be able to navigate as I described just now, but also be able to do so with the expression represented as LaTeX or ASCII Math. And, independent of your personal request, I think that would be a nice feature to have. But I think it should come from MathCAT, so when you press right, it updates your location in the LaTeX string and gives Orca the next LaTeX character to speak.

[RastislavKish]

@joanmarie your understanding of my feature request is pretty correct, except one important detail. The most important reason why I want this feature is being able to work with formulas without switching to expression browsing mode.

I believe it's very convenient and useful, especially if you have a complicated formula you need to familiarize yourself with. However from my experience of university math, it's often pretty fast to just skim a math expression markup from left to right instead of having to walk down a tree, the symbols used by AsciiMath are very efficient and the tree will construct in my head automatically. A bonus point is the markup is compatible with my notes and writing, so I can copy things around and they will fit.

This means I could simply convert all math objects into AsciiMath using MathCAT. That would be doable, however at the same time I'd like to keep the option to browse a formula as a tree, because it can be a very useful option.

From there comes my idea of swapping the objects. If Orca instead used MathCAT commands for moving inside the math expressions, that could work, however it would introduce two problems:

• We would lose integration with the normal caret navigation, i.e. I'd either need to use a shortcut to activate MathCAT, or MathCAT would get autoactivated, however not letting me navigate in the document, as was the case with original Orca's MathCAT implementation.
• And text selection would not work, although I'd imagine this to be less of a problem given MathCAT does have functions for copying things.

I myself do not know yet which approach to the issue would be optimal. When I last studied math on a university, I had AsciiMath and nothing else, so for me this is a pretty interesting experiment. Although I'd also love not to put the implementation burden of my experimentation on the shoulders of others, hence I'm thinking how to achieve this functionality with as little Orca / MathCAT changes as possible.

[NSoiffer]

In case it is still relevent, supporting ASCIIMath and LaTeX input are on my plan. I thought that there were issues for them, but I don't see them. ASCIIMath is probably very easy because it is simple and there appears to be a Rust crate that does this already. So assuming it works, it is just a matter of adding another interface function that takes ASCIIMath as input. The interface would then need to be propogated to the other language interfaces, but that's straightforward.

@RastislavKish, @joanmarie: please comment on whether you think this should be a prioity for MathCAT or whether your thinking has changed.

LaTeX is harder because there is not a fixed definnition and lots of LaTeX packages need to be supported to have something truely useful. I'm dubious the existing Rust crates except a wide range of packages.

[NSoiffer]

@RastislavKish: MathCAT has a character navigation mode, so you can walk through the equation by character (actually words). But you are navigating the displayed equation, not the underlying ASCIIMath or LaTeX.