Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions differences.md
Original file line number Diff line number Diff line change
Expand Up @@ -469,6 +469,12 @@ Backtalk adds voice commands for settings that Android lets apps change:

Say "find [name]", "focus [name]", "search for [name]", or "locate [name]" (for example, "find play button", "focus search", "locate cancel") to search the screen and move accessibility focus directly to the matching control.

### Voice command scripts API

Backtalk scripts can create custom voice commands:
* **Manifest commands**: Add `voice: 'phrase'` or `voice: ['phrase 1', 'phrase 2']` to any entry in `manifest.commands`. When spoken, Backtalk runs the command with `via: 'voice'`, `event.command`, and `event.remainder`.
* **Dynamic voice API**: Use `backtalk.onVoiceCommand(pattern, callback)` to register exact phrases, fuzzy phrases, or wildcard capture patterns (such as `'search for *'`).

## Screen and brightness

### Brightness reading control
Expand Down
31 changes: 25 additions & 6 deletions docs/scripting.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,8 +248,8 @@ A command is something the user runs. Declare it in the manifest, and give it at

```js
commands: [
{ id: 'unread', title: 'Next unread', gesture: 'swipeUpThenRight', keys: 'backtalk+u' },
{ id: 'summary', title: 'Read summary', menu: true },
{ id: 'unread', title: 'Next unread', gesture: 'swipeUpThenRight', keys: 'backtalk+u', voice: 'next unread' },
{ id: 'summary', title: 'Read summary', menu: true, voice: ['read summary', 'summarize'] },
{ id: 'messages', title: 'Messages', control: true },
],
```
Expand All @@ -258,18 +258,37 @@ commands: [
* `keys` is keys or a list. Keys start with `backtalk`, Backtalk's modifier key, then any of `shift`, `ctrl`, `alt` and `meta`, then a key: `'backtalk+shift+n'`. A second key can follow a comma: `'backtalk+t, s'`.
* `menu: true` adds the command to the Backtalk menu.
* `control: true` adds it to the reading controls. Swiping up or down with the control selected runs the command with `direction` set to `'previous'` or `'next'`.
* `voice` is a voice phrase or a list of phrases that trigger the command when spoken.

Commands need the `input` permission. The user can turn each gesture and key off on the script's page, or assign others. While a script runs, its gestures and keys come before Backtalk's own, except for the gesture assigned to **Turn all scripts on or off**.
Commands need the `input` permission. The user can turn each gesture, key, and voice trigger off on the script's page, or assign others. While a script runs, its gestures, keys and voice commands come before Backtalk's built-in actions, except for the gesture assigned to **Turn all scripts on or off**.

```js
backtalk.onCommand('unread', (event) => {
// event.id, event.via ('gesture', 'keys', 'menu' or 'control'), event.direction, event.node
// event.id, event.via ('gesture', 'keys', 'menu', 'control' or 'voice'), event.direction, event.command, event.remainder, event.node
});
```

`event.node` is the focused item. Return `false` to have Backtalk do what the gesture or keys normally do.
`event.node` is the focused item. Return `false` to have Backtalk do what the action normally does.

`backtalk.bind(command, { gesture })` or `backtalk.bind(command, { keys })` adds a gesture or keys to a declared command, and `backtalk.unbind` removes one.
`backtalk.bind(command, { gesture })`, `backtalk.bind(command, { keys })` or `backtalk.bind(command, { voice: 'phrase' })` adds a trigger to a declared command, and `backtalk.unbind` removes one.

#### Custom voice commands

A script can also register dynamic voice commands directly:

```js
// Exact or fuzzy phrase match:
backtalk.onVoiceCommand('cheer', (event) => {
backtalk.speak('You are doing great!');
});

// Wildcard capture pattern:
backtalk.onVoiceCommand('search for *', (event) => {
backtalk.speak('Searching for ' + event.remainder);
});
```

`onVoiceCommand` returns a function that unregisters the command. Return text or `{ speak: text }` to speak feedback.

### Navigation

Expand Down
1 change: 1 addition & 0 deletions lint.xml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
<ignore regexp="voice_commands_unmuted" />
<ignore regexp="voice_commands_found" />
<ignore regexp="voice_commands_element_not_found" />
<ignore regexp="script_binding_voice" />
<ignore regexp="voice_commands_help_call" />
<ignore regexp="voice_commands_help_call_description" />
<ignore regexp="voice_commands_help_call_control" />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@
import com.google.android.accessibility.talkback.TalkBackService;
import com.google.android.accessibility.talkback.actor.DimScreenActor;
import com.google.android.accessibility.talkback.actor.gemini.GeminiConfiguration;
import com.google.android.accessibility.talkback.scripting.Scripts;
import com.google.android.accessibility.talkback.analytics.TalkBackAnalytics;
import com.google.android.accessibility.talkback.analytics.TalkBackAnalytics.VoiceCommandTypeId;
import com.google.android.accessibility.talkback.contextmenu.ContextMenuItem;
Expand Down Expand Up @@ -330,6 +331,12 @@ public boolean handleSpeechCommand(String command, boolean isPartialResult) {
LogUtils.i(LOG_TAG, "handleSpeechCommand() command=\"%s\"", command);
EventId eventId = EVENT_ID_UNTRACKED;

// Check custom script voice commands
if (!isPartialResult && Scripts.onVoiceCommand(command)) {
analytics.onVoiceCommandEvent(VOICE_COMMAND_RECOGNIZED);
return true;
}

// Calls, and the answers to their questions ("say yes to confirm")
if (callVoiceCommands.handleCommand(
VoiceCommandFuzzyMatcher.normalize(command), isPartialResult)) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,8 @@ interface ScriptHost {
fallback: Runnable,
): Boolean

fun onVoiceCommand(command: String): Boolean

fun menuItems(): List<ScriptMenuItem>

fun readingControls(): List<ScriptReadingControl>
Expand Down Expand Up @@ -246,6 +248,9 @@ object Scripts {
@JvmStatic
fun readingControls(): List<ScriptReadingControl> = host?.readingControls().orEmpty()

@JvmStatic
fun onVoiceCommand(command: String): Boolean = host?.onVoiceCommand(command) == true

@JvmStatic
fun itemActions(node: AccessibilityNodeInfoCompat): List<ScriptItemAction> =
host?.itemActions(node).orEmpty()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -147,11 +147,14 @@ object ScriptInput {

const val GESTURE_BINDING = "gesture/"
const val KEYS_BINDING = "keys/"
const val VOICE_BINDING = "voice/"

fun gestureBinding(name: String): String = GESTURE_BINDING + name

fun keysBinding(keys: Keys): String = KEYS_BINDING + keys.id

fun voiceBinding(phrase: String): String = VOICE_BINDING + phrase.trim().lowercase()

val gestureNames: List<String> = (gestures.keys + fingerprintGestures.keys).toList()

fun isGesture(name: String): Boolean = name in gestures || name in fingerprintGestures
Expand Down
2 changes: 2 additions & 0 deletions talkback/src/main/res/values/strings_voicecommands.xml
Original file line number Diff line number Diff line change
Expand Up @@ -437,4 +437,6 @@
<string name="voice_commands_found">Found %1$s</string>
<!-- Spoken when the find voice command finds no matching control; %1$s is what the user asked for. [CHAR_LIMIT=NONE] -->
<string name="voice_commands_element_not_found">Could not find %1$s on screen</string>
<!-- Shown in a script's list of triggers for a voice command; %1$s is the phrase to say. [CHAR_LIMIT=NONE] -->
<string name="script_binding_voice">Voice: "%1$s"</string>
</resources>
53 changes: 50 additions & 3 deletions talkback/src/phone/assets/scripting/prelude.js
Original file line number Diff line number Diff line change
Expand Up @@ -151,10 +151,12 @@
].map((name) => [name, []]),
);
const commands = new Map();
const voiceCommands = new Map();
const syncHooks = () =>
send('hooks', {
names: Object.keys(handlers).filter((name) => handlers[name].length > 0),
commands: [...commands.keys()],
voicePatterns: [...voiceCommands.keys()],
});
const subscribe = (add, remove) => {
add();
Expand Down Expand Up @@ -183,6 +185,31 @@
() => commands.get(id) === fn && commands.delete(id),
);
};
const onVoiceCommand = (pattern, fn) => {
if (typeof fn !== 'function') throw new TypeError('onVoiceCommand needs a function');
const patterns = Array.isArray(pattern) ? pattern : [pattern];
for (const p of patterns) {
if (typeof p !== 'string' || !p.trim()) {
throw new TypeError('onVoiceCommand pattern must be a non-empty string or array of strings');
}
}
const clean = patterns.map((p) => p.trim());
return subscribe(
() => {
for (const p of clean) voiceCommands.set(p.toLowerCase(), fn);
},
() => {
let changed = false;
for (const p of clean) {
if (voiceCommands.get(p.toLowerCase()) === fn) {
voiceCommands.delete(p.toLowerCase());
changed = true;
}
}
return changed;
},
);
};

const notify = (name, ...args) => {
for (const fn of handlers[name]) guard(name, fn, ...args);
Expand All @@ -207,18 +234,36 @@
return text === input ? undefined : { text };
};

const runCommand = ({ id, via, direction, node: snapshot }) => {
const runCommand = ({ id, via, direction, node: snapshot, command, remainder }) => {
const fn = commands.get(id);
if (!fn) return { fallback: true };
const event = Object.freeze({
id,
via,
direction: direction ?? undefined,
command: command ?? undefined,
remainder: remainder ?? undefined,
node: node(snapshot),
});
return guard(`command ${id}`, fn, event) === false ? { fallback: true } : undefined;
};

const runVoiceCommand = ({ command, pattern, remainder, node: snapshot }) => {
const fn = voiceCommands.get(pattern?.toLowerCase());
if (!fn) return { fallback: true };
const event = Object.freeze({
command,
pattern,
remainder: remainder ?? undefined,
node: node(snapshot),
});
const result = guard(`voice ${pattern}`, fn, event);
if (result === false || result?.fallback) return { fallback: true };
if (typeof result === 'string') return { speak: result };
if (typeof result?.speak === 'string') return { speak: result.speak };
return { handled: true };
};

const offers = new Map();
let nextOffer = 1;
const offerActions = ({ node: snapshot }) => {
Expand Down Expand Up @@ -285,6 +330,7 @@
settingChange: (data) => notify('settingChange', data.key, data.value),
button: (data) => notify('button', data.key),
command: runCommand,
voiceCommand: runVoiceCommand,
actions: offerActions,
dialogEvent: (data) => formEvent(data),
runAction,
Expand Down Expand Up @@ -551,8 +597,8 @@
alert: (message, { title } = {}) =>
dialog('ui.alert', { message: String(message), title: text(title) }).then(() => undefined),
});
const binding = (method) => (command, { gesture, keys } = {}) =>
send(method, { command: String(command), gesture, keys });
const binding = (method) => (command, { gesture, keys, voice } = {}) =>
send(method, { command: String(command), gesture, keys, voice });

const backtalk = Object.freeze({
apiVersion: call('apiVersion'),
Expand All @@ -565,6 +611,7 @@
},
...Object.fromEntries(Object.keys(handlers).map((name) => [handlerName(name), listen(name)])),
onCommand,
onVoiceCommand,
get locale() {
return call('locale');
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,8 @@ class LazyScriptHost(
fallback: Runnable,
): Boolean = manager?.onKeys(modifiers, keyCode, withBacktalkModifier, fallback) == true

override fun onVoiceCommand(command: String): Boolean = manager?.onVoiceCommand(command) == true

override fun menuItems(): List<ScriptMenuItem> = manager?.menuItems().orEmpty()

override fun readingControls(): List<ScriptReadingControl> =
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -284,6 +284,7 @@ class ManifestParser(private val strict: Boolean) {
gestures
.firstOrNull { !ScriptInput.isGesture(it) }
?.let { manifestError("Unknown gesture $it. docs/scripting.md lists the gestures.") }
val voice = command.stringList("voice").distinct()
ScriptCommand(
id = id,
title = command.requiredString("title"),
Expand All @@ -294,10 +295,11 @@ class ManifestParser(private val strict: Boolean) {
},
menu = command.optBoolean("menu", false),
control = command.optBoolean("control", false),
voice = voice,
)
.also {
manifestCheck(it.bindings.isNotEmpty()) {
"The command $id needs gesture, keys, menu or control"
"The command $id needs gesture, keys, menu or control, or voice"
}
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,18 +27,20 @@ private constructor(
val menu: List<BoundCommand>,
val controls: List<BoundCommand>,
val navigation: List<BoundNavigation>,
val voice: Map<String, BoundCommand>,
) {
val layers: Set<ScriptInput.Keys> =
keys.keys.filter { it.next != null }.mapTo(HashSet()) { it.first }

companion object {
val EMPTY = ScriptBindings(emptyMap(), emptyMap(), emptyList(), emptyList(), emptyList())
val EMPTY = ScriptBindings(emptyMap(), emptyMap(), emptyList(), emptyList(), emptyList(), emptyMap())

fun of(runtimes: List<ScriptRuntime>, store: ScriptStore): ScriptBindings {
val gestures = HashMap<String, BoundCommand>()
val keys = HashMap<ScriptInput.Keys, BoundCommand>()
val menu = mutableListOf<BoundCommand>()
val controls = mutableListOf<BoundCommand>()
val voice = HashMap<String, BoundCommand>()
for (runtime in runtimes) {
val commands = runtime.script.manifest.commands.filter { it.id in runtime.commands }
if (commands.isEmpty() || !runtime.allowed(ScriptPermission.INPUT, "Commands")) continue
Expand All @@ -49,6 +51,7 @@ private constructor(
when (binding) {
is CommandBinding.Gesture -> gestures.putIfAbsent(binding.name, bound)
is CommandBinding.KeyCombo -> keys.putIfAbsent(binding.keys, bound)
is CommandBinding.Voice -> voice.putIfAbsent(binding.phrase.trim().lowercase(), bound)
CommandBinding.MenuItem -> menu += bound
CommandBinding.ReadingControl -> controls += bound
}
Expand All @@ -66,7 +69,7 @@ private constructor(
BoundNavigation(runtime, index, item)
}
}
return ScriptBindings(gestures, keys, menu, controls, navigation)
return ScriptBindings(gestures, keys, menu, controls, navigation, voice)
}
}
}
Loading
Loading