Skip to content

Commit bdec925

Browse files
authored
test(kernel-language-model-service): Add queue language model service for testing (#710)
This PR adds a queue language model service to the monorepo. The implementation is intended for mocking language model behavior where a remote API or local model execution are both unavailable. The intended streaming response is pushed to the model prior to sampling. Also adds tests to kernel-test which exercise the basic shape of a LanguageModelService.
1 parent 5874acb commit bdec925

File tree

19 files changed

+1139
-4
lines changed

19 files changed

+1139
-4
lines changed

packages/kernel-language-model-service/package.json

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,16 @@
3333
"default": "./dist/ollama/nodejs.cjs"
3434
}
3535
},
36+
"./test-utils": {
37+
"import": {
38+
"types": "./dist/test-utils/index.d.mts",
39+
"default": "./dist/test-utils/index.mjs"
40+
},
41+
"require": {
42+
"types": "./dist/test-utils/index.d.cts",
43+
"default": "./dist/test-utils/index.cjs"
44+
}
45+
},
3646
"./package.json": "./package.json"
3747
},
3848
"files": [
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
export { makeQueueService } from './queue/service.ts';
2+
export { makeQueueModel } from './queue/model.ts';
3+
export type { QueueLanguageModel } from './queue/model.ts';
4+
export type { QueueLanguageModelService } from './queue/service.ts';
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# Queue-based Language Model Service (Testing Utility)
2+
3+
[`makeQueueService`](./service.ts) is a testing utility that creates a `LanguageModelService` implementation for use in tests. It provides a queue-based language model where responses are manually queued using the `push()` method and consumed by `sample()` calls.
4+
5+
## Usage
6+
7+
1. Create a service using `makeQueueService()`
8+
2. Create a model instance using `makeInstance()`
9+
3. Queue responses using `push()` on the model instance
10+
4. Consume responses by calling `sample()`
11+
12+
Note that `makeInstance` and `sample` ignore their arguments, but expect them nonetheless.
13+
14+
## Examples
15+
16+
### Basic Example
17+
18+
```typescript
19+
import { makeQueueService } from '@ocap/kernel-language-model-service/test-utils';
20+
21+
const service = makeQueueService();
22+
const model = await service.makeInstance({ model: 'test' });
23+
24+
// Queue a response
25+
model.push('Hello, world!');
26+
27+
// Consume the response
28+
const result = await model.sample({ prompt: 'Say hello' });
29+
for await (const chunk of result.stream) {
30+
console.log(chunk.response); // 'Hello, world!'
31+
}
32+
```
33+
34+
### Multiple Queued Responses
35+
36+
```typescript
37+
const service = makeQueueService();
38+
const model = await service.makeInstance({ model: 'test' });
39+
40+
// Queue multiple responses
41+
model.push('First response');
42+
model.push('Second response');
43+
44+
// Each sample() call consumes the next queued response
45+
const first = await model.sample({ prompt: 'test' });
46+
const second = await model.sample({ prompt: 'test' });
47+
48+
// Process streams...
49+
```

0 commit comments

Comments
 (0)