[proposal] - Allow CORS Headers for intranet / VPN purposes#94
Open
WebReflection wants to merge 1 commit into
Open
[proposal] - Allow CORS Headers for intranet / VPN purposes#94WebReflection wants to merge 1 commit into
WebReflection wants to merge 1 commit into
Conversation
Author
|
If anyone is interested around "how can I test this" ? test.js import Queue from 'https://esm.run/gen-q';
const { parse, stringify } = JSON;
const decoder = new TextDecoder;
const chatOptions = {
stream: true,
role: 'user',
};
export default class DS4 {
#model;
#url;
constructor({
url = 'http://YOUR_MACHINE_IP:8000',
model = 'deepseek-v4-flash',
version = 'v1',
}) {
this.#model = model;
this.#queue = new Queue();
this.#url = new URL(`${url}/${version}`);
}
async *chat(content, { stream = true, role = 'user' } = chatOptions) {
const items = new Queue;
const { body } = await fetch(`${this.#url}/chat/completions`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: stringify({
model: this.#model,
messages: [
{ role, content }
],
stream,
}),
});
const reader = body.getReader();
new ReadableStream({
async start(controller) {
(function next() {
reader.read().then(({ done, value }) => {
if (done) {
items.splice(0);
controller.close();
return;
}
const text = decoder.decode(value);
if (text.startsWith('data: {'))
items.push(...parse(text.slice(6, text.lastIndexOf('}') + 1)).choices);
next();
});
}());
},
});
for await (const item of items) {
const { finish_reason, delta } = item;
if (finish_reason !== null) break;
const { reasoning_content, content } = delta;
if (content != null) yield content;
// else if (reasoning_content != null) console.info(reasoning_content);
}
}
}index.html <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>DS4</title>
<script type="module">
import DS4 from './test.js';
const ds4 = new DS4({
url: 'http://YOUR_MACHINE_IP:8000',
model: 'deepseek-v4-flash',
version: 'v1',
});
// write the chat content you'd like to read on the body
for await (const chunk of ds4.chat('List three Redis design principles.'))
document.body.append(chunk);
console.log('done');
</script>
</head>
<body>
</body>
</html>The testing library is a WIP and it will be able to consume all channels and do more but with that, and this patch, you'll see results from a |
Author
|
this might be a duplicate of #70 which I've just realized was in already ... my thoughts:
thank you! |
|
Alternatively, #44 |
Author
|
@calvinrp answered in here #70 (comment) |
|
Why not add a proxy on top for all the shenanigans? IMHO this should be kept as simple as possible. |
Author
|
@d3y4n having CORS options backed in is the "as simple as possible" idea indeed ... anything else is not simple anymore. |
|
@WebReflection not my call, just saying you're hardcoding values and tomorrow someone might need different ones (even you). Cheers |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This MR has been successfully tested in my local network with a DGX Spark around the WiFi and I need this variant to be able to query via my
localhostor any other connected device that Spark so that we can all benefit from this project within my house.Thanks for considering this change/update.
To be discussed
Ideally there should be a
--corsflag when starting the server but I'd like to start with this implementation that "just works" ™️ and hear out from others/maintainer if there's anything else I can improve/change but trust me it works already and I am playing around a tiny library that would let me lurk ds4 from anywhere I am in my own apartment, as long as the DGX is up and running.P.S. thanks a lot for this project, I will inevitably try to bring it to ROCm once I have my machine around but so far with Spark it's working wonderfully!