Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 19 additions & 9 deletions docs/BINARY-NOTATION-SPEC.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Binary Links Notation Specification

**Version**: 1.0.0
**Version**: 1.0.1 (documentation revision; wire version remains 1.0)
**Status**: Draft
**Authors**: links-queue contributors

Expand Down Expand Up @@ -53,10 +53,10 @@ Every binary message starts with a fixed 11-byte header:
Each link is encoded with a type byte followed by variable-length fields:

```
┌────────┬────────────┬────────────┬──────────────┐
│ Type │ Source │ Target │ Values (opt) │
│ 1 byte │ Variable │ Variable │ Variable │
└────────┴────────────┴────────────┴──────────────┘
┌────────┬────────────┬────────────┬────────────┬──────────────┐
│ Type │ ID (opt) │ Source │Target (opt)│ Values (opt) │
│ 1 byte │ VarInt │ Variable │ Variable │ Variable │
└────────┴────────────┴────────────┴────────────┴──────────────┘
```

#### Type Byte
Expand All @@ -65,12 +65,16 @@ Each link is encoded with a type byte followed by variable-length fields:
|-----|------|-------------|
| 0 | SOURCE_IS_ID | Source is a link ID reference (vs inline link/literal) |
| 1 | TARGET_IS_ID | Target is a link ID reference (vs inline link/literal) |
| 2 | SELF_REF | Self-referencing link (source == target, only encode once) |
| 2 | SELF_REF | Source and target are equal numeric ID references; encode the source and omit the target |
| 3 | HAS_ID | Link has an explicit ID |
| 4 | HAS_VALUES | Link has additional values array |
| 5-6 | ID_SIZE | 00=varint, 01=4 bytes, 10=8 bytes, 11=reserved |
| 7 | RESERVED | Reserved for future use |

Fields appear in this order: type, explicit ID if `HAS_ID` is set, source, target unless `SELF_REF` is set, and values if `HAS_VALUES` is set. The source is always encoded, including when it equals the explicit ID.

`SELF_REF` only indicates that source equals target; the link's explicit ID may differ from both. For example, `(7: 5 5)` has body bytes `0F 07 05`. `HAS_ID` and `SELF_REF` are independent: combining them does not omit the source or imply that it equals the explicit ID.

### Value Encoding

Values can be:
Expand Down Expand Up @@ -123,6 +127,8 @@ Arrays are encoded as:

## Examples

The sizes below describe link bodies only. They exclude the 11-byte message header and the payload's VarInt link count.

### Simple Link `(1, 2)`

Text notation: `(1 2)` (5 bytes)
Expand All @@ -134,16 +140,19 @@ Target: 0x02 (varint 2)
Total: 3 bytes (40% reduction)
```

### Self-Referencing Link `(5: 5)`
### Self-Referencing Link `(5: 5 5)`

Text notation: `(5: 5 5)` (10 bytes)
Text notation: `(5: 5 5)` (8 bytes)
Binary encoding:
```
Type: 0x0F (SOURCE_IS_ID | TARGET_IS_ID | SELF_REF | HAS_ID)
ID: 0x05 (varint 5)
Total: 2 bytes (80% reduction)
Source: 0x05 (varint 5; target reuses source)
Total: 3 bytes (62.5% reduction)
```

The body bytes are `0F 05 05`. The explicit ID and the shared source/target reference are separate fields, even though both equal 5. This corrects the example without changing the version 1.0 wire format (`0x0100`).

### Named Link `(type: enqueue)`

Text notation: `(type: enqueue)` (15 bytes)
Expand Down Expand Up @@ -336,6 +345,7 @@ socket.on('data', (chunk) => {

| Version | Date | Changes |
|---------|------|---------|
| 1.0.1 | 2026-10 | Clarify independent ID and source fields; correct self-referencing example sizes (wire version unchanged) |
| 1.0.0 | 2024-01 | Initial specification |

## References
Expand Down
5 changes: 5 additions & 0 deletions js/.changeset/self-referencing-binary-spec.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'links-queue-js': patch
---

Correct the binary notation specification's self-referencing example to `0F 05 05` (3 bytes versus 8 text bytes, a 62.5% reduction). Clarify that the explicit ID and shared source/target reference are independent fields, and add specification and wire compatibility tests without changing the version 1.0 encoding.
79 changes: 79 additions & 0 deletions js/tests/binary-notation-spec.test.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
/** Tests that the self-referencing specification example matches the wire format. */

import { describe, it } from 'node:test';
import assert from 'node:assert/strict';
import { readFileSync } from 'node:fs';
import {
BinaryNotation,
HEADER_SIZE,
encodeLinkBody,
decodeLinkBody,
linkBodySize,
} from '../src/protocol/binary-notation.js';

const specification = readFileSync(
new URL('../../docs/BINARY-NOTATION-SPEC.md', import.meta.url),
'utf8'
);
const example = specification
.split('### Self-Referencing Link')[1]
.split('\n### ')[0];
const documentedBody = Uint8Array.from(
example.matchAll(/^(?:Type|ID|Source|Target): 0x([0-9a-f]+)/gim),
([, hex]) => Number.parseInt(hex, 16)
);
const [, text, textBytes] = example.match(
/Text notation: `([^`]+)` \((\d+) bytes\)/
);
const [, bodyBytes, reduction] = example.match(
/Total: (\d+) bytes \(([\d.]+)% reduction\)/
);

describe('Self-referencing specification example', () => {
it('should match the encoded body and decode to the example link', () => {
const link = { id: 5, source: 5, target: 5 };
const frame = BinaryNotation.encode(link);
assert.deepEqual(frame.slice(HEADER_SIZE + 1), documentedBody);
const decoded = decodeLinkBody(documentedBody, 0);
assert.equal(decoded.bytesRead, documentedBody.length);
assert.equal(decoded.link.id, link.id);
assert.equal(decoded.link.source, link.source);
assert.equal(decoded.link.target, link.target);
});

it('should report the UTF-8 text size accurately', () => {
assert.equal(new TextEncoder().encode(text).length, Number(textBytes));
});

it('should report the body size and reduction without framing overhead', () => {
const size = linkBodySize({ id: 5, source: 5, target: 5 });
assert.equal(size, Number(bodyBytes));
assert.equal(size, documentedBody.length);
assert.equal(((1 - size / Number(textBytes)) * 100).toFixed(1), reduction);
});
});

describe('Version 1.0 self-referencing wire format', () => {
const cases = [
{ id: 5, reference: 5, bytes: [0x0f, 0x05, 0x05] },
{ id: 7, reference: 5, bytes: [0x0f, 0x07, 0x05] },
{ id: 128, reference: 128, bytes: [0x0f, 0x80, 0x01, 0x80, 0x01] },
{ id: null, reference: 5, bytes: [0x07, 0x05] },
];

for (const { id, reference, bytes } of cases) {
it(`should encode ID ${id} and shared reference ${reference}`, () => {
const link = { id, source: reference, target: reference };
const body = Uint8Array.from(bytes);
const buffer = new Uint8Array(body.length);
assert.equal(encodeLinkBody(link, buffer, 0), body.length);
assert.equal(linkBodySize(link), body.length);
assert.deepEqual(buffer, body);
const decoded = decodeLinkBody(body, 0);
assert.equal(decoded.bytesRead, body.length);
assert.equal(decoded.link.id, id ?? 0);
assert.equal(decoded.link.source, reference);
assert.equal(decoded.link.target, reference);
});
}
});
4 changes: 4 additions & 0 deletions rust/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,10 @@ module_name_repetitions = "allow"
too_many_lines = "allow"
missing_errors_doc = "allow"
missing_panics_doc = "allow"
# Async implementations retain deferred execution even when no await is needed.
unused_async_trait_impl = "allow"
# is_empty assertions also work for collections of types without PartialEq.
assert_is_empty = "allow"

[profile.release]
lto = true
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
bump: patch
---

### Fixed

- Correct the binary notation specification's self-referencing example to `0F 05 05` (3 bytes versus 8 text bytes, a 62.5% reduction). Clarify independent ID and source fields and verify the documented bytes against the Rust codec while preserving the version 1.0 wire format.
- Restore strict Clippy checks on current Rust toolchains with equivalent helper expressions and explicit allowances for deferred async implementations and collection emptiness assertions.
2 changes: 1 addition & 1 deletion rust/src/backends/binary_notation.rs
Original file line number Diff line number Diff line change
Expand Up @@ -901,7 +901,7 @@ impl BinaryNotation {

/// Checks if data appears to be binary notation.
#[must_use]
pub fn is_binary(data: &[u8]) -> bool {
pub const fn is_binary(data: &[u8]) -> bool {
if data.len() < 4 {
return false;
}
Expand Down
64 changes: 62 additions & 2 deletions rust/src/backends/binary_notation_tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -101,8 +101,9 @@ mod link_tests {
let mut buf = Vec::new();
let size = encode_link_body(&link, &mut buf).unwrap();

// Self-ref should be compact
assert!(size <= 3);
assert_eq!(size, 3);
assert_eq!(buf, vec![0x0F, 0x05, 0x05]);
assert_eq!(link_body_size(&link), size);

let mut cursor = std::io::Cursor::new(buf);
let decoded = decode_link_body(&mut cursor).unwrap();
Expand All @@ -112,6 +113,65 @@ mod link_tests {
assert!(matches!(decoded.target, EncodedValue::Int(5)));
}

#[test]
fn test_self_ref_specification_example() {
let specification = include_str!("../../../docs/BINARY-NOTATION-SPEC.md");
let example = specification
.split("### Self-Referencing Link")
.nth(1)
.unwrap()
.split("\n### ")
.next()
.unwrap();
let documented_body: Vec<u8> = example
.lines()
.filter_map(|line| {
let (field, value) = line.split_once(": 0x")?;
if matches!(field, "Type" | "ID" | "Source" | "Target") {
Some(u8::from_str_radix(value.split_whitespace().next().unwrap(), 16).unwrap())
} else {
None
}
})
.collect();
let link = EncodedLink::new(Some(5), EncodedValue::Int(5), EncodedValue::Int(5));
let frame = BinaryNotation::encode(&[link], None).unwrap();
assert_eq!(&frame[HEADER_SIZE + 1..], documented_body);

let mut cursor = std::io::Cursor::new(documented_body);
let decoded = decode_link_body(&mut cursor).unwrap();
assert_eq!(decoded.id, Some(5));
assert!(matches!(decoded.source, EncodedValue::Int(5)));
assert!(matches!(decoded.target, EncodedValue::Int(5)));
assert_eq!(cursor.position(), 3);
}

#[test]
fn test_self_ref_wire_format() {
let cases: [(Option<u64>, u64, &[u8]); 3] = [
(Some(7), 5, &[0x0F, 0x07, 0x05]),
(Some(128), 128, &[0x0F, 0x80, 0x01, 0x80, 0x01]),
(None, 5, &[0x07, 0x05]),
];
for (id, reference, body) in cases {
let link = EncodedLink::new(
id,
EncodedValue::Int(reference),
EncodedValue::Int(reference),
);
let mut buffer = Vec::new();
assert_eq!(encode_link_body(&link, &mut buffer).unwrap(), body.len());
assert_eq!(link_body_size(&link), body.len());
assert_eq!(buffer, body);
let mut cursor = std::io::Cursor::new(body);
let decoded = decode_link_body(&mut cursor).unwrap();
assert_eq!(decoded.id, id);
assert!(matches!(decoded.source, EncodedValue::Int(n) if n == reference));
assert!(matches!(decoded.target, EncodedValue::Int(n) if n == reference));
assert_eq!(cursor.position(), body.len() as u64);
}
}

#[test]
fn test_encode_decode_link_with_values() {
let link = EncodedLink::with_values(
Expand Down
3 changes: 1 addition & 2 deletions rust/src/backends/link_cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -266,8 +266,7 @@ impl<T: LinkType + FromStr + Display> LinkCliBackend<T> {
fn current_time_ms() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0)
.map_or(0, |d| d.as_millis() as u64)
}

/// Ensures the backend is connected.
Expand Down
3 changes: 1 addition & 2 deletions rust/src/backends/memory_backend.rs
Original file line number Diff line number Diff line change
Expand Up @@ -177,8 +177,7 @@ impl<T: LinkType> MemoryBackend<T> {
fn current_time_ms() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0)
.map_or(0, |d| d.as_millis() as u64)
}

/// Ensures the backend is connected, returning an error if not.
Expand Down
3 changes: 1 addition & 2 deletions rust/src/client/connection.rs
Original file line number Diff line number Diff line change
Expand Up @@ -236,8 +236,7 @@ where
// Use try_read to avoid blocking
self.state
.try_read()
.map(|s| *s == ConnectionState::Connected)
.unwrap_or(false)
.is_ok_and(|s| *s == ConnectionState::Connected)
}

fn server_address(&self) -> Option<String> {
Expand Down
7 changes: 2 additions & 5 deletions rust/src/cluster/node.rs
Original file line number Diff line number Diff line change
Expand Up @@ -295,10 +295,7 @@ impl ClusterNode for Node {

fn status(&self) -> NodeStatus {
// Use try_read to avoid blocking, default to Joining if locked
self.status
.try_read()
.map(|s| *s)
.unwrap_or(NodeStatus::Joining)
self.status.try_read().map_or(NodeStatus::Joining, |s| *s)
}

async fn ping(&self) -> Result<u64, ClusterError> {
Expand Down Expand Up @@ -578,7 +575,7 @@ mod tests {
assert_eq!(node.last_latency(), Duration::ZERO);

node.update_latency(Duration::from_millis(50));
assert_eq!(node.last_latency(), Duration::from_micros(50000));
assert_eq!(node.last_latency(), Duration::from_millis(50));
}
}

Expand Down
4 changes: 2 additions & 2 deletions rust/src/queue/manager.rs
Original file line number Diff line number Diff line change
Expand Up @@ -170,8 +170,8 @@ impl<T: LinkType> MemoryQueueManager<T> {
let dead_letters: Vec<(String, Vec<crate::Link<T>>)> = {
let queues = self.queues.lock().expect("lock poisoned");
queues
.iter()
.filter_map(|(_, entry)| {
.values()
.filter_map(|entry| {
let dlq_name = entry.queue.dead_letter_queue_name()?;
let items = entry.queue.drain_dead_letters();
if items.is_empty() {
Expand Down
3 changes: 1 addition & 2 deletions rust/tests/link_cli_integration.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,7 @@ fn clink_available() -> bool {
std::process::Command::new("clink")
.arg("--version")
.output()
.map(|output| output.status.success())
.unwrap_or(false)
.is_ok_and(|output| output.status.success())
}

/// Checks if tests should be skipped.
Expand Down
Loading