diff --git a/CHANGELOG.md b/CHANGELOG.md index cab347e..3c05aaa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,53 @@ All notable changes to this project are documented in this file. ## Unreleased +- Internal reorganization: the top-level modules that had grown past a + single implementation each are now grouped into themed subpackages — + `core/` (`logger`, `levels`, `records`, `context`, `exceptions`, + `handler`, `opt`), `runtime/` (`worker`, `shutdown`, `toggle`, + `runtime_level`, `serverless`), `tracing/` (`span`, `retry`, `mcp`, + `semconv`), `cli/` (`cli`, `webui`, `trace_tree`, `static/`), and + `privacy/` (`privacy`, `audit`). Every name already exported from the + top-level `logquill` package (`from logquill import Logger`, etc.) is + unaffected; only a direct deep import of one of these modules (e.g. + `from logquill.mcp import inbound, propagate`) needs its path updated + to the new location (`from logquill.tracing.mcp import ...`). The CLI's + packaged entry point moved from `logquill.cli:main` to + `logquill.cli.cli:main`, transparent to anyone invoking the installed + `logquill` command. +- Metrics bridge and graph transports: + - `MetricsPlugin` turns log patterns into metric emissions — the + logs-to-metrics side of the three-pillars observability model. Every + record at `error_at` (default `ERROR`) or above increments an error + counter tagged by `logger`; any record carrying `meta.duration_ms` + (every span-closing record) feeds a duration histogram. Runs in + `after_log`, so a slow or unreachable backend never delays the record + it's reporting on, and a raising backend is caught by the plugin + pipeline the same as any other plugin. `backend` is a small + `MetricsBackend` protocol (`increment`/`observe`), resolved lazily to + `OTelMetricsBackend()` by default so importing `logquill` never pulls + in OpenTelemetry on its own. `StatsDBackend` (stdlib `socket`, + DogStatsD-style tags) and `PrometheusPushgatewayBackend` (stdlib + `urllib`, accumulates in memory and only sends a batched `PUT` on + `.push()`/`.close()`, never per-metric) cover teams not yet on OTel. + - `BaseGraphTransport` writes a run's span tree directly into a graph + database instead of a flat log: every record becomes a `:LogRecord` + node, `meta.parent_span_id` becomes a `PARENT_OF` edge, and + `meta.run_id` (falling back to `meta.trace_id`) groups a run's nodes + under a `:Session` via `HAS_RECORD` — all through one shared, + backend-portable Cypher query, so reconstructing an agent run's exact + call tree is a traversal, not application code walking a JSONL file. + Three concrete backends ship on top of it: `Neo4jTransport` + (`pip install logquill[neo4j]`), `FalkorDBTransport` + (`pip install logquill[falkordb]`), and `NeptuneTransport` (via + `boto3`'s openCypher support, reusing the existing `aws` extra). + Switching backends means swapping the transport, not touching any + call site. `KuzuTransport` was scoped for this release but + deliberately not built: Kuzu's own repository was archived by its + maintainers on 2025-10-10 with no release schedule, and building + against an archived database contradicts the "verify it's still + maintained before committing to it" bar every other dependency here + is held to. - Runtime intelligence: - `FlightRecorderPlugin` generalizes `SamplingPlugin`'s tail-based elevation from "a record sampling happened to drop" to "a record below diff --git a/README.md b/README.md index aadd4e6..60e850f 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ for what's landed so far. - **Plugin pipeline** — `ContextPlugin`, `RedactPlugin` (by key), `PIIRedactPlugin` (by pattern), `SamplingPlugin` (with tail-based elevation), `TamperEvidentPlugin` (hash-chained logs), `TraceContextPlugin` (cross-service trace correlation), and `AlertingPlugin` (`SlackAlertPlugin`/`PagerDutyAlertPlugin`/`EmailAlertPlugin`, deduplicated) out of the box; a broken plugin can't crash logging; `.use()` also accepts a plain function, no subclassing required (see [Plugins](#plugins)) - **Agentic & harness tracing** — `.child()` loggers, `RunPlugin`, `.thought()/.action()/.observation()/.decision()`, `with agent_log.span(...)`, and framework adapters — `LangChainAdapter` (`pip install logquill[langchain]`), `LangGraphAdapter` (`pip install logquill[langgraph]`, adds checkpoint interrupt/resume events on top), `CrewAIAdapter` (`pip install logquill[crewai]`), `LlamaIndexAdapter` (`pip install logquill[llamaindex]`), `AutoGenAdapter` (`pip install logquill[autogen]`), and `OpenAIAgentsAdapter` (`pip install logquill[openai-agents]`) — see [Agentic & harness tracing](#agentic--harness-tracing) - **LLM calls & OpenTelemetry** — `logger.llm_call(model=, tokens_in=, tokens_out=, cost_usd=, ...)` records a first-class `llm` block, `span(capture_state=...)` records what state changed, repeated tool calls get `meta.retry_count` automatically, and `OTLPTransport` (`pip install logquill[otel]`) exports it all as real OpenTelemetry spans named and attributed per the GenAI semantic conventions; `OTelLogsTransport` sends records as OTLP logs — see [LLM calls & OpenTelemetry](#llm-calls--opentelemetry) -- **Auto-instrumentation & MCP** — `logquill.instrument.anthropic(logger)`/`.openai(logger)`/`.litellm(logger)` patch a provider SDK so every LLM call logs itself, no call-site changes; `OpenAIAgentsAdapter` covers the OpenAI Agents SDK; `logquill.mcp` propagates trace context over MCP requests and stamps `meta.mcp.*` — see [Auto-instrumentation and MCP](#auto-instrumentation-and-mcp) +- **Auto-instrumentation & MCP** — `logquill.instrument.anthropic(logger)`/`.openai(logger)`/`.litellm(logger)` patch a provider SDK so every LLM call logs itself, no call-site changes; `OpenAIAgentsAdapter` covers the OpenAI Agents SDK; `logquill.tracing.mcp` propagates trace context over MCP requests and stamps `meta.mcp.*` — see [Auto-instrumentation and MCP](#auto-instrumentation-and-mcp) - **Non-blocking async dispatch** — `Logger(async_dispatch=True)` moves transport writes onto a background thread with a bounded queue and a configurable backpressure policy (`drop_oldest`/`drop_newest`/`block`); `flush()`/`flush_async()` and a `with_lambda`/`with_cloud_function`/`with_azure_function` decorator make serverless shutdown safe — see [Async dispatch & serverless safety](#async-dispatch--serverless-safety) - **Zero required runtime dependencies** — stdlib only; `aiohttp` (`logquill[http]`) is opt-in, for keep-alive HTTP delivery - **Typed throughout** — `mypy --strict` clean on the public API @@ -38,6 +38,8 @@ for what's landed so far. - **CLI** — `logquill tail app.log --level=warn --json -f` for filtering/following a JSONL log file in local dev, no extra install — see [CLI](#cli) - **Local-first trace viewer** — `logquill trace --file logs.jsonl` prints an annotated span tree (streamed, bounded memory at gigabyte scale); `logquill serve` runs a small offline web UI (stdlib only — run list, span tree/waterfall, search, level filter) reading JSONL or a `SQLiteTransport` database; `logquill dev` live-renders the current run as it happens — no account, nothing leaves your machine - **Runtime intelligence** — `FlightRecorderPlugin` ships a run's buffered DEBUG trail only if it errors; `RunSummaryPlugin` emits one record with a run's total tokens/cost/tool calls/retries/errors; `AdaptiveSamplingPlugin` always keeps errors and slow spans while capping ordinary traffic to a bytes-per-second budget; `install_signal_level_handler`/`LevelFileWatcher`/`LevelEnvWatcher` change a logger's level without a restart — see [Runtime level control](#runtime-level-control) +- **Metrics bridge** — `MetricsPlugin` turns ERROR/FATAL records into a counter and any `meta.duration_ms` into a histogram, tagged by logger; ships with `OTelMetricsBackend` (default), `StatsDBackend`, and `PrometheusPushgatewayBackend` — see [Metrics bridge](#metrics-bridge) +- **Graph transports** — `Neo4jTransport`, `FalkorDBTransport`, `NeptuneTransport` write every record as a node and `parent_span_id` as a `PARENT_OF` edge, so a run's span tree is a Cypher traversal away with zero manual graph-writing code; one shared `BaseGraphTransport` means switching backends only means swapping the transport — see [Graph transports](#graph-transports) ## Install @@ -418,6 +420,76 @@ logger = Logger("app", transports=[transport]) logger.error("payment webhook failed") ``` +### Graph transports + +A run's `run_id`/`span_id`/`parent_span_id` already describe a tree — these +transports write that tree directly into a graph database instead of a flat +log, so "reconstruct this agent run's call tree" is a Cypher traversal +instead of application code that walks a JSONL file. `BaseGraphTransport` +owns the one query every backend runs — `MERGE` a `:LogRecord` node per +record, a `PARENT_OF` edge from `parent_span_id`, and a `HAS_RECORD` edge +grouping every node under a `:Session` keyed by `run_id` (or `trace_id`) — +so switching backends means swapping the transport, not rewriting anything: + +```python +from logquill import Logger, Neo4jTransport, RunPlugin + +class FakeSession: + def __init__(self, calls): + self._calls = calls + def run(self, query, parameters): + self._calls.append((query, parameters)) + def __enter__(self): + return self + def __exit__(self, *exc_info): + pass + +class FakeDriver: + def __init__(self): + self.calls = [] + def session(self, **kwargs): + return FakeSession(self.calls) + def close(self): + pass + +driver = FakeDriver() +transport = Neo4jTransport(driver=driver, max_records=10) +logger = Logger("agent", transports=[transport], plugins=[RunPlugin()]) + +with logger.span("run"): + logger.action("call_tool", tool="search") +logger.close() +assert len(driver.calls) == 1 # one batched write for the whole span +``` + +Passing a real `neo4j.GraphDatabase.driver(uri, auth=...)` instead of a fake +works identically — `Neo4jTransport(uri="bolt://localhost:7687", auth=("neo4j", "password"))` +builds one lazily via the optional `neo4j` driver. Once written, the span +tree comes back with one traversal: + +```cypher +MATCH (s:Session {key: ""})-[:HAS_RECORD]->(n) +OPTIONAL MATCH (n)-[:PARENT_OF*0..]->(child) +RETURN n, child +``` + +Each driver is a lazy, injectable dependency, the same shape as the +SQL/NoSQL transports above: + +| Transport | Driver | Extra | +|---|---|---| +| `Neo4jTransport` | `neo4j` | `pip install logquill[neo4j]` | +| `FalkorDBTransport` | `falkordb` | `pip install logquill[falkordb]` | +| `NeptuneTransport` | `boto3` (`neptunedata`, openCypher) | `pip install logquill[aws]` | + +`KuzuTransport` was scoped for this release but deliberately not built — +[Kuzu's own repository](https://github.com/kuzudb/kuzu) was archived by its +maintainers on 2025-10-10 ("Kuzu is working on something new!"), with no +release schedule. Building against an archived database contradicts the +"verify it's still maintained before committing to it" bar every other +dependency in this project is held to, so it's cut rather than shipped +against a dead upstream — revisit only if the project resumes. + ### Encryption-at-rest for file logs `FileTransport(encrypt_key=...)` encrypts each line with @@ -625,7 +697,7 @@ analyzer/anonymizer; Presidio stays a real, opt-in dependency, never a default one. `RedactPlugin` can also target a **field classification** -(`logquill.privacy.FIELD_CLASSES`: `secret` \| `pii` \| `content`) instead of +(`FIELD_CLASSES`: `secret` \| `pii` \| `content`) instead of listing key names one by one: ```python @@ -692,7 +764,7 @@ logquill verify app.log --sign-key --signature Prompt/completion text, tool call arguments/results, and a span's captured state (`meta.input_messages`, `output_messages`, `system_instructions`, `tool_arguments`, `tool_result`, `state_diff` — the `content`-classed fields -in `logquill.privacy.FIELD_CLASSES`) are governed by a per-logger policy, +in `FIELD_CLASSES`) are governed by a per-logger policy, **off by default** — applied before any plugin or transport ever sees the value, not something a plugin has to remember to attach: @@ -786,6 +858,69 @@ logger = Logger( A URL Apprise doesn't recognize raises `ValueError` right away, at startup, instead of silently failing on the first real alert. +### Metrics bridge + +`MetricsPlugin` is the logs-to-metrics side of the three-pillars +observability model: every record at `error_at` (default `ERROR`) or above +increments an error counter tagged by `logger`, and any record carrying +`meta.duration_ms` (every span-closing record) feeds a duration histogram — +so dashboards and alerting don't have to re-derive these from raw log lines. +It runs in `after_log`, after every transport has already seen the record, +so a slow or unreachable metrics backend can never delay the log call it's +reporting on, and a raising backend is caught by the plugin pipeline the +same as any other plugin: + +```python +from logquill import Logger, MetricsPlugin, RunPlugin + +logger = Logger("agent", plugins=[RunPlugin(), MetricsPlugin()]) + +with logger.span("call_llm", duration_ms=420): + pass +logger.error("rate limited") +# logquill.duration_ms histogram gets 420; logquill.errors counter +1, tagged logger="agent" +``` + +`backend` defaults to `OTelMetricsBackend()` (the OTel Metrics SDK, the same +`pip install logquill[otel]` extra `OTLPTransport` already needs, resolved +lazily so importing `logquill` never imports OpenTelemetry on its own). Pass +`StatsDBackend()` or `PrometheusPushgatewayBackend(...)` instead for a team +not yet on OTel: + +```python +from logquill import Logger, MetricsPlugin, StatsDBackend + +logger = Logger("app", plugins=[MetricsPlugin(StatsDBackend(host="localhost", port=8125))]) +logger.error("boom") # sends "logquill.errors:1|c|#logger:app" over UDP +``` + +`PrometheusPushgatewayBackend` only accumulates in-memory counters/histogram +sums on the hot path — nothing reaches the network until `.push()` (or +`.close()`, which calls it once) sends one batched `PUT` to the gateway: + +```python +from logquill import Logger, MetricsPlugin, PrometheusPushgatewayBackend + +class FakeSender: + def __init__(self): + self.calls = [] + def __call__(self, url, body): + self.calls.append((url, body)) + +sender = FakeSender() +backend = PrometheusPushgatewayBackend("http://localhost:9091", job_name="app", sender=sender) +logger = Logger("app", plugins=[MetricsPlugin(backend)]) + +logger.error("boom") +backend.push() # one PUT, batched +assert sender.calls[0][0] == "http://localhost:9091/metrics/job/app" +``` + +Implement the small `MetricsBackend` protocol (`increment`/`observe`) for +any system none of these three cover. `MetricsPlugin.close()` closes the +backend if it has a `close()` method — not called automatically by +`Logger.close()`, the same documented convention as `AlertingPlugin.close()`. + ## Agentic & harness tracing `.child()` makes a namespaced logger that shares the parent's transports — @@ -1184,7 +1319,7 @@ intervening `.uninstrument()` raises, so a call is never wrapped twice. instrumented; see each provider module's docstring (`logquill.instrument.anthropic`, `.openai`, `.litellm`). -**MCP (Model Context Protocol).** `logquill.mcp` propagates trace context +**MCP (Model Context Protocol).** `logquill.tracing.mcp` propagates trace context over an MCP request and stamps `meta.mcp.*` on records logged while handling one — with no dependency on the `mcp` package itself, since both sides just build or read a plain dict (exactly what an MCP client's `meta=` argument and @@ -1192,7 +1327,7 @@ a server's inbound `_meta` are): ```python from logquill import Logger, TraceContextPlugin -from logquill.mcp import inbound, propagate +from logquill.tracing.mcp import inbound, propagate # client, before calling a tool: await session.call_tool("search", {"q": "..."}, meta=propagate(run_id=agent_run_id)) diff --git a/benchmarks/measure.py b/benchmarks/measure.py index 7c02ccc..19758a8 100644 --- a/benchmarks/measure.py +++ b/benchmarks/measure.py @@ -20,7 +20,7 @@ from typing import Callable from logquill import ContextPlugin, Logger, PIIRedactPlugin, Plugin, RedactPlugin -from logquill.records import LogRecord +from logquill.core.records import LogRecord from logquill.transports.transport import Transport #: The most a metric may be before the build fails. Each is a few times what diff --git a/benchmarks/test_trace_memory.py b/benchmarks/test_trace_memory.py index 93ef3cb..745bfb0 100644 --- a/benchmarks/test_trace_memory.py +++ b/benchmarks/test_trace_memory.py @@ -12,7 +12,7 @@ from pathlib import Path from typing import Any, Iterator -from logquill.trace_tree import build_trace +from logquill.cli.trace_tree import build_trace #: Comfortably over 1 GB — the scale this test is meant to prove, without #: depending on exactly hitting it. diff --git a/logquill/__init__.py b/logquill/__init__.py index 4bf7115..feccc53 100644 --- a/logquill/__init__.py +++ b/logquill/__init__.py @@ -1,13 +1,13 @@ from logquill.adapters.base import LogQuillAdapter -from logquill.audit import AuditLogger from logquill.config import load_config, logger_from_env, logger_from_file -from logquill.context import bind_context, current_context -from logquill.exceptions import format_exc_info +from logquill.core.context import bind_context, current_context +from logquill.core.exceptions import format_exc_info +from logquill.core.handler import LogQuillHandler +from logquill.core.levels import Level, parse_level +from logquill.core.logger import Logger +from logquill.core.opt import OptLogger +from logquill.core.records import SCHEMA_VERSION, LLMBlock, LogRecord, parse_record from logquill.formatters import Formatter, JSONFormatter, LogfmtFormatter, TextFormatter -from logquill.handler import LogQuillHandler -from logquill.levels import Level, parse_level -from logquill.logger import Logger -from logquill.opt import OptLogger from logquill.parsing import TEXT_LOG_CASTS, TEXT_LOG_PATTERN, parse, parse_logfmt from logquill.plugins.adaptive_sampling_plugin import AdaptiveSamplingPlugin from logquill.plugins.alerting_plugin import AlertingPlugin @@ -15,15 +15,19 @@ from logquill.plugins.context_plugin import ContextPlugin from logquill.plugins.email_alert_plugin import EmailAlertPlugin from logquill.plugins.flight_recorder_plugin import FlightRecorderPlugin +from logquill.plugins.metrics_plugin import MetricsBackend, MetricsPlugin +from logquill.plugins.otel_metrics_backend import OTelMetricsBackend from logquill.plugins.pagerduty_alert_plugin import PagerDutyAlertPlugin from logquill.plugins.pii_redact_plugin import PIIRedactPlugin from logquill.plugins.plugin import FunctionPlugin, Plugin +from logquill.plugins.prometheus_pushgateway_backend import PrometheusPushgatewayBackend from logquill.plugins.rate_limit_plugin import RateLimitPlugin from logquill.plugins.redact_plugin import RedactPlugin from logquill.plugins.run_plugin import RunPlugin from logquill.plugins.run_summary_plugin import RunSummaryPlugin from logquill.plugins.sampling_plugin import SamplingPlugin from logquill.plugins.slack_alert_plugin import SlackAlertPlugin +from logquill.plugins.statsd_metrics_backend import StatsDBackend from logquill.plugins.tamper_evident_plugin import ( TamperEvidentPlugin, VerificationResult, @@ -33,15 +37,16 @@ verify_signed_chain, ) from logquill.plugins.trace_context_plugin import TraceContextPlugin -from logquill.privacy import CONTENT_FIELDS, FIELD_CLASSES, ContentCapturePolicy, FieldClass -from logquill.records import SCHEMA_VERSION, LLMBlock, LogRecord, parse_record -from logquill.runtime_level import ( +from logquill.privacy.audit import AuditLogger +from logquill.privacy.privacy import CONTENT_FIELDS, FIELD_CLASSES, ContentCapturePolicy, FieldClass +from logquill.runtime.runtime_level import ( LevelEnvWatcher, LevelFileWatcher, install_signal_level_handler, ) -from logquill.serverless import with_azure_function, with_cloud_function, with_lambda -from logquill.toggle import disable, enable, is_enabled +from logquill.runtime.serverless import with_azure_function, with_cloud_function, with_lambda +from logquill.runtime.toggle import disable, enable, is_enabled +from logquill.runtime.worker import AsyncWorker from logquill.transports.batching_transport import BatchingTransport from logquill.transports.cloud.app_insights_transport import AppInsightsTransport from logquill.transports.cloud.cloud_logging_transport import CloudLoggingTransport @@ -52,6 +57,10 @@ from logquill.transports.cloud.syslog_transport import SyslogTransport from logquill.transports.console_transport import ConsoleTransport from logquill.transports.file_transport import FileTransport +from logquill.transports.graph.base_graph_transport import BaseGraphTransport, GraphRow +from logquill.transports.graph.falkordb_transport import FalkorDBTransport +from logquill.transports.graph.neo4j_transport import Neo4jTransport +from logquill.transports.graph.neptune_transport import NeptuneTransport from logquill.transports.http_transport import HTTPTransport from logquill.transports.nosql.dynamodb_transport import DynamoDBTransport from logquill.transports.nosql.mongodb_transport import MongoDBTransport @@ -68,7 +77,6 @@ from logquill.transports.sql.postgres_transport import PostgresTransport from logquill.transports.sql.sqlite_transport import SQLiteTransport from logquill.transports.transport import CollectingTransport, Transport -from logquill.worker import AsyncWorker __version__ = "1.0.0" @@ -79,6 +87,7 @@ "AppInsightsTransport", "AsyncWorker", "AuditLogger", + "BaseGraphTransport", "BaseQueueTransport", "BaseSQLTransport", "BatchingTransport", @@ -94,11 +103,13 @@ "ElasticsearchTransport", "EmailAlertPlugin", "FIELD_CLASSES", + "FalkorDBTransport", "FieldClass", "FileTransport", "FlightRecorderPlugin", "Formatter", "FunctionPlugin", + "GraphRow", "HTTPTransport", "JSONFormatter", "KafkaTransport", @@ -111,16 +122,22 @@ "LogQuillHandler", "LogRecord", "Logger", + "MetricsBackend", + "MetricsPlugin", "MongoDBTransport", "MySQLTransport", + "Neo4jTransport", + "NeptuneTransport", "NewRelicTransport", "OTLPTransport", "OTelLogsTransport", + "OTelMetricsBackend", "OptLogger", "PIIRedactPlugin", "PagerDutyAlertPlugin", "Plugin", "PostgresTransport", + "PrometheusPushgatewayBackend", "PubSubTransport", "RabbitMQTransport", "RateLimitPlugin", @@ -134,6 +151,7 @@ "SQSTransport", "SamplingPlugin", "SlackAlertPlugin", + "StatsDBackend", "SyslogTransport", "TEXT_LOG_CASTS", "TEXT_LOG_PATTERN", diff --git a/logquill/adapters/autogen.py b/logquill/adapters/autogen.py index 66a64c1..c41cfbf 100644 --- a/logquill/adapters/autogen.py +++ b/logquill/adapters/autogen.py @@ -12,7 +12,7 @@ ) from exc from logquill.adapters.base import LogQuillAdapter -from logquill.logger import Logger +from logquill.core.logger import Logger _ERROR_EVENT_NAMES = { "MessageHandlerException": "message_handler_exception", diff --git a/logquill/adapters/base.py b/logquill/adapters/base.py index 8c365cb..bf90aa4 100644 --- a/logquill/adapters/base.py +++ b/logquill/adapters/base.py @@ -1,6 +1,6 @@ from __future__ import annotations -from logquill.logger import Logger +from logquill.core.logger import Logger class LogQuillAdapter: diff --git a/logquill/adapters/crewai.py b/logquill/adapters/crewai.py index edb5d96..97939ad 100644 --- a/logquill/adapters/crewai.py +++ b/logquill/adapters/crewai.py @@ -29,8 +29,8 @@ ) from exc from logquill.adapters.base import LogQuillAdapter -from logquill.logger import Logger -from logquill.span import SpanContext +from logquill.core.logger import Logger +from logquill.tracing.span import SpanContext def _span_ids(event: Any) -> dict[str, Any]: diff --git a/logquill/adapters/langchain.py b/logquill/adapters/langchain.py index 0c0d55b..7cbcbff 100644 --- a/logquill/adapters/langchain.py +++ b/logquill/adapters/langchain.py @@ -13,8 +13,8 @@ ) from exc from logquill.adapters.base import LogQuillAdapter -from logquill.logger import Logger -from logquill.span import SpanContext +from logquill.core.logger import Logger +from logquill.tracing.span import SpanContext def _span_ids(run_id: UUID, parent_run_id: UUID | None) -> dict[str, Any]: diff --git a/logquill/adapters/llamaindex.py b/logquill/adapters/llamaindex.py index 3758f28..6da17ce 100644 --- a/logquill/adapters/llamaindex.py +++ b/logquill/adapters/llamaindex.py @@ -18,7 +18,7 @@ ) from exc from logquill.adapters.base import LogQuillAdapter -from logquill.logger import Logger +from logquill.core.logger import Logger # Event classes carrying no useful correlation id of their own beyond the # span they fired in — every `XxxStartEvent`/`XxxEndEvent`/`XxxErrorEvent` diff --git a/logquill/adapters/openai_agents.py b/logquill/adapters/openai_agents.py index de89de9..3e5fcdc 100644 --- a/logquill/adapters/openai_agents.py +++ b/logquill/adapters/openai_agents.py @@ -23,8 +23,8 @@ ) from exc from logquill.adapters.base import LogQuillAdapter -from logquill.logger import Logger -from logquill.span import SpanContext, new_span_id +from logquill.core.logger import Logger +from logquill.tracing.span import SpanContext, new_span_id def _agent_name(agent: Any) -> str: diff --git a/logquill/cli/__init__.py b/logquill/cli/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/logquill/cli.py b/logquill/cli/cli.py similarity index 98% rename from logquill/cli.py rename to logquill/cli/cli.py index 8138656..f1cf3a4 100644 --- a/logquill/cli.py +++ b/logquill/cli/cli.py @@ -11,11 +11,11 @@ from pathlib import Path from typing import IO, Any, Callable, Sequence +from logquill.cli.trace_tree import build_trace, render_tree +from logquill.cli.webui import JSONLSource, RecordSource, SQLiteSource, TraceViewerServer +from logquill.core.levels import Level, parse_level from logquill.formatters import format_text -from logquill.levels import Level, parse_level from logquill.plugins.tamper_evident_plugin import verify_chain_detailed, verify_signed_chain -from logquill.trace_tree import build_trace, render_tree -from logquill.webui import JSONLSource, RecordSource, SQLiteSource, TraceViewerServer _COLORS = { Level.TRACE: "\x1b[90m", diff --git a/logquill/static/trace_viewer.html b/logquill/cli/static/trace_viewer.html similarity index 100% rename from logquill/static/trace_viewer.html rename to logquill/cli/static/trace_viewer.html diff --git a/logquill/trace_tree.py b/logquill/cli/trace_tree.py similarity index 100% rename from logquill/trace_tree.py rename to logquill/cli/trace_tree.py diff --git a/logquill/webui.py b/logquill/cli/webui.py similarity index 99% rename from logquill/webui.py rename to logquill/cli/webui.py index 9633a6e..d846b15 100644 --- a/logquill/webui.py +++ b/logquill/cli/webui.py @@ -25,7 +25,7 @@ from typing import Any, Protocol from urllib.parse import parse_qs, urlsplit -from logquill.trace_tree import build_trace +from logquill.cli.trace_tree import build_trace _logger = logging.getLogger("logquill") diff --git a/logquill/config.py b/logquill/config.py index 71c748e..25fbd56 100644 --- a/logquill/config.py +++ b/logquill/config.py @@ -6,8 +6,8 @@ from pathlib import Path from typing import Any +from logquill.core.logger import Logger from logquill.formatters import Formatter, JSONFormatter, LogfmtFormatter, TextFormatter -from logquill.logger import Logger from logquill.plugins.context_plugin import ContextPlugin from logquill.plugins.pii_redact_plugin import PIIRedactPlugin from logquill.plugins.plugin import Plugin diff --git a/logquill/core/__init__.py b/logquill/core/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/logquill/context.py b/logquill/core/context.py similarity index 100% rename from logquill/context.py rename to logquill/core/context.py diff --git a/logquill/exceptions.py b/logquill/core/exceptions.py similarity index 100% rename from logquill/exceptions.py rename to logquill/core/exceptions.py diff --git a/logquill/handler.py b/logquill/core/handler.py similarity index 97% rename from logquill/handler.py rename to logquill/core/handler.py index 3d29796..fd0bbe8 100644 --- a/logquill/handler.py +++ b/logquill/core/handler.py @@ -3,10 +3,10 @@ import logging from typing import TYPE_CHECKING, Any -from logquill.levels import Level +from logquill.core.levels import Level if TYPE_CHECKING: - from logquill.logger import Logger + from logquill.core.logger import Logger # Attributes every stdlib `logging.LogRecord` carries for its own bookkeeping # (formatting, source location, timing) — excluded from the `meta` a bridged diff --git a/logquill/levels.py b/logquill/core/levels.py similarity index 100% rename from logquill/levels.py rename to logquill/core/levels.py diff --git a/logquill/logger.py b/logquill/core/logger.py similarity index 96% rename from logquill/logger.py rename to logquill/core/logger.py index c04c14a..822b5f1 100644 --- a/logquill/logger.py +++ b/logquill/core/logger.py @@ -4,20 +4,24 @@ import logging from typing import Any, Callable -from logquill import shutdown -from logquill.context import current_context -from logquill.exceptions import LocalRedactor, format_exc_info -from logquill.levels import Level, parse_level -from logquill.opt import OptLogger, caller_info, resolve_lazy +from logquill.core.context import current_context +from logquill.core.exceptions import LocalRedactor, format_exc_info +from logquill.core.levels import Level, parse_level +from logquill.core.opt import OptLogger, caller_info, resolve_lazy +from logquill.core.records import LLMBlock, LogRecord, build_llm_block, create_record from logquill.plugins.context_plugin import ContextPlugin from logquill.plugins.plugin import FunctionPlugin, MiddlewareFunc, Plugin -from logquill.privacy import ContentCapturePolicy, apply_content_policy, parse_content_policy -from logquill.records import LLMBlock, LogRecord, build_llm_block, create_record -from logquill.retry import RetryTracker -from logquill.span import SpanContext, current_span_id -from logquill.toggle import is_enabled +from logquill.privacy.privacy import ( + ContentCapturePolicy, + apply_content_policy, + parse_content_policy, +) +from logquill.runtime import shutdown +from logquill.runtime.toggle import is_enabled +from logquill.runtime.worker import AsyncWorker, BackpressurePolicy +from logquill.tracing.retry import RetryTracker +from logquill.tracing.span import SpanContext, current_span_id from logquill.transports.transport import Transport -from logquill.worker import AsyncWorker, BackpressurePolicy _logger = logging.getLogger("logquill") diff --git a/logquill/opt.py b/logquill/core/opt.py similarity index 97% rename from logquill/opt.py rename to logquill/core/opt.py index 4bf27f4..bf7efe9 100644 --- a/logquill/opt.py +++ b/logquill/core/opt.py @@ -3,11 +3,11 @@ import sys from typing import TYPE_CHECKING, Any -from logquill.levels import Level -from logquill.records import LogRecord, build_llm_block +from logquill.core.levels import Level +from logquill.core.records import LogRecord, build_llm_block if TYPE_CHECKING: - from logquill.logger import Logger + from logquill.core.logger import Logger def resolve_lazy(meta: dict[str, Any]) -> dict[str, Any]: diff --git a/logquill/records.py b/logquill/core/records.py similarity index 99% rename from logquill/records.py rename to logquill/core/records.py index 01848f4..571aa13 100644 --- a/logquill/records.py +++ b/logquill/core/records.py @@ -6,7 +6,7 @@ from datetime import datetime, timezone from typing import Any, TypedDict, cast -from logquill.levels import Level +from logquill.core.levels import Level _logger = logging.getLogger("logquill") diff --git a/logquill/formatters/base.py b/logquill/formatters/base.py index 3349d01..337d5a5 100644 --- a/logquill/formatters/base.py +++ b/logquill/formatters/base.py @@ -2,7 +2,7 @@ from typing import Protocol -from logquill.records import LogRecord +from logquill.core.records import LogRecord class Formatter(Protocol): diff --git a/logquill/formatters/json_formatter.py b/logquill/formatters/json_formatter.py index 3bf2d13..84210cd 100644 --- a/logquill/formatters/json_formatter.py +++ b/logquill/formatters/json_formatter.py @@ -2,7 +2,7 @@ import json -from logquill.records import LogRecord +from logquill.core.records import LogRecord class JSONFormatter: diff --git a/logquill/formatters/logfmt_formatter.py b/logquill/formatters/logfmt_formatter.py index 4e977d4..9e71d64 100644 --- a/logquill/formatters/logfmt_formatter.py +++ b/logquill/formatters/logfmt_formatter.py @@ -5,7 +5,7 @@ from collections.abc import Mapping from typing import Any -from logquill.records import LogRecord +from logquill.core.records import LogRecord #: The core record fields, always emitted first. A `meta` key with one of #: these names is emitted as `meta.` so it can't shadow the field the diff --git a/logquill/formatters/text_formatter.py b/logquill/formatters/text_formatter.py index b74507b..ba6e669 100644 --- a/logquill/formatters/text_formatter.py +++ b/logquill/formatters/text_formatter.py @@ -4,7 +4,7 @@ from collections.abc import Mapping from typing import Any -from logquill.records import LogRecord +from logquill.core.records import LogRecord def format_text(record: Mapping[str, Any]) -> str: diff --git a/logquill/instrument/anthropic.py b/logquill/instrument/anthropic.py index 3ecf1f9..29f14a5 100644 --- a/logquill/instrument/anthropic.py +++ b/logquill/instrument/anthropic.py @@ -20,8 +20,8 @@ import time from typing import Any +from logquill.core.logger import Logger from logquill.instrument._util import Instrumenter, Patch, apply, record_safely -from logquill.logger import Logger def _record(logger: Logger, kwargs: dict[str, Any], response: Any, duration_ms: float) -> None: diff --git a/logquill/instrument/litellm.py b/logquill/instrument/litellm.py index ff6514c..4bd9421 100644 --- a/logquill/instrument/litellm.py +++ b/logquill/instrument/litellm.py @@ -17,8 +17,8 @@ import time from typing import Any +from logquill.core.logger import Logger from logquill.instrument._util import Instrumenter, Patch, apply, record_safely -from logquill.logger import Logger def _provider(response: Any) -> str | None: diff --git a/logquill/instrument/openai.py b/logquill/instrument/openai.py index 821677f..6db0735 100644 --- a/logquill/instrument/openai.py +++ b/logquill/instrument/openai.py @@ -16,8 +16,8 @@ import time from typing import Any +from logquill.core.logger import Logger from logquill.instrument._util import Instrumenter, Patch, apply, record_safely -from logquill.logger import Logger def _chat_finish_reason(response: Any) -> str | None: diff --git a/logquill/plugins/adaptive_sampling_plugin.py b/logquill/plugins/adaptive_sampling_plugin.py index a38bcc1..fbe6325 100644 --- a/logquill/plugins/adaptive_sampling_plugin.py +++ b/logquill/plugins/adaptive_sampling_plugin.py @@ -5,9 +5,9 @@ import time from typing import Callable -from logquill.levels import Level, parse_level +from logquill.core.levels import Level, parse_level +from logquill.core.records import LogRecord from logquill.plugins.plugin import Plugin -from logquill.records import LogRecord class AdaptiveSamplingPlugin(Plugin): diff --git a/logquill/plugins/alerting_plugin.py b/logquill/plugins/alerting_plugin.py index 107ba5b..844050c 100644 --- a/logquill/plugins/alerting_plugin.py +++ b/logquill/plugins/alerting_plugin.py @@ -4,9 +4,9 @@ import threading from typing import Callable -from logquill.levels import Level, parse_level +from logquill.core.levels import Level, parse_level +from logquill.core.records import LogRecord from logquill.plugins.plugin import Plugin -from logquill.records import LogRecord class _Window: diff --git a/logquill/plugins/apprise_alert_plugin.py b/logquill/plugins/apprise_alert_plugin.py index 6186a55..5b6f76b 100644 --- a/logquill/plugins/apprise_alert_plugin.py +++ b/logquill/plugins/apprise_alert_plugin.py @@ -4,8 +4,8 @@ from collections.abc import Sequence from typing import Any +from logquill.core.records import LogRecord from logquill.plugins.alerting_plugin import AlertingPlugin -from logquill.records import LogRecord # Apprise's own notification types, which each service maps onto its native # severity (a color, an emoji, a priority). diff --git a/logquill/plugins/context_plugin.py b/logquill/plugins/context_plugin.py index 5e21fc1..bd7b73a 100644 --- a/logquill/plugins/context_plugin.py +++ b/logquill/plugins/context_plugin.py @@ -2,8 +2,8 @@ from typing import Any +from logquill.core.records import LogRecord from logquill.plugins.plugin import Plugin -from logquill.records import LogRecord class ContextPlugin(Plugin): diff --git a/logquill/plugins/email_alert_plugin.py b/logquill/plugins/email_alert_plugin.py index aff6787..0ae39f3 100644 --- a/logquill/plugins/email_alert_plugin.py +++ b/logquill/plugins/email_alert_plugin.py @@ -4,8 +4,8 @@ from email.message import EmailMessage from typing import Any +from logquill.core.records import LogRecord from logquill.plugins.alerting_plugin import AlertingPlugin -from logquill.records import LogRecord class EmailAlertPlugin(AlertingPlugin): diff --git a/logquill/plugins/flight_recorder_plugin.py b/logquill/plugins/flight_recorder_plugin.py index a85b050..c877359 100644 --- a/logquill/plugins/flight_recorder_plugin.py +++ b/logquill/plugins/flight_recorder_plugin.py @@ -2,9 +2,9 @@ from collections import OrderedDict -from logquill.levels import Level, parse_level +from logquill.core.levels import Level, parse_level +from logquill.core.records import LogRecord from logquill.plugins.plugin import Plugin -from logquill.records import LogRecord from logquill.transports.transport import Transport diff --git a/logquill/plugins/metrics_plugin.py b/logquill/plugins/metrics_plugin.py new file mode 100644 index 0000000..8209d0d --- /dev/null +++ b/logquill/plugins/metrics_plugin.py @@ -0,0 +1,96 @@ +from __future__ import annotations + +from collections.abc import Mapping +from typing import Protocol + +from logquill.core.levels import Level, parse_level +from logquill.core.records import LogRecord +from logquill.plugins.plugin import Plugin + + +class MetricsBackend(Protocol): + """What `MetricsPlugin` needs from a metrics backend — implement this + shape for your own (a fake in tests, or a system none of the built-in + backends cover).""" + + def increment(self, name: str, *, tags: Mapping[str, str]) -> None: + """Adds 1 to the named counter, tagged with `tags`.""" + ... + + def observe(self, name: str, value: float, *, tags: Mapping[str, str]) -> None: + """Records one value into the named histogram, tagged with `tags`.""" + ... + + +class MetricsPlugin(Plugin): + """Turns specific log patterns into metric emissions — the + logs-to-metrics correlation the three-pillars observability model + expects, done as an opt-in plugin rather than a required part of core. + + - Every record at `error_at` (default `ERROR`) or above increments a + counter (`error_metric`, default `"logquill.errors"`), tagged by + `logger`. + - Any record carrying `meta.duration_ms` (every span-closing record, + and anything else that happens to set it) feeds a histogram + (`duration_metric`, default `"logquill.duration_ms"`), tagged by + `logger`. + + Runs in `after_log`, after the record has already been dispatched to + every transport — a slow or failing metrics backend can delay the next + log call processed by this plugin, but never the one it's reporting on, + and (like every plugin hook) a raising backend is caught by the + pipeline and routed to `on_error`, never crashing the caller. + + `backend` defaults to `OTelMetricsBackend()` — the OTel Metrics SDK, + the same optional dependency `OTLPTransport` already needs + (`pip install logquill[otel]`), so this is additive, not a new one. + Pass `StatsDBackend()`/`PrometheusPushgatewayBackend(...)` instead for + a team not yet on OTel. + """ + + def __init__( + self, + backend: MetricsBackend | None = None, + *, + error_metric: str = "logquill.errors", + duration_metric: str = "logquill.duration_ms", + error_at: int | str | Level = Level.ERROR, + ) -> None: + """`backend` is constructed lazily (an `OTelMetricsBackend()`) if + not given, so importing this module never imports the OTel SDK + unless you actually use the default.""" + self._backend = backend + self.error_metric = error_metric + self.duration_metric = duration_metric + self.error_at = parse_level(error_at) + + @property + def backend(self) -> MetricsBackend: + """The backend this plugin reports to — resolved on first use if + none was given at construction.""" + if self._backend is None: + from logquill.plugins.otel_metrics_backend import OTelMetricsBackend + + self._backend = OTelMetricsBackend() + return self._backend + + def after_log(self, record: LogRecord) -> None: + """Increments `error_metric` for a record at `error_at` or above; + feeds `duration_metric` for a record carrying `meta.duration_ms`.""" + tags = {"logger": record["logger"]} + + if Level[record["level"]] >= self.error_at: + self.backend.increment(self.error_metric, tags=tags) + + duration = record["meta"].get("duration_ms") + if isinstance(duration, (int, float)) and not isinstance(duration, bool): + self.backend.observe(self.duration_metric, float(duration), tags=tags) + + def close(self) -> None: + """Closes the backend, if it has a `close()` method (e.g. + `PrometheusPushgatewayBackend`, which flushes its accumulated + metrics on close). Not called automatically by `Logger.close()` — + call this yourself at shutdown, the same as `AlertingPlugin.close()`.""" + closer = getattr(self._backend, "close", None) + if callable(closer): + closer() diff --git a/logquill/plugins/otel_metrics_backend.py b/logquill/plugins/otel_metrics_backend.py new file mode 100644 index 0000000..c577481 --- /dev/null +++ b/logquill/plugins/otel_metrics_backend.py @@ -0,0 +1,59 @@ +from __future__ import annotations + +from collections.abc import Mapping +from typing import Any + + +class OTelMetricsBackend: + """`MetricsPlugin`'s default backend: records through the OpenTelemetry + Metrics SDK, the same optional dependency `OTLPTransport` already needs + (`pip install logquill[otel]`) — this is additive, not a new one. + + Lazily creates one `Counter`/`Histogram` instrument per metric name on + first use (the SDK requires creating an instrument before recording to + it, and there's no way to know every name a caller will ever pass in + advance), then just calls `.add()`/`.record()` — the SDK's own + `MeterProvider` and whatever reader/exporter it's configured with + (typically the OTLP Metrics exporter) controls batching and export, + the same as every other OTel instrument in a process. + """ + + def __init__(self, *, meter_name: str = "logquill", meter_provider: Any = None) -> None: + """`meter_provider` defaults to whatever + `opentelemetry.metrics.get_meter_provider()` currently returns — + usually set once, globally, at process startup; pass one explicitly + to scope this backend to its own provider instead (tests, or more + than one in the same process).""" + try: + from opentelemetry import metrics + except ImportError as exc: + raise ImportError( + "OTelMetricsBackend requires the OpenTelemetry SDK — install with " + "`pip install logquill[otel]`." + ) from exc + provider = meter_provider if meter_provider is not None else metrics.get_meter_provider() + self._meter = provider.get_meter(meter_name) + self._counters: dict[str, Any] = {} + self._histograms: dict[str, Any] = {} + + def _counter(self, name: str) -> Any: + counter = self._counters.get(name) + if counter is None: + counter = self._meter.create_counter(name) + self._counters[name] = counter + return counter + + def _histogram(self, name: str) -> Any: + histogram = self._histograms.get(name) + if histogram is None: + histogram = self._meter.create_histogram(name) + self._histograms[name] = histogram + return histogram + + def increment(self, name: str, *, tags: Mapping[str, str]) -> None: + """Adds 1 to the named counter.""" + self._counter(name).add(1, attributes=dict(tags)) + + def observe(self, name: str, value: float, *, tags: Mapping[str, str]) -> None: + """Records one value into the named histogram.""" + self._histogram(name).record(value, attributes=dict(tags)) diff --git a/logquill/plugins/pagerduty_alert_plugin.py b/logquill/plugins/pagerduty_alert_plugin.py index 5d249e7..36871c2 100644 --- a/logquill/plugins/pagerduty_alert_plugin.py +++ b/logquill/plugins/pagerduty_alert_plugin.py @@ -4,8 +4,8 @@ import urllib.request from typing import Any +from logquill.core.records import LogRecord from logquill.plugins.alerting_plugin import AlertingPlugin -from logquill.records import LogRecord _ENDPOINT = "https://events.pagerduty.com/v2/enqueue" _SEVERITY = {"ERROR": "error", "FATAL": "critical"} diff --git a/logquill/plugins/pii_redact_plugin.py b/logquill/plugins/pii_redact_plugin.py index 5c806fe..ad1f706 100644 --- a/logquill/plugins/pii_redact_plugin.py +++ b/logquill/plugins/pii_redact_plugin.py @@ -4,8 +4,8 @@ from collections.abc import Mapping, Sequence from typing import Any +from logquill.core.records import LogRecord from logquill.plugins.plugin import Plugin -from logquill.records import LogRecord #: Syntactic (not semantic) patterns — matched on shape, so both false #: positives (a random 9-digit number) and false negatives (anything that diff --git a/logquill/plugins/plugin.py b/logquill/plugins/plugin.py index 1bb4187..45adc7e 100644 --- a/logquill/plugins/plugin.py +++ b/logquill/plugins/plugin.py @@ -2,7 +2,7 @@ from typing import Callable -from logquill.records import LogRecord +from logquill.core.records import LogRecord MiddlewareFunc = Callable[[LogRecord], "LogRecord | None"] diff --git a/logquill/plugins/prometheus_pushgateway_backend.py b/logquill/plugins/prometheus_pushgateway_backend.py new file mode 100644 index 0000000..d091a00 --- /dev/null +++ b/logquill/plugins/prometheus_pushgateway_backend.py @@ -0,0 +1,117 @@ +from __future__ import annotations + +import re +import threading +import urllib.request +from collections.abc import Mapping +from typing import Callable + +_INVALID_NAME_CHARS = re.compile(r"[^a-zA-Z0-9_:]") +_LEADING_DIGIT = re.compile(r"^[0-9]") + + +def _sanitize(name: str) -> str: + sanitized = _INVALID_NAME_CHARS.sub("_", name) + return f"_{sanitized}" if _LEADING_DIGIT.match(sanitized) else sanitized + + +def _labels(tags: tuple[tuple[str, str], ...]) -> str: + if not tags: + return "" + pairs = ",".join(f'{key}="{value}"' for key, value in tags) + return f"{{{pairs}}}" + + +class PrometheusPushgatewayBackend: + """A `MetricsPlugin` backend for teams running a Prometheus Pushgateway + instead of OTel — stdlib `urllib` only, no new dependency (not even + `prometheus_client`). + + Pushgateway is designed around periodic batch pushes, not one HTTP + request per metric — so `increment()`/`observe()` only accumulate + in-memory counters/histogram sums; nothing reaches the network until + `push()` is called (by hand, on your own timer, or via `close()`, + which calls it once). This also means metrics recorded on the hot log + path never pay for an HTTP round trip. + + Metric names are sanitized to Prometheus's `[a-zA-Z_:][a-zA-Z0-9_:]*` + rule (e.g. `logquill.errors` becomes `logquill_errors`); `tags` become + labels. + """ + + def __init__( + self, + gateway_url: str, + *, + job_name: str = "logquill", + timeout: float = 5.0, + sender: Callable[[str, bytes], None] | None = None, + ) -> None: + """`sender(url, body)` defaults to a stdlib `urllib`-based `PUT` + (Pushgateway's "replace this job's metrics" semantics); override + for a fake in tests.""" + self._gateway_url = gateway_url.rstrip("/") + self._job_name = job_name + self._timeout = timeout + self._sender = sender or self._urllib_put + self._lock = threading.Lock() + self._counters: dict[tuple[str, tuple[tuple[str, str], ...]], float] = {} + self._histogram_sums: dict[tuple[str, tuple[tuple[str, str], ...]], float] = {} + self._histogram_counts: dict[tuple[str, tuple[tuple[str, str], ...]], int] = {} + + def _urllib_put(self, url: str, body: bytes) -> None: + request = urllib.request.Request(url, data=body, method="PUT") + with urllib.request.urlopen(request, timeout=self._timeout) as response: # noqa: S310 + if response.status >= 400: + raise RuntimeError( + f"PrometheusPushgatewayBackend: pushgateway returned HTTP " + f"{response.status} — check the gateway URL is reachable and correct" + ) + + @staticmethod + def _key(name: str, tags: Mapping[str, str]) -> tuple[str, tuple[tuple[str, str], ...]]: + return name, tuple(sorted(tags.items())) + + def increment(self, name: str, *, tags: Mapping[str, str]) -> None: + """Adds 1 to the named counter's in-memory total.""" + key = self._key(name, tags) + with self._lock: + self._counters[key] = self._counters.get(key, 0.0) + 1 + + def observe(self, name: str, value: float, *, tags: Mapping[str, str]) -> None: + """Adds one sample to the named histogram's in-memory sum/count.""" + key = self._key(name, tags) + with self._lock: + self._histogram_sums[key] = self._histogram_sums.get(key, 0.0) + value + self._histogram_counts[key] = self._histogram_counts.get(key, 0) + 1 + + def _render(self) -> str: + with self._lock: + counters = dict(self._counters) + sums = dict(self._histogram_sums) + counts = dict(self._histogram_counts) + + lines = [] + for (name, tags), value in counters.items(): + lines.append(f"{_sanitize(name)}{_labels(tags)} {value}") + for key, total in sums.items(): + name, tags = key + metric = _sanitize(name) + lines.append(f"{metric}_sum{_labels(tags)} {total}") + lines.append(f"{metric}_count{_labels(tags)} {counts[key]}") + return "\n".join(lines) + "\n" if lines else "" + + def push(self) -> None: + """Sends every metric accumulated so far to the pushgateway, as one + request — the in-memory totals are *not* cleared, since Pushgateway + itself expects each push to carry the job's current cumulative + state, the same way a `/metrics` scrape would.""" + body = self._render() + if not body: + return + url = f"{self._gateway_url}/metrics/job/{self._job_name}" + self._sender(url, body.encode("utf-8")) + + def close(self) -> None: + """Pushes one final time.""" + self.push() diff --git a/logquill/plugins/rate_limit_plugin.py b/logquill/plugins/rate_limit_plugin.py index f084cb7..45059a5 100644 --- a/logquill/plugins/rate_limit_plugin.py +++ b/logquill/plugins/rate_limit_plugin.py @@ -4,8 +4,8 @@ from collections import OrderedDict from typing import Callable, Hashable +from logquill.core.records import LogRecord from logquill.plugins.plugin import Plugin -from logquill.records import LogRecord RateLimitKeyFunc = Callable[[LogRecord], Hashable] diff --git a/logquill/plugins/redact_plugin.py b/logquill/plugins/redact_plugin.py index f2b4d4b..2dfc473 100644 --- a/logquill/plugins/redact_plugin.py +++ b/logquill/plugins/redact_plugin.py @@ -2,9 +2,9 @@ from collections.abc import Iterable +from logquill.core.records import LogRecord from logquill.plugins.plugin import Plugin -from logquill.privacy import FieldClass, keys_in_class -from logquill.records import LogRecord +from logquill.privacy.privacy import FieldClass, keys_in_class #: The `secret`-classed keys in `logquill.privacy.FIELD_CLASSES` — kept as its #: own name since it's also `RedactPlugin`'s default, and existing code may diff --git a/logquill/plugins/run_plugin.py b/logquill/plugins/run_plugin.py index d6b7e1f..d7c00af 100644 --- a/logquill/plugins/run_plugin.py +++ b/logquill/plugins/run_plugin.py @@ -2,8 +2,8 @@ import uuid +from logquill.core.records import LogRecord from logquill.plugins.plugin import Plugin -from logquill.records import LogRecord class RunPlugin(Plugin): diff --git a/logquill/plugins/run_summary_plugin.py b/logquill/plugins/run_summary_plugin.py index 3a237e6..2522ed5 100644 --- a/logquill/plugins/run_summary_plugin.py +++ b/logquill/plugins/run_summary_plugin.py @@ -4,9 +4,9 @@ from dataclasses import dataclass, field from typing import Any -from logquill.levels import Level +from logquill.core.levels import Level +from logquill.core.records import LogRecord, create_record from logquill.plugins.plugin import Plugin -from logquill.records import LogRecord, create_record from logquill.transports.transport import Transport diff --git a/logquill/plugins/sampling_plugin.py b/logquill/plugins/sampling_plugin.py index 95427b1..1331e2d 100644 --- a/logquill/plugins/sampling_plugin.py +++ b/logquill/plugins/sampling_plugin.py @@ -4,9 +4,9 @@ from collections import OrderedDict from typing import Callable -from logquill.levels import Level, parse_level +from logquill.core.levels import Level, parse_level +from logquill.core.records import LogRecord from logquill.plugins.plugin import Plugin -from logquill.records import LogRecord from logquill.transports.transport import Transport diff --git a/logquill/plugins/slack_alert_plugin.py b/logquill/plugins/slack_alert_plugin.py index 2e0b916..5253a37 100644 --- a/logquill/plugins/slack_alert_plugin.py +++ b/logquill/plugins/slack_alert_plugin.py @@ -4,8 +4,8 @@ import urllib.request from typing import Any +from logquill.core.records import LogRecord from logquill.plugins.alerting_plugin import AlertingPlugin -from logquill.records import LogRecord class SlackAlertPlugin(AlertingPlugin): diff --git a/logquill/plugins/statsd_metrics_backend.py b/logquill/plugins/statsd_metrics_backend.py new file mode 100644 index 0000000..74bd6c3 --- /dev/null +++ b/logquill/plugins/statsd_metrics_backend.py @@ -0,0 +1,80 @@ +from __future__ import annotations + +import logging +import socket +from collections.abc import Mapping +from typing import Protocol + +_logger = logging.getLogger("logquill") + + +class UDPSocketLike(Protocol): + """The subset of a `socket.socket` this backend calls — implement this + shape to inject a fake in tests without sending real UDP packets.""" + + def sendto(self, data: bytes, address: tuple[str, int]) -> int: + """Sends `data` to `address`.""" + ... + + def close(self) -> None: + """Releases the socket.""" + ... + + +class StatsDBackend: + """A `MetricsPlugin` backend for StatsD (or a DogStatsD-compatible + agent) — stdlib `socket` only, no new dependency. + + Sends the StatsD wire format over UDP: `name:value|type`, with tags + appended as `|#key:value,key:value` — the DogStatsD extension, since + plain StatsD has no tag concept at all and this extension is the + closest thing to a de facto standard (Datadog's agent, Telegraf, and + most StatsD-compatible collectors understand it; a collector that + doesn't just ignores the trailing segment it doesn't recognize). + + UDP sends never block waiting for an ack and are fire-and-forget by + design — a send that fails outright (no route, closed socket) is + logged and dropped, never raised into the caller. + """ + + def __init__( + self, + *, + host: str = "localhost", + port: int = 8125, + prefix: str | None = None, + sock: UDPSocketLike | None = None, + ) -> None: + """`prefix`, if given, is prepended to every metric name as + `f"{prefix}.{name}"`. `sock` is injectable for tests; defaults to a + real UDP `socket.socket`.""" + self._host = host + self._port = port + self._prefix = prefix + self._sock: UDPSocketLike = sock or socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + + def _full_name(self, name: str) -> str: + return f"{self._prefix}.{name}" if self._prefix else name + + def _tag_suffix(self, tags: Mapping[str, str]) -> str: + if not tags: + return "" + return "|#" + ",".join(f"{key}:{value}" for key, value in tags.items()) + + def _send(self, line: str) -> None: + try: + self._sock.sendto(line.encode("utf-8"), (self._host, self._port)) + except OSError: + _logger.debug("StatsDBackend: failed to send a metric to %s:%d", self._host, self._port) + + def increment(self, name: str, *, tags: Mapping[str, str]) -> None: + """Sends a StatsD counter increment (`|c`) of 1.""" + self._send(f"{self._full_name(name)}:1|c{self._tag_suffix(tags)}") + + def observe(self, name: str, value: float, *, tags: Mapping[str, str]) -> None: + """Sends a StatsD histogram sample (`|h`).""" + self._send(f"{self._full_name(name)}:{value}|h{self._tag_suffix(tags)}") + + def close(self) -> None: + """Releases the underlying socket.""" + self._sock.close() diff --git a/logquill/plugins/tamper_evident_plugin.py b/logquill/plugins/tamper_evident_plugin.py index 49d6c89..19b0e34 100644 --- a/logquill/plugins/tamper_evident_plugin.py +++ b/logquill/plugins/tamper_evident_plugin.py @@ -7,8 +7,8 @@ from dataclasses import dataclass from typing import Any +from logquill.core.records import LEGACY_SCHEMA_VERSION, LogRecord from logquill.plugins.plugin import Plugin -from logquill.records import LEGACY_SCHEMA_VERSION, LogRecord GENESIS_HASH = "0" * 64 diff --git a/logquill/plugins/trace_context_plugin.py b/logquill/plugins/trace_context_plugin.py index 675f656..4c86f59 100644 --- a/logquill/plugins/trace_context_plugin.py +++ b/logquill/plugins/trace_context_plugin.py @@ -4,8 +4,8 @@ import secrets from contextvars import ContextVar, Token +from logquill.core.records import LogRecord from logquill.plugins.plugin import Plugin -from logquill.records import LogRecord _current_traceparent: ContextVar[str | None] = ContextVar("logquill_traceparent", default=None) diff --git a/logquill/privacy/__init__.py b/logquill/privacy/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/logquill/audit.py b/logquill/privacy/audit.py similarity index 94% rename from logquill/audit.py rename to logquill/privacy/audit.py index 05f5176..cc20000 100644 --- a/logquill/audit.py +++ b/logquill/privacy/audit.py @@ -14,15 +14,17 @@ from typing import Any -from logquill.levels import Level -from logquill.logger import Logger +from logquill.core.levels import Level +from logquill.core.logger import Logger +from logquill.core.records import ( + LogRecord, # noqa: F401 — resolves MiddlewareFunc's forward reference +) from logquill.plugins.context_plugin import ContextPlugin from logquill.plugins.pii_redact_plugin import PIIRedactPlugin from logquill.plugins.plugin import MiddlewareFunc, Plugin from logquill.plugins.redact_plugin import RedactPlugin from logquill.plugins.tamper_evident_plugin import TamperEvidentPlugin -from logquill.privacy import ContentCapturePolicy -from logquill.records import LogRecord # noqa: F401 — resolves MiddlewareFunc's forward reference +from logquill.privacy.privacy import ContentCapturePolicy from logquill.transports.transport import Transport diff --git a/logquill/privacy.py b/logquill/privacy/privacy.py similarity index 100% rename from logquill/privacy.py rename to logquill/privacy/privacy.py diff --git a/logquill/runtime/__init__.py b/logquill/runtime/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/logquill/runtime_level.py b/logquill/runtime/runtime_level.py similarity index 99% rename from logquill/runtime_level.py rename to logquill/runtime/runtime_level.py index 8896b88..1d14857 100644 --- a/logquill/runtime_level.py +++ b/logquill/runtime/runtime_level.py @@ -29,7 +29,7 @@ from pathlib import Path from typing import Callable -from logquill.logger import Logger +from logquill.core.logger import Logger _logger = logging.getLogger("logquill") diff --git a/logquill/serverless.py b/logquill/runtime/serverless.py similarity index 98% rename from logquill/serverless.py rename to logquill/runtime/serverless.py index 3ac37d6..7502363 100644 --- a/logquill/serverless.py +++ b/logquill/runtime/serverless.py @@ -4,7 +4,7 @@ import inspect from typing import Any, Callable, Sequence, TypeVar, cast -from logquill.logger import Logger +from logquill.core.logger import Logger F = TypeVar("F", bound=Callable[..., Any]) diff --git a/logquill/shutdown.py b/logquill/runtime/shutdown.py similarity index 97% rename from logquill/shutdown.py rename to logquill/runtime/shutdown.py index 5b11421..a7250c2 100644 --- a/logquill/shutdown.py +++ b/logquill/runtime/shutdown.py @@ -6,11 +6,11 @@ import weakref from typing import TYPE_CHECKING +from logquill.runtime.worker import AsyncWorker from logquill.transports.transport import Transport -from logquill.worker import AsyncWorker if TYPE_CHECKING: - from logquill.logger import Logger + from logquill.core.logger import Logger _logger = logging.getLogger("logquill") diff --git a/logquill/toggle.py b/logquill/runtime/toggle.py similarity index 100% rename from logquill/toggle.py rename to logquill/runtime/toggle.py diff --git a/logquill/worker.py b/logquill/runtime/worker.py similarity index 100% rename from logquill/worker.py rename to logquill/runtime/worker.py diff --git a/logquill/tracing/__init__.py b/logquill/tracing/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/logquill/mcp.py b/logquill/tracing/mcp.py similarity index 98% rename from logquill/mcp.py rename to logquill/tracing/mcp.py index b8a6eef..c46f301 100644 --- a/logquill/mcp.py +++ b/logquill/tracing/mcp.py @@ -20,7 +20,7 @@ from contextlib import contextmanager from typing import Any -from logquill.context import bind_context +from logquill.core.context import bind_context from logquill.plugins.trace_context_plugin import ( generate_trace_id, reset_traceparent, diff --git a/logquill/retry.py b/logquill/tracing/retry.py similarity index 100% rename from logquill/retry.py rename to logquill/tracing/retry.py diff --git a/logquill/semconv.py b/logquill/tracing/semconv.py similarity index 99% rename from logquill/semconv.py rename to logquill/tracing/semconv.py index 8d95250..cd558d8 100644 --- a/logquill/semconv.py +++ b/logquill/tracing/semconv.py @@ -26,7 +26,7 @@ from datetime import datetime from typing import Any, Literal -from logquill.records import LogRecord +from logquill.core.records import LogRecord _logger = logging.getLogger("logquill") diff --git a/logquill/span.py b/logquill/tracing/span.py similarity index 98% rename from logquill/span.py rename to logquill/tracing/span.py index 8dd0fd9..5b416a1 100644 --- a/logquill/span.py +++ b/logquill/tracing/span.py @@ -8,10 +8,10 @@ from types import TracebackType from typing import TYPE_CHECKING, Any, Callable -from logquill.levels import Level +from logquill.core.levels import Level if TYPE_CHECKING: - from logquill.logger import Logger + from logquill.core.logger import Logger _logger = logging.getLogger("logquill") diff --git a/logquill/transports/batching_transport.py b/logquill/transports/batching_transport.py index 863dfc0..c95b6af 100644 --- a/logquill/transports/batching_transport.py +++ b/logquill/transports/batching_transport.py @@ -5,8 +5,8 @@ from abc import abstractmethod from typing import Generic, Sequence, TypeVar, cast +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.records import LogRecord from logquill.transports.transport import Transport T = TypeVar("T") diff --git a/logquill/transports/cloud/app_insights_transport.py b/logquill/transports/cloud/app_insights_transport.py index 38bb460..86c8bb5 100644 --- a/logquill/transports/cloud/app_insights_transport.py +++ b/logquill/transports/cloud/app_insights_transport.py @@ -4,8 +4,8 @@ import urllib.request from typing import Callable, Sequence +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.records import LogRecord from logquill.transports.batching_transport import BatchingTransport Sender = Callable[[str, Sequence[str]], None] diff --git a/logquill/transports/cloud/cloud_logging_transport.py b/logquill/transports/cloud/cloud_logging_transport.py index a984df3..5e38758 100644 --- a/logquill/transports/cloud/cloud_logging_transport.py +++ b/logquill/transports/cloud/cloud_logging_transport.py @@ -2,8 +2,8 @@ from typing import Any, Protocol, Sequence, cast +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.records import LogRecord from logquill.transports.batching_transport import BatchingTransport _SEVERITY = { diff --git a/logquill/transports/cloud/cloudwatch_transport.py b/logquill/transports/cloud/cloudwatch_transport.py index b29c806..a3a8509 100644 --- a/logquill/transports/cloud/cloudwatch_transport.py +++ b/logquill/transports/cloud/cloudwatch_transport.py @@ -2,8 +2,8 @@ from typing import Any, Protocol, Sequence, cast +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.records import LogRecord from logquill.transports.batching_transport import BatchingTransport diff --git a/logquill/transports/cloud/datadog_transport.py b/logquill/transports/cloud/datadog_transport.py index a3cf6f0..2f45c3a 100644 --- a/logquill/transports/cloud/datadog_transport.py +++ b/logquill/transports/cloud/datadog_transport.py @@ -4,8 +4,8 @@ import urllib.request from typing import Callable, Sequence +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.records import LogRecord from logquill.transports.batching_transport import BatchingTransport DatadogSender = Callable[[str, str, Sequence[str]], None] diff --git a/logquill/transports/cloud/elasticsearch_transport.py b/logquill/transports/cloud/elasticsearch_transport.py index dffb7f2..6efe673 100644 --- a/logquill/transports/cloud/elasticsearch_transport.py +++ b/logquill/transports/cloud/elasticsearch_transport.py @@ -5,8 +5,8 @@ import urllib.request from typing import Callable, Sequence +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.records import LogRecord from logquill.transports.batching_transport import BatchingTransport ElasticsearchSender = Callable[[str, Sequence[str]], None] diff --git a/logquill/transports/cloud/new_relic_transport.py b/logquill/transports/cloud/new_relic_transport.py index 0a54df1..6886d87 100644 --- a/logquill/transports/cloud/new_relic_transport.py +++ b/logquill/transports/cloud/new_relic_transport.py @@ -9,8 +9,8 @@ from email.utils import parsedate_to_datetime from typing import Callable, Literal, Sequence, TypedDict, cast +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.records import LogRecord from logquill.transports.batching_transport import BatchingTransport NewRelicRegion = Literal["US", "EU"] diff --git a/logquill/transports/cloud/syslog_transport.py b/logquill/transports/cloud/syslog_transport.py index 4745c6f..f5f9993 100644 --- a/logquill/transports/cloud/syslog_transport.py +++ b/logquill/transports/cloud/syslog_transport.py @@ -4,9 +4,9 @@ import socket from typing import Callable +from logquill.core.levels import Level, parse_level +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.levels import Level, parse_level -from logquill.records import LogRecord from logquill.transports.transport import Transport SyslogSender = Callable[[bytes], None] diff --git a/logquill/transports/console_transport.py b/logquill/transports/console_transport.py index a631160..cc0cee4 100644 --- a/logquill/transports/console_transport.py +++ b/logquill/transports/console_transport.py @@ -3,9 +3,9 @@ import sys from typing import TextIO +from logquill.core.levels import Level, parse_level +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.levels import Level, parse_level -from logquill.records import LogRecord from logquill.transports.transport import Transport _COLORS = { diff --git a/logquill/transports/file_transport.py b/logquill/transports/file_transport.py index e4786eb..bab6f3f 100644 --- a/logquill/transports/file_transport.py +++ b/logquill/transports/file_transport.py @@ -3,8 +3,8 @@ from pathlib import Path from typing import Any, BinaryIO, TextIO, cast +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.records import LogRecord from logquill.transports.transport import Transport diff --git a/logquill/transports/graph/__init__.py b/logquill/transports/graph/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/logquill/transports/graph/base_graph_transport.py b/logquill/transports/graph/base_graph_transport.py new file mode 100644 index 0000000..a545d67 --- /dev/null +++ b/logquill/transports/graph/base_graph_transport.py @@ -0,0 +1,147 @@ +from __future__ import annotations + +import json +import uuid +from abc import abstractmethod +from dataclasses import dataclass +from typing import Any, Sequence + +from logquill.core.records import LogRecord +from logquill.formatters import Formatter +from logquill.transports.batching_transport import BatchingTransport + +#: One shared Cypher statement, built once here, that every concrete graph +#: backend executes as-is against its own driver. Written in the portable +#: `FOREACH`-with-`CASE` idiom rather than Neo4j's newer `CALL {}` subquery +#: syntax, since `FOREACH` has been part of Cypher since the beginning and +#: is far more likely to be supported by every openCypher-compatible +#: backend this category covers (including Neptune's), not just Neo4j's own +#: more recent extensions. +CYPHER = ( + "UNWIND $rows AS row " + "MERGE (n:LogRecord {key: row.key}) " + "SET n += row.props " + "FOREACH (_ IN CASE WHEN row.session_key IS NULL THEN [] ELSE [1] END | " + " MERGE (s:Session {key: row.session_key}) " + " MERGE (s)-[:HAS_RECORD]->(n) " + ") " + "FOREACH (_ IN CASE WHEN row.parent_key IS NULL THEN [] ELSE [1] END | " + " MERGE (p:LogRecord {key: row.parent_key}) " + " MERGE (p)-[:PARENT_OF]->(n) " + ")" +) + + +@dataclass(frozen=True) +class GraphRow: + """One record, already reduced to exactly what `CYPHER`'s `$rows` + parameter needs — a plain, JSON-serializable shape every backend's + driver can pass through its own parameter-binding call unchanged.""" + + key: str + props: dict[str, Any] + session_key: str | None + parent_key: str | None + + +class BaseGraphTransport(BatchingTransport[GraphRow]): + """Shared base for every graph-database sink: the one piece of logic + every backend needs is identical regardless of which database it talks + to — record → node, `parent_span_id` → a `PARENT_OF` edge, `run_id`/ + `trace_id` → a `Session` node every record in that run/trace is linked + to via `HAS_RECORD` — so that logic lives here, once, as `CYPHER`. + A concrete backend only has to know how to execute one parameterized + Cypher/openCypher query against its own driver; see `_execute`. + + **Node identity.** A record that carries its own `span_id` (a span's + closing record) is keyed by that id — stable and globally unique, so + re-delivering the same span's record twice merges onto the same node + rather than duplicating it. A record with no `span_id` of its own (an + ordinary `.action()`/`.thought()`/... line) has no naturally stable + identity, so it's keyed by a fresh random id each time — it still + becomes its own node and still gets a `PARENT_OF` edge from whichever + span it's nested in, it just isn't idempotent under a retry the way a + span's own node is. + + **Reconstructing the tree.** Because every node's `PARENT_OF` edges + point from `meta.parent_span_id` to the record's own key, a Cypher + traversal from any root (a node with no incoming `PARENT_OF` edge) + reproduces the exact span hierarchy that was logged — no manual + graph-writing code at any call site, and switching to a different + concrete backend never touches a call site either, only which + transport class is constructed. + + **Session grouping.** `run_key`/`trace_key` name which `meta` fields + this instance groups records under; `run_key` is tried first, falling + back to `trace_key` — a record with neither is still its own node, it + just isn't linked to any `Session`. + + Batched the same way every other sink in this category is: one + multi-row Cypher statement per flush (`UNWIND $rows`), never one query + per record. + """ + + def __init__( + self, + *, + formatter: Formatter | None = None, + max_records: int = 100, + max_bytes: int = 1_000_000, + run_key: str = "run_id", + trace_key: str = "trace_id", + ) -> None: + super().__init__(formatter=formatter, max_records=max_records, max_bytes=max_bytes) + self.run_key = run_key + self.trace_key = trace_key + + def _to_item(self, formatted: str, record: LogRecord) -> GraphRow: + meta = record["meta"] + span_id = meta.get("span_id") + key = span_id if isinstance(span_id, str) and span_id else f"leaf-{uuid.uuid4().hex}" + + session_key = meta.get(self.run_key) + if not isinstance(session_key, str) or not session_key: + session_key = meta.get(self.trace_key) + if not isinstance(session_key, str) or not session_key: + session_key = None + + parent_key = meta.get("parent_span_id") + if not isinstance(parent_key, str) or not parent_key or parent_key == key: + parent_key = None + + props: dict[str, Any] = { + "timestamp": record["timestamp"], + "level": record["level"], + "logger": record["logger"], + "message": record["message"], + "meta": json.dumps(meta, default=str, separators=(",", ":")), + } + for field in ("run_id", "trace_id", "span_id", "parent_span_id"): + value = meta.get(field) + if isinstance(value, str) and value: + props[field] = value + + return GraphRow(key=key, props=props, session_key=session_key, parent_key=parent_key) + + def _size_of(self, item: GraphRow) -> int: + return len(json.dumps(item.props, default=str).encode("utf-8")) + len(item.key) + 32 + + def _send_batch(self, batch: Sequence[GraphRow]) -> None: + rows = [ + { + "key": row.key, + "props": row.props, + "session_key": row.session_key, + "parent_key": row.parent_key, + } + for row in batch + ] + self._execute(CYPHER, {"rows": rows}) + + @abstractmethod + def _execute(self, query: str, parameters: dict[str, Any]) -> None: + """Runs `query` (always `CYPHER`) against this backend's own + driver with `parameters`. Called only with a non-empty batch; a + raised exception is caught by `BatchingTransport.flush()` and + logged, not propagated to the caller.""" + ... diff --git a/logquill/transports/graph/falkordb_transport.py b/logquill/transports/graph/falkordb_transport.py new file mode 100644 index 0000000..7c0f44d --- /dev/null +++ b/logquill/transports/graph/falkordb_transport.py @@ -0,0 +1,102 @@ +from __future__ import annotations + +from typing import Any, Protocol + +from logquill.formatters import Formatter +from logquill.transports.graph.base_graph_transport import BaseGraphTransport + + +class FalkorDBGraphLike(Protocol): + """The subset of `falkordb-py`'s selected-graph object this transport + calls — implement this shape to inject a fake in tests without + installing the real driver.""" + + def query(self, q: str, params: dict[str, Any] | None = None) -> object: + """Run one parameterized Cypher query against this graph.""" + ... + + +class FalkorDBClientLike(Protocol): + """The subset of `falkordb-py`'s client this transport calls — + implement this shape to inject a fake in tests without installing the + real driver.""" + + def select_graph(self, name: str) -> FalkorDBGraphLike: + """Selects (creating if needed) the named graph.""" + ... + + def close(self) -> None: + """Release the client's connection.""" + ... + + +class FalkorDBTransport(BaseGraphTransport): + """Writes the span tree to FalkorDB — Redis-based and Cypher-compatible, + the lightweight, self-hosted-friendly option in this category. + + Pass `client` to inject an already-constructed `falkordb.FalkorDB` + client (or a fake, for tests), or `host`/`port`/`graph_name` to let this + transport connect its own via the optional `falkordb` peer dependency + (`pip install logquill[falkordb]`). + """ + + def __init__( + self, + *, + client: FalkorDBClientLike | None = None, + host: str = "localhost", + port: int = 6379, + graph_name: str = "logquill", + formatter: Formatter | None = None, + max_records: int = 100, + max_bytes: int = 1_000_000, + run_key: str = "run_id", + trace_key: str = "trace_id", + ) -> None: + """`host`/`port`/`graph_name` are used only when this transport + connects its own `falkordb` client (ignored if `client` is given).""" + super().__init__( + formatter=formatter, + max_records=max_records, + max_bytes=max_bytes, + run_key=run_key, + trace_key=trace_key, + ) + self._injected = client + self._host = host + self._port = port + self._graph_name = graph_name + self._client: FalkorDBClientLike | None = None + self._graph: FalkorDBGraphLike | None = None + + def _resolved_graph(self) -> FalkorDBGraphLike: + if self._graph is not None: + return self._graph + client = self._injected + if client is None: + if self._client is None: + self._client = self._connect() + client = self._client + self._graph = client.select_graph(self._graph_name) + return self._graph + + def _connect(self) -> FalkorDBClientLike: + try: + from falkordb import FalkorDB # type: ignore[import-not-found] + except ImportError as exc: + raise ImportError( + "FalkorDBTransport requires the optional `falkordb` package — install with " + "`pip install logquill[falkordb]`." + ) from exc + return FalkorDB(host=self._host, port=self._port) # type: ignore[no-any-return] + + def _execute(self, query: str, parameters: dict[str, Any]) -> None: + self._resolved_graph().query(query, parameters) + + def close(self) -> None: + """Flushes any remaining buffered records, then releases the + client's connection (only if this transport connected it itself — + an injected `client` is left for its owner to close).""" + super().close() + if self._injected is None and self._client is not None: + self._client.close() diff --git a/logquill/transports/graph/neo4j_transport.py b/logquill/transports/graph/neo4j_transport.py new file mode 100644 index 0000000..0504a44 --- /dev/null +++ b/logquill/transports/graph/neo4j_transport.py @@ -0,0 +1,99 @@ +from __future__ import annotations + +from typing import Any, ContextManager, Protocol + +from logquill.formatters import Formatter +from logquill.transports.graph.base_graph_transport import BaseGraphTransport + + +class Neo4jSessionLike(Protocol): + """The subset of the `neo4j` driver's `Session` this transport calls — + implement this shape to inject a fake in tests without installing the + real driver.""" + + def run(self, query: str, parameters: dict[str, Any]) -> object: + """Run one parameterized Cypher query.""" + ... + + +class Neo4jDriverLike(Protocol): + """The subset of the `neo4j` driver's `Driver` this transport calls — + implement this shape to inject a fake in tests without installing the + real driver.""" + + def session(self, **kwargs: Any) -> ContextManager[Neo4jSessionLike]: + """Open a session scoped to one database/transaction context.""" + ... + + def close(self) -> None: + """Release the driver's connection pool.""" + ... + + +class Neo4jTransport(BaseGraphTransport): + """Writes the span tree to Neo4j — the reference/most widely deployed + backend in this category, and the first one implemented here. + + Pass `driver` to inject an already-constructed `neo4j.GraphDatabase` + driver (or a fake, for tests), or `uri`/`auth`/`database` to let this + transport connect its own via the optional `neo4j` peer dependency + (`pip install logquill[neo4j]`). + """ + + def __init__( + self, + *, + driver: Neo4jDriverLike | None = None, + uri: str = "bolt://localhost:7687", + auth: tuple[str, str] | None = None, + database: str | None = None, + formatter: Formatter | None = None, + max_records: int = 100, + max_bytes: int = 1_000_000, + run_key: str = "run_id", + trace_key: str = "trace_id", + ) -> None: + """`uri`/`auth`/`database` are used only when this transport + connects its own `neo4j` driver (ignored if `driver` is given).""" + super().__init__( + formatter=formatter, + max_records=max_records, + max_bytes=max_bytes, + run_key=run_key, + trace_key=trace_key, + ) + self._injected = driver + self._uri = uri + self._auth = auth + self._database = database + self._driver: Neo4jDriverLike | None = None + + def _resolved_driver(self) -> Neo4jDriverLike: + if self._injected is not None: + return self._injected + if self._driver is None: + self._driver = self._connect() + return self._driver + + def _connect(self) -> Neo4jDriverLike: + try: + from neo4j import GraphDatabase # type: ignore[import-not-found] + except ImportError as exc: + raise ImportError( + "Neo4jTransport requires the optional `neo4j` package — install with " + "`pip install logquill[neo4j]`." + ) from exc + return GraphDatabase.driver(self._uri, auth=self._auth) # type: ignore[no-any-return] + + def _execute(self, query: str, parameters: dict[str, Any]) -> None: + session_kwargs = {"database": self._database} if self._database else {} + with self._resolved_driver().session(**session_kwargs) as session: + session.run(query, parameters) + + def close(self) -> None: + """Flushes any remaining buffered records, then releases the + driver's connection pool (only if this transport connected it + itself — an injected `driver` is left for its owner to close).""" + super().close() + if self._injected is None and self._driver is not None: + self._driver.close() diff --git a/logquill/transports/graph/neptune_transport.py b/logquill/transports/graph/neptune_transport.py new file mode 100644 index 0000000..d67202a --- /dev/null +++ b/logquill/transports/graph/neptune_transport.py @@ -0,0 +1,83 @@ +from __future__ import annotations + +import json +from typing import Any, Protocol + +from logquill.formatters import Formatter +from logquill.transports.graph.base_graph_transport import BaseGraphTransport + + +class NeptuneDataClientLike(Protocol): + """The subset of `boto3`'s `neptunedata` client this transport calls — + implement this shape to inject a fake in tests without installing the + real SDK or reaching a real cluster.""" + + def execute_open_cypher_query(self, *, openCypherQuery: str, parameters: str) -> object: # noqa: N803 + """Runs one openCypher query; `parameters` is JSON-encoded, per the + `neptunedata` API's own shape — not a dict.""" + ... + + +class NeptuneTransport(BaseGraphTransport): + """Writes the span tree to Amazon Neptune via its openCypher HTTPS data + API (the `neptunedata` service), sitting alongside this project's other + AWS-native transports (`CloudWatchTransport`, `DynamoDBTransport`, ...). + + Pass `client` to inject an already-constructed `boto3` `neptunedata` + client (or a fake, for tests), or `endpoint_url`/`region_name` to let + this transport build its own via the optional `boto3` peer dependency + (`pip install logquill[aws]` — the same extra the other AWS transports + use). Reaching `endpoint_url` normally requires running from inside the + cluster's VPC (or wherever has network access to it) — that's a Neptune + deployment detail, not something this transport can work around. + """ + + def __init__( + self, + *, + client: NeptuneDataClientLike | None = None, + endpoint_url: str | None = None, + region_name: str | None = None, + formatter: Formatter | None = None, + max_records: int = 100, + max_bytes: int = 1_000_000, + run_key: str = "run_id", + trace_key: str = "trace_id", + ) -> None: + """`endpoint_url`/`region_name` are used only when this transport + builds its own `boto3` client (ignored if `client` is given).""" + super().__init__( + formatter=formatter, + max_records=max_records, + max_bytes=max_bytes, + run_key=run_key, + trace_key=trace_key, + ) + self._injected = client + self._endpoint_url = endpoint_url + self._region_name = region_name + self._client: NeptuneDataClientLike | None = None + + def _resolved_client(self) -> NeptuneDataClientLike: + if self._injected is not None: + return self._injected + if self._client is None: + self._client = self._connect() + return self._client + + def _connect(self) -> NeptuneDataClientLike: + try: + import boto3 # type: ignore[import-not-found] + except ImportError as exc: + raise ImportError( + "NeptuneTransport requires the optional `boto3` package — install with " + "`pip install logquill[aws]`." + ) from exc + return boto3.client( # type: ignore[no-any-return] + "neptunedata", endpoint_url=self._endpoint_url, region_name=self._region_name + ) + + def _execute(self, query: str, parameters: dict[str, Any]) -> None: + self._resolved_client().execute_open_cypher_query( + openCypherQuery=query, parameters=json.dumps(parameters, default=str) + ) diff --git a/logquill/transports/http_transport.py b/logquill/transports/http_transport.py index 1fb2261..1be5add 100644 --- a/logquill/transports/http_transport.py +++ b/logquill/transports/http_transport.py @@ -4,8 +4,8 @@ import urllib.request from typing import Callable, Literal, Sequence +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.records import LogRecord from logquill.transports.transport import Transport Sender = Callable[[str, Sequence[str]], None] diff --git a/logquill/transports/nosql/dynamodb_transport.py b/logquill/transports/nosql/dynamodb_transport.py index aa36cff..f520a1f 100644 --- a/logquill/transports/nosql/dynamodb_transport.py +++ b/logquill/transports/nosql/dynamodb_transport.py @@ -2,8 +2,8 @@ from typing import Any, ContextManager, Protocol, Sequence, cast +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.records import LogRecord from logquill.transports.batching_transport import BatchingTransport diff --git a/logquill/transports/nosql/mongodb_transport.py b/logquill/transports/nosql/mongodb_transport.py index ff37835..719791e 100644 --- a/logquill/transports/nosql/mongodb_transport.py +++ b/logquill/transports/nosql/mongodb_transport.py @@ -2,8 +2,8 @@ from typing import Any, Protocol, Sequence, cast +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.records import LogRecord from logquill.transports.batching_transport import BatchingTransport diff --git a/logquill/transports/nosql/redis_transport.py b/logquill/transports/nosql/redis_transport.py index 4fcdb17..9b79700 100644 --- a/logquill/transports/nosql/redis_transport.py +++ b/logquill/transports/nosql/redis_transport.py @@ -3,8 +3,8 @@ import json from typing import Any, Protocol, Sequence +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.records import LogRecord from logquill.transports.batching_transport import BatchingTransport diff --git a/logquill/transports/otel/logs_transport.py b/logquill/transports/otel/logs_transport.py index eba6401..3cf25d1 100644 --- a/logquill/transports/otel/logs_transport.py +++ b/logquill/transports/otel/logs_transport.py @@ -3,9 +3,9 @@ import json from typing import Any -from logquill import semconv +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.records import LogRecord +from logquill.tracing import semconv from logquill.transports.otel import _ids from logquill.transports.transport import Transport diff --git a/logquill/transports/otel/otlp_transport.py b/logquill/transports/otel/otlp_transport.py index 1b83f8c..4074cd7 100644 --- a/logquill/transports/otel/otlp_transport.py +++ b/logquill/transports/otel/otlp_transport.py @@ -4,9 +4,9 @@ import threading from typing import Any -from logquill import semconv +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.records import LogRecord +from logquill.tracing import semconv from logquill.transports.otel import _ids from logquill.transports.transport import Transport diff --git a/logquill/transports/queue/base_queue_transport.py b/logquill/transports/queue/base_queue_transport.py index 57aaf84..767579f 100644 --- a/logquill/transports/queue/base_queue_transport.py +++ b/logquill/transports/queue/base_queue_transport.py @@ -3,8 +3,8 @@ from abc import abstractmethod from typing import Sequence +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.records import LogRecord from logquill.transports.batching_transport import BatchingTransport diff --git a/logquill/transports/queue/kafka_transport.py b/logquill/transports/queue/kafka_transport.py index eec8a48..5322b19 100644 --- a/logquill/transports/queue/kafka_transport.py +++ b/logquill/transports/queue/kafka_transport.py @@ -3,8 +3,8 @@ import json from typing import Protocol, Sequence, cast +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.records import LogRecord from logquill.transports.queue.base_queue_transport import BaseQueueTransport diff --git a/logquill/transports/queue/pubsub_transport.py b/logquill/transports/queue/pubsub_transport.py index d5ba608..73eb208 100644 --- a/logquill/transports/queue/pubsub_transport.py +++ b/logquill/transports/queue/pubsub_transport.py @@ -3,8 +3,8 @@ import json from typing import Protocol, Sequence, cast +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.records import LogRecord from logquill.transports.queue.base_queue_transport import BaseQueueTransport diff --git a/logquill/transports/queue/rabbitmq_transport.py b/logquill/transports/queue/rabbitmq_transport.py index c7a1096..04cd759 100644 --- a/logquill/transports/queue/rabbitmq_transport.py +++ b/logquill/transports/queue/rabbitmq_transport.py @@ -3,8 +3,8 @@ import json from typing import Protocol, Sequence, cast +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.records import LogRecord from logquill.transports.queue.base_queue_transport import BaseQueueTransport diff --git a/logquill/transports/queue/sqs_transport.py b/logquill/transports/queue/sqs_transport.py index 9e341f5..15318b8 100644 --- a/logquill/transports/queue/sqs_transport.py +++ b/logquill/transports/queue/sqs_transport.py @@ -3,8 +3,8 @@ import json from typing import Protocol, Sequence, cast +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.records import LogRecord from logquill.transports.queue.base_queue_transport import BaseQueueTransport _SQS_BATCH_LIMIT = 10 diff --git a/logquill/transports/sql/base_sql_transport.py b/logquill/transports/sql/base_sql_transport.py index 08b5a7b..8f99e4d 100644 --- a/logquill/transports/sql/base_sql_transport.py +++ b/logquill/transports/sql/base_sql_transport.py @@ -4,8 +4,8 @@ from abc import abstractmethod from typing import Any, Sequence, TypedDict +from logquill.core.records import LogRecord from logquill.formatters import Formatter -from logquill.records import LogRecord from logquill.transports.batching_transport import BatchingTransport diff --git a/logquill/transports/transport.py b/logquill/transports/transport.py index 1427b9e..eceaa5a 100644 --- a/logquill/transports/transport.py +++ b/logquill/transports/transport.py @@ -2,8 +2,8 @@ from abc import ABC, abstractmethod +from logquill.core.records import LogRecord from logquill.formatters import Formatter, JSONFormatter -from logquill.records import LogRecord class Transport(ABC): diff --git a/pyproject.toml b/pyproject.toml index e8c18dd..c7615a1 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -58,6 +58,8 @@ rabbitmq = ["pika>=1.3"] pubsub = ["google-cloud-pubsub>=2.21"] gcp-logging = ["google-cloud-logging>=3.10"] aws = ["boto3>=1.34"] +neo4j = ["neo4j>=5.20"] +falkordb = ["falkordb>=1.0"] presidio = ["presidio-analyzer>=2.2", "presidio-anonymizer>=2.2"] crypto = ["cryptography>=41"] langchain = ["langchain-core>=0.3"] @@ -88,7 +90,7 @@ docs = [ ] [project.scripts] -logquill = "logquill.cli:main" +logquill = "logquill.cli.cli:main" [project.urls] Homepage = "https://github.com/nikhilvdev/logquill-python" diff --git a/tests/conftest.py b/tests/conftest.py index be8e962..a90c3f3 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -4,7 +4,7 @@ import pytest -from logquill import toggle +from logquill.runtime import toggle @pytest.fixture(autouse=True) diff --git a/tests/test_adapters/test_autogen_adapter.py b/tests/test_adapters/test_autogen_adapter.py index e0af245..6fbce12 100644 --- a/tests/test_adapters/test_autogen_adapter.py +++ b/tests/test_adapters/test_autogen_adapter.py @@ -8,7 +8,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.transport import CollectingTransport _EVENT_LOGGER_NAME = "autogen_core.events" diff --git a/tests/test_adapters/test_base.py b/tests/test_adapters/test_base.py index 4bdbdef..25a4098 100644 --- a/tests/test_adapters/test_base.py +++ b/tests/test_adapters/test_base.py @@ -1,5 +1,5 @@ from logquill.adapters.base import LogQuillAdapter -from logquill.logger import Logger +from logquill.core.logger import Logger def test_holds_a_reference_to_the_given_logger() -> None: diff --git a/tests/test_adapters/test_crewai_adapter.py b/tests/test_adapters/test_crewai_adapter.py index 7ffd9c7..7550fb7 100644 --- a/tests/test_adapters/test_crewai_adapter.py +++ b/tests/test_adapters/test_crewai_adapter.py @@ -9,7 +9,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.plugins.run_plugin import RunPlugin from logquill.transports.transport import CollectingTransport diff --git a/tests/test_adapters/test_langchain_adapter.py b/tests/test_adapters/test_langchain_adapter.py index 741cda8..92e55b9 100644 --- a/tests/test_adapters/test_langchain_adapter.py +++ b/tests/test_adapters/test_langchain_adapter.py @@ -6,7 +6,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.plugins.run_plugin import RunPlugin from logquill.transports.transport import CollectingTransport diff --git a/tests/test_adapters/test_langgraph_adapter.py b/tests/test_adapters/test_langgraph_adapter.py index 6264ba5..0afeb5f 100644 --- a/tests/test_adapters/test_langgraph_adapter.py +++ b/tests/test_adapters/test_langgraph_adapter.py @@ -9,7 +9,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.transport import CollectingTransport diff --git a/tests/test_adapters/test_llamaindex_adapter.py b/tests/test_adapters/test_llamaindex_adapter.py index 52b0e1e..2813ab0 100644 --- a/tests/test_adapters/test_llamaindex_adapter.py +++ b/tests/test_adapters/test_llamaindex_adapter.py @@ -8,7 +8,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.transport import CollectingTransport diff --git a/tests/test_adapters/test_openai_agents_adapter.py b/tests/test_adapters/test_openai_agents_adapter.py index ed532d5..56a8860 100644 --- a/tests/test_adapters/test_openai_agents_adapter.py +++ b/tests/test_adapters/test_openai_agents_adapter.py @@ -8,7 +8,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.transport import CollectingTransport diff --git a/tests/test_audit.py b/tests/test_audit.py index b50f31e..60ecfc2 100644 --- a/tests/test_audit.py +++ b/tests/test_audit.py @@ -133,6 +133,6 @@ def test_child_inherits_content_policy() -> None: def test_kwargs_are_forwarded_to_logger_init() -> None: logger, _ = _audit_logger(flush_at_exit=False) - from logquill import shutdown + from logquill.runtime import shutdown assert logger not in shutdown._loggers diff --git a/tests/test_backpressure_burst.py b/tests/test_backpressure_burst.py index 512d05e..2eb955a 100644 --- a/tests/test_backpressure_burst.py +++ b/tests/test_backpressure_burst.py @@ -11,7 +11,7 @@ import pytest from logquill import Logger -from logquill.records import LogRecord +from logquill.core.records import LogRecord from logquill.transports.transport import CollectingTransport BURST = 30_000 diff --git a/tests/test_cli.py b/tests/test_cli.py index 375f68b..655af26 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -6,8 +6,8 @@ import pytest -from logquill.cli import _follow, _read_existing, _run_tail, build_parser, main -from logquill.levels import Level +from logquill.cli.cli import _follow, _read_existing, _run_tail, build_parser, main +from logquill.core.levels import Level def _write_lines(path: Path, records: list[dict]) -> None: diff --git a/tests/test_cli_dev.py b/tests/test_cli_dev.py index 3f2f81b..2ecada2 100644 --- a/tests/test_cli_dev.py +++ b/tests/test_cli_dev.py @@ -4,7 +4,7 @@ from pathlib import Path from logquill import Logger, RunPlugin -from logquill.cli import _CLEAR_SCREEN, _run_dev, build_parser +from logquill.cli.cli import _CLEAR_SCREEN, _run_dev, build_parser from logquill.transports.file_transport import FileTransport @@ -60,7 +60,7 @@ def append_more(_: object = None) -> None: logger.action("second", tool="search") # patch time.sleep so the poll loop's single iteration does real work - import logquill.cli as cli_module + import logquill.cli.cli as cli_module original_sleep = cli_module.time.sleep cli_module.time.sleep = lambda _s: append_more() # type: ignore[assignment] @@ -83,7 +83,7 @@ def test_dev_follows_a_new_run_once_the_old_one_finishes(tmp_path: Path) -> None args = build_parser().parse_args(["dev", str(path)]) out = io.StringIO() - import logquill.cli as cli_module + import logquill.cli.cli as cli_module second = _logger_for(path, "run-2") @@ -109,7 +109,7 @@ def test_an_explicit_run_id_is_never_overridden_by_a_newer_run(tmp_path: Path) - args = build_parser().parse_args(["dev", str(path), "--run-id", "run-1"]) out = io.StringIO() - import logquill.cli as cli_module + import logquill.cli.cli as cli_module second = _logger_for(path, "run-2") diff --git a/tests/test_cli_serve.py b/tests/test_cli_serve.py index ac1abf6..9847882 100644 --- a/tests/test_cli_serve.py +++ b/tests/test_cli_serve.py @@ -10,7 +10,7 @@ import pytest from logquill import Logger -from logquill.cli import _run_serve, build_parser +from logquill.cli.cli import _run_serve, build_parser from logquill.transports.file_transport import FileTransport @@ -64,7 +64,7 @@ def test_serve_starts_a_real_server_and_prints_its_url(tmp_path: Path) -> None: def test_serve_via_main_reports_the_error_for_a_bad_source( tmp_path: Path, capsys: pytest.CaptureFixture[str] ) -> None: - from logquill.cli import main + from logquill.cli.cli import main exit_code = main(["serve", "--db", str(tmp_path / "missing.sqlite")]) diff --git a/tests/test_cli_trace.py b/tests/test_cli_trace.py index 557ceea..d6221c2 100644 --- a/tests/test_cli_trace.py +++ b/tests/test_cli_trace.py @@ -7,7 +7,7 @@ import pytest from logquill import Logger, RunPlugin -from logquill.cli import _run_trace, build_parser, main +from logquill.cli.cli import _run_trace, build_parser, main from logquill.transports.file_transport import FileTransport diff --git a/tests/test_cli_verify.py b/tests/test_cli_verify.py index dae7738..d112c62 100644 --- a/tests/test_cli_verify.py +++ b/tests/test_cli_verify.py @@ -4,7 +4,7 @@ from pathlib import Path from logquill import AuditLogger -from logquill.cli import _run_verify, build_parser, main +from logquill.cli.cli import _run_verify, build_parser, main from logquill.transports.file_transport import FileTransport diff --git a/tests/test_config.py b/tests/test_config.py index d65aaf3..bf5069b 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -6,7 +6,7 @@ import pytest from logquill.config import load_config, logger_from_env, logger_from_file -from logquill.levels import Level +from logquill.core.levels import Level from logquill.plugins.context_plugin import ContextPlugin from logquill.transports.console_transport import ConsoleTransport from logquill.transports.file_transport import FileTransport @@ -204,7 +204,7 @@ def test_an_unknown_formatter_name_lists_the_valid_ones() -> None: def test_flush_at_exit_can_be_set_from_config() -> None: - from logquill import shutdown + from logquill.runtime import shutdown opted_out = load_config({"flush_at_exit": False}) default = load_config({}) diff --git a/tests/test_context.py b/tests/test_context.py index be51211..bbb695d 100644 --- a/tests/test_context.py +++ b/tests/test_context.py @@ -3,8 +3,8 @@ import asyncio import concurrent.futures -from logquill.context import bind_context, current_context -from logquill.logger import Logger +from logquill.core.context import bind_context, current_context +from logquill.core.logger import Logger def test_bound_context_appears_in_meta_without_manual_passing() -> None: diff --git a/tests/test_contract.py b/tests/test_contract.py index 0635478..00ee83b 100644 --- a/tests/test_contract.py +++ b/tests/test_contract.py @@ -21,7 +21,7 @@ TraceContextPlugin, parse_record, ) -from logquill.records import LEGACY_SCHEMA_VERSION, SCHEMA_VERSION, create_record +from logquill.core.records import LEGACY_SCHEMA_VERSION, SCHEMA_VERSION, create_record from logquill.transports.transport import CollectingTransport SCHEMA_DIR = Path(__file__).resolve().parent.parent / "schema" diff --git a/tests/test_diagnose.py b/tests/test_diagnose.py index 34f0c0b..5845328 100644 --- a/tests/test_diagnose.py +++ b/tests/test_diagnose.py @@ -5,8 +5,8 @@ import pytest from logquill import Logger, PIIRedactPlugin, Plugin, RedactPlugin -from logquill import exceptions as exceptions_module -from logquill.exceptions import format_exc_info +from logquill.core import exceptions as exceptions_module +from logquill.core.exceptions import format_exc_info from logquill.transports.transport import CollectingTransport # Obviously fake values, kept in constants rather than assigned as literals to diff --git a/tests/test_exceptions.py b/tests/test_exceptions.py index 1317f87..9a8ad48 100644 --- a/tests/test_exceptions.py +++ b/tests/test_exceptions.py @@ -2,8 +2,8 @@ import pytest -from logquill.exceptions import format_exc_info -from logquill.logger import Logger +from logquill.core.exceptions import format_exc_info +from logquill.core.logger import Logger def test_format_exc_info_from_exception_instance() -> None: diff --git a/tests/test_formatter.py b/tests/test_formatter.py index edf2861..ace09a7 100644 --- a/tests/test_formatter.py +++ b/tests/test_formatter.py @@ -1,7 +1,7 @@ import json +from logquill.core.logger import Logger from logquill.formatter import JSONFormatter -from logquill.logger import Logger def test_json_formatter_round_trips_record() -> None: diff --git a/tests/test_formatter_properties.py b/tests/test_formatter_properties.py index b17213e..4d3b804 100644 --- a/tests/test_formatter_properties.py +++ b/tests/test_formatter_properties.py @@ -9,7 +9,7 @@ from hypothesis import strategies as st from logquill import JSONFormatter, LogfmtFormatter, TextFormatter, parse_logfmt -from logquill.records import LogRecord +from logquill.core.records import LogRecord _settings = settings(max_examples=150, suppress_health_check=[HealthCheck.too_slow]) diff --git a/tests/test_formatters.py b/tests/test_formatters.py index 743f61b..c4ed32c 100644 --- a/tests/test_formatters.py +++ b/tests/test_formatters.py @@ -9,8 +9,8 @@ TextFormatter, parse_logfmt, ) -from logquill.levels import Level -from logquill.records import LogRecord, create_record +from logquill.core.levels import Level +from logquill.core.records import LogRecord, create_record def _record(message: str = "user signed up", **meta: object) -> LogRecord: diff --git a/tests/test_handler.py b/tests/test_handler.py index 326d487..a119398 100644 --- a/tests/test_handler.py +++ b/tests/test_handler.py @@ -2,8 +2,8 @@ import logging -from logquill.handler import LogQuillHandler -from logquill.logger import Logger +from logquill.core.handler import LogQuillHandler +from logquill.core.logger import Logger from logquill.transports.transport import CollectingTransport diff --git a/tests/test_levels.py b/tests/test_levels.py index c033688..ca19bd0 100644 --- a/tests/test_levels.py +++ b/tests/test_levels.py @@ -2,7 +2,7 @@ import pytest -from logquill.levels import Level, parse_level +from logquill.core.levels import Level, parse_level def test_level_values() -> None: diff --git a/tests/test_logger.py b/tests/test_logger.py index e50d1c5..7d574f7 100644 --- a/tests/test_logger.py +++ b/tests/test_logger.py @@ -1,5 +1,5 @@ -from logquill.levels import Level -from logquill.logger import Logger +from logquill.core.levels import Level +from logquill.core.logger import Logger def test_info_produces_well_formed_record() -> None: diff --git a/tests/test_logger_async.py b/tests/test_logger_async.py index 7f70060..d5318c8 100644 --- a/tests/test_logger_async.py +++ b/tests/test_logger_async.py @@ -4,8 +4,8 @@ import time from typing import Any -from logquill.logger import Logger -from logquill.records import LogRecord +from logquill.core.logger import Logger +from logquill.core.records import LogRecord from logquill.transports.transport import CollectingTransport _POLL_TIMEOUT = 2.0 diff --git a/tests/test_mcp.py b/tests/test_mcp.py index 5d29cf4..55561a4 100644 --- a/tests/test_mcp.py +++ b/tests/test_mcp.py @@ -3,7 +3,7 @@ import json from logquill import Logger, RunPlugin, TraceContextPlugin -from logquill.mcp import RUN_ID_KEY, TRACEPARENT_KEY, inbound, propagate +from logquill.tracing.mcp import RUN_ID_KEY, TRACEPARENT_KEY, inbound, propagate from logquill.transports.transport import CollectingTransport diff --git a/tests/test_plugin_pipeline_properties.py b/tests/test_plugin_pipeline_properties.py index 8ab20b6..1cc3005 100644 --- a/tests/test_plugin_pipeline_properties.py +++ b/tests/test_plugin_pipeline_properties.py @@ -5,7 +5,7 @@ from adversarial import meta_dicts from hypothesis import HealthCheck, given, settings -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.plugins.context_plugin import ContextPlugin from logquill.plugins.pii_redact_plugin import PIIRedactPlugin from logquill.plugins.redact_plugin import RedactPlugin diff --git a/tests/test_plugins/test_alerting_plugin.py b/tests/test_plugins/test_alerting_plugin.py index a9951e6..bd7732d 100644 --- a/tests/test_plugins/test_alerting_plugin.py +++ b/tests/test_plugins/test_alerting_plugin.py @@ -3,9 +3,9 @@ import threading import time -from logquill.logger import Logger +from logquill.core.logger import Logger +from logquill.core.records import LogRecord from logquill.plugins.alerting_plugin import AlertingPlugin -from logquill.records import LogRecord _POLL_TIMEOUT = 2.0 _POLL_INTERVAL = 0.01 diff --git a/tests/test_plugins/test_apprise_alert_plugin.py b/tests/test_plugins/test_apprise_alert_plugin.py index 93e1651..c1947bf 100644 --- a/tests/test_plugins/test_apprise_alert_plugin.py +++ b/tests/test_plugins/test_apprise_alert_plugin.py @@ -8,7 +8,7 @@ import pytest from logquill import AppriseAlertPlugin, Level, Logger -from logquill.records import LogRecord, create_record +from logquill.core.records import LogRecord, create_record class FakeApprise: diff --git a/tests/test_plugins/test_context_plugin.py b/tests/test_plugins/test_context_plugin.py index f7f1baf..55585ec 100644 --- a/tests/test_plugins/test_context_plugin.py +++ b/tests/test_plugins/test_context_plugin.py @@ -1,4 +1,4 @@ -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.plugins.context_plugin import ContextPlugin diff --git a/tests/test_plugins/test_email_alert_plugin.py b/tests/test_plugins/test_email_alert_plugin.py index c4e5a76..0423566 100644 --- a/tests/test_plugins/test_email_alert_plugin.py +++ b/tests/test_plugins/test_email_alert_plugin.py @@ -2,9 +2,9 @@ from unittest.mock import MagicMock, patch -from logquill.levels import Level +from logquill.core.levels import Level +from logquill.core.records import create_record from logquill.plugins.email_alert_plugin import EmailAlertPlugin -from logquill.records import create_record def _record(message: str = "boom") -> object: diff --git a/tests/test_plugins/test_flight_recorder_plugin.py b/tests/test_plugins/test_flight_recorder_plugin.py index 0a7bc7a..e5dbb46 100644 --- a/tests/test_plugins/test_flight_recorder_plugin.py +++ b/tests/test_plugins/test_flight_recorder_plugin.py @@ -1,6 +1,6 @@ from __future__ import annotations -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.plugins.flight_recorder_plugin import FlightRecorderPlugin from logquill.transports.transport import CollectingTransport diff --git a/tests/test_plugins/test_metrics_plugin.py b/tests/test_plugins/test_metrics_plugin.py new file mode 100644 index 0000000..f6774b3 --- /dev/null +++ b/tests/test_plugins/test_metrics_plugin.py @@ -0,0 +1,179 @@ +from __future__ import annotations + +import sys +import types +from collections.abc import Mapping + +import pytest + +from logquill import Logger, MetricsPlugin +from logquill.transports.transport import CollectingTransport + + +class FakeBackend: + def __init__(self) -> None: + self.increments: list[tuple[str, dict[str, str]]] = [] + self.observations: list[tuple[str, float, dict[str, str]]] = [] + self.closed = False + + def increment(self, name: str, *, tags: Mapping[str, str]) -> None: + self.increments.append((name, dict(tags))) + + def observe(self, name: str, value: float, *, tags: Mapping[str, str]) -> None: + self.observations.append((name, value, dict(tags))) + + def close(self) -> None: + self.closed = True + + +def _logger(backend: FakeBackend, **kwargs: object) -> tuple[Logger, CollectingTransport]: + sink = CollectingTransport() + logger = Logger( + "app.agent", + transports=[sink], + plugins=[MetricsPlugin(backend, **kwargs)], # type: ignore[arg-type] + ) + return logger, sink + + +def test_an_error_increments_the_error_counter_tagged_by_logger() -> None: + backend = FakeBackend() + logger, _ = _logger(backend) + + logger.error("boom") + + assert backend.increments == [("logquill.errors", {"logger": "app.agent"})] + + +def test_fatal_also_increments_the_error_counter() -> None: + backend = FakeBackend() + logger, _ = _logger(backend) + + logger.fatal("boom") + + assert backend.increments == [("logquill.errors", {"logger": "app.agent"})] + + +def test_info_and_warn_do_not_increment_the_error_counter_by_default() -> None: + backend = FakeBackend() + logger, _ = _logger(backend) + + logger.info("fine") + logger.warn("careful") + + assert backend.increments == [] + + +def test_error_at_is_configurable() -> None: + backend = FakeBackend() + logger, _ = _logger(backend, error_at="WARN") + + logger.warn("careful") + + assert backend.increments == [("logquill.errors", {"logger": "app.agent"})] + + +def test_a_duration_ms_feeds_the_histogram() -> None: + backend = FakeBackend() + logger, _ = _logger(backend) + + with logger.span("work", duration_ms=12.5): + pass + + assert ("logquill.duration_ms", 12.5, {"logger": "app.agent"}) in backend.observations + + +def test_a_record_with_no_duration_ms_does_not_feed_the_histogram() -> None: + backend = FakeBackend() + logger, _ = _logger(backend) + + logger.info("no duration here") + + assert backend.observations == [] + + +def test_an_error_record_with_a_duration_does_both() -> None: + backend = FakeBackend() + logger, _ = _logger(backend) + + try: + with logger.span("work"): + raise ValueError("boom") + except ValueError: + pass + + assert backend.increments == [("logquill.errors", {"logger": "app.agent"})] + assert len(backend.observations) == 1 + + +def test_metric_names_are_configurable() -> None: + backend = FakeBackend() + logger, _ = _logger(backend, error_metric="my.errors", duration_metric="my.duration") + + logger.error("boom") + with logger.span("work", duration_ms=1.0): + pass + + assert backend.increments == [("my.errors", {"logger": "app.agent"})] + assert ("my.duration", 1.0, {"logger": "app.agent"}) in backend.observations + + +def test_a_raising_backend_does_not_crash_the_caller() -> None: + class Broken: + def increment(self, name: str, *, tags: Mapping[str, str]) -> None: + raise RuntimeError("backend is down") + + def observe(self, name: str, value: float, *, tags: Mapping[str, str]) -> None: + raise RuntimeError("backend is down") + + sink = CollectingTransport() + logger = Logger("app", transports=[sink], plugins=[MetricsPlugin(Broken())]) # type: ignore[arg-type] + + record = logger.error("boom") + + assert record is not None # the caller's own call still succeeded + assert sink.records == [record] + + +def test_close_closes_the_backend_if_it_has_one() -> None: + backend = FakeBackend() + plugin = MetricsPlugin(backend) + + plugin.close() + + assert backend.closed is True + + +def test_close_is_safe_when_the_backend_has_no_close_method() -> None: + class NoClose: + def increment(self, name: str, *, tags: Mapping[str, str]) -> None: + pass + + def observe(self, name: str, value: float, *, tags: Mapping[str, str]) -> None: + pass + + plugin = MetricsPlugin(NoClose()) # type: ignore[arg-type] + + plugin.close() # must not raise + + +def test_the_backend_defaults_to_otel_and_is_resolved_lazily( + monkeypatch: pytest.MonkeyPatch, +) -> None: + fake_meter = types.SimpleNamespace( + create_counter=lambda name: types.SimpleNamespace(add=lambda *a, **k: None), + create_histogram=lambda name: types.SimpleNamespace(record=lambda *a, **k: None), + ) + fake_provider = types.SimpleNamespace(get_meter=lambda name: fake_meter) + fake_metrics_module = types.ModuleType("opentelemetry.metrics") + fake_metrics_module.get_meter_provider = lambda: fake_provider # type: ignore[attr-defined] + opentelemetry_module = types.ModuleType("opentelemetry") + opentelemetry_module.metrics = fake_metrics_module # type: ignore[attr-defined] + monkeypatch.setitem(sys.modules, "opentelemetry", opentelemetry_module) + monkeypatch.setitem(sys.modules, "opentelemetry.metrics", fake_metrics_module) + + plugin = MetricsPlugin() + assert plugin._backend is None # not resolved yet + logger = Logger("app", plugins=[plugin]) + logger.error("boom") + assert plugin._backend is not None # resolved on first use diff --git a/tests/test_plugins/test_otel_metrics_backend.py b/tests/test_plugins/test_otel_metrics_backend.py new file mode 100644 index 0000000..245feea --- /dev/null +++ b/tests/test_plugins/test_otel_metrics_backend.py @@ -0,0 +1,113 @@ +from __future__ import annotations + +import sys +import types +from typing import Any + +import pytest + +from logquill.plugins.otel_metrics_backend import OTelMetricsBackend + + +class FakeInstrument: + def __init__(self) -> None: + self.calls: list[tuple[float, dict[str, str]]] = [] + + def add(self, value: float, attributes: dict[str, str]) -> None: + self.calls.append((value, attributes)) + + def record(self, value: float, attributes: dict[str, str]) -> None: + self.calls.append((value, attributes)) + + +class FakeMeter: + def __init__(self) -> None: + self.counters: dict[str, FakeInstrument] = {} + self.histograms: dict[str, FakeInstrument] = {} + + def create_counter(self, name: str) -> FakeInstrument: + instrument = FakeInstrument() + self.counters[name] = instrument + return instrument + + def create_histogram(self, name: str) -> FakeInstrument: + instrument = FakeInstrument() + self.histograms[name] = instrument + return instrument + + +class FakeMeterProvider: + def __init__(self) -> None: + self.meter = FakeMeter() + + def get_meter(self, name: str) -> FakeMeter: + return self.meter + + +@pytest.fixture +def fake_opentelemetry(monkeypatch: pytest.MonkeyPatch) -> Any: + fake_metrics_module = types.ModuleType("opentelemetry.metrics") + default_provider = FakeMeterProvider() + fake_metrics_module.get_meter_provider = lambda: default_provider # type: ignore[attr-defined] + opentelemetry_module = types.ModuleType("opentelemetry") + opentelemetry_module.metrics = fake_metrics_module # type: ignore[attr-defined] + monkeypatch.setitem(sys.modules, "opentelemetry", opentelemetry_module) + monkeypatch.setitem(sys.modules, "opentelemetry.metrics", fake_metrics_module) + return default_provider + + +def test_increment_adds_one_to_a_lazily_created_counter( + fake_opentelemetry: FakeMeterProvider, +) -> None: + backend = OTelMetricsBackend() + + backend.increment("logquill.errors", tags={"logger": "app"}) + + instrument = fake_opentelemetry.meter.counters["logquill.errors"] + assert instrument.calls == [(1, {"logger": "app"})] + + +def test_observe_records_a_value_on_a_lazily_created_histogram( + fake_opentelemetry: FakeMeterProvider, +) -> None: + backend = OTelMetricsBackend() + + backend.observe("logquill.duration_ms", 12.5, tags={"logger": "app"}) + + instrument = fake_opentelemetry.meter.histograms["logquill.duration_ms"] + assert instrument.calls == [(12.5, {"logger": "app"})] + + +def test_the_same_instrument_is_reused_across_calls(fake_opentelemetry: FakeMeterProvider) -> None: + backend = OTelMetricsBackend() + + backend.increment("logquill.errors", tags={}) + backend.increment("logquill.errors", tags={}) + + assert len(fake_opentelemetry.meter.counters) == 1 + assert fake_opentelemetry.meter.counters["logquill.errors"].calls == [ + (1, {}), + (1, {}), + ] + + +def test_an_explicit_meter_provider_overrides_the_global_default( + fake_opentelemetry: FakeMeterProvider, +) -> None: + explicit_provider = FakeMeterProvider() + + backend = OTelMetricsBackend(meter_provider=explicit_provider) + backend.increment("logquill.errors", tags={}) + + assert "logquill.errors" in explicit_provider.meter.counters + assert fake_opentelemetry.meter.counters == {} + + +def test_a_missing_opentelemetry_package_gives_an_install_hint( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setitem(sys.modules, "opentelemetry", None) + monkeypatch.setitem(sys.modules, "opentelemetry.metrics", None) + + with pytest.raises(ImportError, match=r"pip install logquill\[otel\]"): + OTelMetricsBackend() diff --git a/tests/test_plugins/test_pagerduty_alert_plugin.py b/tests/test_plugins/test_pagerduty_alert_plugin.py index 97331da..64680c9 100644 --- a/tests/test_plugins/test_pagerduty_alert_plugin.py +++ b/tests/test_plugins/test_pagerduty_alert_plugin.py @@ -3,9 +3,9 @@ import json from unittest.mock import MagicMock, patch -from logquill.levels import Level +from logquill.core.levels import Level +from logquill.core.records import create_record from logquill.plugins.pagerduty_alert_plugin import PagerDutyAlertPlugin -from logquill.records import create_record def _record(level: Level = Level.ERROR, message: str = "boom") -> object: diff --git a/tests/test_plugins/test_pii_redact_plugin.py b/tests/test_plugins/test_pii_redact_plugin.py index ec68688..f63b942 100644 --- a/tests/test_plugins/test_pii_redact_plugin.py +++ b/tests/test_plugins/test_pii_redact_plugin.py @@ -7,7 +7,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.plugins.pii_redact_plugin import PIIRedactPlugin diff --git a/tests/test_plugins/test_plugin.py b/tests/test_plugins/test_plugin.py index 5359620..ca83442 100644 --- a/tests/test_plugins/test_plugin.py +++ b/tests/test_plugins/test_plugin.py @@ -1,8 +1,8 @@ from __future__ import annotations -from logquill.logger import Logger +from logquill.core.logger import Logger +from logquill.core.records import LogRecord from logquill.plugins.plugin import Plugin -from logquill.records import LogRecord from logquill.transports.transport import CollectingTransport diff --git a/tests/test_plugins/test_prometheus_pushgateway_backend.py b/tests/test_plugins/test_prometheus_pushgateway_backend.py new file mode 100644 index 0000000..cbbddf6 --- /dev/null +++ b/tests/test_plugins/test_prometheus_pushgateway_backend.py @@ -0,0 +1,108 @@ +from __future__ import annotations + +from logquill.plugins.prometheus_pushgateway_backend import PrometheusPushgatewayBackend, _sanitize + + +def test_sanitize_replaces_invalid_characters_with_underscores() -> None: + assert _sanitize("logquill.errors") == "logquill_errors" + + +def test_sanitize_prefixes_a_leading_digit() -> None: + assert _sanitize("2xx.count") == "_2xx_count" + + +def test_increment_and_observe_do_not_send_anything_until_push() -> None: + sent: list[tuple[str, str]] = [] + backend = PrometheusPushgatewayBackend( + "http://localhost:9091", sender=lambda url, body: sent.append((url, body.decode())) + ) + + backend.increment("logquill.errors", tags={"logger": "app"}) + backend.observe("logquill.duration_ms", 10.0, tags={"logger": "app"}) + + assert sent == [] + + +def test_push_sends_a_put_to_the_job_url_with_rendered_metrics() -> None: + sent: list[tuple[str, str]] = [] + backend = PrometheusPushgatewayBackend( + "http://localhost:9091", + job_name="myjob", + sender=lambda url, body: sent.append((url, body.decode())), + ) + + backend.increment("logquill.errors", tags={"logger": "app"}) + backend.push() + + ((url, body),) = sent + assert url == "http://localhost:9091/metrics/job/myjob" + assert body == 'logquill_errors{logger="app"} 1.0\n' + + +def test_repeated_increments_accumulate_into_one_cumulative_total() -> None: + sent: list[tuple[str, str]] = [] + backend = PrometheusPushgatewayBackend( + "http://localhost:9091", sender=lambda url, body: sent.append((url, body.decode())) + ) + + backend.increment("logquill.errors", tags={"logger": "app"}) + backend.increment("logquill.errors", tags={"logger": "app"}) + backend.increment("logquill.errors", tags={"logger": "app"}) + backend.push() + + (_url, body) = sent[0] + assert body == 'logquill_errors{logger="app"} 3.0\n' + + +def test_observe_accumulates_a_histogram_sum_and_count() -> None: + sent: list[tuple[str, str]] = [] + backend = PrometheusPushgatewayBackend( + "http://localhost:9091", sender=lambda url, body: sent.append((url, body.decode())) + ) + + backend.observe("logquill.duration_ms", 10.0, tags={"logger": "app"}) + backend.observe("logquill.duration_ms", 20.0, tags={"logger": "app"}) + backend.push() + + (_url, body) = sent[0] + lines = body.splitlines() + assert 'logquill_duration_ms_sum{logger="app"} 30.0' in lines + assert 'logquill_duration_ms_count{logger="app"} 2' in lines + + +def test_different_tag_sets_are_tracked_as_separate_series() -> None: + sent: list[tuple[str, str]] = [] + backend = PrometheusPushgatewayBackend( + "http://localhost:9091", sender=lambda url, body: sent.append((url, body.decode())) + ) + + backend.increment("logquill.errors", tags={"logger": "app.a"}) + backend.increment("logquill.errors", tags={"logger": "app.b"}) + backend.push() + + (_url, body) = sent[0] + assert 'logquill_errors{logger="app.a"} 1.0' in body + assert 'logquill_errors{logger="app.b"} 1.0' in body + + +def test_pushing_with_nothing_accumulated_sends_nothing() -> None: + sent: list[tuple[str, str]] = [] + backend = PrometheusPushgatewayBackend( + "http://localhost:9091", sender=lambda url, body: sent.append((url, body.decode())) + ) + + backend.push() + + assert sent == [] + + +def test_close_pushes_exactly_once() -> None: + sent: list[tuple[str, str]] = [] + backend = PrometheusPushgatewayBackend( + "http://localhost:9091", sender=lambda url, body: sent.append((url, body.decode())) + ) + + backend.increment("logquill.errors", tags={}) + backend.close() + + assert len(sent) == 1 diff --git a/tests/test_plugins/test_rate_limit_plugin.py b/tests/test_plugins/test_rate_limit_plugin.py index 7caeedd..1382392 100644 --- a/tests/test_plugins/test_rate_limit_plugin.py +++ b/tests/test_plugins/test_rate_limit_plugin.py @@ -2,7 +2,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.plugins.rate_limit_plugin import RateLimitPlugin diff --git a/tests/test_plugins/test_redact_plugin.py b/tests/test_plugins/test_redact_plugin.py index 39c02ca..3995cb8 100644 --- a/tests/test_plugins/test_redact_plugin.py +++ b/tests/test_plugins/test_redact_plugin.py @@ -1,4 +1,4 @@ -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.plugins.redact_plugin import RedactPlugin @@ -32,7 +32,7 @@ def test_custom_keys_and_replacement() -> None: def test_classes_adds_every_key_tagged_with_that_class() -> None: - from logquill.privacy import keys_in_class + from logquill.privacy.privacy import keys_in_class plugin = RedactPlugin(keys=(), classes=["secret"]) diff --git a/tests/test_plugins/test_run_plugin.py b/tests/test_plugins/test_run_plugin.py index b507482..51d144d 100644 --- a/tests/test_plugins/test_run_plugin.py +++ b/tests/test_plugins/test_run_plugin.py @@ -1,4 +1,4 @@ -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.plugins.run_plugin import RunPlugin diff --git a/tests/test_plugins/test_sampling_plugin.py b/tests/test_plugins/test_sampling_plugin.py index 0b4d526..63f3d3d 100644 --- a/tests/test_plugins/test_sampling_plugin.py +++ b/tests/test_plugins/test_sampling_plugin.py @@ -1,6 +1,6 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.plugins.sampling_plugin import SamplingPlugin from logquill.transports.transport import CollectingTransport diff --git a/tests/test_plugins/test_slack_alert_plugin.py b/tests/test_plugins/test_slack_alert_plugin.py index aaa1ee7..f68063b 100644 --- a/tests/test_plugins/test_slack_alert_plugin.py +++ b/tests/test_plugins/test_slack_alert_plugin.py @@ -3,9 +3,9 @@ import json from unittest.mock import MagicMock, patch -from logquill.levels import Level +from logquill.core.levels import Level +from logquill.core.records import create_record from logquill.plugins.slack_alert_plugin import SlackAlertPlugin -from logquill.records import create_record def _record(message: str = "boom") -> object: diff --git a/tests/test_plugins/test_statsd_metrics_backend.py b/tests/test_plugins/test_statsd_metrics_backend.py new file mode 100644 index 0000000..0551dd5 --- /dev/null +++ b/tests/test_plugins/test_statsd_metrics_backend.py @@ -0,0 +1,86 @@ +from __future__ import annotations + +from logquill.plugins.statsd_metrics_backend import StatsDBackend + + +class FakeSocket: + def __init__(self, *, fail: bool = False) -> None: + self.sent: list[tuple[bytes, tuple[str, int]]] = [] + self.closed = False + self._fail = fail + + def sendto(self, data: bytes, address: tuple[str, int]) -> int: + if self._fail: + raise OSError("network unreachable") + self.sent.append((data, address)) + return len(data) + + def close(self) -> None: + self.closed = True + + +def test_increment_sends_a_counter_line() -> None: + sock = FakeSocket() + backend = StatsDBackend(sock=sock) + + backend.increment("logquill.errors", tags={}) + + (data, address) = sock.sent[0] + assert data == b"logquill.errors:1|c" + assert address == ("localhost", 8125) + + +def test_observe_sends_a_histogram_line() -> None: + sock = FakeSocket() + backend = StatsDBackend(sock=sock) + + backend.observe("logquill.duration_ms", 12.5, tags={}) + + (data, _address) = sock.sent[0] + assert data == b"logquill.duration_ms:12.5|h" + + +def test_tags_are_rendered_as_a_dogstatsd_suffix() -> None: + sock = FakeSocket() + backend = StatsDBackend(sock=sock) + + backend.increment("logquill.errors", tags={"logger": "app.agent"}) + + (data, _address) = sock.sent[0] + assert data == b"logquill.errors:1|c|#logger:app.agent" + + +def test_prefix_is_prepended_to_the_metric_name() -> None: + sock = FakeSocket() + backend = StatsDBackend(sock=sock, prefix="myapp") + + backend.increment("errors", tags={}) + + (data, _address) = sock.sent[0] + assert data == b"myapp.errors:1|c" + + +def test_host_and_port_are_forwarded_to_the_socket_address() -> None: + sock = FakeSocket() + backend = StatsDBackend(sock=sock, host="metrics.internal", port=9125) + + backend.increment("errors", tags={}) + + (_data, address) = sock.sent[0] + assert address == ("metrics.internal", 9125) + + +def test_a_failed_send_is_swallowed_not_raised() -> None: + sock = FakeSocket(fail=True) + backend = StatsDBackend(sock=sock) + + backend.increment("errors", tags={}) # must not raise + + +def test_close_closes_the_underlying_socket() -> None: + sock = FakeSocket() + backend = StatsDBackend(sock=sock) + + backend.close() + + assert sock.closed is True diff --git a/tests/test_plugins/test_tamper_evident_plugin.py b/tests/test_plugins/test_tamper_evident_plugin.py index e5fff1c..222846b 100644 --- a/tests/test_plugins/test_tamper_evident_plugin.py +++ b/tests/test_plugins/test_tamper_evident_plugin.py @@ -2,7 +2,7 @@ import copy -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.plugins.tamper_evident_plugin import TamperEvidentPlugin from logquill.transports.transport import CollectingTransport diff --git a/tests/test_plugins/test_trace_context_plugin.py b/tests/test_plugins/test_trace_context_plugin.py index 720db4f..ccdc96a 100644 --- a/tests/test_plugins/test_trace_context_plugin.py +++ b/tests/test_plugins/test_trace_context_plugin.py @@ -3,7 +3,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.plugins.trace_context_plugin import ( TraceContextPlugin, parse_trace_header, diff --git a/tests/test_privacy.py b/tests/test_privacy.py index 3b0df6a..4233553 100644 --- a/tests/test_privacy.py +++ b/tests/test_privacy.py @@ -3,7 +3,7 @@ import pytest from logquill import Logger -from logquill.privacy import ( +from logquill.privacy.privacy import ( CONTENT_FIELDS, FIELD_CLASSES, apply_content_policy, diff --git a/tests/test_retry.py b/tests/test_retry.py index b804af9..bf00d01 100644 --- a/tests/test_retry.py +++ b/tests/test_retry.py @@ -3,7 +3,7 @@ import pytest from logquill import Logger -from logquill.retry import RetryTracker +from logquill.tracing.retry import RetryTracker from logquill.transports.transport import CollectingTransport diff --git a/tests/test_runtime_level.py b/tests/test_runtime_level.py index 32b93e7..3cf00b3 100644 --- a/tests/test_runtime_level.py +++ b/tests/test_runtime_level.py @@ -9,7 +9,7 @@ import pytest from logquill import Level, Logger -from logquill.runtime_level import ( +from logquill.runtime.runtime_level import ( LevelEnvWatcher, LevelFileWatcher, install_signal_level_handler, @@ -85,7 +85,7 @@ def test_an_invalid_level_value_warns_and_leaves_the_level_unchanged( def test_without_sigusr1_on_this_platform_raises_unless_an_explicit_signum_is_given( monkeypatch: pytest.MonkeyPatch, ) -> None: - import logquill.runtime_level as module + import logquill.runtime.runtime_level as module class NoSigUsr1: def getsignal(self, *a: object) -> object: diff --git a/tests/test_semconv.py b/tests/test_semconv.py index 5e4d0cb..49524aa 100644 --- a/tests/test_semconv.py +++ b/tests/test_semconv.py @@ -7,8 +7,9 @@ import pytest -from logquill import Level, semconv -from logquill.records import create_record +from logquill import Level +from logquill.core.records import create_record +from logquill.tracing import semconv def _record(message: str = "m", *, llm: dict[str, Any] | None = None, **meta: Any) -> Any: diff --git a/tests/test_serverless.py b/tests/test_serverless.py index 8d01b33..c857988 100644 --- a/tests/test_serverless.py +++ b/tests/test_serverless.py @@ -2,8 +2,8 @@ import pytest -from logquill.logger import Logger -from logquill.serverless import with_azure_function, with_cloud_function, with_lambda +from logquill.core.logger import Logger +from logquill.runtime.serverless import with_azure_function, with_cloud_function, with_lambda from logquill.transports.transport import CollectingTransport diff --git a/tests/test_shutdown.py b/tests/test_shutdown.py index 9fb5ae7..b387f8e 100644 --- a/tests/test_shutdown.py +++ b/tests/test_shutdown.py @@ -5,7 +5,8 @@ import textwrap from pathlib import Path -from logquill import Logger, shutdown +from logquill import Logger +from logquill.runtime import shutdown from logquill.transports.transport import CollectingTransport _SCRIPT = """ diff --git a/tests/test_trace_tree.py b/tests/test_trace_tree.py index b9220f4..2ddaf3b 100644 --- a/tests/test_trace_tree.py +++ b/tests/test_trace_tree.py @@ -3,7 +3,7 @@ from typing import Any from logquill import Logger, RunPlugin -from logquill.trace_tree import Rollup, TraceBuilder, build_trace, render_tree +from logquill.cli.trace_tree import Rollup, TraceBuilder, build_trace, render_tree from logquill.transports.transport import CollectingTransport diff --git a/tests/test_tracing.py b/tests/test_tracing.py index 55cfc26..ef85a39 100644 --- a/tests/test_tracing.py +++ b/tests/test_tracing.py @@ -1,6 +1,6 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.transport import CollectingTransport diff --git a/tests/test_tracing_end_to_end.py b/tests/test_tracing_end_to_end.py index 7fb39df..c03d790 100644 --- a/tests/test_tracing_end_to_end.py +++ b/tests/test_tracing_end_to_end.py @@ -10,7 +10,7 @@ optional-dependency faking setup that lives there. """ -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.plugins.run_plugin import RunPlugin from logquill.plugins.trace_context_plugin import ( TraceContextPlugin, diff --git a/tests/test_transports/cloud/test_app_insights_transport.py b/tests/test_transports/cloud/test_app_insights_transport.py index a0a9235..6ff93cc 100644 --- a/tests/test_transports/cloud/test_app_insights_transport.py +++ b/tests/test_transports/cloud/test_app_insights_transport.py @@ -3,7 +3,7 @@ import json from typing import Sequence -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.cloud.app_insights_transport import AppInsightsTransport diff --git a/tests/test_transports/cloud/test_cloud_logging_transport.py b/tests/test_transports/cloud/test_cloud_logging_transport.py index eecdb69..28bb2bc 100644 --- a/tests/test_transports/cloud/test_cloud_logging_transport.py +++ b/tests/test_transports/cloud/test_cloud_logging_transport.py @@ -5,7 +5,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.cloud.cloud_logging_transport import CloudLoggingTransport diff --git a/tests/test_transports/cloud/test_cloudwatch_transport.py b/tests/test_transports/cloud/test_cloudwatch_transport.py index c061d2a..006c864 100644 --- a/tests/test_transports/cloud/test_cloudwatch_transport.py +++ b/tests/test_transports/cloud/test_cloudwatch_transport.py @@ -5,7 +5,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.cloud.cloudwatch_transport import CloudWatchTransport diff --git a/tests/test_transports/cloud/test_datadog_transport.py b/tests/test_transports/cloud/test_datadog_transport.py index 54bc018..ea562bc 100644 --- a/tests/test_transports/cloud/test_datadog_transport.py +++ b/tests/test_transports/cloud/test_datadog_transport.py @@ -5,7 +5,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.cloud.datadog_transport import DatadogTransport diff --git a/tests/test_transports/cloud/test_elasticsearch_transport.py b/tests/test_transports/cloud/test_elasticsearch_transport.py index 9b2b56d..327b18c 100644 --- a/tests/test_transports/cloud/test_elasticsearch_transport.py +++ b/tests/test_transports/cloud/test_elasticsearch_transport.py @@ -3,7 +3,7 @@ import json from typing import Sequence -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.cloud.elasticsearch_transport import ElasticsearchTransport diff --git a/tests/test_transports/cloud/test_new_relic_transport.py b/tests/test_transports/cloud/test_new_relic_transport.py index f27b80c..b7feedf 100644 --- a/tests/test_transports/cloud/test_new_relic_transport.py +++ b/tests/test_transports/cloud/test_new_relic_transport.py @@ -4,7 +4,7 @@ import json from typing import Callable -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.cloud.new_relic_transport import ( NewRelicSenderResult, NewRelicTransport, @@ -50,7 +50,7 @@ def test_region_url_gzip_body_and_headers() -> None: def test_stripping_event_type_keeps_the_schema_version_and_llm_block() -> None: from logquill import Level - from logquill.records import create_record + from logquill.core.records import create_record sender = FakeSender() transport = NewRelicTransport(license_key="lk-1", sender=sender, max_records=1) diff --git a/tests/test_transports/cloud/test_syslog_transport.py b/tests/test_transports/cloud/test_syslog_transport.py index 837b47e..a7a5952 100644 --- a/tests/test_transports/cloud/test_syslog_transport.py +++ b/tests/test_transports/cloud/test_syslog_transport.py @@ -2,7 +2,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.cloud.syslog_transport import SyslogTransport diff --git a/tests/test_transports/graph/__init__.py b/tests/test_transports/graph/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/test_transports/graph/test_base_graph_transport.py b/tests/test_transports/graph/test_base_graph_transport.py new file mode 100644 index 0000000..69899e6 --- /dev/null +++ b/tests/test_transports/graph/test_base_graph_transport.py @@ -0,0 +1,185 @@ +from __future__ import annotations + +from collections import defaultdict +from typing import Any + +from logquill import Logger, RunPlugin +from logquill.transports.graph.base_graph_transport import CYPHER, BaseGraphTransport + + +class FakeGraphTransport(BaseGraphTransport): + """Records every Cypher call it would have executed, instead of + talking to a real backend.""" + + def __init__(self, **kwargs: Any) -> None: + super().__init__(**kwargs) + self.calls: list[tuple[str, dict[str, Any]]] = [] + + def _execute(self, query: str, parameters: dict[str, Any]) -> None: + self.calls.append((query, parameters)) + + +class MiniGraph: + """A tiny in-memory openCypher-shaped engine for test purposes: applies + exactly the semantics `CYPHER` describes (MERGE a node by key, link it + to a session and/or a parent) so a real traversal can be checked + against, without needing a live database.""" + + def __init__(self) -> None: + self.nodes: dict[str, dict[str, Any]] = {} + self.parent_of: dict[str, str] = {} # child_key -> parent_key + self.session_of: defaultdict[str, set[str]] = defaultdict(set) + + def apply(self, query: str, parameters: dict[str, Any]) -> None: + assert query == CYPHER + for row in parameters["rows"]: + self.nodes[row["key"]] = row["props"] + if row["session_key"] is not None: + self.session_of[row["session_key"]].add(row["key"]) + if row["parent_key"] is not None: + self.parent_of[row["key"]] = row["parent_key"] + + def children_of(self, key: str) -> list[str]: + return [child for child, parent in self.parent_of.items() if parent == key] + + def roots(self) -> list[str]: + return [key for key in self.nodes if key not in self.parent_of] + + def reconstruct_tree(self, key: str) -> dict[str, Any]: + """A traversal: walk PARENT_OF from `key`, the same thing a Cypher + `MATCH (n {key: $key})-[:PARENT_OF*0..]->(d) RETURN d` would do.""" + return { + "message": self.nodes[key]["message"], + "children": sorted( + (self.reconstruct_tree(child) for child in self.children_of(key)), + key=lambda node: node["message"], + ), + } + + +def _run_agent(logger: Logger) -> None: + with logger.span("run", operation="invoke_agent", agent_name="planner"): + logger.thought("plan") + with logger.span("step"): + logger.llm_call("chat", model="m", tokens_in=10, tokens_out=5, cost_usd=0.01) + logger.action("call", tool="search") + + +def test_every_record_becomes_a_row_with_the_shared_cypher_query() -> None: + sink = FakeGraphTransport(max_records=100) + logger = Logger("app.agent", transports=[sink], plugins=[RunPlugin(run_id="run-1")]) + _run_agent(logger) + logger.close() + + ((query, parameters),) = sink.calls + assert query == CYPHER + assert len(parameters["rows"]) == 5 # plan, chat, step, call, run (closed together on close()) + + +def test_a_span_is_keyed_by_its_own_span_id_a_leaf_is_keyed_randomly() -> None: + sink = FakeGraphTransport(max_records=100) + logger = Logger("app.agent", transports=[sink]) + with logger.span("work", span_id="a" * 16): + logger.info("leaf") + logger.close() + + ((query, parameters),) = sink.calls + leaf, root = parameters["rows"] + assert root["key"] == "a" * 16 + assert leaf["key"] != "a" * 16 + assert leaf["key"].startswith("leaf-") + + +def test_a_root_span_has_no_parent_key() -> None: + sink = FakeGraphTransport(max_records=100) + logger = Logger("app.agent", transports=[sink]) + with logger.span("root"): + pass + logger.close() + + ((query, parameters),) = sink.calls + (row,) = parameters["rows"] + assert row["parent_key"] is None + + +def test_run_id_is_preferred_over_trace_id_for_session_grouping() -> None: + sink = FakeGraphTransport(max_records=100) + logger = Logger("app.agent", transports=[sink]) + logger.info("x", run_id="run-1", trace_id="trace-1") + logger.close() + + ((query, parameters),) = sink.calls + (row,) = parameters["rows"] + assert row["session_key"] == "run-1" + + +def test_trace_id_is_used_when_there_is_no_run_id() -> None: + sink = FakeGraphTransport(max_records=100) + logger = Logger("app.agent", transports=[sink]) + logger.info("x", trace_id="trace-1") + logger.close() + + ((query, parameters),) = sink.calls + (row,) = parameters["rows"] + assert row["session_key"] == "trace-1" + + +def test_a_record_with_neither_has_no_session_key() -> None: + sink = FakeGraphTransport(max_records=100) + logger = Logger("app.agent", transports=[sink]) + logger.info("loose") + logger.close() + + ((query, parameters),) = sink.calls + (row,) = parameters["rows"] + assert row["session_key"] is None + + +def test_meta_is_json_encoded_and_key_fields_are_lifted_onto_props() -> None: + sink = FakeGraphTransport(max_records=100) + logger = Logger("app.agent", transports=[sink]) + logger.info("x", run_id="run-1", user_id=42) + logger.close() + + ((query, parameters),) = sink.calls + (row,) = parameters["rows"] + assert row["props"]["run_id"] == "run-1" + assert '"user_id":42' in row["props"]["meta"] + + +def test_a_cypher_traversal_reproduces_the_exact_span_tree() -> None: + """The exit criterion: an agent run logged normally produces a + queryable node/edge tree reconstructing the exact span hierarchy via a + Cypher traversal, with no manual graph-writing code at any call site. + """ + sink = FakeGraphTransport(max_records=100) + logger = Logger("app.agent", transports=[sink], plugins=[RunPlugin(run_id="run-1")]) + _run_agent(logger) + logger.close() + + graph = MiniGraph() + for query, parameters in sink.calls: + graph.apply(query, parameters) + + (root_key,) = graph.roots() + tree = graph.reconstruct_tree(root_key) + assert tree == { + "message": "run", + "children": [ + {"message": "call", "children": []}, + {"message": "plan", "children": []}, + {"message": "step", "children": [{"message": "chat", "children": []}]}, + ], + } + assert graph.session_of["run-1"] == set(graph.nodes) # every node is in the run's session + + +def test_buffering_is_batched_not_one_query_per_record() -> None: + sink = FakeGraphTransport(max_records=1000) + logger = Logger("app.agent", transports=[sink]) + for i in range(50): + logger.info(f"step {i}") + logger.close() + + assert len(sink.calls) == 1 + assert len(sink.calls[0][1]["rows"]) == 50 diff --git a/tests/test_transports/graph/test_falkordb_transport.py b/tests/test_transports/graph/test_falkordb_transport.py new file mode 100644 index 0000000..3aba2f8 --- /dev/null +++ b/tests/test_transports/graph/test_falkordb_transport.py @@ -0,0 +1,73 @@ +from __future__ import annotations + +import sys +from typing import Any + +import pytest + +from logquill import FalkorDBTransport, Logger +from logquill.transports.graph.base_graph_transport import CYPHER + + +class FakeGraph: + def __init__(self, calls: list[tuple[str, dict[str, Any]]]) -> None: + self._calls = calls + + def query(self, q: str, params: dict[str, Any] | None = None) -> None: + self._calls.append((q, params or {})) + + +class FakeClient: + def __init__(self) -> None: + self.calls: list[tuple[str, dict[str, Any]]] = [] + self.closed = False + self.selected: list[str] = [] + + def select_graph(self, name: str) -> FakeGraph: + self.selected.append(name) + return FakeGraph(self.calls) + + def close(self) -> None: + self.closed = True + + +def test_writes_go_through_the_selected_graph_with_the_shared_cypher() -> None: + client = FakeClient() + transport = FalkorDBTransport(client=client, graph_name="agents", max_records=100) + logger = Logger("app", transports=[transport]) + + logger.info("hello") + logger.close() + + ((query, parameters),) = client.calls + assert query == CYPHER + assert len(parameters["rows"]) == 1 + assert client.selected == ["agents"] + + +def test_the_graph_is_selected_only_once() -> None: + client = FakeClient() + transport = FalkorDBTransport(client=client, max_records=1) + logger = Logger("app", transports=[transport]) + + logger.info("one") + logger.info("two") + + assert client.selected == ["logquill"] + + +def test_close_does_not_close_an_injected_client() -> None: + client = FakeClient() + transport = FalkorDBTransport(client=client) + + transport.close() + + assert client.closed is False + + +def test_a_missing_falkordb_package_gives_an_install_hint(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setitem(sys.modules, "falkordb", None) + transport = FalkorDBTransport() + + with pytest.raises(ImportError, match=r"pip install logquill\[falkordb\]"): + transport._resolved_graph() diff --git a/tests/test_transports/graph/test_neo4j_transport.py b/tests/test_transports/graph/test_neo4j_transport.py new file mode 100644 index 0000000..fff769b --- /dev/null +++ b/tests/test_transports/graph/test_neo4j_transport.py @@ -0,0 +1,83 @@ +from __future__ import annotations + +import sys +from typing import Any + +import pytest + +from logquill import Logger, Neo4jTransport +from logquill.transports.graph.base_graph_transport import CYPHER + + +class FakeSession: + def __init__(self, calls: list[tuple[str, dict[str, Any]]]) -> None: + self._calls = calls + + def run(self, query: str, parameters: dict[str, Any]) -> None: + self._calls.append((query, parameters)) + + def __enter__(self) -> FakeSession: + return self + + def __exit__(self, *exc_info: object) -> None: + pass + + +class FakeDriver: + def __init__(self) -> None: + self.calls: list[tuple[str, dict[str, Any]]] = [] + self.closed = False + self.session_kwargs: list[dict[str, Any]] = [] + + def session(self, **kwargs: Any) -> FakeSession: + self.session_kwargs.append(kwargs) + return FakeSession(self.calls) + + def close(self) -> None: + self.closed = True + + +def test_writes_go_through_a_session_with_the_shared_cypher() -> None: + driver = FakeDriver() + transport = Neo4jTransport(driver=driver, max_records=100) + logger = Logger("app", transports=[transport]) + + logger.info("hello") + logger.close() + + ((query, parameters),) = driver.calls + assert query == CYPHER + assert len(parameters["rows"]) == 1 + + +def test_database_is_forwarded_to_the_session() -> None: + driver = FakeDriver() + transport = Neo4jTransport(driver=driver, database="agents", max_records=1) + Logger("app", transports=[transport]).info("hello") + + assert driver.session_kwargs == [{"database": "agents"}] + + +def test_no_database_means_no_database_kwarg() -> None: + driver = FakeDriver() + transport = Neo4jTransport(driver=driver, max_records=1) + Logger("app", transports=[transport]).info("hello") + + assert driver.session_kwargs == [{}] + + +def test_close_does_not_close_an_injected_driver() -> None: + driver = FakeDriver() + transport = Neo4jTransport(driver=driver) + + transport.close() + + assert driver.closed is False + + +def test_a_missing_neo4j_package_gives_an_install_hint(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setitem(sys.modules, "neo4j", None) + transport = Neo4jTransport() + + with pytest.raises(ImportError, match=r"pip install logquill\[neo4j\]"): + transport._resolved_driver() diff --git a/tests/test_transports/graph/test_neptune_transport.py b/tests/test_transports/graph/test_neptune_transport.py new file mode 100644 index 0000000..80ac9b7 --- /dev/null +++ b/tests/test_transports/graph/test_neptune_transport.py @@ -0,0 +1,41 @@ +from __future__ import annotations + +import json +import sys +from typing import Any + +import pytest + +from logquill import Logger, NeptuneTransport +from logquill.transports.graph.base_graph_transport import CYPHER + + +class FakeNeptuneClient: + def __init__(self) -> None: + self.calls: list[tuple[str, str]] = [] + + def execute_open_cypher_query(self, *, openCypherQuery: str, parameters: str) -> dict[str, Any]: # noqa: N803 + self.calls.append((openCypherQuery, parameters)) + return {"results": []} + + +def test_writes_use_the_opencypher_data_api_with_json_encoded_parameters() -> None: + client = FakeNeptuneClient() + transport = NeptuneTransport(client=client, max_records=100) + logger = Logger("app", transports=[transport]) + + logger.info("hello") + logger.close() + + ((query, parameters),) = client.calls + assert query == CYPHER + decoded = json.loads(parameters) + assert len(decoded["rows"]) == 1 + + +def test_a_missing_boto3_package_gives_an_install_hint(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setitem(sys.modules, "boto3", None) + transport = NeptuneTransport() + + with pytest.raises(ImportError, match=r"pip install logquill\[aws\]"): + transport._resolved_client() diff --git a/tests/test_transports/nosql/test_dynamodb_transport.py b/tests/test_transports/nosql/test_dynamodb_transport.py index 14b678c..cd788ab 100644 --- a/tests/test_transports/nosql/test_dynamodb_transport.py +++ b/tests/test_transports/nosql/test_dynamodb_transport.py @@ -5,7 +5,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.nosql.dynamodb_transport import DynamoDBTransport diff --git a/tests/test_transports/nosql/test_mongodb_transport.py b/tests/test_transports/nosql/test_mongodb_transport.py index bccb5c1..f068385 100644 --- a/tests/test_transports/nosql/test_mongodb_transport.py +++ b/tests/test_transports/nosql/test_mongodb_transport.py @@ -5,7 +5,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.nosql.mongodb_transport import MongoDBTransport diff --git a/tests/test_transports/nosql/test_redis_transport.py b/tests/test_transports/nosql/test_redis_transport.py index dc5340d..2523f10 100644 --- a/tests/test_transports/nosql/test_redis_transport.py +++ b/tests/test_transports/nosql/test_redis_transport.py @@ -6,7 +6,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.nosql.redis_transport import RedisTransport diff --git a/tests/test_transports/queue/test_kafka_transport.py b/tests/test_transports/queue/test_kafka_transport.py index 56d9db3..a1dae6b 100644 --- a/tests/test_transports/queue/test_kafka_transport.py +++ b/tests/test_transports/queue/test_kafka_transport.py @@ -5,7 +5,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.queue.kafka_transport import KafkaTransport diff --git a/tests/test_transports/queue/test_pubsub_transport.py b/tests/test_transports/queue/test_pubsub_transport.py index 422190c..dffd69c 100644 --- a/tests/test_transports/queue/test_pubsub_transport.py +++ b/tests/test_transports/queue/test_pubsub_transport.py @@ -4,7 +4,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.queue.pubsub_transport import PubSubTransport diff --git a/tests/test_transports/queue/test_rabbitmq_transport.py b/tests/test_transports/queue/test_rabbitmq_transport.py index 30cbff3..7dabfad 100644 --- a/tests/test_transports/queue/test_rabbitmq_transport.py +++ b/tests/test_transports/queue/test_rabbitmq_transport.py @@ -4,7 +4,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.queue.rabbitmq_transport import RabbitMQTransport diff --git a/tests/test_transports/queue/test_sqs_transport.py b/tests/test_transports/queue/test_sqs_transport.py index f46bdd4..78543a4 100644 --- a/tests/test_transports/queue/test_sqs_transport.py +++ b/tests/test_transports/queue/test_sqs_transport.py @@ -4,7 +4,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.queue.sqs_transport import SQSTransport diff --git a/tests/test_transports/sql/test_mysql_transport.py b/tests/test_transports/sql/test_mysql_transport.py index 2aac347..213a2f2 100644 --- a/tests/test_transports/sql/test_mysql_transport.py +++ b/tests/test_transports/sql/test_mysql_transport.py @@ -4,7 +4,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.sql.mysql_transport import MySQLCursorLike, MySQLTransport diff --git a/tests/test_transports/sql/test_postgres_transport.py b/tests/test_transports/sql/test_postgres_transport.py index 11db140..80ef780 100644 --- a/tests/test_transports/sql/test_postgres_transport.py +++ b/tests/test_transports/sql/test_postgres_transport.py @@ -4,7 +4,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.sql.postgres_transport import PostgresCursorLike, PostgresTransport diff --git a/tests/test_transports/sql/test_sqlite_transport.py b/tests/test_transports/sql/test_sqlite_transport.py index 0afcf49..da7bf6c 100644 --- a/tests/test_transports/sql/test_sqlite_transport.py +++ b/tests/test_transports/sql/test_sqlite_transport.py @@ -2,7 +2,7 @@ from typing import Iterable, Sequence -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.sql.sqlite_transport import SQLiteConnectionLike, SQLiteTransport diff --git a/tests/test_transports/test_batching_transport.py b/tests/test_transports/test_batching_transport.py index e6b914b..3a7a96e 100644 --- a/tests/test_transports/test_batching_transport.py +++ b/tests/test_transports/test_batching_transport.py @@ -2,8 +2,8 @@ from typing import Sequence -from logquill.levels import Level -from logquill.records import LogRecord, create_record +from logquill.core.levels import Level +from logquill.core.records import LogRecord, create_record from logquill.transports.batching_transport import BatchingTransport diff --git a/tests/test_transports/test_console_transport.py b/tests/test_transports/test_console_transport.py index dcb0479..9d13af3 100644 --- a/tests/test_transports/test_console_transport.py +++ b/tests/test_transports/test_console_transport.py @@ -1,6 +1,6 @@ import io -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.console_transport import ConsoleTransport diff --git a/tests/test_transports/test_file_transport.py b/tests/test_transports/test_file_transport.py index 30787a8..392bbdc 100644 --- a/tests/test_transports/test_file_transport.py +++ b/tests/test_transports/test_file_transport.py @@ -3,9 +3,9 @@ import pytest -from logquill.levels import Level -from logquill.logger import Logger -from logquill.records import create_record +from logquill.core.levels import Level +from logquill.core.logger import Logger +from logquill.core.records import create_record from logquill.transports.file_transport import FileTransport diff --git a/tests/test_transports/test_http_transport.py b/tests/test_transports/test_http_transport.py index edcda65..a5b61c3 100644 --- a/tests/test_transports/test_http_transport.py +++ b/tests/test_transports/test_http_transport.py @@ -4,7 +4,7 @@ import pytest -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.http_transport import HTTPTransport diff --git a/tests/test_transports/test_transport.py b/tests/test_transports/test_transport.py index c0d9c11..92333e1 100644 --- a/tests/test_transports/test_transport.py +++ b/tests/test_transports/test_transport.py @@ -1,4 +1,4 @@ -from logquill.logger import Logger +from logquill.core.logger import Logger from logquill.transports.transport import CollectingTransport diff --git a/tests/test_webui.py b/tests/test_webui.py index 3ad3d84..f1947f3 100644 --- a/tests/test_webui.py +++ b/tests/test_webui.py @@ -12,9 +12,7 @@ import pytest from logquill import Logger, RunPlugin -from logquill.transports.file_transport import FileTransport -from logquill.transports.sql.sqlite_transport import SQLiteTransport -from logquill.webui import ( +from logquill.cli.webui import ( STATIC_DIR, JSONLSource, SQLiteSource, @@ -22,6 +20,8 @@ search_records, summarize_runs, ) +from logquill.transports.file_transport import FileTransport +from logquill.transports.sql.sqlite_transport import SQLiteTransport def _write_a_run(path: Path, run_id: str = "run-1") -> None: diff --git a/tests/test_worker.py b/tests/test_worker.py index 50726fa..0d9c370 100644 --- a/tests/test_worker.py +++ b/tests/test_worker.py @@ -6,7 +6,7 @@ import pytest -from logquill.worker import AsyncWorker +from logquill.runtime.worker import AsyncWorker _POLL_TIMEOUT = 2.0 _POLL_INTERVAL = 0.005 @@ -211,7 +211,7 @@ def test_the_first_drop_warns_even_when_the_monotonic_clock_is_still_near_zero( ) -> None: # the monotonic clock's zero point is arbitrary (boot time on many # platforms), so a process started shortly after boot must still warn - monkeypatch.setattr("logquill.worker.time.monotonic", lambda: 5.0) + monkeypatch.setattr("logquill.runtime.worker.time.monotonic", lambda: 5.0) gate = threading.Event() worker = AsyncWorker(max_queue_size=1, backpressure="drop_newest")