diff --git a/CHANGELOG.md b/CHANGELOG.md index 394c73d..5fbf795 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,728 +1,42 @@ -# CHANGELOG - +# Changelog ## Unreleased -### Breaking Changes - -- `SplunkAOLogger.flush()` and `async_flush()` now return `None` and only drain - completed spans already queued for OTLP export. They no longer conclude open - spans or return uploaded proprietary trace objects. - -- **`OpenAIGalileo` renamed to `OpenAISplunkAO`** (HYBIM-726): The OpenAI wrapper class in - `splunk_ao.openai` has been renamed as part of the Galileo → Splunk AO rebrand. Any code - directly importing this class by name must update to - `from splunk_ao.openai import OpenAISplunkAO`. - -### Security - -- **deps (VULN-88813)**: Raise the optional `starlette` floor to `>=1.3.1` (from `>=0.27.0`) to remediate urlencoded form limit bypass in `request.form()`. Starlette 1.x removes APIs deprecated since 0.x; consumers of the `middleware`, `openai`, or `crewai` extras who rely on removed Starlette 0.x APIs may need to update custom middleware. - -### Features - -- **New `generated_output` field**: Add `generated_output` field to `DatasetRecord` for storing model-generated outputs separately from ground truth. This allows you to track both the expected output (ground truth) and the actual model output in the same dataset record. In the UI, this field is displayed as "Generated Output". - - Example: - ```python - from galileo.schema.datasets import DatasetRecord - - record = DatasetRecord( - input="What is 2+2?", - output="4", # Ground truth - generated_output="The answer is 4" # Model-generated output - ) - ``` - -- **Ground Truth naming support**: The existing `output` field is now displayed as "Ground Truth" in the Galileo UI for better clarity. The SDK supports both `output` and `ground_truth` field names when creating records - both are normalized to `output` internally, ensuring full backward compatibility. You can use either field name, and access the value via the `ground_truth` property. - - Example: - ```python - from galileo.schema.datasets import DatasetRecord - - # Using 'output' (backward compatible) - record1 = DatasetRecord(input="What is 2+2?", output="4") - assert record1.ground_truth == "4" # Property accessor - - # Using 'ground_truth' (new recommended way) - record2 = DatasetRecord(input="What is 2+2?", ground_truth="4") - assert record2.output == "4" # Normalized internally - assert record2.ground_truth == "4" # Property accessor - ``` - - -## v0.10.0 (2025-05-29) - -### Bug Fixes - -- Langchain on_chain_start with_kwargs name and serialised is none - ([#138](https://github.com/rungalileo/galileo-python/pull/138), - [`132955c`](https://github.com/rungalileo/galileo-python/commit/132955cd88f369e99c203262dfd77c394a4e02a7)) - -### Chores - -- **deps**: Bump langchain-core from 0.3.40 to 0.3.58 - ([#122](https://github.com/rungalileo/galileo-python/pull/122), - [`e2e6a49`](https://github.com/rungalileo/galileo-python/commit/e2e6a49cca0657ebd6483a3d361505422fc754ca)) - -### Features - -- Add codeflash optimizer on our repo - ([#151](https://github.com/rungalileo/galileo-python/pull/151), - [`de1897b`](https://github.com/rungalileo/galileo-python/commit/de1897bf9aa46aa2fba0f9f7870a27ce766985b5)) - -- Enable setting session ID ([#139](https://github.com/rungalileo/galileo-python/pull/139), - [`e646df3`](https://github.com/rungalileo/galileo-python/commit/e646df34606a3949543364c15c44c36b5b697bdb)) - -This changes the `start_session` function to return the session ID, and adds a new `set_session` - function. - -- **langchain**: Implement duration_ns metric for langchain callback async handler - ([#150](https://github.com/rungalileo/galileo-python/pull/150), - [`eed7a81`](https://github.com/rungalileo/galileo-python/commit/eed7a81f89e4e15801fa95e65229d08ea9824cab)) - -- **langchain**: Implement duration_ns metric for langchain callback handler - ([#145](https://github.com/rungalileo/galileo-python/pull/145), - [`55f3cdf`](https://github.com/rungalileo/galileo-python/commit/55f3cdf63cbc254f4d8804ef61081c877b4c61b7)) - - -## v0.9.2 (2025-05-22) - -### Bug Fixes - -- Creating empty dataset ([#134](https://github.com/rungalileo/galileo-python/pull/134), - [`3b13f52`](https://github.com/rungalileo/galileo-python/commit/3b13f522de20ba977915a310e1b7aea184b867f5)) - -- Runtimewarning: coroutine AsyncMockMixin._execute_mock_call was never awaited - ([#135](https://github.com/rungalileo/galileo-python/pull/135), - [`bd76cff`](https://github.com/rungalileo/galileo-python/commit/bd76cffb4098f38e064c720305ddd2f2bd368bed)) - -- Updating Langchain handler issues ([#118](https://github.com/rungalileo/galileo-python/pull/118), - [`4304d16`](https://github.com/rungalileo/galileo-python/commit/4304d16d727ef797103e1a8efcb6c7d2531021f6)) - -### Chores - -- **deps-dev**: Bump mkdocs-material from 9.6.13 to 9.6.14 - ([#136](https://github.com/rungalileo/galileo-python/pull/136), - [`71d6676`](https://github.com/rungalileo/galileo-python/commit/71d667610930967b77fa687cc691a1a9cc735322)) - -- **release**: V0.9.2 - ([`12ad78c`](https://github.com/rungalileo/galileo-python/commit/12ad78c78a55e2c1db0f7bc85ffcbe307b15337a)) - -Automatically generated by python-semantic-release - - -## v0.9.1 (2025-05-15) - -### Bug Fixes - -- Reverting a change from a previous context manager fix - ([#133](https://github.com/rungalileo/galileo-python/pull/133), - [`b3bd345`](https://github.com/rungalileo/galileo-python/commit/b3bd34554f4094fe6dc178242bb2af3f8218efcf)) - -### Chores - -- **deps-dev**: Bump mkdocs-material from 9.6.12 to 9.6.13 - ([#131](https://github.com/rungalileo/galileo-python/pull/131), - [`f44be0c`](https://github.com/rungalileo/galileo-python/commit/f44be0ca98e7468d207fe0b9eafae38d325ffd60)) - -- **deps-dev**: Bump pytest-asyncio from 0.25.3 to 0.26.0 - ([#129](https://github.com/rungalileo/galileo-python/pull/129), - [`43e2af7`](https://github.com/rungalileo/galileo-python/commit/43e2af726be1b35c3594f987598bae2242f911f6)) - -- **release**: V0.9.1 - ([`b80a218`](https://github.com/rungalileo/galileo-python/commit/b80a218735d3092c0748c1eb348d34a100363b63)) - -Automatically generated by python-semantic-release - -### Continuous Integration - -- Run tests on all OSes ([#127](https://github.com/rungalileo/galileo-python/pull/127), - [`d6f3d1b`](https://github.com/rungalileo/galileo-python/commit/d6f3d1bf494d7568abe776b87c7f4be4d750ee9c)) - - -## v0.9.0 (2025-05-10) - -### Bug Fixes - -- Using context var stacks for saving context with nested _call_ s - ([#126](https://github.com/rungalileo/galileo-python/pull/126), - [`42c1424`](https://github.com/rungalileo/galileo-python/commit/42c14245d7099f1fb73c93c5d2e19b94d8061a12)) - -### Chores - -- **deps**: Bump codecov/codecov-action from 5.4.0 to 5.4.2 - ([#119](https://github.com/rungalileo/galileo-python/pull/119), - [`ba53ebf`](https://github.com/rungalileo/galileo-python/commit/ba53ebf3976e6f92a415611d088c368d2262e10c)) - -- **deps**: Bump openai-agents from 0.0.7 to 0.0.12 - ([#123](https://github.com/rungalileo/galileo-python/pull/123), - [`0150eee`](https://github.com/rungalileo/galileo-python/commit/0150eee62b4763fc9d3acf6e233a8b9e8d7cbe31)) - -- **deps-dev**: Bump mkdocs-material from 9.6.5 to 9.6.12 - ([#125](https://github.com/rungalileo/galileo-python/pull/125), - [`707149a`](https://github.com/rungalileo/galileo-python/commit/707149a4614709548bf1714f770099a44f9fe9a7)) - -- **deps-dev**: Bump mkdocstrings from 0.27.0 to 0.29.1 - ([#124](https://github.com/rungalileo/galileo-python/pull/124), - [`bdb2c47`](https://github.com/rungalileo/galileo-python/commit/bdb2c4736fac595640b5723d62f90738726503fd)) - -- **deps-dev**: Bump pytest from 8.3.4 to 8.3.5 - ([#121](https://github.com/rungalileo/galileo-python/pull/121), - [`7e5ab19`](https://github.com/rungalileo/galileo-python/commit/7e5ab19c8e43775ce0e9d3085637b27781635afd)) - -- **release**: V0.9.0 - ([`eeb6ddf`](https://github.com/rungalileo/galileo-python/commit/eeb6ddffa9ab2292142a603d0eadb02f71bee603)) - -Automatically generated by python-semantic-release - -### Features - -- Local scorers for local runner experiments - ([#116](https://github.com/rungalileo/galileo-python/pull/116), - [`4f7d8e0`](https://github.com/rungalileo/galileo-python/commit/4f7d8e02e40a2513b1e2a7b3a35a200d7db799e7)) - -- Session management ([#120](https://github.com/rungalileo/galileo-python/pull/120), - [`fed6f65`](https://github.com/rungalileo/galileo-python/commit/fed6f65c7f1887bbf1e8bc89388d064a57a9cc72)) - - -## v0.8.1 (2025-04-30) - -### Chores - -- Restrict galileo-core version to only float forward for patch releases - ([#115](https://github.com/rungalileo/galileo-python/pull/115), - [`37a4c36`](https://github.com/rungalileo/galileo-python/commit/37a4c3670fe6c0e23d17e3238228bb3fc9d04919)) - -- **release**: V0.8.1 - ([`a308655`](https://github.com/rungalileo/galileo-python/commit/a308655d77a6f2beaad98c8c515b97f264db84f5)) - -Automatically generated by python-semantic-release - - -## v0.8.0 (2025-04-30) - -### Bug Fixes - -- Use correct logging schemas from `galileo_core.schemas.logging` - ([#114](https://github.com/rungalileo/galileo-python/pull/114), - [`fa387d3`](https://github.com/rungalileo/galileo-python/commit/fa387d3698dac559ad56e6d5b0e4a21ee82554d2)) - -### Chores - -- Pulling the latest openapi.yaml and regenerating the API client - ([#112](https://github.com/rungalileo/galileo-python/pull/112), - [`bfa38fb`](https://github.com/rungalileo/galileo-python/commit/bfa38fb9a4ead7a0df8dc6a02e77945f4d1578d7)) - -- **deps**: Bump h11 from 0.14.0 to 0.16.0 - ([#113](https://github.com/rungalileo/galileo-python/pull/113), - [`1336bef`](https://github.com/rungalileo/galileo-python/commit/1336bef29f32b36e1663f77051841e6004699676)) - -- **release**: V0.8.0 - ([`3335cb9`](https://github.com/rungalileo/galileo-python/commit/3335cb92c6f4904fdc5c05f40f37abd3657e264d)) - -Automatically generated by python-semantic-release - -### Documentation - -- Add supported py versions, refactor urls - ([#111](https://github.com/rungalileo/galileo-python/pull/111), - [`d6e7169`](https://github.com/rungalileo/galileo-python/commit/d6e7169a05e4c59115923b0d7a0623e76336f385)) - -### Features - -- Enable mypy on ci ([#108](https://github.com/rungalileo/galileo-python/pull/108), - [`c445a09`](https://github.com/rungalileo/galileo-python/commit/c445a09587c701ac65cbfce90406e53db3a2b0fb)) - - -## v0.7.0 (2025-04-17) - -### Bug Fixes - -- Adding list_prompt_templates ([#102](https://github.com/rungalileo/galileo-python/pull/102), - [`5814c90`](https://github.com/rungalileo/galileo-python/commit/5814c90fadf1a0c534cf168764d4871940ad20e7)) - -- Error occurred during execution: shutdown: 'GalileoTracingProcessor' object has no attribute - '_commit' ([#105](https://github.com/rungalileo/galileo-python/pull/105), - [`e2ece22`](https://github.com/rungalileo/galileo-python/commit/e2ece2210c453c5058bfe2d3b4164be4644beeef)) - -- Fix functionality to add new rows to dataset - ([#103](https://github.com/rungalileo/galileo-python/pull/103), - [`5f2d633`](https://github.com/rungalileo/galileo-python/commit/5f2d6337c440af83ae54d9809dc0a510833fc58d)) - -- Fixing the OpenAI wrapper when it's used in an active trace - ([#109](https://github.com/rungalileo/galileo-python/pull/109), - [`10bb6f6`](https://github.com/rungalileo/galileo-python/commit/10bb6f6b46dffb9089f4ec4bd2fce701c5433ab5)) - -### Chores - -- **release**: V0.7.0 - ([`76ab22d`](https://github.com/rungalileo/galileo-python/commit/76ab22d9f8b6a3466da0ec2264225e3fa7e7b1df)) - -Automatically generated by python-semantic-release - -### Features - -- Bubble up openai status_code ([#107](https://github.com/rungalileo/galileo-python/pull/107), - [`8074ea3`](https://github.com/rungalileo/galileo-python/commit/8074ea352501e4f8e80d1569eeec1cf4b9f67068)) - -### Testing - -- Bump codecov target ([#106](https://github.com/rungalileo/galileo-python/pull/106), - [`04ee6eb`](https://github.com/rungalileo/galileo-python/commit/04ee6ebe64930a9b8f9dabd03bfb227079e99011)) - - -## v0.6.0 (2025-04-10) - -### Bug Fixes - -- Adding tools sent to the chat model in the Langchain callback handlers - ([#95](https://github.com/rungalileo/galileo-python/pull/95), - [`125de12`](https://github.com/rungalileo/galileo-python/commit/125de12b889975250000dfb34d522d498b4ba668)) - -Co-authored-by: Andrii Soldatenko - -- Json datasets are not being passed into runner functions correctly - ([#100](https://github.com/rungalileo/galileo-python/pull/100), - [`e833481`](https://github.com/rungalileo/galileo-python/commit/e8334811084d0e74fb98341509d221851dae20aa)) - -- Updating README snippets to point to the correct OpenAI client wrapper - ([#92](https://github.com/rungalileo/galileo-python/pull/92), - [`ffd2a68`](https://github.com/rungalileo/galileo-python/commit/ffd2a68d5db198c76f2add24d46e46a805f426d9)) - -- Updating the Langchain handler retriever, tool, async chain callbacks - ([#101](https://github.com/rungalileo/galileo-python/pull/101), - [`ce29a91`](https://github.com/rungalileo/galileo-python/commit/ce29a91bb581c78fc01b1ee97ab2f2b6a043b17a)) - -- Use `Message` and `MessageRole` models from `galileo-core` - ([#99](https://github.com/rungalileo/galileo-python/pull/99), - [`c9ad656`](https://github.com/rungalileo/galileo-python/commit/c9ad656a702c1b1950c349fc43ca839c4e5cd799)) - -### Chores - -- Enum name normalization ([#91](https://github.com/rungalileo/galileo-python/pull/91), - [`e084dac`](https://github.com/rungalileo/galileo-python/commit/e084daccc3ad46318ac4a02d3adc50fd3e2d5781)) - -- Increase codecov target, since we alredy > 75% - ([#93](https://github.com/rungalileo/galileo-python/pull/93), - [`4a09189`](https://github.com/rungalileo/galileo-python/commit/4a091897edeff3aa0d0311101c5e57f9ebc97071)) - -- Revert "Enum name normalization" ([#98](https://github.com/rungalileo/galileo-python/pull/98), - [`508af9e`](https://github.com/rungalileo/galileo-python/commit/508af9e8ab9040ca59d3986b1368d3dc0111f6d9)) - -- **release**: V0.6.0 - ([`ff166d8`](https://github.com/rungalileo/galileo-python/commit/ff166d86fa0786401172369e6ff5f4ef6993fadd)) - -Automatically generated by python-semantic-release - -### Features - -- Add dataset version methods ([#94](https://github.com/rungalileo/galileo-python/pull/94), - [`4081828`](https://github.com/rungalileo/galileo-python/commit/408182825fde179f4a40b1ca3114116b60fa5d7a)) - -- Allowing custom hosted scorers to be used - ([#87](https://github.com/rungalileo/galileo-python/pull/87), - [`e908d87`](https://github.com/rungalileo/galileo-python/commit/e908d8707c65ee1e11b0868ebfc67479b3cef59a)) - -Co-authored-by: Andrii Soldatenko - -- Error if a non existent metric is specified - ([#97](https://github.com/rungalileo/galileo-python/pull/97), - [`93a4914`](https://github.com/rungalileo/galileo-python/commit/93a4914399cca4013af2e1e77cba1ad5a9478f88)) - -- Improve error handling for create_dataset - ([#96](https://github.com/rungalileo/galileo-python/pull/96), - [`0dda7ba`](https://github.com/rungalileo/galileo-python/commit/0dda7baf67ec1f86d40a71794279b3e860255d23)) - - -## v0.5.0 (2025-04-04) - -### Chores - -- **release**: V0.5.0 - ([`2203995`](https://github.com/rungalileo/galileo-python/commit/2203995932c767a8df1eabc8320d4b083a734299)) - -Automatically generated by python-semantic-release - -### Features - -- **integration**: Add support for openai agents (with tree) - ([#88](https://github.com/rungalileo/galileo-python/pull/88), - [`78b0c56`](https://github.com/rungalileo/galileo-python/commit/78b0c56681065fc6db7a3944aa7a5ae5f43ba3f5)) - -Co-authored-by: Andrii Soldatenko - - -## v0.4.0 (2025-04-04) - -### Chores - -- **release**: V0.4.0 - ([`8d700bc`](https://github.com/rungalileo/galileo-python/commit/8d700bcc321e3476b132e6b5a65cad2a19365f0a)) - -Automatically generated by python-semantic-release - -### Features - -- Langchain async callback handler ([#89](https://github.com/rungalileo/galileo-python/pull/89), - [`ea8d4f8`](https://github.com/rungalileo/galileo-python/commit/ea8d4f81ecf9d26fb6aec849354bd461736195c9)) - -Co-authored-by: Andrii Soldatenko - - -## v0.3.0 (2025-04-02) - -### Bug Fixes - -- Ierror handling in Experiments.get ([#84](https://github.com/rungalileo/galileo-python/pull/84), - [`0d36b8c`](https://github.com/rungalileo/galileo-python/commit/0d36b8c86d8034da4a4daf7e763136cc9c9b2e69)) - -- Improve error message when galileo api key is empty or invalid - ([#82](https://github.com/rungalileo/galileo-python/pull/82), - [`31593c4`](https://github.com/rungalileo/galileo-python/commit/31593c4b82acaf76e41ea9cab5361b6b6e634ef4)) - -- Make No traces to flush as info ([#83](https://github.com/rungalileo/galileo-python/pull/83), - [`babcd2f`](https://github.com/rungalileo/galileo-python/commit/babcd2fd40f2517310249c34fb1ab1bd02c69c9d)) - -- Making the run_experiment return object consistent - ([#86](https://github.com/rungalileo/galileo-python/pull/86), - [`4a3059f`](https://github.com/rungalileo/galileo-python/commit/4a3059f13fcfbd06de8b8f607dfd980edaa27d4d)) - -### Chores - -- **release**: V0.3.0 - ([`8fee183`](https://github.com/rungalileo/galileo-python/commit/8fee183ffa1d4f49ed582a50acfc255400db8ea7)) - -Automatically generated by python-semantic-release - -### Features - -- Add ability to disable logger ([#68](https://github.com/rungalileo/galileo-python/pull/68), - [`b417bd6`](https://github.com/rungalileo/galileo-python/commit/b417bd6d78ab2468ce43429a6ea9524f63d8285f)) - -- Experiments name should not be re used - ([#79](https://github.com/rungalileo/galileo-python/pull/79), - [`483367b`](https://github.com/rungalileo/galileo-python/commit/483367b213cafcb963fdac54ca7dfe0557ba1aca)) - - -## v0.2.4 (2025-03-28) - -### Bug Fixes - -- Improve error handling and print better error when create prompt template fails - ([#81](https://github.com/rungalileo/galileo-python/pull/81), - [`8839f8b`](https://github.com/rungalileo/galileo-python/commit/8839f8b037e342b037f58916fe99a81273ce4ef1)) - -- Set correct openai status code for successful run - ([#80](https://github.com/rungalileo/galileo-python/pull/80), - [`6d8224f`](https://github.com/rungalileo/galileo-python/commit/6d8224f2478855f273afb6be8e16191bb2e6bad2)) - -- Update explanation for Context Relevance - ([#77](https://github.com/rungalileo/galileo-python/pull/77), - [`43e8230`](https://github.com/rungalileo/galileo-python/commit/43e8230730abdfb557a705ea8867ed402ce36e92)) - -### Chores - -- **deps**: Bump `galileo-core` to v3.19.0 - ([#78](https://github.com/rungalileo/galileo-python/pull/78), - [`8391fb3`](https://github.com/rungalileo/galileo-python/commit/8391fb33e81f6ae764e79b19ce49336b7bac722b)) - -- **release**: V0.2.4 - ([`4e68e33`](https://github.com/rungalileo/galileo-python/commit/4e68e33fc738faf6953909f891c7aa99f48e364a)) - -Automatically generated by python-semantic-release - -### Documentation - -- Add badges (codecov and pypi) to readme - ([#75](https://github.com/rungalileo/galileo-python/pull/75), - [`9f268fe`](https://github.com/rungalileo/galileo-python/commit/9f268fe70b5e4fd7f66732db9ed9636711d7f9d4)) - -- Remove extra brackets ([#76](https://github.com/rungalileo/galileo-python/pull/76), - [`9f0ccc0`](https://github.com/rungalileo/galileo-python/commit/9f0ccc0a89db7c7169e31fa8cb227b1c09095ebf)) - -### Testing - -- Add unit tests for experiments ([#71](https://github.com/rungalileo/galileo-python/pull/71), - [`19870ee`](https://github.com/rungalileo/galileo-python/commit/19870ee044a2a76c0dffa5cfc1082f629b832643)) - - -## v0.2.3 (2025-03-21) - -### Bug Fixes - -- Move from packaging.version import Version to openai section - ([#74](https://github.com/rungalileo/galileo-python/pull/74), - [`085e989`](https://github.com/rungalileo/galileo-python/commit/085e989e2f548ee016a1c6b153b8c5b429364a13)) - -### Chores - -- **release**: V0.2.3 - ([`5aa9b41`](https://github.com/rungalileo/galileo-python/commit/5aa9b41dbf7e0fcd7db492d095702e23780c611a)) - -Automatically generated by python-semantic-release - - -## v0.2.2 (2025-03-18) - -### Bug Fixes - -- Run_experiment() ([#73](https://github.com/rungalileo/galileo-python/pull/73), - [`209ef0f`](https://github.com/rungalileo/galileo-python/commit/209ef0f0c696c3c24a8d2bdbae3297f62984c16e)) - -### Chores - -- **release**: V0.2.2 - ([`2a00be2`](https://github.com/rungalileo/galileo-python/commit/2a00be2635efb6206f861925438c16a7d9569df0)) - -Automatically generated by python-semantic-release - - -## v0.2.1 (2025-03-18) - -### Bug Fixes - -- Fixup after refactoring: PromptTemplates - ([#72](https://github.com/rungalileo/galileo-python/pull/72), - [`cae8e70`](https://github.com/rungalileo/galileo-python/commit/cae8e7097425ef72b6f137e7b5aa0fccbd28dde1)) - -### Chores - -- Add more informative console messages after running an experiment - ([#70](https://github.com/rungalileo/galileo-python/pull/70), - [`fff4d0c`](https://github.com/rungalileo/galileo-python/commit/fff4d0c28af0e0d0ca593485d67a2d89ed21defe)) - -- **release**: V0.2.1 - ([`780b1b9`](https://github.com/rungalileo/galileo-python/commit/780b1b98ac902bdca96ed85fa6fd055500b0d8e5)) - -Automatically generated by python-semantic-release - - -## v0.2.0 (2025-03-18) - -### Bug Fixes - -- Langgraph metadata errors in the Langchain handler - ([#62](https://github.com/rungalileo/galileo-python/pull/62), - [`a8f8133`](https://github.com/rungalileo/galileo-python/commit/a8f8133c196c2c8ac233574fc87fcab520a76f99)) - -Co-authored-by: Andrii Soldatenko - -- Log tools on LLM span in OpenAI decorator - ([#47](https://github.com/rungalileo/galileo-python/pull/47), - [`d2ef9aa`](https://github.com/rungalileo/galileo-python/commit/d2ef9aae94ecf9a68d268eaad7483512fe541026)) - -Co-authored-by: ajaynayak - -- Making sure the final span output bubbles up to the trace during flush - ([#64](https://github.com/rungalileo/galileo-python/pull/64), - [`bb70952`](https://github.com/rungalileo/galileo-python/commit/bb7095265b53b9c3c5fe121c80a83fcd4430b1b3)) - -Co-authored-by: Andrii Soldatenko - -### Chores - -- **release**: V0.2.0 - ([`7d39fcb`](https://github.com/rungalileo/galileo-python/commit/7d39fcb17f7bebc4fde0ff6f9ee1b8a20b642f42)) - -Automatically generated by python-semantic-release - -### Features - -- Add prompt_settings to run_experiment - ([#67](https://github.com/rungalileo/galileo-python/pull/67), - [`02de6ea`](https://github.com/rungalileo/galileo-python/commit/02de6eabec3c83288fcdbcc78c64388a5c2ef528)) - -- Adding a test and updating the sdk version - ([#69](https://github.com/rungalileo/galileo-python/pull/69), - [`180e8b4`](https://github.com/rungalileo/galileo-python/commit/180e8b4c26510caa7e8ec7d23a862b8f3b643fe8)) - -- Implement properly run_experiments with datasets - ([#66](https://github.com/rungalileo/galileo-python/pull/66), - [`17879ef`](https://github.com/rungalileo/galileo-python/commit/17879efd75f7d0207d798476ef58d8a579ea9ff1)) - -- Use @log with ThreadPoolExecutor ([#65](https://github.com/rungalileo/galileo-python/pull/65), - [`f13c746`](https://github.com/rungalileo/galileo-python/commit/f13c746ca253fed21c4de9eaa5e512c7c9427e58)) - -### Refactoring - -- Newly added classes to align with our naming convention - ([#63](https://github.com/rungalileo/galileo-python/pull/63), - [`c540c2a`](https://github.com/rungalileo/galileo-python/commit/c540c2a8e6f3e99ea75314cb443ff9a38d632971)) - -Co-authored-by: Andrii Soldatenko - - -## v0.1.0 (2025-03-14) - -### Bug Fixes - -- Adding init, reset, and flush_all methods to galileo_context, adding tests, fixing existing tests - ([#28](https://github.com/rungalileo/galileo-python/pull/28), - [`569d1d0`](https://github.com/rungalileo/galileo-python/commit/569d1d07686f54b434081f1f79fe5c318e8873f9)) - -- Change job name according to new api version - ([#52](https://github.com/rungalileo/galileo-python/pull/52), - [`9c9352d`](https://github.com/rungalileo/galileo-python/commit/9c9352dc51f8591a33cf411d580707bfc18162d8)) - -- Fixing an issue with parsing OpenAI tool calls outputs - ([#36](https://github.com/rungalileo/galileo-python/pull/36), - [`55c0fe4`](https://github.com/rungalileo/galileo-python/commit/55c0fe44445b6a3b3858978853dd48057bc0452f)) - -- Fixing get log_stream by name ([#40](https://github.com/rungalileo/galileo-python/pull/40), - [`2571024`](https://github.com/rungalileo/galileo-python/commit/257102433d2ab765a11526481d6b2f7c729b6b61)) - -- Output parsing for retriever spans ([#34](https://github.com/rungalileo/galileo-python/pull/34), - [`c87f926`](https://github.com/rungalileo/galileo-python/commit/c87f9268b8ef681ed8535ec0aad5e5b612a71243)) - -- Serialization of non-serialized types or classes - ([#48](https://github.com/rungalileo/galileo-python/pull/48), - [`39d7611`](https://github.com/rungalileo/galileo-python/commit/39d761164b4659f4313cbe05bcf0246371520bbe)) - -- Serializing trace, workflow, and tool span inputs and outputs - ([#41](https://github.com/rungalileo/galileo-python/pull/41), - [`e54cb95`](https://github.com/rungalileo/galileo-python/commit/e54cb9585ffbf6cdbb9e553f76c1fc836b37e36b)) - -- Set min Python version for ruff to py39 - ([#23](https://github.com/rungalileo/galileo-python/pull/23), - [`fcfc0d1`](https://github.com/rungalileo/galileo-python/commit/fcfc0d11937bcde62f38d0e21c2319afa9af3d8f)) - -- Typo inside ./scripts/auto-generate-api-client.sh - ([#32](https://github.com/rungalileo/galileo-python/pull/32), - [`252c061`](https://github.com/rungalileo/galileo-python/commit/252c061d7798cef822b24dc6819f49e5864a7bf9)) - -- Update add_llm_span example ([#39](https://github.com/rungalileo/galileo-python/pull/39), - [`e01838f`](https://github.com/rungalileo/galileo-python/commit/e01838ffe95181133c455f21bc57cc5f1aacd297)) - -### Chores - -- Add missing keys to `pyproject.toml` ([#57](https://github.com/rungalileo/galileo-python/pull/57), - [`500ac21`](https://github.com/rungalileo/galileo-python/commit/500ac21f6201c738a59546b8d602ca947575992f)) - -- Fix path to `__init__` ([#61](https://github.com/rungalileo/galileo-python/pull/61), - [`8d6425d`](https://github.com/rungalileo/galileo-python/commit/8d6425df53510f67f543192e014f694a894651f3)) - -- Remove js dir and files ([#56](https://github.com/rungalileo/galileo-python/pull/56), - [`962ae5d`](https://github.com/rungalileo/galileo-python/commit/962ae5da831da3d10f5748aa1333ef6e6fe1b6fe)) - -- Set version field in `pyproject.toml` correctly - ([#60](https://github.com/rungalileo/galileo-python/pull/60), - [`1560b76`](https://github.com/rungalileo/galileo-python/commit/1560b7615df8bf0320ab531bcd3f87c5fa41495d)) - -- Setup repo + package similarly to our other Python repos - ([#25](https://github.com/rungalileo/galileo-python/pull/25), - [`4daac02`](https://github.com/rungalileo/galileo-python/commit/4daac020776875d2d1e174b300f558f68448671e)) - -- **deps**: Bump `galileo-core` to v3.2+ - ([#29](https://github.com/rungalileo/galileo-python/pull/29), - [`ed234e3`](https://github.com/rungalileo/galileo-python/commit/ed234e3495281409095db3243cff8b01143cd41d)) - -- **deps**: Bump codecov/codecov-action from 5.3.1 to 5.4.0 - ([#42](https://github.com/rungalileo/galileo-python/pull/42), - [`41d7955`](https://github.com/rungalileo/galileo-python/commit/41d79556106db802b2013b40bbf7ac2a65b5899f)) - -- **deps**: Bump python-semantic-release/python-semantic-release from 9.20.0 to 9.21.0 - ([#43](https://github.com/rungalileo/galileo-python/pull/43), - [`95543ce`](https://github.com/rungalileo/galileo-python/commit/95543cec5afb5df96b75439ab7888065f805c4e8)) - -- **release**: V0.1.0 - ([`e443300`](https://github.com/rungalileo/galileo-python/commit/e4433000f2be7a6c2a707a6d369f2f55d5028bb1)) - -Automatically generated by python-semantic-release - -### Continuous Integration - -- Bump python-semantic-release/python-semantic-release from 9.17.0 to 9.20.0 - ([#26](https://github.com/rungalileo/galileo-python/pull/26), - [`ac9aaab`](https://github.com/rungalileo/galileo-python/commit/ac9aaab3539e1ff3e1603ea8371064e6913ddf36)) - -### Documentation - -- Add reference docs to the more client-facing pages - ([#37](https://github.com/rungalileo/galileo-python/pull/37), - [`08b3457`](https://github.com/rungalileo/galileo-python/commit/08b3457d9957523d77e480b4c2dce9e63bb0f5f6)) - -Co-authored-by: ajaynayak - -- Add small note about poetry shell ([#12](https://github.com/rungalileo/galileo-python/pull/12), - [`d598aae`](https://github.com/rungalileo/galileo-python/commit/d598aae0fb61c700e9ea56fdeb72154aeb343aea)) - -### Features - -- Add streaming support to openai wrapper - ([#31](https://github.com/rungalileo/galileo-python/pull/31), - [`78687c7`](https://github.com/rungalileo/galileo-python/commit/78687c72cbc4ac10df5630209ffe9b09a4dd56f2)) - -- Adding a client-type header to all requests - ([#54](https://github.com/rungalileo/galileo-python/pull/54), - [`764ee2d`](https://github.com/rungalileo/galileo-python/commit/764ee2d00cb3493304c2d1af51ce1f1b97a5d6e1)) - -- Adding a way to conclude all spans in a trace; restoring defaults in the decorator - ([#35](https://github.com/rungalileo/galileo-python/pull/35), - [`130fe02`](https://github.com/rungalileo/galileo-python/commit/130fe0218f2df201b86366959b165034a44c74e9)) - -- Catch and handle errors throughout the client - ([#46](https://github.com/rungalileo/galileo-python/pull/46), - [`d6a82ee`](https://github.com/rungalileo/galileo-python/commit/d6a82eed09b1923d8d25c792e804857419b62a98)) - -- Changes to support the new core logging schemas - ([#30](https://github.com/rungalileo/galileo-python/pull/30), - [`a2c6ca8`](https://github.com/rungalileo/galileo-python/commit/a2c6ca85efb7ad1d5e1e581219c88c91484fd3b6)) - -Changes to the client based on the following core and api changes: rungalileo/core#232 - rungalileo/api#3489 - -There were some DX changes I made in this PR that will need to get moved to core: - renaming - user_metadata to metadata for the logging functions - allowing more flexible types to be used for - documents in the add_retriever_span() method. The current traces_logger method is too restrictive - (forces a user to specify a list of dicts or a list of Documents, else throws an error). Since - we're using function decorators, we need to be more permissive of method outputs which will map to - the retriever documents field. - -- Decorator should create trace but reraise original exception - ([#45](https://github.com/rungalileo/galileo-python/pull/45), - [`d3dbb7a`](https://github.com/rungalileo/galileo-python/commit/d3dbb7a0cb2a6e37c0684ec18cd55ee1113e173f)) - -- Implement get/create for prompt templates - ([#49](https://github.com/rungalileo/galileo-python/pull/49), - [`c647da3`](https://github.com/rungalileo/galileo-python/commit/c647da3bd1ac44033b493d7ca7f8ef70877a7e68)) - -- Langchain callback ([#44](https://github.com/rungalileo/galileo-python/pull/44), - [`b6dd9a3`](https://github.com/rungalileo/galileo-python/commit/b6dd9a3514b3dba7e2a48beff42342b352b2e60e)) - -- Replace app.galileo.ai with api.galileo.ai if users specify it incorrectly - ([#53](https://github.com/rungalileo/galileo-python/pull/53), - [`9a7ac9e`](https://github.com/rungalileo/galileo-python/commit/9a7ac9e623808341c9bf72ffc58543943bf6d27f)) - -- Run experiment with a runner function and hosted metrics - ([#58](https://github.com/rungalileo/galileo-python/pull/58), - [`5c37ff7`](https://github.com/rungalileo/galileo-python/commit/5c37ff7516d020a8195c23fe30907e9839f2d9fe)) - -Co-authored-by: Andrii Soldatenko - -Co-authored-by: ajaynayak - -- Run experiment with run_prompt and hosted metrics - ([#50](https://github.com/rungalileo/galileo-python/pull/50), - [`c78908f`](https://github.com/rungalileo/galileo-python/commit/c78908f699006311a1249c0341ed9f34f589dcde)) - -- Updating the readme and pyproject for release - ([#59](https://github.com/rungalileo/galileo-python/pull/59), - [`867a327`](https://github.com/rungalileo/galileo-python/commit/867a327e92395451feb3e2a76b301a10a3cb8f99)) - -### Refactoring - -- Removing `project` and `log_stream` from the `log` decorator - ([#55](https://github.com/rungalileo/galileo-python/pull/55), - [`d348dbd`](https://github.com/rungalileo/galileo-python/commit/d348dbdc1dd23595c8bfbaee736112681e11ea21)) - -### Testing - -- Add tests to emulate openai errors and galileo api errors - ([#38](https://github.com/rungalileo/galileo-python/pull/38), - [`16f73bb`](https://github.com/rungalileo/galileo-python/commit/16f73bb6f63a755a8d4ee3a907b4d938a22519f9)) - -- Adding unit tests for openai wrapper ([#27](https://github.com/rungalileo/galileo-python/pull/27), - [`fdc15d1`](https://github.com/rungalileo/galileo-python/commit/fdc15d144b1e10199d64d3d5bc706f7da80e2a94)) - -- Dont ignore async test ([#33](https://github.com/rungalileo/galileo-python/pull/33), - [`6f75fa8`](https://github.com/rungalileo/galileo-python/commit/6f75fa8e134cc1e8ec82e2856680e354386b5414)) +### Added + +- Added deployment-aware configuration and authentication for Splunk + Observability Cloud and standalone Agent Observability deployments. +- Added O11y Cloud configuration through `SPLUNK_AO_REALM`, + `SPLUNK_AO_O11Y_TOKEN`, and the optional `SPLUNK_AO_O11Y_API_TOKEN`. +- Added native OTLP trace export for SDK handlers and standard OpenTelemetry or + OpenInference instrumentations. +- Added Project and Agent Stream routing by name or ID in both OTLP request + headers and OpenTelemetry Resource attributes. +- Added standard OpenTelemetry Resource detection, including a valid + `service.name` fallback and support for `OTEL_SERVICE_NAME`. +- Added structured input and output capture for workflow, agent, LLM, tool, and + retriever spans, including supported multimodal content. + +### Changed + +- Completed spans are queued immediately in an OpenTelemetry + `BatchSpanProcessor` and exported on its configured schedule. +- `flush()` and `async_flush()` now drain completed spans without concluding an + active operation. +- Independent top-level decorated calls own separate traces, while nested calls + remain children of their outer operation. +- Exceptions from decorated functions, synchronous generators, and asynchronous + generators are re-raised after telemetry finalization. +- Reserved Agent Observability routing keys in `OTEL_RESOURCE_ATTRIBUTES` are + removed so SDK routing configuration remains consistent with request headers. + +### Diagnostics + +- Standard OpenTelemetry exporter logging reports transport, + authentication, retry, and non-2xx HTTP failures. +- Successful HTTP responses that reject telemetry produce sanitized, + rate-limited SDK error logs and a recovery log after a later accepted + response. +- Exporters expose bounded, read-only receiver-acknowledgement health. Health is + unknown before an acknowledgement and after ordinary transport or non-2xx + failures; it is not a delivery guarantee. diff --git a/README.md b/README.md index 6f4f491..f1661be 100644 --- a/README.md +++ b/README.md @@ -6,21 +6,32 @@ [![PyPI][pypi-badge]][pypi-url] [![Python Version][python-badge]][python-url] -![codecov.io][codecov-url] -[pypi-badge]: https://img.shields.io/pypi/v/galileo.svg -[pypi-url]: https://pypi.org/project/galileo/ -[python-badge]: https://img.shields.io/pypi/pyversions/galileo.svg +[pypi-badge]: https://img.shields.io/pypi/v/splunk-ao.svg +[pypi-url]: https://pypi.org/project/splunk-ao/ +[python-badge]: https://img.shields.io/pypi/pyversions/splunk-ao.svg [python-url]: https://www.python.org/downloads/ -[codecov-url]: https://codecov.io/github/rungalileo/galileo-python/coverage.svg?branch=main ## Getting Started ### Installation -`pip install splunk-ao` +Install the SDK: + +```shell +pip install splunk-ao +``` + +Install the optional integration dependencies used by your application: + +```shell +pip install "splunk-ao[openai]" +pip install "splunk-ao[langchain]" langchain-openai +``` + +Other available extras include `crewai`, `middleware`, and `all`. ### Setup @@ -30,11 +41,17 @@ from the variables that are present and rejects ambiguous configurations. #### O11y Cloud user -Set your Splunk Observability Cloud realm and access token: +| Environment variable | Description | +| --- | --- | +| `SPLUNK_AO_REALM` | Splunk Observability Cloud realm (required) | +| `SPLUNK_AO_O11Y_TOKEN` | O11y ingest token used for OTLP export (required for telemetry) | +| `SPLUNK_AO_O11Y_API_TOKEN` | Dedicated O11y API token used for CRUD operations (optional) | + +For telemetry export, set your realm and O11y ingest token: ```shell export SPLUNK_AO_REALM="us1" -export SPLUNK_AO_O11Y_TOKEN="your-splunk-ingest-token" +export SPLUNK_AO_O11Y_TOKEN="your-o11y-ingest-token" ``` `SPLUNK_AO_O11Y_TOKEN` is required to export telemetry. It is also used for CRUD @@ -44,14 +61,14 @@ token is configured. You may configure a separate token for CRUD operations: ```shell -export SPLUNK_AO_O11Y_API_TOKEN="your-splunk-api-token" +export SPLUNK_AO_O11Y_API_TOKEN="your-o11y-api-token" ``` When both tokens are set, `SPLUNK_AO_O11Y_API_TOKEN` is preferred for CRUD and -`SPLUNK_AO_O11Y_TOKEN` is used for telemetry ingestion. For CRUD only use, you -may set `SPLUNK_AO_REALM` and `SPLUNK_AO_O11Y_API_TOKEN` without setting -`SPLUNK_AO_O11Y_TOKEN`. Note that attempting to export telemetry without -`SPLUNK_AO_O11Y_TOKEN` raises a configuration error. +`SPLUNK_AO_O11Y_TOKEN` is used for telemetry ingestion. For CRUD-only use, set +`SPLUNK_AO_REALM` and `SPLUNK_AO_O11Y_API_TOKEN`; an ingest token is not +required until the application exports telemetry. Attempting to construct an +OTLP exporter without `SPLUNK_AO_O11Y_TOKEN` raises a configuration error. The SDK derives the console, API and OTLP ingest endpoints from the realm. Do not set `SPLUNK_AO_CONSOLE_URL` or `SPLUNK_AO_API_URL` for O11y @@ -59,6 +76,12 @@ Cloud. #### Standalone Agent Observability user +| Environment variable | Description | +| --- | --- | +| `SPLUNK_AO_API_KEY` | Agent Observability API key (required) | +| `SPLUNK_AO_CONSOLE_URL` | Agent Observability console URL (required) | +| `SPLUNK_AO_API_URL` | Explicit API URL when it cannot be derived from the console URL (optional) | + Set your Agent Observability API key and console URL: ```shell @@ -74,18 +97,46 @@ cannot be derived from the console URL: export SPLUNK_AO_API_URL="https://api.galileo.ai" ``` -For either deployment, project and agent-stream routing can be supplied through -the SDK APIs or environment variables: +#### Routing + +Routing configuration is shared by both deployments: + +| Environment variable | Description | +| --- | --- | +| `SPLUNK_AO_PROJECT` | Project name | +| `SPLUNK_AO_AGENT_STREAM` | Agent Stream name | + +Configure Project and Agent Stream routing by name: ```shell export SPLUNK_AO_PROJECT="your-project-name" export SPLUNK_AO_AGENT_STREAM="your-agent-stream-name" ``` -When using `SplunkAOSpanProcessor`, routing is captured when its exporter is -constructed and remains fixed for that exporter's lifetime. This matches the -OpenTelemetry Resource model and keeps request headers consistent for batched -spans. Use a separate processor and exporter for each additional destination. +Explicit SDK arguments take precedence over the active `splunk_ao_context`, +which takes precedence over environment variables. The deprecated +`SPLUNK_AO_LOG_STREAM` variable remains a compatibility alias; use +`SPLUNK_AO_AGENT_STREAM` in new applications. + +Routing is captured when an OTLP exporter is constructed and remains fixed for +that exporter's lifetime. Use a separate processor and exporter for each +additional destination. + +Routing is SDK configuration, not a general OpenTelemetry Resource setting. Do +not set Agent Observability routing in `OTEL_RESOURCE_ATTRIBUTES`. Use the +supported SDK arguments, context, or `SPLUNK_AO_*` environment variables. + +Set a meaningful OpenTelemetry service name when you want traces to appear +under a recognizable service in APM: + +```shell +export OTEL_SERVICE_NAME="checkout-agent" +export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,service.version=1.2.0" +``` + +`OTEL_SERVICE_NAME` is optional. If it is absent, the standard OpenTelemetry +unknown-service fallback is used. Other non-routing +`OTEL_RESOURCE_ATTRIBUTES` are preserved. Set `SPLUNK_AO_LOGGING_DISABLED=true` to disable telemetry collection and export. @@ -100,128 +151,129 @@ import os from splunk_ao import splunk_ao_context from splunk_ao.openai import openai -# If you've set your SPLUNK_AO_PROJECT and SPLUNK_AO_AGENT_STREAM env vars, you can skip this step -splunk_ao_context.init(project="your-project-name", agent_stream="your-log-stream-name") +# Skip this when SPLUNK_AO_PROJECT and SPLUNK_AO_AGENT_STREAM are set. +splunk_ao_context.init(project="your-project-name", agent_stream="your-agent-stream-name") -# Initialize the Agent Observability wrapped OpenAI client client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY")) -def call_openai(): +def call_openai() -> str | None: chat_completion = client.chat.completions.create( messages=[{"role": "user", "content": "Say this is a test"}], model="gpt-4o" ) - return chat_completion.choices[0].message.content +print(call_openai()) -# This will create a single span trace with the OpenAI call -call_openai() - -# This will upload the trace to Agent Observability +# This finite example waits for its completed span to be delivered. splunk_ao_context.flush() ``` -You can also use the `@log` decorator to log spans. Here's how to create a workflow span with two nested LLM spans: +Completed spans enter the SDK's OpenTelemetry batch processor immediately and +are exported on its schedule. Long-running applications do not need to flush +after each operation. The explicit flush above is only to make the finite +example wait for delivery before it exits. + +Use the `@log` decorator to capture application operations. Every independent +top-level decorated call owns and concludes its trace. Nested decorated calls +become children of the outer operation. Captured function arguments and return +values are preserved as the operation input and output, including supported +structured and multimodal content: ```python -from splunk_ao import log +from splunk_ao import log, splunk_ao_context @log -def make_nested_call(): +def make_nested_call() -> None: call_openai() call_openai() -# If you've set your SPLUNK_AO_PROJECT and SPLUNK_AO_AGENT_STREAM env vars, you can skip this step -splunk_ao_context.init(project="your-project-name", agent_stream="your-log-stream-name") +splunk_ao_context.init( + project="your-project-name", + agent_stream="your-agent-stream-name", +) -# This will create a trace with a workflow span and two nested LLM spans containing the OpenAI calls make_nested_call() ``` -Here's how to create a retriever span using the decorator: +If a decorated function, synchronous generator, or asynchronous generator +raises, the SDK finalizes its telemetry and re-raises the original exception. + +Create a retriever span: ```python from splunk_ao import log @log(span_type="retriever") -def retrieve_documents(query: str): +def retrieve_documents(query: str) -> list[str]: return ["doc1", "doc2"] -# This will create a trace with a retriever span containing the documents in the output retrieve_documents(query="history") ``` -Here's how to create a tool span using the decorator: +Create a tool span: ```python from splunk_ao import log @log(span_type="tool") -def tool_call(input: str = "tool call input"): - return "tool call output" - -# This will create a trace with a tool span containing the tool call output -tool_call(input="question") - -# This will upload the trace to Agent Observability -splunk_ao_context.flush() -``` +def lookup_weather(city: str) -> str: + return f"The weather in {city} is sunny." -In some cases, you may want to wrap a block of code to start and flush a trace automatically. You can do this using the `splunk_ao_context` context manager: - -```python -from splunk_ao import splunk_ao_context - -# This will log a block of code to the project and log stream specified in the context manager -with splunk_ao_context(): - content = make_nested_call() - print(content) +lookup_weather("Seattle") ``` -`splunk_ao_context` also allows you specify a separate project and log stream for the trace: +Use `splunk_ao_context` to scope routing or session state for a block. It does +not create an extra operation around the block: ```python from splunk_ao import splunk_ao_context -# This will log to the project and log stream specified in the context manager -with splunk_ao_context(project="gen-ai-project", agent_stream="test2"): - content = make_nested_call() - print(content) +with splunk_ao_context(project="gen-ai-project", agent_stream="production"): + make_nested_call() ``` -You can also use the `SplunkAOLogger` for manual logging scenarios: +For manual logging, conclude every trace or span that you start. In finite +scripts and jobs, call `terminate()` during teardown so the final queued batch +is delivered: ```python from splunk_ao.logger import SplunkAOLogger -# This will log to the project and log stream specified in the logger constructor -logger = SplunkAOLogger(project="gen-ai-project", agent_stream="test3") -trace = logger.start_trace("Say this is a test") - -logger.add_llm_span( - input="Say this is a test", - output="Hello, this is a test", - model="gpt-4o", - num_input_tokens=10, - num_output_tokens=3, - total_tokens=13, - duration_ns=1000, -) - -logger.conclude(output="Hello, this is a test", duration_ns=1000) -logger.flush() # This will upload the trace to Agent Observability +logger = SplunkAOLogger(project="gen-ai-project", agent_stream="production") +try: + logger.start_trace("Say this is a test") + logger.add_llm_span( + input="Say this is a test", + output="Hello, this is a test", + model="gpt-4o", + num_input_tokens=10, + num_output_tokens=3, + total_tokens=13, + duration_ns=1000, + ) + logger.conclude(output="Hello, this is a test", duration_ns=1000) +finally: + logger.terminate() ``` +`flush()` and `async_flush()` only drain completed spans already waiting in the +batch processor. They do not conclude an active operation and are not required +for normal scheduled export. `terminate()` drains completed spans and shuts +down resources owned by `SplunkAOLogger`; it does not turn unfinished manual +spans into completed spans. + #### Using Agent Observability context with Agent Control If you use Agent Control hosted by Splunk, initialize Agent Control with the -current Agent Observability agent stream as the runtime target: +current Agent Observability Agent Stream as the runtime target: ```python import agent_control + from splunk_ao import splunk_ao_context, get_agent_control_target -splunk_ao_context.init(project="my-project", agent_stream="prod") +# Configure SPLUNK_AO_PROJECT and SPLUNK_AO_AGENT_STREAM_ID before startup. +splunk_ao_context.init() target = get_agent_control_target() @@ -234,10 +286,12 @@ agent_control.init( ) ``` -The helper resolves an explicit agent stream ID, `SPLUNK_AO_AGENT_STREAM_ID` (or the -deprecated `SPLUNK_AO_LOG_STREAM_ID` alias), or an already-initialized -`splunk_ao_context` logger. It does not import the Agent Control SDK or resolve -agent stream names over the network. If you use a direct +Agent Control requires the Agent Stream target ID; the project can still be +configured by name with `SPLUNK_AO_PROJECT`. The helper resolves an explicit +Agent Stream ID, `SPLUNK_AO_AGENT_STREAM_ID` (or the deprecated +`SPLUNK_AO_LOG_STREAM_ID` alias), or an already-initialized +`splunk_ao_context` logger with resolved IDs. It does not import the Agent +Control SDK or resolve Agent Stream names over the network. If you use a direct Agent Control client instead of `agent_control.init(...)`, pass `target.target_type` and `target.target_id` on each evaluation call. @@ -263,55 +317,99 @@ for chunk in stream: print(chunk.choices[0].delta.content or "", end="") ``` -In some cases (like long-running processes), it may be necessary to explicitly flush the trace to upload it to Agent Observability: +#### Using OpenTelemetry instrumentations + +You can connect standard OpenTelemetry or OpenInference instrumentations to +Agent Observability without constructing an exporter yourself: ```python -import os +from opentelemetry import trace +from opentelemetry.sdk import trace as trace_sdk +from splunk_ao import otel -from splunk_ao import splunk_ao_context -from splunk_ao.openai import openai +tracer_provider = trace_sdk.TracerProvider() +processor = otel.add_splunk_ao_span_processor(tracer_provider) +trace.set_tracer_provider(tracer_provider) -splunk_ao_context.init(project="your-project-name", agent_stream="your-log-stream-name") +# Configure each instrumentation library with this provider, or let it use the +# global provider set above. -# Initialize the Agent Observability wrapped OpenAI client -client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY")) +# At application shutdown: +tracer_provider.shutdown() +``` -def call_openai(): - chat_completion = client.chat.completions.create( - messages=[{"role": "user", "content": "Say this is a test"}], model="gpt-4o" - ) +`add_splunk_ao_span_processor()` reads the same deployment and routing +configuration described in [Setup](#setup). It registers a +`SplunkAOSpanProcessor`, which uses a standard OpenTelemetry +`BatchSpanProcessor`. - return chat_completion.choices[0].message.content +If your application needs another processor or exporter, register it through +the standard OpenTelemetry extension points. For example, to send the same +spans to an additional custom exporter: +```python +from opentelemetry.sdk.trace.export import BatchSpanProcessor -# This will create a single span trace with the OpenAI call -call_openai() +# custom_exporter is your SpanExporter implementation. +tracer_provider.add_span_processor(BatchSpanProcessor(custom_exporter)) +``` -# This will upload the trace to Agent Observability -splunk_ao_context.flush() +The application owns a `TracerProvider` that it constructs, so it must call +`tracer_provider.shutdown()` during teardown. A `SplunkAOLogger` created +elsewhere owns a separate provider and should be terminated separately. + +#### Export diagnostics + +The SDK does not change your application's global logging configuration. +Configure Python logging to surface standard OpenTelemetry transport, +authentication, retry, and HTTP errors, as well as Agent Observability +successful-response rejection diagnostics: + +```python +import logging + +logging.basicConfig(level=logging.INFO) +logging.getLogger("opentelemetry.exporter.otlp.proto.http.trace_exporter").setLevel(logging.WARNING) +logging.getLogger("splunk_ao.exporter").setLevel(logging.INFO) ``` -Using the Langchain callback handler: +A successful HTTP response can still acknowledge that some or all spans were +rejected. The SDK emits a sanitized, rate-limited error for those +acknowledgements and one informational message after recovery. Export failures +never fail the instrumented application or business request. + +The processor returned by `add_splunk_ao_span_processor()` exposes the latest +receiver acknowledgement as `processor.export_health`. Its `healthy` value is: + +- `None` before an acknowledgement, or after a transport or non-2xx failure. +- `True` after an accepted 2xx response. +- `False` after a 2xx response that rejects telemetry. + +This is advisory diagnostic state, not a delivery guarantee. + +#### Framework handlers + +Using the LangChain callback handler: ```python -from splunk_ao.handlers.langchain import SplunkAOCallback -from langchain.schema import HumanMessage +from langchain_core.messages import HumanMessage from langchain_openai import ChatOpenAI +from splunk_ao.handlers.langchain import SplunkAOCallback -# You can optionally pass a SplunkAOLogger instance to the callback if you don't want to use the default context callback = SplunkAOCallback() - llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0.7, callbacks=[callback]) -# Create a message with the user's query messages = [HumanMessage(content="What is LangChain and how is it used with OpenAI?")] - -# Make the API call response = llm.invoke(messages) print(response.content) ``` +Handlers conclude traces that they own when their framework run ends. They do +not close a trace that was started by the caller. In short-lived jobs, ensure +the owning logger or provider is terminated or shut down after framework work +finishes. + #### Datasets Create a dataset: @@ -359,7 +457,7 @@ datasets = list_datasets() > ) > ``` > -> - **`output` / `ground_truth`**: The existing `output` field is now displayed as "Ground Truth" in the Agent Observability UI for better clarity. The SDK supports both `output` and `ground_truth` field names when creating records - both are normalized to `output` internally, ensuring full backward compatibility. You can use either field name, and access the value via the `ground_truth` property. +> - **`output` / `ground_truth`**: The existing `output` field is displayed as "Ground Truth" in the Agent Observability UI. The SDK accepts both `output` and `ground_truth` when creating records. Both are stored as `output` internally, and the value is also accessible through the `ground_truth` property. > > Example: > ```python @@ -369,7 +467,7 @@ datasets = list_datasets() > record1 = DatasetRecord(input="What is 2+2?", output="4") > assert record1.ground_truth == "4" # Property accessor > -> # Using 'ground_truth' (new recommended way) +> # Using the equivalent 'ground_truth' input name > record2 = DatasetRecord(input="What is 2+2?", ground_truth="4") > assert record2.output == "4" # Normalized internally > assert record2.ground_truth == "4" # Property accessor @@ -383,12 +481,12 @@ Run an experiment with a prompt template: from splunk_ao import Message, MessageRole from splunk_ao.datasets import get_dataset from splunk_ao.experiments import run_experiment -from splunk_ao.prompts import create_prompt_template +from splunk_ao.prompts import create_prompt -prompt = create_prompt_template( +prompt = create_prompt( name="my-prompt", - project="new-project", - messages=[ + project_name="new-project", + template=[ Message(role=MessageRole.system, content="you are a helpful assistant"), Message(role=MessageRole.user, content="why is sky blue?") ] @@ -396,10 +494,10 @@ prompt = create_prompt_template( results = run_experiment( "my-experiment", - dataset=get_dataset(name="storyteller-dataset"), - prompt=prompt, + dataset=get_dataset(name="storyteller-dataset", project_name="new-project"), + prompt_template=prompt, metrics=["correctness"], - project="andrii-new-project", + project="new-project", ) ``` @@ -434,45 +532,65 @@ run_experiment( ### Sessions -Sessions allow you to group related traces together. By default, a session is created for each trace and a session name is auto-generated. If you would like to override this, you can explicitly start a session: +Sessions group related traces under the same conversation. To explicitly create +a named session: ```python from splunk_ao import SplunkAOLogger -logger = SplunkAOLogger(project="gen-ai-project", agent_stream="my-log-stream") -session_id =logger.start_session(name="my-session-name") - -... - -logger.conclude() -logger.flush() +logger = SplunkAOLogger(project="gen-ai-project", agent_stream="my-agent-stream") +try: + session_id = logger.start_session(name="my-session-name") + # Log and conclude operations here. +finally: + logger.terminate() ``` -You can continue a previous session by using the same session ID that was previously generated: +`start_session()` uses session CRUD in both deployment modes and requires +complete Project plus Agent Stream or experiment routing. The O11y Cloud +telemetry logger requires `SPLUNK_AO_O11Y_TOKEN`; session CRUD uses +`SPLUNK_AO_O11Y_API_TOKEN` when configured, otherwise the ingest token must +also include API permissions. Standalone uses its configured API credentials. + +You can continue a previous session by using the same session ID that was +previously generated: ```python from splunk_ao import SplunkAOLogger -logger = SplunkAOLogger(project="gen-ai-project", agent_stream="my-log-stream") -logger.set_session(session_id="123e4567-e89b-12d3-a456-426614174000") - -... - -logger.conclude() -logger.flush() +logger = SplunkAOLogger(project="gen-ai-project", agent_stream="my-agent-stream") +try: + logger.set_session(session_id="123e4567-e89b-12d3-a456-426614174000") + # Log and conclude operations here. +finally: + logger.terminate() ``` -All of this can also be done using the `splunk_ao_context` context manager: +`set_session()` selects an existing session ID locally; it does not create or +validate a backend session. + +You can create a session through the shared `splunk_ao_context` helper, which +uses the active context or environment routing: ```python from splunk_ao import splunk_ao_context session_id = splunk_ao_context.start_session(name="my-session-name") +``` -# OR +To scope SDK logging and caller-wired OpenTelemetry spans to an existing +session, provide the session ID on the context: -splunk_ao_context.set_session(session_id=session_id) +```python +from splunk_ao import splunk_ao_context +with splunk_ao_context( + project="gen-ai-project", + agent_stream="my-agent-stream", + session_id="123e4567-e89b-12d3-a456-426614174000", +): + # Run related operations here. + ... ``` ## Contributing diff --git a/splunk-ao-a2a/CHANGELOG.md b/splunk-ao-a2a/CHANGELOG.md index 6e7815c..fb98ccb 100644 --- a/splunk-ao-a2a/CHANGELOG.md +++ b/splunk-ao-a2a/CHANGELOG.md @@ -1,7 +1,14 @@ -# CHANGELOG +# Changelog - +## Unreleased -## v1.0.0 (2026-04-23) +### Added -- Initial Release +- Added automatic OpenTelemetry instrumentation for A2A client and server + operations. +- Added W3C trace-context propagation through A2A message metadata so remote + agents participate in one distributed trace. +- Added A2A task and context correlation, session grouping, and optional + message-content capture. +- Added deployment-aware OTLP export for Splunk Observability Cloud and + standalone Agent Observability through the core `splunk-ao` SDK. diff --git a/splunk-ao-a2a/README.md b/splunk-ao-a2a/README.md index 87cbc45..ea83cb0 100644 --- a/splunk-ao-a2a/README.md +++ b/splunk-ao-a2a/README.md @@ -75,21 +75,20 @@ For standalone Splunk AO: | `SPLUNK_AO_API_KEY` | Splunk AO API key (required) | | `SPLUNK_AO_CONSOLE_URL` | Splunk AO console URL (required for self-hosted deployments, e.g. `http://localhost:8088`) | | `SPLUNK_AO_API_URL` | Explicit API URL (optional; otherwise derived from the console URL) | -| `SPLUNK_AO_PROJECT` / `SPLUNK_AO_PROJECT_ID` | Project name or ID | -| `SPLUNK_AO_AGENT_STREAM` / `SPLUNK_AO_AGENT_STREAM_ID` | Agent-stream name or ID | +| `SPLUNK_AO_PROJECT` | Project name | +| `SPLUNK_AO_AGENT_STREAM` | Agent Stream name | For Splunk Observability Cloud: | Environment Variable | Description | |---------------------|-------------| | `SPLUNK_AO_REALM` | Observability Cloud realm (required) | -| `SPLUNK_AO_O11Y_TOKEN` | Splunk O11y ingest token used for OTLP export (required) | -| `SPLUNK_AO_PROJECT` / `SPLUNK_AO_PROJECT_ID` | Optional project routing | -| `SPLUNK_AO_AGENT_STREAM` / `SPLUNK_AO_AGENT_STREAM_ID` | Optional agent-stream routing | +| `SPLUNK_AO_O11Y_TOKEN` | O11y ingest token used for OTLP export (required) | +| `SPLUNK_AO_O11Y_API_TOKEN` | Dedicated O11y API token for core SDK CRUD operations (optional) | +| `SPLUNK_AO_PROJECT` | Project name | +| `SPLUNK_AO_AGENT_STREAM` | Agent Stream name | -Applications should configure project and agent-stream routing. If it is -accidentally absent, export remains non-blocking and ingestion may assign the -trace to the unknown-project bucket. +Applications should configure Project and Agent Stream routing by name. The same Python setup works for both deployments; only the environment changes. ## Features diff --git a/splunk-ao-adk/CHANGELOG.md b/splunk-ao-adk/CHANGELOG.md index 922bc9e..b3f08bf 100644 --- a/splunk-ao-adk/CHANGELOG.md +++ b/splunk-ao-adk/CHANGELOG.md @@ -1,403 +1,17 @@ -# CHANGELOG - - - -## v2.0.1 (2026-05-14) - -### Bug Fixes - -- Bump galileo dep rev constraint in galileo-adk - ([#583](https://github.com/rungalileo/galileo-python/pull/583), - [`8b45a91`](https://github.com/rungalileo/galileo-python/commit/8b45a91676a647f0acdd993ba52c2b5e97ff881e)) - - -## v2.0.0 (2026-05-14) - -### Bug Fixes - -- Add default console url value ([#490](https://github.com/rungalileo/galileo-python/pull/490), - [`7360299`](https://github.com/rungalileo/galileo-python/commit/7360299a4d7fc6078b17910f5e1d194da4cbe53b)) - -- Add session start handling in new traces client - ([#543](https://github.com/rungalileo/galileo-python/pull/543), - [`42b8201`](https://github.com/rungalileo/galileo-python/commit/42b82017c850c575f0daba9d3d09c4876ad4f7a9)) - -- Add tqdm as explicit required dependency - ([#492](https://github.com/rungalileo/galileo-python/pull/492), - [`ac9d5d5`](https://github.com/rungalileo/galileo-python/commit/ac9d5d5710e057e71b10a304dd6a4213ab642563)) - -- Adjust LogStream.list() to throw NotFoundError when project is unknown - ([#577](https://github.com/rungalileo/galileo-python/pull/577), - [`52ab01b`](https://github.com/rungalileo/galileo-python/commit/52ab01ba3457c23f33807bd263d4269a7c04a4b4)) - -- Attach dataset ground truth fields to OTEL spans - ([#496](https://github.com/rungalileo/galileo-python/pull/496), - [`aa3994e`](https://github.com/rungalileo/galileo-python/commit/aa3994eb3209f810ad63660c974ff6438e226ca2)) - -- Auto-convert non-string metadata values to strings - ([#488](https://github.com/rungalileo/galileo-python/pull/488), - [`9b60fa3`](https://github.com/rungalileo/galileo-python/commit/9b60fa3f63d769d245c5d3db62f3509bc32893cf)) - -- Cannot pass prompt_settings as dict to run_experiment - ([#527](https://github.com/rungalileo/galileo-python/pull/527), - [`a32ea66`](https://github.com/rungalileo/galileo-python/commit/a32ea66e3d874c98ff0476b66ca2ed32f0aebf33)) - -- Clarify error message when GALILEO_API_KEY is not detected - ([#509](https://github.com/rungalileo/galileo-python/pull/509), - [`c2bf734`](https://github.com/rungalileo/galileo-python/commit/c2bf73424102cc5d1b52e228bf7f8b506bdf7373)) - -- Fix unformatted %s placeholder in APIError message in Project.get() - ([#523](https://github.com/rungalileo/galileo-python/pull/523), - [`979f2b7`](https://github.com/rungalileo/galileo-python/commit/979f2b78f55ba13ae8935a3ebf359375b5920677)) - -- Get card fixes ([#552](https://github.com/rungalileo/galileo-python/pull/552), - [`a1cc7a2`](https://github.com/rungalileo/galileo-python/commit/a1cc7a2998b111f6e168d92fc0b33d4b888f921d)) - -- Handle legacy token fields from OpenAI usage - ([#517](https://github.com/rungalileo/galileo-python/pull/517), - [`6db840f`](https://github.com/rungalileo/galileo-python/commit/6db840fe5b561d42799b87935376d31ca87e828d)) - -- Missing tasks in hierarchical CrewAI crew traces - ([#532](https://github.com/rungalileo/galileo-python/pull/532), - [`6afaa04`](https://github.com/rungalileo/galileo-python/commit/6afaa04e92b0c531ced76b2823a8f98c9ff12fed)) - -- Move dataset context to decorator to remove otel import from experiments - ([#502](https://github.com/rungalileo/galileo-python/pull/502), - [`6e4f193`](https://github.com/rungalileo/galileo-python/commit/6e4f1935a9f069fe443c41074314ff0268d1f870)) - -- NewSDK: get_version* are [very] confusing - ([#530](https://github.com/rungalileo/galileo-python/pull/530), - [`cf606c4`](https://github.com/rungalileo/galileo-python/commit/cf606c4a53dd3c5fc4b769cfe6eeb7516b708c96)) - -- Prioritize experiment over logstream in OTEL export - ([#498](https://github.com/rungalileo/galileo-python/pull/498), - [`12c736a`](https://github.com/rungalileo/galileo-python/commit/12c736a034cbda50b3675b1fe5ffa88ab199cd6f)) - -- Python SDK tests often hang ([#531](https://github.com/rungalileo/galileo-python/pull/531), - [`7dff69d`](https://github.com/rungalileo/galileo-python/commit/7dff69ddd48fc77aa554fc724ce97e376f943385)) - -- Raise consistent ResourceNotFoundError when project cannot be resolved - ([#510](https://github.com/rungalileo/galileo-python/pull/510), - [`ec6e2d0`](https://github.com/rungalileo/galileo-python/commit/ec6e2d039219f2027d70f7fc237eddc2de7c3cf8)) - -- Raise error when extend/generate job fails silently - ([#528](https://github.com/rungalileo/galileo-python/pull/528), - [`d7b9da1`](https://github.com/rungalileo/galileo-python/commit/d7b9da1c1b0c40a70efe5f205a4a557478999b70)) - -- Regression issues on experiments ([#511](https://github.com/rungalileo/galileo-python/pull/511), - [`b7cc9bd`](https://github.com/rungalileo/galileo-python/commit/b7cc9bdc10591ed4b7e9ca49874fd1612546974a)) - -- Resolve OpenAI test failures when Azure env vars are set - ([#540](https://github.com/rungalileo/galileo-python/pull/540), - [`5322d63`](https://github.com/rungalileo/galileo-python/commit/5322d6362d70f2089e2ed82eb507567c0d856a86)) - -- Schema patches ([#573](https://github.com/rungalileo/galileo-python/pull/573), - [`05a8d7f`](https://github.com/rungalileo/galileo-python/commit/05a8d7f4304e146db60e13e3e997ceea795bd14c)) - -- The filter schema patches ([#568](https://github.com/rungalileo/galileo-python/pull/568), - [`da22838`](https://github.com/rungalileo/galileo-python/commit/da228383d9b64eddaddfba298876a2ec62430aae)) - -- Url ([#571](https://github.com/rungalileo/galileo-python/pull/571), - [`bc36762`](https://github.com/rungalileo/galileo-python/commit/bc367626d1160c93b791f28eb333579ab409ce35)) - -- **config**: Extend auth guard to recognize all supported auth methods - ([#579](https://github.com/rungalileo/galileo-python/pull/579), - [`83488f6`](https://github.com/rungalileo/galileo-python/commit/83488f62420c4e9949033d14f4e477b32f24d827)) - -- **dataset**: Extend now adds generated rows to the existing dataset - ([#544](https://github.com/rungalileo/galileo-python/pull/544), - [`2f0db14`](https://github.com/rungalileo/galileo-python/commit/2f0db1432cdc8c55fc7b570070dd560b0634b246)) - -- **dataset**: Normalize ground_truth to output in add_rows - ([#545](https://github.com/rungalileo/galileo-python/pull/545), - [`c262a27`](https://github.com/rungalileo/galileo-python/commit/c262a27c8a850f8d641056a1473cd7ee004dddc7)) - -- **datasets**: Adjust default model alias name - ([#535](https://github.com/rungalileo/galileo-python/pull/535), - [`95a4e45`](https://github.com/rungalileo/galileo-python/commit/95a4e459f65fca9b63f8cff5856f233e741d6e68)) - -- **datasets**: Preserve all prompt_settings fields in extend_dataset - ([#549](https://github.com/rungalileo/galileo-python/pull/549), - [`4b290c8`](https://github.com/rungalileo/galileo-python/commit/4b290c81106314b1a64995e7bb603f646b21c4a7)) - -- **datasets**: Remap 'output' to 'ground_truth' in get_content() response - ([#555](https://github.com/rungalileo/galileo-python/pull/555), - [`be7666d`](https://github.com/rungalileo/galileo-python/commit/be7666dda4c88af3dc0673d85768bb220a97911f)) - -- **deprecations**: Remove @deprecated decorators from legacy SDK functions - ([#562](https://github.com/rungalileo/galileo-python/pull/562), - [`6c0d200`](https://github.com/rungalileo/galileo-python/commit/6c0d200a9b9bf3af332cb296df4e939424d8d434)) - -- **deps**: Bump pyjwt to >=2.12.0 (fixes CVE-2026-32597 / GHSA-752w-5fwx-jx9f) - ([#558](https://github.com/rungalileo/galileo-python/pull/558), - [`548addf`](https://github.com/rungalileo/galileo-python/commit/548addf39b447c973e4700b5870fa5376f7ad203)) - -- **deps**: Upgrade openai to 2.x and litellm to >=1.83.0 (fixes CVE GHSA-69x8-hrgq-fjj8) - ([#557](https://github.com/rungalileo/galileo-python/pull/557), - [`f5e90da`](https://github.com/rungalileo/galileo-python/commit/f5e90dae720a675f79031f3157c66edd211b5593)) - -- **docs**: Update span docstrings with accurate input formats - ([#541](https://github.com/rungalileo/galileo-python/pull/541), - [`63197c1`](https://github.com/rungalileo/galileo-python/commit/63197c15695c2d3071f3256b07c996cf2c56fe96)) - -- **errors**: Context-aware project-not-found error messages - ([#569](https://github.com/rungalileo/galileo-python/pull/569), - [`0c8efb8`](https://github.com/rungalileo/galileo-python/commit/0c8efb88af614b5326e48b294759660f4bf5ed8a)) - -- **experiment**: Make prompt optional in Experiment.__init__ - ([#563](https://github.com/rungalileo/galileo-python/pull/563), - [`e2ad2c2`](https://github.com/rungalileo/galileo-python/commit/e2ad2c240b4b7c61a5b887eb463770add08dcdc1)) - -- **experiment**: Preserve user prompt_settings when overriding model - ([#547](https://github.com/rungalileo/galileo-python/pull/547), - [`50987ce`](https://github.com/rungalileo/galileo-python/commit/50987cee8bacce2949f615bfcc2ff625add7173d)) - -- **experiment**: Reset _run_result_consumed flag in create() - ([#548](https://github.com/rungalileo/galileo-python/pull/548), - [`6ea5380`](https://github.com/rungalileo/galileo-python/commit/6ea538034ac2b7164e4cd45dad709a0c73b52fc5)) - -- **experiments,jobs**: Surface validation errors on invalid model alias - ([#550](https://github.com/rungalileo/galileo-python/pull/550), - [`74443ad`](https://github.com/rungalileo/galileo-python/commit/74443add47ec8880dfad2a18d235c290d59d0932)) - -- **log_stream**: Convert ProjectsAPIException to ResourceNotFoundError in get/list - ([#561](https://github.com/rungalileo/galileo-python/pull/561), - [`4a45976`](https://github.com/rungalileo/galileo-python/commit/4a4597684b8bfb1780f59e24d15b5656b27277f7)) - -- **logger**: Add explicit str to start_trace input type hints and docstrings - ([#574](https://github.com/rungalileo/galileo-python/pull/574), - [`b186595`](https://github.com/rungalileo/galileo-python/commit/b186595f8c3ee3a25a8a11e3e8574a644723b457)) - -- **logger**: Add list[dict] to start_trace input docs - ([#565](https://github.com/rungalileo/galileo-python/pull/565), - [`2cbf848`](https://github.com/rungalileo/galileo-python/commit/2cbf8487dae71d76244949d29984f5253b610b17)) - -- **step_type**: Add CONTROL to StepType enum - ([#570](https://github.com/rungalileo/galileo-python/pull/570), - [`dd17a21`](https://github.com/rungalileo/galileo-python/commit/dd17a21335de74a845c47c22934eed2e2b8b17fc)) - -- **validation**: Handle plain-string detail in HTTPValidationError - ([#564](https://github.com/rungalileo/galileo-python/pull/564), - [`2f67cfb`](https://github.com/rungalileo/galileo-python/commit/2f67cfbb1d5643f9faf32980afd4e48da8f338b1)) - -### Chores - -- Bump galileo-core to ^4.1.2 ([#499](https://github.com/rungalileo/galileo-python/pull/499), - [`f0a561a`](https://github.com/rungalileo/galileo-python/commit/f0a561a86fee61f299213ccba19f5a75e2c70da5)) - -- Bump galileo-core to ^4.2.1 ([#507](https://github.com/rungalileo/galileo-python/pull/507), - [`66e86cd`](https://github.com/rungalileo/galileo-python/commit/66e86cd758f08a76fee6904a149bf57d7e660e9b)) - -- Migrating remaining files to root ([#512](https://github.com/rungalileo/galileo-python/pull/512), - [`c86c53a`](https://github.com/rungalileo/galileo-python/commit/c86c53a4f7446598c7aed2459a5ba133bcadea60)) - -- Release version 2.1.2 - ([`edecb41`](https://github.com/rungalileo/galileo-python/commit/edecb412e4e303d2c1d7ab17154222e12ec703b3)) - -- Release version 2.1.3 - ([`207c9ce`](https://github.com/rungalileo/galileo-python/commit/207c9ce931269b4cf482a23280bf84d70a982e6f)) - -- Remove CODEOWNERS ([#533](https://github.com/rungalileo/galileo-python/pull/533), - [`354491b`](https://github.com/rungalileo/galileo-python/commit/354491b828aae4a83d86f7db45d0aafd8e63cacb)) - -- Update CONTRIBUTING.md ([#582](https://github.com/rungalileo/galileo-python/pull/582), - [`9435453`](https://github.com/rungalileo/galileo-python/commit/943545377df0200cf2fcf4c4ea8500ed6b47eff8)) - -- Update dev environment to recommend Python 3.13 - ([#518](https://github.com/rungalileo/galileo-python/pull/518), - [`4b197ab`](https://github.com/rungalileo/galileo-python/commit/4b197ab8bc4c94120f852d1ae5161af46f49aba1)) - -- Update ruff target-version and fix mypy pre-commit hook - ([#536](https://github.com/rungalileo/galileo-python/pull/536), - [`4b6bffb`](https://github.com/rungalileo/galileo-python/commit/4b6bffbf9e54f96479fe2308e5f0ac100ec55594)) - -- **release**: Galileo-a2a v1.0.0 - ([`bc515b3`](https://github.com/rungalileo/galileo-python/commit/bc515b346a00071f1ed2c38dba17db321350a3b6)) - -- **release**: V1.48.0 - ([`bdcdede`](https://github.com/rungalileo/galileo-python/commit/bdcdede246a5c885e3afd8c8d48f02a3f3630ed0)) - -- **release**: V1.49.0 - ([`5a7c747`](https://github.com/rungalileo/galileo-python/commit/5a7c74796e3063f8fe255acccfb98db7cc19488c)) - -- **release**: V1.49.1 - ([`beb1a4a`](https://github.com/rungalileo/galileo-python/commit/beb1a4acb038c7f207619f01591d013815a06b76)) - -- **release**: V1.49.2 - ([`459457d`](https://github.com/rungalileo/galileo-python/commit/459457d22710a00acf796984b9e9972d2c3ec6f3)) - -- **release**: V1.50.0 - ([`a8ac26f`](https://github.com/rungalileo/galileo-python/commit/a8ac26ff4c952e2385a77a4ae06b6cc452c66ebd)) - -- **release**: V1.50.1 - ([`30b381a`](https://github.com/rungalileo/galileo-python/commit/30b381ad496d2c8631924977ee49c1458bda3897)) - -- **release**: V1.51.0 - ([`a6dadc6`](https://github.com/rungalileo/galileo-python/commit/a6dadc6b25dddb81082e4b02cefa9f6a24b6089e)) - -- **release**: V2.0.0 - ([`0dcb7d9`](https://github.com/rungalileo/galileo-python/commit/0dcb7d9a1139380b4e9f3c6bc14a5041e28f8ee4)) - -- **release**: V2.1.0 - ([`eedf91f`](https://github.com/rungalileo/galileo-python/commit/eedf91fc674d341e8cfce620c19ad806cc895adb)) - -- **release**: V2.1.1 - ([`ba8c8d2`](https://github.com/rungalileo/galileo-python/commit/ba8c8d2e94b1bc5261ce2feb1370d9eabef7f4b4)) - -- **release**: V2.2.0 - ([`10a8a90`](https://github.com/rungalileo/galileo-python/commit/10a8a90745160ad43779c860aa83a078712f6f13)) - -### Features - -- Add Agent Control target helper ([#581](https://github.com/rungalileo/galileo-python/pull/581), - [`8ef084f`](https://github.com/rungalileo/galileo-python/commit/8ef084ff92e1108927cef88dfe5f563b2efe6788)) - -- Add agentcontrol logger bridge ([#566](https://github.com/rungalileo/galileo-python/pull/566), - [`861c2e3`](https://github.com/rungalileo/galileo-python/commit/861c2e3bfe43c1a4cab1501884c3cd9efe0c4ce7)) - -- Add metric_aggregates + experiment_columns, deprecate aggregate_metrics - ([#578](https://github.com/rungalileo/galileo-python/pull/578), - [`9b3c424`](https://github.com/rungalileo/galileo-python/commit/9b3c42403fb943a40c24ecbfa7fe35b1224c6eb9)) - -- Add proto-plus message serialization support in EventSerializer… - ([#485](https://github.com/rungalileo/galileo-python/pull/485), - [`5cabda5`](https://github.com/rungalileo/galileo-python/commit/5cabda5557020bf5d345e8faddbbd733aad4b42c)) - -- Add support for serializing WorkflowSpan input and output - ([#472](https://github.com/rungalileo/galileo-python/pull/472), - [`ed49115`](https://github.com/rungalileo/galileo-python/commit/ed491150331ec93a19471d582e9171ba278a4913)) - -- Add support for ToolSpan attributes - ([#475](https://github.com/rungalileo/galileo-python/pull/475), - [`aa8ba2e`](https://github.com/rungalileo/galileo-python/commit/aa8ba2eea2d1cf978156c493980abd068bbdd96f)) - -- Creating custom input and output types for multimodal ingestion - ([#500](https://github.com/rungalileo/galileo-python/pull/500), - [`3685658`](https://github.com/rungalileo/galileo-python/commit/368565852e04f00247836fcdba339112e71ce5e2)) - -- Harden a2a distributed tracing and improve test coverage - ([#546](https://github.com/rungalileo/galileo-python/pull/546), - [`263a79b`](https://github.com/rungalileo/galileo-python/commit/263a79b383f5b4753911535b355ec3ba1a90a179)) - -- Implement google a2a instrumentor ([#529](https://github.com/rungalileo/galileo-python/pull/529), - [`ca24de6`](https://github.com/rungalileo/galileo-python/commit/ca24de66ef93d610f5e0e719b67c1ed2f08f830a)) - -- Implement Metric.update() via PATCH /scorers/{id} - ([#505](https://github.com/rungalileo/galileo-python/pull/505), - [`f181ab6`](https://github.com/rungalileo/galileo-python/commit/f181ab69d3bf7d56f865ef70cc7aabb4527dd08f)) - -- Implement Project.save() and Dataset.save() - ([#503](https://github.com/rungalileo/galileo-python/pull/503), - [`5d473ed`](https://github.com/rungalileo/galileo-python/commit/5d473edf0e4b27fa593eacc68b9433e17ddf309b)) - -- Implement Prompt.save() with automatic dirty tracking for name changes - ([#504](https://github.com/rungalileo/galileo-python/pull/504), - [`e8cc58f`](https://github.com/rungalileo/galileo-python/commit/e8cc58f6f00dd71418bf601899da3fbce287fd8f)) - -- Improve metric naming conventions and optimize scorer retrieval - ([#522](https://github.com/rungalileo/galileo-python/pull/522), - [`65e28c7`](https://github.com/rungalileo/galileo-python/commit/65e28c77ab983af1a4cc258392dd08f408331136)) - -- Serialize trace output in distributed mode decorator path - ([#514](https://github.com/rungalileo/galileo-python/pull/514), - [`444ce9b`](https://github.com/rungalileo/galileo-python/commit/444ce9be5dece6fa352c321246e6a7530b6493e2)) - -- Support crewai 1.9.3 ([#484](https://github.com/rungalileo/galileo-python/pull/484), - [`dba40f6`](https://github.com/rungalileo/galileo-python/commit/dba40f62d09d5e1e820082b732b14da6d23a76a8)) - -- Support experiment groups (V1) ([#575](https://github.com/rungalileo/galileo-python/pull/575), - [`46c1b35`](https://github.com/rungalileo/galileo-python/commit/46c1b35832468a30e8912a2a06244756b6a68856)) - -- Support generated-output flow in run_experiment (no prompt_temp… - ([#486](https://github.com/rungalileo/galileo-python/pull/486), - [`222acf8`](https://github.com/rungalileo/galileo-python/commit/222acf8fd3300773058052d6f029e31af2140ecc)) - -- Switch run_experiment to trigger=True for generated output flow - ([#497](https://github.com/rungalileo/galileo-python/pull/497), - [`837d9b5`](https://github.com/rungalileo/galileo-python/commit/837d9b546cac3b47db824b13df5bf4585156bed0)) - -- **experiments**: Add on_error callback to flush and experiment run - ([#537](https://github.com/rungalileo/galileo-python/pull/537), - [`e08596c`](https://github.com/rungalileo/galileo-python/commit/e08596c6ac7ce7dd4a708110d8148fd35363d5cc)) - -- **langchain**: Add multimodal content support to LangChain handler - ([#515](https://github.com/rungalileo/galileo-python/pull/515), - [`fb6c1a1`](https://github.com/rungalileo/galileo-python/commit/fb6c1a1f87c94fb396ed19e795cc4712315ff313)) - -- **log-stream**: Add limit parameter to LogStream.list() - ([#580](https://github.com/rungalileo/galileo-python/pull/580), - [`cccd15e`](https://github.com/rungalileo/galileo-python/commit/cccd15e4370aeedfe64b9d4d9a37cd84c693c5ec)) - -- **logger**: Validate input types in start_trace - ([#539](https://github.com/rungalileo/galileo-python/pull/539), - [`e07ab69`](https://github.com/rungalileo/galileo-python/commit/e07ab69290f1a67229bca59d4f3ab63961fbaff0)) - -### Refactoring - -- Add @deprecated decorators to legacy service functions in datasets, experiments, prompts, - log_streams ([#524](https://github.com/rungalileo/galileo-python/pull/524), - [`546e00f`](https://github.com/rungalileo/galileo-python/commit/546e00fa1d00f9a5f25b55aabbfe0343733417ed)) - -- Adding deprecated on __future__ re export - ([#525](https://github.com/rungalileo/galileo-python/pull/525), - [`69d0c22`](https://github.com/rungalileo/galileo-python/commit/69d0c2207cb13393510ddb3002e8e38e62aa7392)) - -- Move tests from tests/future/ to tests/ directory - ([#526](https://github.com/rungalileo/galileo-python/pull/526), - [`48bc50c`](https://github.com/rungalileo/galileo-python/commit/48bc50cbee9819048a37b834ae9a6133b171dced)) - -- Moving project.py to root, away from __future__ - ([#506](https://github.com/rungalileo/galileo-python/pull/506), - [`1dd4b4c`](https://github.com/rungalileo/galileo-python/commit/1dd4b4c4eaca11ffb33e9478da38b143cb4fb4b2)) - -- Streamline project and logstream resolution in GalileoOTLPE… - ([#493](https://github.com/rungalileo/galileo-python/pull/493), - [`8be1413`](https://github.com/rungalileo/galileo-python/commit/8be141376bca9114d27492b5eea503812da07dcc)) - - -## v1.0.0 (2026-02-23) - -### Bug Fixes - -- Also send metadata on async handler - ([#476](https://github.com/rungalileo/galileo-python/pull/476), - [`7a85819`](https://github.com/rungalileo/galileo-python/commit/7a8581976c00499bffc36b5d03e95f04d4e12bab)) - -- Normalize ground_truth to output in create_dataset - ([#481](https://github.com/rungalileo/galileo-python/pull/481), - [`545073a`](https://github.com/rungalileo/galileo-python/commit/545073acc8dd59c2d656f36ca11bfd8b46319686)) - -- **handlers**: Pass ingestion_hook through handler → logger chain - ([#479](https://github.com/rungalileo/galileo-python/pull/479), - [`4cd3c10`](https://github.com/rungalileo/galileo-python/commit/4cd3c106aca157ae95d5dbbd8ce09fa793eaf44c)) - -### Chores - -- Change crew ai imports to be lazy ([#477](https://github.com/rungalileo/galileo-python/pull/477), - [`d1f987a`](https://github.com/rungalileo/galileo-python/commit/d1f987a530e966f9134b8c6f9f2fdbc19bc68b9b)) - -- **release**: Galileo-adk v1.0.0 - ([`0e364eb`](https://github.com/rungalileo/galileo-python/commit/0e364eb1158d2e6e920c5afa519a26fe35befcac)) - -- **release**: V1.46.1 - ([`25c51cd`](https://github.com/rungalileo/galileo-python/commit/25c51cdae90ffa3f165a49e1b7046e2a2f382120)) - -- **release**: V1.46.2 - ([`17c8b0f`](https://github.com/rungalileo/galileo-python/commit/17c8b0f3ec5db18d859c9728aa2d64354d893624)) - -- **release**: V1.47.0 - ([`a3106fd`](https://github.com/rungalileo/galileo-python/commit/a3106fd6cb8e68c7bed73516a8881bc1837ee8b4)) - -### Features - -- Allow session metadata tagging ([#480](https://github.com/rungalileo/galileo-python/pull/480), - [`114e24d`](https://github.com/rungalileo/galileo-python/commit/114e24db31eef7a16f966eb6c9238f13c88a66ca)) - -- Galileo adk to natively support custom retriever tools - ([#482](https://github.com/rungalileo/galileo-python/pull/482), - [`f11138a`](https://github.com/rungalileo/galileo-python/commit/f11138aaf5dd944f4cd3b5dfe486755e7c1c65a1)) - - -## v1.0.0-beta.1 (2026-02-11) - -- Initial Release +# Changelog + +## Unreleased + +### Added + +- Added automatic tracing for Google ADK agent runs, model calls, tool calls, + and retriever operations. +- Added session correlation through ADK session IDs and support for ADK + `RunConfig.custom_metadata`. +- Added plugin and callback integration modes with structured and multimodal + content capture. +- Added deployment-aware OTLP export for Splunk Observability Cloud and + standalone Agent Observability through the core `splunk-ao` SDK. +- Retained the proprietary ingestion hook as deprecated migration + compatibility; new custom pipelines should use OpenTelemetry extension + points. diff --git a/splunk-ao-adk/README.md b/splunk-ao-adk/README.md index fac8342..1a5afbd 100644 --- a/splunk-ao-adk/README.md +++ b/splunk-ao-adk/README.md @@ -12,7 +12,8 @@ Splunk AO observability for [Google ADK](https://github.com/google/adk-python) a pip install splunk-ao-adk ``` -**Requirements:** Python 3.11+, a [Splunk AO API key](https://www.splunk.com/), and a [Google AI API key](https://aistudio.google.com/apikey) +**Requirements:** Python 3.11+, Splunk AO standalone or Splunk Observability +Cloud credentials, and a [Google AI API key](https://aistudio.google.com/apikey). ## Quick Start @@ -34,17 +35,41 @@ async def main(): print(event.content.parts[0].text) if __name__ == "__main__": - # Set environment variables: SPLUNK_AO_API_KEY, GOOGLE_API_KEY + # Configure one Splunk AO deployment below, plus GOOGLE_API_KEY. asyncio.run(main()) ``` ## Configuration -| Parameter | Environment Variable | Description | -|-----------|---------------------|-------------| -| `project` | `SPLUNK_AO_PROJECT` | Project name (required unless `ingestion_hook` provided) | -| `log_stream` | `SPLUNK_AO_AGENT_STREAM` | Log stream name (required unless `ingestion_hook` provided) | -| `ingestion_hook` | - | Custom callback for trace data (bypasses Splunk AO backend) | +| Parameter | Description | +|-----------|-------------| +| `project` | Project name. Explicit arguments override environment routing. | +| `agent_stream` | Agent Stream name. Explicit arguments override environment routing. | +| `ingestion_hook` | Deprecated compatibility callback that receives proprietary trace requests and bypasses normal OTLP export. | + +For standalone Splunk AO: + +| Environment Variable | Description | +|---------------------|-------------| +| `SPLUNK_AO_API_KEY` | Splunk AO API key (required) | +| `SPLUNK_AO_CONSOLE_URL` | Splunk AO console URL (required for self-hosted deployments) | +| `SPLUNK_AO_API_URL` | Explicit API URL (optional; otherwise derived from the console URL) | +| `SPLUNK_AO_PROJECT` | Project name | +| `SPLUNK_AO_AGENT_STREAM` | Agent Stream name | + +For Splunk Observability Cloud: + +| Environment Variable | Description | +|---------------------|-------------| +| `SPLUNK_AO_REALM` | Observability Cloud realm (required) | +| `SPLUNK_AO_O11Y_TOKEN` | O11y ingest token used for OTLP export (required) | +| `SPLUNK_AO_O11Y_API_TOKEN` | Dedicated O11y API token used for session and other CRUD operations (optional) | +| `SPLUNK_AO_PROJECT` | Project name | +| `SPLUNK_AO_AGENT_STREAM` | Agent Stream name | + +When both O11y tokens are configured, the API token is preferred for CRUD and +the ingest token is used for telemetry. A combined token can perform both when +it includes both permissions. ## Features @@ -80,7 +105,7 @@ async def main(): print(f"Response 2: {event.content.parts[0].text}") if __name__ == "__main__": - # Set environment variables: SPLUNK_AO_API_KEY, GOOGLE_API_KEY + # Configure one Splunk AO deployment above, plus GOOGLE_API_KEY. asyncio.run(main()) ``` @@ -121,7 +146,7 @@ async def main(): print(event.content.parts[0].text) if __name__ == "__main__": - # Set environment variables: SPLUNK_AO_API_KEY, GOOGLE_API_KEY + # Configure one Splunk AO deployment above, plus GOOGLE_API_KEY. asyncio.run(main()) ``` @@ -158,7 +183,7 @@ async def main(): print(event.content.parts[0].text) if __name__ == "__main__": - # Set environment variables: SPLUNK_AO_API_KEY, GOOGLE_API_KEY + # Configure one Splunk AO deployment above, plus GOOGLE_API_KEY. asyncio.run(main()) ``` @@ -181,7 +206,10 @@ tool = FunctionTool(search_docs) ### Ingestion Hook -Intercept traces for custom processing before forwarding to Splunk AO: +The proprietary ingestion hook remains available as deprecated migration +compatibility. It bypasses the normal OTLP export path. New custom telemetry +pipelines should use OpenTelemetry `SpanProcessor` and `SpanExporter` +extension points instead. ```python import asyncio @@ -198,7 +226,7 @@ logger = SplunkAOLogger( ) def my_ingestion_hook(request): - """Hook that captures traces locally and forwards to Galileo with session management.""" + """Capture traces locally and forward them with session management.""" if hasattr(request, "traces") and request.traces: print(f"\n[Ingestion Hook] Intercepted {len(request.traces)} trace(s)") for trace in request.traces: @@ -206,11 +234,11 @@ def my_ingestion_hook(request): span_types = [getattr(s, "type", "unknown") for s in spans] print(f" - Trace with {len(spans)} span(s): {span_types}") - # Session management: same external_id returns the same Galileo session - galileo_session_id = logger.start_session(external_id=request.session_external_id) - request.session_id = galileo_session_id + # The same external ID returns the same Agent Observability session. + session_id = logger.start_session(external_id=request.session_external_id) + request.session_id = session_id - # Forward traces to Galileo + # Forward traces through the legacy proprietary endpoint. logger.ingest_traces(request) async def main(): @@ -224,13 +252,13 @@ async def main(): print(event.content.parts[0].text) if __name__ == "__main__": - # Set environment variables: SPLUNK_AO_API_KEY, GOOGLE_API_KEY + # Configure one Splunk AO deployment above, plus GOOGLE_API_KEY. asyncio.run(main()) ``` ## Resources -- [Splunk AO Documentation](https://docs.rungalileo.io/) +- [Splunk AO Documentation](https://agent-observability-docs.splunk.com) - [Google ADK Documentation](https://google.github.io/adk-docs/) ## License diff --git a/src/splunk_ao/README_API_CLIENT.md b/src/splunk_ao/README_API_CLIENT.md index b5d3d1c..5b7fb4a 100644 --- a/src/splunk_ao/README_API_CLIENT.md +++ b/src/splunk_ao/README_API_CLIENT.md @@ -4,14 +4,21 @@ A client library for accessing the Splunk AO platform API ## Usage -First, create a client: +This is the low-level client generated from the platform API schema. It does +not read `SPLUNK_AO_*` environment variables or perform deployment-aware +configuration. Most applications should use the public Splunk AO SDK APIs, +which configure authentication and endpoints automatically. + +When using the generated client directly, provide its base URL and any +required authentication headers explicitly: ```python from splunk_ao.resources.client import Client -# Make sure you've set the SPLUNK_AO_CONSOLE_URL and SPLUNK_AO_API_KEY env vars -# Optionally, you can specify both base_url and api_key -client = Client(base_url="https://your-splunk-ao-instance") +client = Client( + base_url="https://your-splunk-ao-instance", + headers={"your-auth-header": "your-token"}, +) ``` Now call your endpoint and use your models: