Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 3 additions & 6 deletions .github/workflows/ci-tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,8 @@ jobs:
run: |
poetry env use "${{ steps.setup-python.outputs.python-path }}"

- name: Install invoke
run: pipx install invoke

- name: Install Dependencies
run: invoke install
run: poetry install --all-extras --no-root
Comment on lines 55 to +56

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 minor (design): This CI workflow change (removing the pipx install invoke step, switching invoke installpoetry install --all-extras --no-root, and prefixing poetry run to the invoke calls) is unrelated to the PR's stated purpose (docs(examples): fix doc link paths) and is not mentioned anywhere in the description. The change itself looks functionally correct — invoke is declared in the Poetry dev group, so poetry run invoke type-check / poetry run invoke test-report-xml resolve after poetry install. But this scope creep should either be split into its own PR or called out explicitly in the description so reviewers know to evaluate it.

🤖 Generated by the Astra agent

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the PR description to match the changes. The CI workflow is to fix one of the failed pipeline for this PR.


- name: Verify Poetry Python version
run: |
Expand All @@ -70,8 +67,8 @@ jobs:

- name: Validate Types
if: always()
run: invoke type-check
run: poetry run invoke type-check

- name: Run Tests
if: always()
run: invoke test-report-xml
run: poetry run invoke test-report-xml
2 changes: 1 addition & 1 deletion examples/agent/crewAI/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ This repo contains examples of how to use Splunk AO to instrument [CrewAI](https
The [resarch-crew](./research_crew/) is a quickstart tutorial. It is a completed version of the [CrewAI quickstart](https://docs.crewai.com/en/quickstart) and adds the
Splunk AO's [CrewAIEventListener](https://agent-observability-docs.splunk.com/sdk-api/python/reference/handlers/crewai/handler),
an event handler implemented on top of OpenTelemetry (OTel). For more information, see
Splunk AO’s [Add Splunk AO to a CrewAI Application](https://agent-observability-docs.splunk.com/how-to-guides/third-party-integrations/add-galileo-to-crewai/add-galileo-to-crewai)
Splunk AO’s [Add Splunk AO to a CrewAI Application](https://agent-observability-docs.splunk.com/how-to-guides/third-party-integrations/add-splunk-ao-to-crewai/add-splunk-ao-to-crewai)
how-to guide.

See the [README.md](./research_crew/README.md) for detailed setup instructions.
4 changes: 2 additions & 2 deletions examples/agent/crewAI/research_crew/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
This tutorial is a completed version of the [CrewAI quickstart](https://docs.crewai.com/en/quickstart) and adds the
Splunk AO [CrewAIEventListener](https://agent-observability-docs.splunk.com/sdk-api/python/reference/handlers/crewai/handler),
an event handler implemented on top of OpenTelemetry (OTel). For more information, see
Splunk AO’s [Add Splunk AO to a CrewAI Application](https://agent-observability-docs.splunk.com/how-to-guides/third-party-integrations/add-galileo-to-crewai/add-galileo-to-crewai)
Splunk AO’s [Add Splunk AO to a CrewAI Application](https://agent-observability-docs.splunk.com/how-to-guides/third-party-integrations/add-splunk-ao-to-crewai/add-splunk-ao-to-crewai)
how-to guide.

Below you'll find concise setup steps, configuration notes, and a short primer on Splunk AO so you know why we add it and what it provides.
Expand All @@ -12,7 +12,7 @@ Below you'll find concise setup steps, configuration notes, and a short primer o

The relevant guides:

- [Add Splunk AO to CrewAI](https://agent-observability-docs.splunk.com/how-to-guides/third-party-integrations/add-galileo-to-crewai/add-galileo-to-crewai)
- [Add Splunk AO to CrewAI](https://agent-observability-docs.splunk.com/how-to-guides/third-party-integrations/add-splunk-ao-to-crewai/add-splunk-ao-to-crewai)
- [CrewAI quickstart](https://docs.crewai.com/en/guides/crews/first-crew)

## What is Splunk AO?
Expand Down
2 changes: 1 addition & 1 deletion examples/agent/langgraph-fsi-agent/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ From there you can configure the metrics you are interested in. Once metrics are

This project also includes a unit test to run the chatbot with a set of defined prompts, evaluating the prompts for action advancement, action completion, tool selection quality, and tool errors, only passing the test if both metrics score an average of 100% (or 0% for tool errors) over all the entries in the dataset.

This is run using the [Splunk AO experiments framework](https://agent-observability-docs.splunk.com/concepts/experiments/overview) - allowing you to run any code as an experiment against a fixed dataset of prompts. This mechanism allows you to run AI applications, from simple to complex, under test conditions with a defined set of inputs. You can then use the results of evaluations run against your app to help with model selection or prompt engineering, as well as validating your application as part of a CI/CD pipeline.
This is run using the [Splunk AO experiments framework](https://agent-observability-docs.splunk.com/sdk-api/experiments/experiments) - allowing you to run any code as an experiment against a fixed dataset of prompts. This mechanism allows you to run AI applications, from simple to complex, under test conditions with a defined set of inputs. You can then use the results of evaluations run against your app to help with model selection or prompt engineering, as well as validating your application as part of a CI/CD pipeline.

You can run the unit test by running the following command inside your virtual environment:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ These moons are significant for their unique geological features and potential f

Every run of the app is logged as a new session in Splunk AO, with each prompt and response a separate trace.

To evaluate the chatbot, head to your project and Log stream in the Splunk AO console. Turn on [instruction adherence](https://agent-observability-docs.splunk.com/concepts/metrics/response-quality/instruction-adherence) and [correctness](https://agent-observability-docs.splunk.com/concepts/metrics/response-quality/correctness) for your Log stream.
To evaluate the chatbot, head to your project and Log stream in the Splunk AO console. Turn on [instruction adherence](https://agent-observability-docs.splunk.com/concepts/evaluators/response-quality/instruction-adherence) and [correctness](https://agent-observability-docs.splunk.com/concepts/evaluators/response-quality/correctness) for your Log stream.

Then when you run the chatbot, these metrics will be evaluated.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ These moons are significant for their unique geological features and potential f

Every run of the app is logged as a new session in Splunk AO, with each prompt and response a separate trace.

To evaluate the chatbot, head to your project and Log stream in the Splunk AO console. Turn on [instruction adherence](https://agent-observability-docs.splunk.com/concepts/metrics/response-quality/instruction-adherence) and [correctness](https://agent-observability-docs.splunk.com/concepts/metrics/response-quality/correctness) for your Log stream.
To evaluate the chatbot, head to your project and Log stream in the Splunk AO console. Turn on [instruction adherence](https://agent-observability-docs.splunk.com/concepts/evaluators/response-quality/instruction-adherence) and [correctness](https://agent-observability-docs.splunk.com/concepts/evaluators/response-quality/correctness) for your Log stream.

Then when you run the chatbot, these metrics will be evaluated.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ These moons are significant for their unique geological features and potential f

Every run of the app is logged as a new session in Splunk AO, with each prompt and response a separate trace.

To evaluate the chatbot, head to your project and Log stream in the Splunk AO console. Turn on [instruction adherence](https://agent-observability-docs.splunk.com/concepts/metrics/response-quality/instruction-adherence) and [correctness](https://agent-observability-docs.splunk.com/concepts/metrics/response-quality/correctness) for your Log stream.
To evaluate the chatbot, head to your project and Log stream in the Splunk AO console. Turn on [instruction adherence](https://agent-observability-docs.splunk.com/concepts/evaluators/response-quality/instruction-adherence) and [correctness](https://agent-observability-docs.splunk.com/concepts/evaluators/response-quality/correctness) for your Log stream.

Then when you run the chatbot, these metrics will be evaluated.

Expand Down
2 changes: 1 addition & 1 deletion examples/experiments/multi-turn/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ Pre-defined session-level metrics include:
- `SplunkAOMetrics.context_relevance`
- `SplunkAOMetrics.tool_error_rate`

Related documentation: [Metrics Comparison](https://agent-observability-docs.splunk.com/concepts/metrics/metric-comparison)
Related documentation: [Metrics Comparison](https://agent-observability-docs.splunk.com/concepts/evaluators/evaluator-comparison)

Optionally, you can define your own custom session-level metric in the Splunk AO Console UI, and then add the custom metric name.

Expand Down
2 changes: 1 addition & 1 deletion examples/experiments/upload_experiment/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,5 +172,5 @@ After running the script, your Splunk AO project will contain:

- [Splunk AO Documentation](https://agent-observability-docs.splunk.com/what-is-splunk-agent-observability)
- [Splunk AO SDK Reference](https://agent-observability-docs.splunk.com/sdk-api/overview)
- [Creating Custom Metrics](https://agent-observability-docs.splunk.com/concepts/metrics/custom-metrics/custom-metrics-ui-llm)
- [Creating Custom Metrics](https://agent-observability-docs.splunk.com/concepts/evaluators/custom-evaluators/custom-evaluators-ui-llm)
- [Understanding Experiments](https://agent-observability-docs.splunk.com/sdk-api/experiments/experiments)
2 changes: 1 addition & 1 deletion examples/logging-samples/log-mcp-calls/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

This is an example project demonstrating how to use Splunk AO to log MCP server calls as tool spans. This code is described in the [Log MCP Server Tool Calls how-to guide](https://agent-observability-docs.splunk.com/how-to-guides/basics/log-mcp-server-calls/log-mcp-server-calls) in the Splunk AO documentation.

The MCP server used by default is the [Splunk AO MCP server](https://agent-observability-docs.splunk.com/getting-started/mcp/setup-galileo-mcp), but you can use this code with any MCP server that supports a streamable HTTP connection by changing the `MCP_SERVER_URL` environment variable.
The MCP server used by default is the [Splunk AO MCP server](https://agent-observability-docs.splunk.com/getting-started/mcp/setup-splunk-ao-mcp), but you can use this code with any MCP server that supports a streamable HTTP connection by changing the `MCP_SERVER_URL` environment variable.

## Getting Started

Expand Down
Loading