Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ Note: version releases in the 0.x.y range may introduce breaking changes.
- patch: Add filtering, timestamp explosion, insert/count/remove/update operations to vector store API. Add top-level vector_size config to VectorStoreConfig.
- patch: add csv table smoke tests
- patch: add manual release instructions
- patch: add streamming to the two first workflows
- patch: add streaming to the two first workflows
- patch: add support for cosmosdb output
- patch: create_communities streaming
- patch: create_final_documents streaming
Expand Down Expand Up @@ -138,7 +138,7 @@ Note: version releases in the 0.x.y range may introduce breaking changes.
- minor: Remove Dynamic Max Retries support. Refactor typer typing in cli interface
- minor: Update fnllm to latest. Update default graphrag configuration
- patch: A few fixes and enhancements for better reuse and flow.
- patch: Add full llm response to LLM PRovider output
- patch: Add full llm response to LLM Provider output
- patch: Fix Drift Reduce Response for non streaming calls
- patch: Fix global search prompt to include missing formatting key
- patch: Upgrade pyarrow dependency to >=17.0.0 to fix CVE-2024-52338
Expand All @@ -165,7 +165,7 @@ Note: version releases in the 0.x.y range may introduce breaking changes.
## 2.1.0

- minor: Add support for JSON input files.
- minor: Updated the prompt tunning client to support csv-metadata injection and updated output file types to match the new naming convention.
- minor: Updated the prompt tuning client to support csv-metadata injection and updated output file types to match the new naming convention.
- patch: Add check for custom model types while config loading
- patch: Adds general-purpose pipeline run state object.

Expand Down Expand Up @@ -271,7 +271,7 @@ Note: version releases in the 0.x.y range may introduce breaking changes.
## 0.5.0

- minor: Data model changes.
- patch: Add Parquet as part of the default emitters when not pressent
- patch: Add Parquet as part of the default emitters when not present
- patch: Centralized prompts and export all for easier injection.
- patch: Cleanup of artifact outputs/schemas.
- patch: Config and docs updates.
Expand Down Expand Up @@ -331,7 +331,7 @@ Note: version releases in the 0.x.y range may introduce breaking changes.
- patch: Optimize Create Base Documents subflow
- patch: Optimize text unit relationship count
- patch: Perf optimizations in map_query_to_entities()
- patch: Remove aggregate_df from final coomunities and final text units
- patch: Remove aggregate_df from final communities and final text units
- patch: Remove duplicated relationships and nodes
- patch: Remove unused column from final entities
- patch: Reorganized api,reporter,callback code into separate components. Defined debug profiles.
Expand Down Expand Up @@ -422,7 +422,7 @@ Note: version releases in the 0.x.y range may introduce breaking changes.
## 0.2.2

- patch: Add a check if there is no community record added in local search context
- patch: Add sepparate workflow for Python Tests
- patch: Add separate workflow for Python Tests
- patch: Docs updates
- patch: Run smoke tests on 4o

Expand Down
2 changes: 1 addition & 1 deletion DEVELOPING.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ or by simply running `azurite` in the terminal if already installed globally. Se

# Lifecycle Scripts

Our Python package utilizes uv to manage dependencies and [poethepoet](https://pypi.org/project/poethepoet/) to manage custom build scripts.
Our Python package uses uv to manage dependencies and [poethepoet](https://pypi.org/project/poethepoet/) to manage custom build scripts.

Available scripts are:
- `uv run poe index` - Run the Indexing CLI
Expand Down
44 changes: 22 additions & 22 deletions RAI_TRANSPARENCY.md
Original file line number Diff line number Diff line change
@@ -1,41 +1,41 @@
# GraphRAG: Responsible AI FAQ
# GraphRAG: Responsible AI FAQ

## What is GraphRAG?
## What is GraphRAG?

GraphRAG is an AI-based content interpretation and search capability. Using LLMs, it parses data to create a knowledge graph and answer user questions about a user-provided private dataset.
GraphRAG is an AI-based content interpretation and search capability. Using LLMs, it parses data to create a knowledge graph and answer user questions about a user-provided private dataset.

## What can GraphRAG do?
## What can GraphRAG do?

GraphRAG is able to connect information across large volumes of information and use these connections to answer questions that are difficult or impossible to answer using keyword and vector-based search mechanisms. Building on the previous question, provide semi-technical, high-level information on how the system offers functionality for various uses. This lets a system using GraphRAG to answer questions where the answers span many documents as well as thematic questions such as “what are the top themes in this dataset?.
GraphRAG can connect information across large volumes of data and use these connections to answer questions that are difficult or impossible to answer using keyword- and vector-based search mechanisms. This enables a system using GraphRAG to answer questions whose answers span many documents, as well as thematic questions such as “What are the top themes in this dataset?”

## What are GraphRAG’s intended use(s)?
## What are GraphRAG’s intended use(s)?

* GraphRAG is intended to support critical information discovery and analysis use cases where the information required to arrive at a useful insight spans many documents, is noisy, is mixed with mis and/or dis-information, or when the questions users aim to answer are more abstract or thematic than the underlying data can directly answer.
* GraphRAG is designed to be used in settings where users are already trained on responsible analytic approaches and critical reasoning is expected. GraphRAG is capable of providing high degrees of insight on complex information topics, however human analysis by a domain expert of the answers is needed in order to verify and augment GraphRAG’s generated responses.
* GraphRAG is intended to be deployed and used with a domain specific corpus of text data. GraphRAG itself does not collect user data, but users are encouraged to verify data privacy policies of the chosen LLM used to configure GraphRAG.
* GraphRAG is intended to support critical information discovery and analysis use cases where the information required to arrive at a useful insight spans many documents, is noisy, is mixed with misinformation or disinformation, or involves questions that are more abstract or thematic than the underlying data can directly answer.
* GraphRAG is designed for settings where users are trained in responsible analytic approaches and expected to apply critical reasoning. GraphRAG can provide deep insight into complex information topics; however, a domain expert should analyze its answers to verify and augment the generated responses.
* GraphRAG is intended to be deployed and used with a domain-specific corpus of text data. GraphRAG itself does not collect user data, but users are encouraged to verify the data privacy policies of the LLM used to configure GraphRAG.

## How was GraphRAG evaluated? What metrics are used to measure performance?
## How was GraphRAG evaluated? What metrics are used to measure performance?

GraphRAG has been evaluated in multiple ways. The primary concerns are 1) accurate representation of the data set, 2) providing transparency and groundedness of responses, 3) resilience to prompt and data corpus injection attacks, and 4) low hallucination rates. Details on how each of these has been evaluated is outlined below by number.
GraphRAG has been evaluated in multiple ways. The primary concerns are 1) accurate representation of the dataset, 2) transparency and groundedness of responses, 3) resilience to prompt and data corpus injection attacks, and 4) low hallucination rates. Details on how each concern was evaluated are outlined below.

1) Accurate representation of the dataset has been tested by both manual inspection and automated testing against a “gold answer” that is created from randomly selected subsets of a test corpus.
1) Accurate representation of the dataset has been tested through both manual inspection and automated testing against a “gold answer” created from randomly selected subsets of a test corpus.

2) Transparency and groundedness of responses is tested via automated answer coverage evaluation and human inspection of the underlying context returned.
2) Transparency and groundedness of responses are tested through automated answer coverage evaluation and human inspection of the returned context.

3) We test both user prompt injection attacks (“jailbreaks”) and cross prompt injection attacks (“data attacks”) using manual and semi-automated techniques.
3) We test both user prompt injection attacks (“jailbreaks”) and cross-prompt injection attacks (“data attacks”) using manual and semi-automated techniques.

4) Hallucination rates are evaluated using claim coverage metrics, manual inspection of answer and source, and adversarial attacks to attempt a forced hallucination through adversarial and exceptionally challenging datasets.
4) Hallucination rates are evaluated using claim coverage metrics, manual inspection of answers and sources, and attempts to force hallucinations through adversarial and exceptionally challenging datasets.

## What are the limitations of GraphRAG? How can users minimize the impact of GraphRAG’s limitations when using the system?
## What are the limitations of GraphRAG? How can users minimize the impact of GraphRAG’s limitations when using the system?

GraphRAG depends on a well-constructed indexing examples. For general applications (e.g. content oriented around people, places, organizations, things, etc.) we provide example indexing prompts. For unique datasets effective indexing can depend on proper identification of domain-specific concepts.
GraphRAG depends on well-constructed indexing prompts. For general applications (for example, content about people, places, organizations, or things), we provide example prompts. For unique datasets, effective indexing can depend on properly identifying domain-specific concepts.

Indexing is a relatively expensive operation; a best practice to mitigate indexing is to create a small test dataset in the target domain to ensure indexer performance prior to large indexing operations.
Indexing is a relatively expensive operation. To mitigate its cost, create a small test dataset in the target domain to evaluate indexer performance before running large indexing operations.

## What operational factors and settings allow for effective and responsible use of GraphRAG?
## What operational factors and settings allow for effective and responsible use of GraphRAG?

GraphRAG is designed for use by users with domain sophistication and experience working through difficult information challenges. While the approach is generally robust to injection attacks and identifying conflicting sources of information, the system is designed for trusted users. Proper human analysis of responses is important to generate reliable insights, and the provenance of information should be traced to ensure human agreement with the inferences made as part of the answer generation.
GraphRAG is designed for users with domain expertise and experience working through difficult information challenges. While the approach is generally robust to injection attacks and can identify conflicting sources of information, the system is designed for trusted users. Human analysis of responses is important for generating reliable insights, and information provenance should be traced to verify the inferences made during answer generation.

GraphRAG yields the most effective results on natural language text data that is collectively focused on an overall topic or theme, and that is entity rich entities being people, places, things, or objects that can be uniquely identified.
GraphRAG yields the most effective results on natural-language text data focused on an overall topic or theme and rich in identifiable entities such as people, places, or objects.

While GraphRAG has been evaluated for its resilience to prompt and data corpus injection attacks, and has been probed for specific types of harms, the LLM that the user configures with GraphRAG may produce inappropriate or offensive content, which may make it inappropriate to deploy for sensitive contexts without additional mitigations that are specific to the use case and model. Developers should assess outputs for their context and use available safety classifiers, model specific safety filters and features (such as https://azure.microsoft.com/en-us/products/ai-services/ai-content-safety), or custom solutions appropriate for their use case.
While GraphRAG has been evaluated for resilience to prompt and data corpus injection attacks and probed for specific types of harm, the configured LLM may produce inappropriate or offensive content. This may make GraphRAG unsuitable for sensitive contexts without additional mitigations specific to the use case and model. Developers should assess outputs for their context and use available safety classifiers, model-specific safety filters and features (such as [Azure AI Content Safety](https://azure.microsoft.com/en-us/products/ai-services/ai-content-safety)), or custom solutions appropriate for their use case.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ We strongly recommend to fine-tune your prompts following the [Prompt Tuning Gui

Please see the [breaking changes](./breaking-changes.md) document for notes on our approach to versioning the project.

*Always run `graphrag init --root [path] --force` between minor version bumps to ensure you have the latest config format. Run the provided migration notebook between major version bumps if you want to avoid re-indexing prior datasets. Note that this will overwrite your configuration and prompts, so backup if necessary.*
*Always run `graphrag init --root [path] --force` between minor version bumps to ensure you have the latest config format. Run the provided migration notebook between major version bumps if you want to avoid re-indexing prior datasets. Note that this will overwrite your configuration and prompts, so back them up if necessary.*

## Responsible AI FAQ

Expand Down
4 changes: 2 additions & 2 deletions breaking-changes.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@ There are five surface areas that may be impacted on any given release. They are
- [CLI](https://microsoft.github.io/graphrag/cli/) - The CLI is the interface most project consumers are using. **Changes to the CLI will conform to standard semver.**
- [API](https://github.com/microsoft/graphrag/tree/main/graphrag/api) - The API layer is the primary interface we expect developers to use if they are consuming the project as a library in their own codebases. **Changes to the API layer modules will conform to standard semver.**
- Internals - Any code modules behind the CLI and API layers are considered "internal" and may change at any time without conforming to strict semver. This is intended to give the research team high flexibility to change our underlying implementation rapidly. We are not enforcing access via tightly controlled `__init__.py` files, so please understand that if you utilize modules other than the index or query API, they may break between releases in a non-semver-compliant manner.
- [settings.yaml](https://microsoft.github.io/graphrag/config/yaml/) - The settings.yaml file may have changes made to it as we adjust configurability. **Changes that affect the settings.yml will result in a minor version bump**. `graphrag init` will always emit compatible starter config, so we recommend always running the command when updating GraphRAG between minor versions, and copying your endpoint information or other customizations over to the new file.
- [settings.yaml](https://microsoft.github.io/graphrag/config/yaml/) - The settings.yaml file may have changes made to it as we adjust configurability. **Changes that affect settings.yaml will result in a minor version bump**. `graphrag init` will always emit a compatible starter config, so we recommend running the command when updating GraphRAG between minor versions and copying your endpoint information or other customizations to the new file.
- [Data model](https://microsoft.github.io/graphrag/index/outputs/) - The output data model may change over time as we adjust our approach. **Changes to the data model will conform to standard semver.** Any changes to the output tables will be shimmed for backwards compatibility between major releases, and we'll provide a migration notebook for folks to upgrade without requiring a re-index.

> TL;DR: Always run `graphrag init --path [path] --force` between minor version bumps to ensure you have the latest config format. Run the provided migration notebook between major version bumps if you want to avoid re-indexing prior datasets. Note that this will overwrite your configuration and prompts, so backup if necessary.
> TL;DR: Always run `graphrag init --path [path] --force` between minor version bumps to ensure you have the latest config format. Run the provided migration notebook between major version bumps if you want to avoid re-indexing prior datasets. Note that this will overwrite your configuration and prompts, so back them up if necessary.

# v3
Run the [migration notebook](./docs/examples_notebooks/index_migration_to_v3.ipynb) to convert older tables to the v3 format. Our main goals with v3 were to slim down the core library to minimize long-term maintenance of features that are either largely unused or should have been out of scope for a long time anyway.
Expand Down
1 change: 1 addition & 0 deletions cspell.config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ ignorePaths:
- cspell.config.yaml
- node_modules
- _site
- site/
- /project-words.txt
- default_pipeline.yml
- .turbo
Expand Down
2 changes: 1 addition & 1 deletion docs/developing.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ or by simply running `azurite` in the terminal if already installed globally. Se

# Lifecycle Scripts

Our Python package utilize uv to manage dependencies and [poethepoet](https://pypi.org/project/poethepoet/) to manage build scripts.
Our Python package uses uv to manage dependencies and [poethepoet](https://pypi.org/project/poethepoet/) to manage build scripts.

Available scripts are:

Expand Down
2 changes: 1 addition & 1 deletion docs/examples_notebooks/api_overview.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"The response object is the official reponse from graphrag while the context object holds various metadata regarding the querying process used to obtain the final response.\n"
"The response object contains GraphRAG's response, while the context object holds metadata about the querying process used to obtain the final response."
]
},
{
Expand Down
Loading
Loading