First pass at docs (#19)

General first pass at the documentation

Author: Sam Partee

Makefile

@@ -58,7 +58,7 @@ mypy:
 # help: docs                           - generate project documentation
 .PHONY: docs
 docs:
-	@cd doc; make html
+	@cd docs; make html
 
 # help:
 # help: Test

README.md

@@ -1,100 +1,128 @@
-# RedisVL
+# RedisVL: Python Client Library for Redis as a Vector Database
 
-> DISCLAIMER: This project is still under signifigant development and should not be used in any production settings. We would love input/contributions as we finalize what the CLI and library interfaces should look like.
 
-A CLI and Library to help with loading data into Redis specifically for
-usage with RediSearch and Redis Vector Search capabilities
+[![Codecov](https://img.shields.io/codecov/c/github/RedisVentures/RedisVL/dev?label=Codecov&logo=codecov&token=E30WxqBeJJ)](https://codecov.io/gh/RedisVentures/RedisVL)
+[![License](https://img.shields.io/badge/License-BSD-3--blue.svg)](https://opensource.org/licenses/mit/)
 
-### Usage
 
-```
-usage: redisvl <command> [<args>]
+RedisVL provides a powerful Python client library for using Redis as a Vector Database. Leverage the speed and reliability of Redis along with vector-based semantic search capabilities to supercharge your application!
 
-Commands:
-        load        Load vector data into redis
-        index       Index manipulation (create, delete, etc.)
-        query       Query an existing index
+**Note:** This project is rapidly evolving, and the API may change frequently. Always refer to the most recent [documentation](https://redisvl.com/docs).
+## 🚀 What is RedisVL?
 
-Redis Vector load CLI
+Vector databases have become increasingly popular in recent years due to their ability to store and retrieve vectors efficiently. However, most vector databases are complex to use and require a lot of time and effort to set up. RedisVL aims to solve this problem by providing a simple and intuitive interface for using Redis as a vector database.
 
-positional arguments:
-  command     Subcommand to run
+RedisVL provides a client library that enables you to harness the power of Redis as a vector database. This library simplifies the process of storing, retrieving, and performing semantic searches on vectors in Redis. It also provides a robust index management system that allows you to create, update, and delete indices with ease.
 
-optional arguments:
-  -h, --help  show this help message and exit
 
-```
+### Capabilities
+
+RedisVL has a host of powerful features designed to streamline your vector database operations.
+
+1. **Index Management**: RedisVL allows for indices to be created, updated, and deleted with ease. A schema for each index can be defined in yaml or directly in python code and used throughout the lifetime of the index.
+
+2. **Vector Creation**: RedisVL integrates with OpenAI and other embedding providers to make the process of creating vectors straightforward.
+
+3. **Vector Search**: RedisVL provides robust search capabilities that enable you to query vectors synchronously and asynchronously. Hybrid queries that utilize tag, geographic, numeric, and other filters like full-text search are also supported.
+
+4. **Semantic Caching**: ``LLMCache`` is a semantic caching interface built directly into RedisVL. It allows for the caching of generated output from LLM models like GPT-3 and others. As semantic search is used to check the cache, a threshold can be set to determine if the cached result is relevant enough to be returned. If not, the model is called and the result is cached for future use. This can increase the QPS and reduce the cost of using LLM models.
+
+
+## 😊 Quick Start
 
-For any of the above commands, you will need to have an index schema written
-into a yaml file for the cli to read. The format of the schema is as follows
+Please note that this library is still under heavy development, and while you can quickly try RedisVL and deploy it in a production environment, the API may be subject to change at any time.
+
+`pip install redisvl`
+
+## Example Usage
+
+### Index Management
+
+Indices can be defined through yaml specification that corresponds directly to the RediSearch field names and arguments in redis-py
 
 ```yaml
 index:
-  name: sample # index name used for querying
+  name: users
   storage_type: hash
-  key_field: "id" # column name to use for key in redis
-  prefix: vector  # prefix used for all loaded docs
+  prefix: "user:"
+  key_field: "id"
 
-# all fields to create index with
-# sub-items correspond to redis-py Field arguments
 fields:
+  # define tag fields
   tag:
-    categories: # name of a tag field used for queries
-      separator: "|"
-    year: # name of a tag field used for queries
-      separator: "|"
+  - name: users
+  - name: job
+  - name: credit_store
+  # define numeric fields
+  numeric:
+  - name: age
+  # define vector fields
   vector:
-    vector: # name of the vector field used for queries
-      datatype: "float32"
-      algorithm: "flat" # flat or HSNW
-      dims: 768
-      distance_metric: "cosine" # ip, L2, cosine
+  - name: user_embedding
+    algorithm: hnsw
+    distance_metric: cosine
 ```
 
-#### Example Usage
-These examples reference [provided sample data](sample-data/).
+This would correspond to a dataset that looked something like
 
-```bash
-# load in a pickled dataframe with
-redisvl load -s sample-data/sample.yml -d sample-data/pandas-sample.pkl
-```
+| users | age |     job    | credit_score |           user_embedding          |
+|-------|-----|------------|--------------|-----------------------------------|
+| john  |  1  |  engineer  |     high     | \x3f\x8c\xcc\x3f\x8c\xcc?@         |
+| mary  |  2  |   doctor   |     low      | \x3f\x8c\xcc\x3f\x8c\xcc?@         |
+|  joe  |  3  |  dentist   |    medium    | \x3f\xab\xcc?\xab\xcc?@         |
 
-```bash
-# load in a pickled dataframe to a specific address and port
-redisvl load -s sample-data/sample.yml -d sample-data/pandas-sample.pkl -h 127.0.0.1 -p 6379
-```
 
-```bash
-# load in a pickled dataframe to a specific
-# address and port and with password
-redisvl load -s sample-data/sample.yml -d sample-data/pandas-sample.pkl -h 127.0.0.1 -p 6379 -p supersecret
-```
+With the schema, the RedisVL library can be used to create, load vectors and perform vector searches
+```python
+from redisvl.index import SearchIndex
+from redisvl.query import create_vector_query
 
-### Support
+# define and create the index
+index = SearchIndex.from_yaml("./users_schema.yml"))
+index.connect("redis://localhost:6379")
+index.create()
 
-#### Supported Index Fields
+index.load(pd.read_csv("./users.csv").to_records())
 
-  - ``geo``
-  - ``tag``
-  - ``numeric``
-  - ``vector``
-  - ``text``
-#### Supported Data Types
- - Pandas DataFrame (pickled)
-#### Supported Redis Data Types
- - Hash
- - JSON (soon)
+query = create_vector_query(
+    ["users", "age", "job", "credit_score"],
+    number_of_results=2,
+    vector_field_name="user_embedding",
+)
 
-### Install
-Install the Python requirements listed in `requirements.txt`.
+query_vector = np.array([0.1, 0.1, 0.5]).tobytes()
+results = index.search(query, query_params={"vector": query_vector})
+
+```
 
-```bash
-git clone https://github.com/RedisVentures/data-loader.git
-cd redisvl
-pip install .
+### Semantic cache
+
+The ``LLMCache`` Interface in RedisVL can be used as follows.
+
+```python
+# init open ai client
+import openai
+openai.api_key = "sk-xxx"
+
+from redisvl.llmcache.semantic import SemanticCache
+cache = SemanticCache(redis_host="localhost", redis_port=6379, redis_password=None)
+
+def ask_gpt3(question):
+    response = openai.Completion.create(
+      engine="text-davinci-003",
+      prompt=question,
+      max_tokens=100
+    )
+    return response.choices[0].text.strip()
+
+def answer_question(question: str):
+    results = cache.check(question)
+    if results:
+        return results[0]
+    else:
+        answer = ask_gpt3(question)
+        cache.store(question, answer)
+        return answer
 ```
 
-### Creating Input Data
-#### Pandas DataFrame
 
-  more to come, see tests and sample-data for usage

docs/Makefile

@@ -0,0 +1,22 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+
+	# build docs
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/_extension/gallery_directive.py

@@ -0,0 +1,161 @@
+"""A directive to generate a gallery of images from structured data.
+
+Generating a gallery of images that are all the same size is a common
+pattern in documentation, and this can be cumbersome if the gallery is
+generated programmatically. This directive wraps this particular use-case
+in a helper-directive to generate it with a single YAML configuration file.
+
+It currently exists for maintainers of the pydata-sphinx-theme,
+but might be abstracted into a standalone package if it proves useful.
+"""
+from pathlib import Path
+from typing import Any, Dict, List
+
+from docutils import nodes
+from docutils.parsers.rst import directives
+from sphinx.application import Sphinx
+from sphinx.util import logging
+from sphinx.util.docutils import SphinxDirective
+from yaml import safe_load
+
+logger = logging.getLogger(__name__)
+
+
+TEMPLATE_GRID = """
+`````{{grid}} {grid_columns}
+{container_options}
+
+{content}
+
+`````
+"""
+
+GRID_CARD = """
+````{{grid-item-card}} {title}
+{card_options}
+
+{content}
+````
+"""
+
+
+class GalleryDirective(SphinxDirective):
+    """A directive to show a gallery of images and links in a grid."""
+
+    name = "gallery-grid"
+    has_content = True
+    required_arguments = 0
+    optional_arguments = 1
+    final_argument_whitespace = True
+    option_spec = {
+        # A class to be added to the resulting container
+        "grid-columns": directives.unchanged,
+        "class-container": directives.unchanged,
+        "class-card": directives.unchanged,
+    }
+
+    def run(self) -> List[nodes.Node]:
+        if self.arguments:
+            # If an argument is given, assume it's a path to a YAML file
+            # Parse it and load it into the directive content
+            path_data_rel = Path(self.arguments[0])
+            path_doc, _ = self.get_source_info()
+            path_doc = Path(path_doc).parent
+            path_data = (path_doc / path_data_rel).resolve()
+            if not path_data.exists():
+                logger.warn(f"Could not find grid data at {path_data}.")
+                nodes.text("No grid data found at {path_data}.")
+                return
+            yaml_string = path_data.read_text()
+        else:
+            yaml_string = "\n".join(self.content)
+
+        # Read in YAML so we can generate the gallery
+        grid_data = safe_load(yaml_string)
+
+        grid_items = []
+        for item in grid_data:
+            # Grid card parameters
+            options = {}
+            if "website" in item:
+                options["link"] = item["website"]
+
+            if "class-card" in self.options:
+                options["class-card"] = self.options["class-card"]
+
+            if "img-background" in item:
+                options["img-background"] = item["img-background"]
+
+            if "img-top" in item:
+                options["img-top"] = item["img-top"]
+
+            if "img-bottom" in item:
+                options["img-bottom"] = item["img-bottom"]
+
+            options_str = "\n".join(f":{k}: {v}" for k, v in options.items()) + "\n\n"
+
+            # Grid card content
+            content_str = ""
+            if "header" in item:
+                content_str += f"{item['header']}\n\n^^^\n\n"
+
+            if "image" in item:
+                content_str += f"![Gallery image]({item['image']})\n\n"
+
+            if "content" in item:
+                content_str += f"{item['content']}\n\n"
+
+            if "footer" in item:
+                content_str += f"+++\n\n{item['footer']}\n\n"
+
+            title = item.get("title", "")
+            content_str += "\n"
+            grid_items.append(
+                GRID_CARD.format(
+                    card_options=options_str, content=content_str, title=title
+                )
+            )
+
+        # Parse the template with Sphinx Design to create an output
+        container = nodes.container()
+        # Prep the options for the template grid
+        container_options = {"gutter": 2, "class-container": "gallery-directive"}
+        if "class-container" in self.options:
+            container_options[
+                "class-container"
+            ] += f' {self.options["class-container"]}'
+        container_options_str = "\n".join(
+            f":{k}: {v}" for k, v in container_options.items()
+        )
+
+        # Create the directive string for the grid
+        grid_directive = TEMPLATE_GRID.format(
+            grid_columns=self.options.get("grid-columns", "1 2 3 4"),
+            container_options=container_options_str,
+            content="\n".join(grid_items),
+        )
+        # Parse content as a directive so Sphinx Design processes it
+        self.state.nested_parse([grid_directive], 0, container)
+        # Sphinx Design outputs a container too, so just use that
+        container = container.children[0]
+
+        # Add extra classes
+        if self.options.get("container-class", []):
+            container.attributes["classes"] += self.options.get("class", [])
+        return [container]
+
+
+def setup(app: Sphinx) -> Dict[str, Any]:
+    """Add custom configuration to sphinx app.
+
+    Args:
+        app: the Sphinx application
+    Returns:
+        the 2 parallel parameters set to ``True``.
+    """
+    app.add_directive("gallery-grid", GalleryDirective)
+
+    return {
+        "parallel_read_safe": True,
+        "parallel_write_safe": True,
+    }