Skip to content

Commit d50cde4

Browse files
raffaemraffaem
committed
Improve setup process
1 parent 44c077b commit d50cde4

File tree

10 files changed

+637
-95
lines changed

10 files changed

+637
-95
lines changed

‎.gitignore‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
*.bin
2+
*.onnx
3+
*.log
4+
__pycache__/
5+
/chroma_db
6+
vcvarsall.sh
7+
search_context.txt

‎README.md‎

Lines changed: 32 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -105,19 +105,43 @@ Example: I converted [this article](https://www.theatlantic.com/newsletters/arch
105105

106106
> The script will:
107107
> - Pull the SearxNG Docker image
108-
> - Create and activate a Python virtual environment
109-
> - **USER ACTION NEEDED** Set your `GOOGLE_API_KEY` (edit the script to use your real key). [Obtain your API key (Currently Gemini, OpenAI and ollama is supported)](https://ai.google.dev/gemini-api/docs/api-key) from your preferred LLM provider. (Only needed when google mode is set, else set in model_config.py)
110-
> - Start the SearxNG Docker container
108+
> - Install [infinity](https://github.com/michaelfeil/infinity) in a dedicated Python environment `infinity_env`
109+
> - Create a Python virtual environment for CoexistAI in `coexist_env`
111110
> - Install Python dependencies
112-
> - Start the FastAPI server
113111

114-
4. **That’s it!**
115-
The FastAPI and MCP server will start automatically and you’re ready to go.
112+
4. **Run!**
113+
114+
Call `zsh quick_start.sh` or `bash quick_start.sh` to start the FastAPI server and MCP server.
116115

117116
**Note:**
118117
- Make sure Docker, Python 3, and pip are installed on your system.
119-
- Edit quick_setup.sh to set your real `GOOGLE_API_KEY` before running (needed if using google models)
120-
- Windows users can use [WSL](https://docs.microsoft.com/en-us/windows/wsl/) or Git Bash to run the script, or follow manual setup steps.
118+
- Edit `quick_start.sh` to set your real `GOOGLE_API_KEY` before running (needed if using google models)
119+
120+
121+
### Windows user
122+
123+
On Windows:
124+
125+
1. Run a Valkey container with instructions found [here](https://github.com/valkey-io/valkey-py)
126+
127+
```
128+
docker run -p 6379:6379 -it valkey/valkey:latest
129+
```
130+
131+
2. Run a SearXNG instance with the instructions [here](https://www.tanyongsheng.com/note/setting-up-searxng-on-windows-localhost-your-private-customizable-search-engine/).
132+
Make sure to enable JSON output in SearXNG (otherwise you will get a 403 Forbidden error each time).
133+
Make sure the instance work by visiting [this](http://localhost:8080/search?q=When%20was%20Napoleon%20born?&engines=google,brave&format=json) (you may need to change the port).
134+
135+
3. In `model_config.py`, set `START_SEARXNG = 0` (since we are running our own instance) and the correct port in `PORT_NUM_SEARXNG`
136+
137+
4. To install, run the following in Git Bash:
138+
```
139+
wget -q https://raw.githubusercontent.com/nathan818fr/vcvars-bash/refs/heads/main/vcvarsall.sh -O vcvarsall.sh
140+
eval "$(./vcvarsall.sh x64)"
141+
bash quick_setup.sh
142+
```
143+
144+
5. To run, run `bash quick_start.sh` in Git Bash
121145
122146
---
123147

‎model_config.py‎

Lines changed: 30 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import os
2+
23
"""
34
This module defines the configuration for language model (LLM) and embedding models.
45
Attributes:
@@ -10,7 +11,6 @@
1011
for ollama and other local models use "others" with base_url updated in openai_compatible.
1112
- If you using others llm type, then check the openai_compatible url dict for others key, you can generally
1213
find it by "googling YOUR provider name openai api base compatilble url"
13-
- "llm_base_url" (str): Base URL for the LLM API endpoint.
1414
- "llm_tools" (list): List of tools or plugins to use with the LLM.
1515
- "llm_kwargs" (dict): Additional keyword arguments for LLM initialization.
1616
- "temperature" (float): Sampling temperature for generation.
@@ -23,73 +23,72 @@
2323
- "cross_encoder_name" (str): Name of the cross-encoder model for reranking.
2424
"""
2525
############## PORT and HOST SETTINGS
26-
PORT_NUM_SEARXNG = 8085
26+
PORT_NUM_SEARXNG = 8080
2727
PORT_NUM_APP = 8000
2828
HOST_APP = "localhost"
2929
HOST_SEARXNG = "localhost"
30+
START_SEARXNG = 0
3031
###############
3132

3233
## USER INPUTS NEEDED
33-
#for open source model you can replace it by 'DUMMY' (for both llm and embed), else respective providers
34-
llm_api_key = os.environ.get('GOOGLE_API_KEY', 'DUMMY') # either paste llm key, based on provider (for an instance, Google) here directly or export it in the env, else dummy for local
35-
embed_api_key = os.environ.get('GOOGLE_API_KEY', 'DUMMY') # either paste embeder key, based on provider (for an instance, Google) here directly or export it in the env, else dummy for local
34+
# for open source model you can replace it by 'DUMMY' (for both llm and embed), else respective providers
35+
llm_api_key = os.environ.get(
36+
"GOOGLE_API_KEY", "DUMMY"
37+
) # either paste llm key, based on provider (for an instance, Google) here directly or export it in the env, else dummy for local
38+
embed_api_key = os.environ.get(
39+
"GOOGLE_API_KEY", "DUMMY"
40+
) # either paste embeder key, based on provider (for an instance, Google) here directly or export it in the env, else dummy for local
3641

3742
model_config = {
3843
# Name of the LLM model to use. For local models, use the model name served by your local server.
39-
"llm_model_name": "gemini-2.0-flash",
40-
41-
# LLM provider type: choose from 'google', 'local', 'groq', or 'openai' or 'others'
42-
# in case of 'others' (base url needs to be updated in openai_compatible given below accordingly).
44+
"llm_model_name": "yasserrmd/jan-nano-4b",
45+
# LLM provider type: choose from 'google', 'local', 'groq', or 'openai' or 'others'
46+
# in case of 'others' (base url needs to be updated in the `openai_compatible` dictionary below).
4347
# Make sure to update the api_key variable above to match the provider.
4448
# "local" is for lmstudio, for ollama and other local models use "others" with base_url updated in openai_compatible.
4549
# You can generally find it by "googling YOUR PROVIDER (example ollama) name openai api base compatible url"
46-
"llm_type": "google",
47-
50+
"llm_type": "others",
4851
# List of tools or plugins to use with the LLM, if any. Set to None if not used.
4952
"llm_tools": None,
50-
5153
# Additional keyword arguments for LLM initialization.
5254
"llm_kwargs": {
5355
"temperature": 0.1, # Sampling temperature for generation.
5456
"max_tokens": None, # Maximum number of tokens to generate (None for default).
55-
"timeout": None, # Timeout for API requests (None for default).
56-
"max_retries": 2, # Maximum number of retries for failed requests.
57+
"timeout": None, # Timeout for API requests (None for default).
58+
"max_retries": 2, # Maximum number of retries for failed requests.
5759
"api_key": llm_api_key, # API key for authentication.
5860
},
59-
6061
# Name of the embedding model to use.
6162
# For Google, use their embedding model names. For local/HuggingFace, use the model path or name.
62-
"embedding_model_name": "models/embedding-001",
63-
64-
"embed_kwargs":{"google_api_key": embed_api_key}, #optional additional kwargs for embedding model initialization
65-
63+
# Tested models can be found at https://github.com/michaelfeil/infinity?tab=readme-ov-file#supported-tasks-and-models-by-infinity
64+
"embedding_model_name": "mixedbread-ai/mxbai-embed-large-v1",
65+
"embed_kwargs": {}, # optional additional kwargs for embedding model initialization
6666
# Embedding backend: 'google' for Google, 'infinity_emb' for local/HuggingFace models.
67-
"embed_mode": "google",
68-
67+
"embed_mode": "infinity_emb",
6968
# Name of the cross-encoder model for reranking, typically a HuggingFace model.
70-
"cross_encoder_name": "BAAI/bge-reranker-base"
69+
"cross_encoder_name": "BAAI/bge-reranker-base",
7170
}
7271

7372

7473
# NO CHANGE NEEDED UNLESS PROVIDER CHANGES THE BASE URLS, OR YOU WANT TO USE DIFFERENT PROVIDER UNDER "others"
7574
openai_compatible = {
76-
'google': "https://generativelanguage.googleapis.com/v1beta/openai/",
77-
'local': "http://127.0.0.1:1234/v1",
78-
'groq': 'https://api.groq.com/openai/v1',
79-
'openai':'https://api.openai.com/v1',
80-
'others': 'https://openrouter.ai/api/v1' # for an example I have added here the openrouter api, since its openai compatible
75+
"google": "https://generativelanguage.googleapis.com/v1beta/openai/",
76+
"local": "http://127.0.0.1:1234/v1",
77+
"groq": "https://api.groq.com/openai/v1",
78+
"openai": "https://api.openai.com/v1",
79+
"others": "http://localhost:11434/v1", # Ollama default port
8180
}
8281

83-
#####IF YOU WANT TO GO ALL LOCAL
82+
#####IF YOU WANT TO GO ALL LOCAL
8483

8584
# model_config = {
8685
# # Name of the LLM model to use. For local models, use the model name served by your local server.
8786
# "llm_model_name": "google/gemma-3-12b",
8887

89-
# # LLM provider type: choose from 'google', 'local', 'groq', or 'openai' or 'others'
88+
# # LLM provider type: choose from 'google', 'local', 'groq', or 'openai' or 'others'
9089
# # in case of 'others' (base url needs to be updated in openai_compatible given below accordingly).
9190
# # Make sure to update the api_key variable above to match the provider.
92-
# "llm_type": "local",
91+
# "llm_type": "local",
9392

9493
# # List of tools or plugins to use with the LLM, if any. Set to None if not used.
9594
# "llm_tools": None,
@@ -114,4 +113,4 @@
114113

115114
# # Name of the cross-encoder model for reranking, typically a HuggingFace model.
116115
# "cross_encoder_name": "BAAI/bge-reranker-base"
117-
# }
116+
# }

‎quick_setup.sh‎

Lines changed: 63 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -1,62 +1,80 @@
1-
#!/bin/zsh
2-
# Quick Shell Setup for CoexistAI (macOS/zsh)
3-
4-
echo "Pulling SearxNG Docker image..."
5-
docker pull searxng/searxng
1+
#!/bin/sh
2+
# Quick Shell Setup for CoexistAI (linux/sh)
63

4+
# Install infinity in its own virtual environment
75
echo "📚 Installing infinity_emb in separate environment..."
8-
echo "Creating and activating Python virtual environment..."
9-
python3.13 -m venv infinity_env
10-
source infinity_env/bin/activate
11-
pip install 'infinity_emb[all]'
12-
pip install --upgrade "transformers<4.49"
13-
pip install --upgrade "typer==0.19.1" "click>=8.1.3"
14-
deactivate
15-
echo "✅ Infinity environment setup complete"
166

17-
# (Optional) Create and activate a Python virtual environment
18-
echo "Creating and activating Python virtual environment..."
19-
python3.13 -m venv coexistaienv
20-
source coexistaienv/bin/activate
7+
echo "Remove old infinity virtual environment"
8+
rm -rf infinity_env
219

22-
pip install 'markitdown[all]'
10+
echo "Creating infinity virtual environment"
11+
python3 -m venv infinity_env
2312

24-
# You can neglect this if you dont want to use google models (either llm or embedding)
25-
echo "Setting GOOGLE_API_KEY, add any other keys which you want to store in environment (edit this script to use your real key)"
26-
export GOOGLE_API_KEY=REPLACE_YOUR_API_KEY_HERE_WITHOUT_QUOTES_AND_SPACES
13+
echo "Activating infinity virtual environment"
14+
if [ -d infinity_env/Scripts ]; then
15+
ENV_DIR=infinity_env/Scripts
16+
else
17+
ENV_DIR=infinity_env/bin
18+
fi
19+
source $ENV_DIR/activate
20+
if [ $? -ne 0 ]; then
21+
echo "Error activating infinity environment"
22+
return 1
23+
fi
2724

25+
echo "Installing infinity"
26+
python3 -m pip install 'infinity_emb[all]'
27+
if [ $? -ne 0 ]; then
28+
echo "Error installing infinity in the environment"
29+
return 1
30+
fi
31+
python -m pip install --upgrade "transformers<4.49"
32+
python -m pip install --upgrade "typer==0.19.1" "click>=8.1.3"
2833

29-
# Spin up the SearxNG Docker container
30-
echo "Starting SearxNG Docker container..."
31-
PORT_NUM_SEARXNG=$(python3.13 -c "from model_config import PORT_NUM_SEARXNG; print(PORT_NUM_SEARXNG)")
32-
HOST_SEARXNG=$(python3.13 -c "from model_config import HOST_SEARXNG; print(HOST_SEARXNG)")
34+
echo "Deactivating infinity virtual environment"
35+
deactivate
36+
echo "✅ Infinity environment setup complete"
3337

34-
# Stop and remove existing searxng container if it exists
35-
if [ "$(docker ps -aq -f name=searxng)" ]; then
36-
echo "Stopping and removing existing SearxNG container..."
37-
docker stop searxng 2>/dev/null || true
38-
docker rm searxng 2>/dev/null || true
39-
fi
38+
# (Optional) Create and activate a Python virtual environment
39+
echo "Remove old coexistai environment"
40+
rm -rf coexistaienv
4041

41-
# Start new SearxNG container
42-
docker run -d \
43-
--name searxng \
44-
-p ${PORT_NUM_SEARXNG}:8080 \
45-
-v $(pwd)/searxng:/etc/searxng:rw \
46-
-e SEARXNG_BASE_URL=http://${HOST_SEARXNG}:${PORT_NUM_SEARXNG}/ \
47-
-e SEARXNG_PORT=${PORT_NUM_SEARXNG} \
48-
-e SEARXNG_BIND_ADDRESS=${HOST_SEARXNG} \
49-
--restart unless-stopped \
50-
searxng/searxng:latest
42+
echo "Creating coexistai virtual environment..."
43+
python -m venv coexistaienv
5144

52-
echo "SearxNG container started successfully!"
45+
echo "Activating coexistai virtual environment"
46+
if [ -d coexistaienv/Scripts ]; then
47+
ENV_DIR=coexistaienv/Scripts
48+
else
49+
ENV_DIR=coexistaienv/bin
50+
fi
51+
source $ENV_DIR/activate
52+
if [ $? -ne 0 ]; then
53+
echo "Error activating coexistai virtual environment"
54+
return 1
55+
fi
5356

5457
# Install Python dependencies
55-
echo "Installing Python dependencies..."
56-
pip install -r ./requirements.txt
58+
echo "Installing Python dependencies in coexistai virtual environment"
59+
python -m pip install -r ./requirements.txt
60+
61+
# Installing SearXNG
62+
START_SEARXNG=$(python -c "from model_config import START_SEARXNG; print(START_SEARXNG)")
63+
if [ $START_SEARXNG == 0 ]; then
64+
echo "Skipping SearxNG startup as per configuration"
65+
elif [ $START_SEARXNG == 1 ]; then
66+
echo "Pulling SearxNG Docker image..."
67+
docker pull searxng/searxng
68+
else
69+
echo "Invalid value for START_SEARXNG in model_config.py. Use 0 or 1."
70+
exit 1
71+
fi
72+
73+
# Deactivate coexistai virtual environment
74+
echo "Deactivating coexistai virtual environment"
75+
deactivate
5776

5877
# Adding tts files
59-
# Check if wget is installed
6078
# Check if wget or curl is installed
6179
if command -v wget &> /dev/null; then
6280
DOWNLOADER_CMD="wget"
@@ -82,12 +100,3 @@ if [ ! -f voices-v1.0.bin ]; then
82100
else
83101
echo "voices-v1.0.bin already exists, skipping download."
84102
fi
85-
86-
87-
# 8. Start the FastAPI app
88-
echo "Starting FastAPI app..."
89-
cd . || exit 1
90-
# Get port and host values from model_config
91-
PORT_NUM_APP=$(python3.13 -c "from model_config import PORT_NUM_APP; print(PORT_NUM_APP)")
92-
HOST_APP=$(python3.13 -c "from model_config import HOST_APP; print(HOST_APP)")
93-
uvicorn app:app --host ${HOST_APP} --port ${PORT_NUM_APP} --reload

‎quick_start.sh‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
#!/bin/sh
2+
# Quick Shell Startup for CoexistAI (linux/sh)
3+
4+
# For Git bash run
5+
#export PATH=$(pwd)/infinity_env/Scripts:$PATH
6+
7+
# (Optional) Activate a Python virtual environment
8+
echo "Activating Python virtual environment..."
9+
if [ -d coexistaienv/Scripts ]; then
10+
ENV_DIR=coexistaienv/Scripts
11+
else
12+
ENV_DIR=coexistaienv/bin
13+
fi
14+
source $ENV_DIR/activate
15+
if [ $? -ne 0 ]; then
16+
echo "Error activating coexistai environment"
17+
return 1
18+
fi
19+
20+
# You can neglect this if you dont want to use google models (either llm or embedding)
21+
echo "Setting GOOGLE_API_KEY, add any other keys which you want to store in environment (edit this script to use your real key)"
22+
export GOOGLE_API_KEY=REPLACE_YOUR_API_KEY_HERE_WITHOUT_QUOTES_AND_SPACES
23+
24+
# Spin up the SearxNG Docker container
25+
START_SEARXNG=$(python -c "from model_config import START_SEARXNG; print(START_SEARXNG)")
26+
if [ $START_SEARXNG == 0 ]; then
27+
echo "Skipping SearxNG startup as per configuration"
28+
elif [ $START_SEARXNG == 1 ]; then
29+
echo "Starting SearxNG Docker container..."
30+
PORT_NUM_SEARXNG=$(python -c "from model_config import PORT_NUM_SEARXNG; print(PORT_NUM_SEARXNG)")
31+
HOST_SEARXNG=$(python -c "from model_config import HOST_SEARXNG; print(HOST_SEARXNG)")
32+
33+
# Stop and remove existing searxng container if it exists
34+
if [ "$(docker ps -aq -f name=searxng)" ]; then
35+
echo "Stopping and removing existing SearxNG container..."
36+
docker stop searxng 2>/dev/null || true
37+
docker rm searxng 2>/dev/null || true
38+
fi
39+
40+
# Start new SearxNG container
41+
docker run -d \
42+
--name searxng \
43+
-p ${PORT_NUM_SEARXNG}:8080 \
44+
-v $(pwd)/searxng:/etc/searxng:rw \
45+
-e SEARXNG_BASE_URL=http://${HOST_SEARXNG}:${PORT_NUM_SEARXNG}/ \
46+
-e SEARXNG_PORT=${PORT_NUM_SEARXNG} \
47+
-e SEARXNG_BIND_ADDRESS=${HOST_SEARXNG} \
48+
--restart unless-stopped \
49+
searxng/searxng:latest
50+
echo "SearxNG container started successfully!"
51+
else
52+
echo "Invalid value for START_SEARXNG in model_config.py. Use 0 or 1."
53+
exit 1
54+
fi
55+
56+
# Start the FastAPI app
57+
echo "Starting FastAPI app..."
58+
cd . || exit 1
59+
# Get port and host values from model_config
60+
PORT_NUM_APP=$(python -c "from model_config import PORT_NUM_APP; print(PORT_NUM_APP)")
61+
HOST_APP=$(python -c "from model_config import HOST_APP; print(HOST_APP)")
62+
python -m uvicorn app:app --host ${HOST_APP} --port ${PORT_NUM_APP} --reload

‎quick_start_infinity.sh‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
#!/bin/sh
2+
# Quick Shell Startup for CoexistAI (linux/sh)
3+
4+
INFINITY_MODEL=$(python -c "from model_config import model_config; print(model_config['embedding_model_name'])")
5+
./infinity_env/Scripts/infinity_emb v2 --model-id $INFINITY_MODEL

‎requirements.txt‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,4 +39,5 @@ youtube_transcript_api>=1.1.0
3939
slowapi
4040
charset-normalizer
4141
kokoro-onnx
42-
soundfile
42+
soundfile
43+
markitdown[all]

0 commit comments

Comments
 (0)