diff --git a/Basics/Open_Source_Guides/README.md b/Basics/Open_Source_Guides/README.md new file mode 100644 index 00000000..481f2c89 --- /dev/null +++ b/Basics/Open_Source_Guides/README.md @@ -0,0 +1,65 @@ +# Open Source Guides + +Open source software is built by people who collaborate, share knowledge, and improve projects together. If you are new to open source, the guides below are a useful starting point. + +These resources are based on the [Open Source Guides](https://opensource.guide/) maintained by GitHub and the open source community. + +## Getting Started + +- [How to Contribute to Open Source](https://opensource.guide/how-to-contribute/) - Learn how to find projects, choose a contribution, and submit changes. +- [Starting an Open Source Project](https://opensource.guide/starting-a-project/) - Learn the basics of launching an open source project. +- [Finding Users for Your Project](https://opensource.guide/finding-users/) - Ideas for helping people discover and use your project. + +## Building a Healthy Project + +- [Building Welcoming Communities](https://opensource.guide/building-community/) - Learn how to create an inclusive and welcoming community. +- [Best Practices for Maintainers](https://opensource.guide/best-practices/) - Practical guidance for maintaining an open source project. +- [Your Code of Conduct](https://opensource.guide/code-of-conduct/) - Understand the role of a code of conduct in a project community. +- [Leadership and Governance](https://opensource.guide/leadership-and-governance/) - Learn how projects can make decisions and establish governance. +- [Maintaining Balance for Open Source Maintainers](https://opensource.guide/maintaining-balance-for-open-source/) - Tips for sustainable open source maintenance. + +## Security, Accessibility, and Sustainability + +- [Accessibility Best Practices for Your Project](https://opensource.guide/accessibility-best-practices-for-your-project/) - Practical steps for making projects more accessible. +- [Security Best Practices for Your Project](https://opensource.guide/security-best-practices-for-your-project/) - Learn about security practices that help protect open source projects. +- [Open Source Metrics](https://opensource.guide/metrics/) - Use project metrics to understand activity and growth. +- [Getting Paid for Open Source Work](https://opensource.guide/getting-paid/) - Explore ways to make open source work sustainable. +- [The Legal Side of Open Source](https://opensource.guide/legal/) - An introduction to licensing and other legal considerations. + +## A Simple Contribution Workflow + +A typical contribution to a GitHub project looks like this: + +1. **Find a project** that interests you. +2. **Read the README and contribution guidelines.** +3. **Fork the repository** to your GitHub account. +4. **Clone your fork** to your computer. +5. **Create a new branch** for your change. +6. **Make and test your changes.** +7. **Commit the changes** with a clear message. +8. **Push the branch** to your fork. +9. **Open a pull request** and clearly explain what you changed and why. +10. **Respond to review feedback** and update your branch if needed. + +For beginners, documentation fixes, typo corrections, examples, tests, and small improvements can all be valuable contributions. + +## Useful Git Commands + +```bash +git clone https://github.com/YOUR-USERNAME/REPOSITORY.git +cd REPOSITORY +git checkout -b my-contribution + +git status +git add . +git commit -m "Add open source contribution guide" +git push -u origin my-contribution +``` + +After pushing the branch, open a pull request from your fork to the original repository. + +## Source and Attribution + +This page is a concise learning index based on the topics covered by [Open Source Guides](https://opensource.guide/). The original Open Source Guides content is released under the [CC-BY-4.0 license](https://creativecommons.org/licenses/by/4.0/). + +For the complete and current guides, visit [opensource.guide](https://opensource.guide/). diff --git a/DataScience/.gitignore b/DataScience/.gitignore new file mode 100644 index 00000000..c6b06805 --- /dev/null +++ b/DataScience/.gitignore @@ -0,0 +1,16 @@ +# Python +__pycache__/ +*.py[cod] +.venv/ +venv/ + +# Local datasets and generated files +datasets/ +images_dataset/ +*.log + +# IDE / OS +.vscode/ +.idea/ +.DS_Store +Thumbs.db diff --git a/DataScience/BangloreHomePrices/readme.md b/DataScience/BangloreHomePrices/readme.md index ad0a76ff..7ab37bd2 100644 --- a/DataScience/BangloreHomePrices/readme.md +++ b/DataScience/BangloreHomePrices/readme.md @@ -1,6 +1,6 @@  -This data science project series walks through step by step process of how to build a real estate price prediction website. We will first build a model using sklearn and linear regression using banglore home prices dataset from kaggle.com. Second step would be to write a python flask server that uses the saved model to serve http requests. Third component is the website built in html, css and javascript that allows user to enter home square ft area, bedrooms etc and it will call python flask server to retrieve the predicted price. During model building we will cover almost all data science concepts such as data load and cleaning, outlier detection and removal, feature engineering, dimensionality reduction, gridsearchcv for hyperparameter tunning, k fold cross validation etc. Technology and tools wise this project covers, +This data science project series walks through step by step process of how to build a real estate price prediction website. We will first build a model using sklearn and linear regression using Bangalore home prices dataset from kaggle.com. Second step would be to write a python flask server that uses the saved model to serve http requests. Third component is the website built in html, css and javascript that allows user to enter home square ft area, bedrooms etc and it will call python flask server to retrieve the predicted price. During model building we will cover almost all data science concepts such as data load and cleaning, outlier detection and removal, feature engineering, dimensionality reduction, gridsearchcv for hyperparameter tuning, k fold cross validation etc. Technology and tools wise this project covers, 1. Python 2. Numpy and Pandas for data cleaning @@ -52,7 +52,7 @@ ssh -i "C:\Users\Viral\.ssh\Banglore.pem" ubuntu@ec2-3-133-88-210.us-east-2.comp ``` 2. Create symlink for this file in /etc/nginx/sites-enabled by running this command, ``` - sudo ln -v -s /etc/nginx/sites-available/bhp.conf + sudo ln -v -s /etc/nginx/sites-available/bhp.conf /etc/nginx/sites-enabled/bhp.conf ``` 3. Remove symlink for default file in /etc/nginx/sites-enabled directory, ``` @@ -66,10 +66,41 @@ ssh -i "C:\Users\Viral\.ssh\Banglore.pem" ubuntu@ec2-3-133-88-210.us-east-2.comp ``` sudo apt-get install python3-pip sudo pip3 install -r /home/ubuntu/BangloreHomePrices/server/requirements.txt -python3 /home/ubuntu/BangloreHomePrices/client/server.py +python3 /home/ubuntu/BangloreHomePrices/server/server.py ``` Running last command above will prompt that server is running on port 5000. 8. Now just load your cloud url in browser (for me it was http://ec2-3-133-88-210.us-east-2.compute.amazonaws.com/) and this will be fully functional website running in production cloud environment + + +## Run locally + +From the `server` directory, install the dependencies and start the Flask API: + +```bash +cd BangloreHomePrices/server +python -m venv .venv +# Windows: .venv\Scripts\activate +# Linux/macOS: source .venv/bin/activate +pip install -r requirements.txt +python server.py +``` + +The API is available at `http://localhost:5000`. The `/health` endpoint can be +used to verify that the server is running. The saved model artifacts must be +placed in `server/artifacts/` before starting the API. + +## API example + +```bash +curl -X POST http://localhost:5000/predict_home_price ^ + -d "total_sqft=1000" -d "location=1st Phase JP Nagar" -d "bhk=2" -d "bath=2" +``` + +## Contributing + +Bug fixes, documentation improvements, tests, and small usability improvements +are welcome. Please keep changes focused and explain the motivation in your +pull request. diff --git a/DataScience/BangloreHomePrices/server/server.py b/DataScience/BangloreHomePrices/server/server.py index 43529263..1a61bde5 100644 --- a/DataScience/BangloreHomePrices/server/server.py +++ b/DataScience/BangloreHomePrices/server/server.py @@ -1,32 +1,52 @@ -from flask import Flask, request, jsonify +from flask import Flask, jsonify, request import util app = Flask(__name__) -@app.route('/get_location_names', methods=['GET']) -def get_location_names(): - response = jsonify({ - 'locations': util.get_location_names() - }) - response.headers.add('Access-Control-Allow-Origin', '*') +def _set_cors(response): + response.headers["Access-Control-Allow-Origin"] = "*" return response -@app.route('/predict_home_price', methods=['GET', 'POST']) + +@app.get("/get_location_names") +def get_location_names(): + return _set_cors(jsonify({"locations": util.get_location_names()})) + + +@app.route("/predict_home_price", methods=["GET", "POST"]) def predict_home_price(): - total_sqft = float(request.form['total_sqft']) - location = request.form['location'] - bhk = int(request.form['bhk']) - bath = int(request.form['bath']) + data = request.form if request.form else request.args - response = jsonify({ - 'estimated_price': util.get_estimated_price(location,total_sqft,bhk,bath) - }) - response.headers.add('Access-Control-Allow-Origin', '*') + try: + total_sqft = float(data["total_sqft"]) + bhk = int(data["bhk"]) + bath = int(data["bath"]) + location = data["location"].strip() + except (KeyError, TypeError, ValueError): + return _set_cors(jsonify({ + "error": "Provide valid total_sqft, bhk, bath, and location values." + })), 400 + + if total_sqft <= 0 or bhk <= 0 or bath <= 0 or not location: + return _set_cors(jsonify({ + "error": "total_sqft, bhk, bath, and location must contain valid positive values." + })), 400 + + try: + estimated_price = util.get_estimated_price(location, total_sqft, bhk, bath) + except RuntimeError as exc: + return _set_cors(jsonify({"error": str(exc)})), 503 + + return _set_cors(jsonify({"estimated_price": estimated_price})) + + +@app.get("/health") +def health(): + return jsonify({"status": "ok"}) - return response if __name__ == "__main__": print("Starting Python Flask Server For Home Price Prediction...") util.load_saved_artifacts() - app.run() \ No newline at end of file + app.run(host="0.0.0.0", port=5000) diff --git a/DataScience/CONTRIBUTING.md b/DataScience/CONTRIBUTING.md new file mode 100644 index 00000000..08d4dd43 --- /dev/null +++ b/DataScience/CONTRIBUTING.md @@ -0,0 +1,21 @@ +# Contributing + +Thanks for contributing to this DataScience project. + +## Before you start + +- Keep pull requests focused on one improvement. +- Avoid committing generated datasets, virtual environments, or local IDE files. +- Update documentation when a change affects setup or usage. +- Test the affected component before opening a pull request. + +## Pull requests + +Please include: + +1. A short description of the problem. +2. A summary of the solution. +3. Testing steps or commands. +4. Any limitations or follow-up work. + +For model changes, include relevant evaluation results when available. diff --git a/DataScience/CelebrityFaceRecognition/google_image_scrapping/image_download.py b/DataScience/CelebrityFaceRecognition/google_image_scrapping/image_download.py index 9bc8379e..2a0385ae 100644 --- a/DataScience/CelebrityFaceRecognition/google_image_scrapping/image_download.py +++ b/DataScience/CelebrityFaceRecognition/google_image_scrapping/image_download.py @@ -1,168 +1,113 @@ #!/usr/bin/env python3 -# -*- coding: utf-8 -*- -""" -Code credit: -https://towardsdatascience.com/image-scraping-with-python-a96feda8af2d -Also thanks for Debjyoti Paul (my friend and data scientist at Amazon) for help with this +"""Download image search results for the celebrity-classification dataset. + +The scraper uses Selenium to collect image URLs and Pillow to validate and +store downloaded images. Only use it where automated image downloading is +permitted by the source website and applicable terms. """ -import time -import requests -import io import hashlib +import io import os -from selenium import webdriver +import time +from pathlib import Path +from urllib.parse import quote_plus -from selenium.webdriver.common.keys import Keys +import requests +from PIL import Image +from selenium import webdriver +from selenium.webdriver.chrome.service import Service -def fetch_image_urls_util(url,driver_path): - images = [] - # Open main window with URL A - with webdriver.Chrome(executable_path=driver_path) as wd: +REQUEST_TIMEOUT = 10 - # Switch to the new window and open URL B - try: - wd.get(url) - except: - return [] - thumbnail_results = wd.find_elements_by_css_selector("img[class ='irc_mi']") +def _build_driver(driver_path=None): + options = webdriver.ChromeOptions() + options.add_argument("--headless=new") + options.add_argument("--disable-gpu") + options.add_argument("--no-sandbox") + if driver_path: + return webdriver.Chrome(service=Service(driver_path), options=options) + return webdriver.Chrome(options=options) - for img in thumbnail_results: - if img.get_attribute('src') and 'http' in img.get_attribute('src'): - images.append(img.get_attribute('src')) - return images +def fetch_image_urls(query, max_links_to_fetch, wd, sleep_between_interactions=1): + """Collect up to ``max_links_to_fetch`` image URLs from image search.""" + if max_links_to_fetch <= 0: + return set() + search_url = ( + "https://www.google.com/search?tbm=isch&q=" + quote_plus(query) + ) + wd.get(search_url) -def fetch_image_urls(query:str, max_links_to_fetch:int, wd, sleep_between_interactions:int=1,driver_path= None, target_path = None, search_term = None): - - target_folder = os.path.join(target_path,'_'.join(search_term.lower().split(' '))) - def scroll_to_end(wd): + image_urls = set() + previous_count = 0 + + while len(image_urls) < max_links_to_fetch: + thumbnails = wd.find_elements("css selector", "img") + for thumbnail in thumbnails: + src = thumbnail.get_attribute("src") + if src and src.startswith("http"): + image_urls.add(src) + if len(image_urls) >= max_links_to_fetch: + break + + if len(image_urls) == previous_count: + wd.execute_script("window.scrollTo(0, document.body.scrollHeight);") + time.sleep(sleep_between_interactions) + thumbnails = wd.find_elements("css selector", "img") + if len(thumbnails) <= previous_count: + break + previous_count = len(image_urls) wd.execute_script("window.scrollTo(0, document.body.scrollHeight);") - time.sleep(sleep_between_interactions) - - # build the google query - search_url = "https://www.google.com/search?safe=off&site=&tbm=isch&source=hp&q={q}&oq={q}&gs_l=img" - - # load the page - wd.get(search_url.format(q=query)) + time.sleep(sleep_between_interactions) - image_urls = set() - image_count = 0 - image_count2 = 0 - results_start = 0 - i = 0 - d = {} - while image_count < max_links_to_fetch: - scroll_to_end(wd) - - # get all image thumbnail results - thumbnail_results = wd.find_elements_by_css_selector("img.Q4LuWd") - number_results = len(thumbnail_results) - - print(f"Found: {number_results} search results. Extracting links from {results_start}:{number_results}") - - for img in thumbnail_results[50:number_results]: - # try to click every thumbnail such that we can get the real image behind it - try: - img.click() - time.sleep(sleep_between_interactions) - except Exception as e: - print(e) - continue - - links = wd.find_elements_by_css_selector("a[jsname='sTFXNd']") - - for link in links: - if link.get_attribute('href') and 'http' in link.get_attribute('href'): - if link.get_attribute('href') not in d: - d[link.get_attribute('href')] = True - getactualurl = fetch_image_urls_util(link.get_attribute('href'),driver_path) - for imageurl in getactualurl: - if imageurl is not None: - #print(imageurl) - image_urls.add(imageurl) - - image_count2 = len(image_urls) - print(image_count2) - if image_count2 >= max_links_to_fetch/10: - print(f"Found: {len(image_urls)} image links, saving!") - try: - for elem in image_urls: - persist_image(target_folder,elem) - except Exception as e: - print(e) - image_urls = set() - d = {} - - image_count += image_count2 - - #image_count = len(image_urls) - - if len(image_urls) >= max_links_to_fetch: - print(f"Found: {len(image_urls)} image links, done!") - break - else: - print("Found:", len(image_urls), "image links, looking for more ...") - time.sleep(30) - return - load_more_button = wd.find_element_by_css_selector(".mye4qd") - if load_more_button: - wd.execute_script("document.querySelector('.mye4qd').click();") - - # move the result startpoint further down - results_start = image_count - - print(len(image_urls)) - return image_urls - - - -def persist_image(folder_path:str,url:str): - try: - image_content = requests.get(url).content + return set(list(image_urls)[:max_links_to_fetch]) - except Exception as e: - print(f"ERROR - Could not download {url} - {e}") +def persist_image(folder_path, url): + """Download, validate, and save one image. Return its path on success.""" try: - image_file = io.BytesIO(image_content) - image = Image.open(image_file).convert('RGB') - file_path = os.path.join(folder_path,hashlib.sha1(image_content).hexdigest()[:10] + '.jpg') - with open(file_path, 'wb') as f: - image.save(f, "JPEG", quality=85) - print(f"SUCCESS - saved {url} - as {file_path}") - except Exception as e: - print(f"ERROR - Could not save {url} - {e}") - - - -def search_and_download(search_term:str,driver_path:str,target_path='./datasets',number_images=50): - target_folder = os.path.join(target_path,'_'.join(search_term.lower().split(' '))) - - if not os.path.exists(target_folder): - os.makedirs(target_folder) - - with webdriver.Chrome(executable_path=driver_path) as wd: - res = fetch_image_urls(search_term, number_images, wd=wd, sleep_between_interactions=0.5,driver_path= driver_path,target_path= target_path,search_term=search_term) - try: - for elem in res: - persist_image(target_folder,elem) - except Exception as e: - print(e) - -import time -import requests -import io -from PIL import Image, ImageDraw -import hashlib -import os -from selenium import webdriver - - -query = ["Serena Williams"] - -for q in query: - search_and_download(q,"./chromedriver.exe") \ No newline at end of file + response = requests.get( + url, timeout=REQUEST_TIMEOUT, headers={"User-Agent": "Mozilla/5.0"} + ) + response.raise_for_status() + image_content = response.content + + with Image.open(io.BytesIO(image_content)) as image: + image = image.convert("RGB") + filename = hashlib.sha1(image_content).hexdigest()[:10] + ".jpg" + path = Path(folder_path) / filename + path.parent.mkdir(parents=True, exist_ok=True) + image.save(path, "JPEG", quality=85) + + print(f"SUCCESS - saved {url} - as {path}") + return path + except (requests.RequestException, OSError, ValueError) as exc: + print(f"ERROR - could not save {url}: {exc}") + return None + + +def search_and_download(search_term, driver_path=None, target_path="./datasets", number_images=50): + """Search for a term and save the requested number of valid images.""" + target_folder = Path(target_path) / "_".join(search_term.lower().split()) + target_folder.mkdir(parents=True, exist_ok=True) + + with _build_driver(driver_path) as driver: + urls = fetch_image_urls(search_term, number_images, driver) + + saved = 0 + for url in urls: + if persist_image(target_folder, url): + saved += 1 + + print(f"Saved {saved} images for '{search_term}'.") + return saved + + +if __name__ == "__main__": + queries = ["Serena Williams"] + for query in queries: + search_and_download(query, "./chromedriver.exe", number_images=50) diff --git a/DataScience/CelebrityFaceRecognition/readme.md b/DataScience/CelebrityFaceRecognition/readme.md index 983b69c9..c2915c43 100644 --- a/DataScience/CelebrityFaceRecognition/readme.md +++ b/DataScience/CelebrityFaceRecognition/readme.md @@ -11,7 +11,7 @@ Here is the folder structure, * UI : This contains ui website code * server: Python flask server * model: Contains python notebook for model building -* google_image_scrapping: code to scrap google for images +* google_image_scrapping: code to scrape Google for images * images_dataset: Dataset used for our model training Technologies used in this project, @@ -25,3 +25,25 @@ Technologies used in this project, Here is the video playlist for entire project: https://www.youtube.com/playlist?list=PLeo1K3hjS3uvaRHZLl-jLovIjBP14QTXc + + +## Running the image scraper + +The scraper requires Python, Selenium, Chrome, and a compatible ChromeDriver. +Install the project dependencies first, then run the script from the +`google_image_scrapping` directory. + +```bash +pip install -r ../model/requirements.txt +python image_download.py +``` + +The scraper validates downloaded files before saving them and uses deterministic +filenames to avoid duplicate downloads. Use automated downloading only when it +is permitted by the source website and applicable terms. + +## Contributing + +Contributions that improve reproducibility, documentation, model evaluation, +accessibility, or code quality are welcome. Keep pull requests focused and +include a short explanation of what changed and how it was tested. diff --git a/matpltlib/10_subplots.ipynb b/matpltlib/10_subplots.ipynb index d6d2accb..594a4a6c 100644 --- a/matpltlib/10_subplots.ipynb +++ b/matpltlib/10_subplots.ipynb @@ -4,7 +4,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "