ScrapeGraphAI

Copying facts from many web pages by hand is slow, while a scraper, code that collects data from pages, can stop working when a page layout changes. ScrapeGraphAI lets you describe the information you want, then combines a browser with a large language model (LLM), the AI system that interprets your request and the page, to produce organised results. It can also read web pages and text-based files stored on your computer. It does not make every site accessible or every result correct: you still manage browser setup, blocked-site handling, proxy servers that pass requests through another address, running many jobs and review of the extracted data. Its current code enables telemetry, automatic usage reporting, by default and can send your prompt, requested output structure, page address, page content, model name and model response to ScrapeGraphAI's telemetry server.

Use it when you need repeatable extraction from public pages or local files and can test the result against the source. It suits prototypes and workflows that you run on your own computer or server, where you want control over the AI model and setup. Check the website's terms and robots.txt instructions, its published directions for automated visitors, before scraping. Avoid collecting personal or restricted data without permission, and do not rely on it as a way around access controls.

1. Check that Python 3.12 or a later 3.x release is installed. Use Python's virtual-environment guide at https://docs.python.org/3/library/venv.html for your computer to create and activate a virtual environment, an isolated folder for this project's Python packages, so the install does not conflict with other projects.
2. With that environment active, paste the command below into the Terminal app, the window where you type commands. It installs ScrapeGraphAI and Playwright's browser files for opening websites.
3. Choose a local Ollama model or configure an outside AI provider. Before you run a test, check the website's terms and robots.txt instructions, use a public page you have permission to collect, and do not gather personal or restricted data or bypass access controls. Telemetry starts enabled. To turn it off for the current run, put `from scrapegraphai.telemetry.telemetry import disable_telemetry` and then `disable_telemetry()` before the code that starts a graph. The README's `SCRAPEGRAPHAI_TELEMETRY_ENABLED=false` setting does not disable telemetry in the current code, so do not rely on it without checking a later release. An outside AI provider may also receive your request and page content and may charge for usage. Check the result against the page before saving or acting on it.