Describe the bug
Using SmartScraperGraph with a plain-language JSON-extraction prompt, the LLM consistently returns "NA" (or blank) for every requested field — even against pages where the requested information is unambiguously present (e.g. a Wikipedia infobox with an explicit founding year and employee count). Verbose logging shows Fetch Node → ParseNode → GenerateAnswer all execute without error, and the run completes successfully, but the answer content doesn't reflect the actual page.
To Reproduce
from scrapegraphai.graphs import SmartScraperGraph
graph_config = {
"llm": {"api_key": "sk-...", "model": "openai/gpt-4o-mini"},
"verbose": True,
"headless": True,
}
scraper = SmartScraperGraph(
prompt="Extract the company founding year and number of employees as JSON with keys years_established and team_size.",
source="https://en.wikipedia.org/wiki/Timpson_(company)",
config=graph_config,
)
print(scraper.run())Actual output:
{'content': {'years_established': 'NA', 'team_size': 'NA'}}
Expected: something close to {'years_established': '1865', 'team_size': '...'} — the Wikipedia infobox states the founding year plainly.
Also noting for anyone hitting the same thing: the answer is returned nested under a content key ({'content': {...}}), not at the top level of the dict run() returns — that part isn't a bug, just worth documenting since it's easy to miss when reading the field values back out.
Environment
- scrapegraphai 2.1.5
- Python 3.14.6
- macOS
- Model: openai/gpt-4o-mini
Additional context
Same result across multiple real-world pages tested (business homepages, an "About Us" page, and the Wikipedia page above) — not isolated to one site. Suspect the page content isn't reaching the LLM intact, possibly lost/truncated during the parsing or chunking step, but haven't traced it further into the library internals.
Describe the bug
Using
SmartScraperGraphwith a plain-language JSON-extraction prompt, the LLM consistently returns"NA"(or blank) for every requested field — even against pages where the requested information is unambiguously present (e.g. a Wikipedia infobox with an explicit founding year and employee count). Verbose logging showsFetch Node→ParseNode→GenerateAnswerall execute without error, and the run completes successfully, but the answer content doesn't reflect the actual page.To Reproduce
Actual output:
Expected: something close to
{'years_established': '1865', 'team_size': '...'}— the Wikipedia infobox states the founding year plainly.Also noting for anyone hitting the same thing: the answer is returned nested under a
contentkey ({'content': {...}}), not at the top level of the dictrun()returns — that part isn't a bug, just worth documenting since it's easy to miss when reading the field values back out.Environment
Additional context
Same result across multiple real-world pages tested (business homepages, an "About Us" page, and the Wikipedia page above) — not isolated to one site. Suspect the page content isn't reaching the LLM intact, possibly lost/truncated during the parsing or chunking step, but haven't traced it further into the library internals.