Skip to content

No results with specific url #196

Closed
Closed

Description

@plompd

Describe the bug
If I try to scrape the following URL, it is successful:
https://www.werkenbijzicht.nl/vacatures.html

If I try to scrape the same page, but with another URL, it returns no result:
https://www.werkenbijzicht.nl/p/1/vacatures.html

It is the same page, but initiated from the pager at the bottom.

So with the original URL, my scraper only does scrape the first 10 items and if I try to go to the next page, I get this new URL like stated above and that results in nothing.

I cannot see what the difference is or how I should fix this?

  • core: 3.0.0
class ZichtSpider extends BasicSpider
{
    public array $startUrls = [
        'https://www.werkenbijzicht.nl/vacatures.html',
    ];

    public array $downloaderMiddleware = [
        RequestDeduplicationMiddleware::class,
        ExecuteJavascriptMiddleware::class,
        [RandomUserAgentMiddleware::class, ['userAgent' => 'Mozilla/5.0 (compatible; RoachPHP/0.1.0)']],
    ];

    public array $spiderMiddleware = [
        //
    ];

    public array $itemProcessors = [
        //
    ];

    public array $extensions = [
        LoggerExtension::class,
        StatsCollectorExtension::class,
    ];

    public int $concurrency = 2;

    public int $requestDelay = 1;

    public function parse(Response $response): Generator
    {
         $items = $response->filter('.actSResContainer .itemContainer')->each(function (Crawler $node) {

            $titleNode = $node->filter('.itemTitle a.cluetips');
            $relAttribute = $titleNode->attr('rel'); // Fetching the rel attribute

            // Regular expression to extract the ID
            $regex = '/id\/(\d+)\//';
            $matches = [];
            $vacancyId = '';

            if (preg_match($regex, $relAttribute, $matches)) {
                $vacancyId = $matches[1]; // The first captured group contains the ID
            }

            return [
                'url' => $titleNode->link()->getUri(),
                'title' => $titleNode->text(),
                'referenceNumber' => $vacancyId,
            ];
        });

        foreach ($items as $item) {
            yield $this->request('GET', $item['url'], 'parseJob', ['item' => $item]);
        }

        try {
            // Attempt to find the next page link. Adjust the selector as needed.
            $nextPageLink = $response->filter('.pageNav a.pnNext');

            if ($nextPageLink->count() > 0) {
                $nextPageUrl = 'https://www.werkenbijzicht.nl' . $nextPageLink->attr('href');
                yield $this->request('GET', $nextPageUrl);
            }
        } catch (\Exception $e) {
         
        }

    }

    public function parseJob(Response $response): Generator
    {

Activity

  1. plompd commented on Jan 25, 2024

    @plompd
    Author

    @ksassnowski Do you have any clue?

  2. ksassnowski commented on Jan 28, 2024

    @ksassnowski
    Contributor

    I had a quick look at their site and I honestly have no clue what they're doing. The same URL sometimes returns results and sometimes doesn't. It might be tied to the current session but I haven't investigated this any further.

    I'm turning this issue into a discussion since it's not really a problem with the library.

  3. locked and limited conversation to collaborators on Jan 28, 2024
  4. converted this issue into a discussion #207 on Jan 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions