Support ShadowDOM

[mantou132]

Use getInnerHTML

https://web.dev/declarative-shadow-dom/

[bodinsamuel]

Thanks for your PR 🚀

Can you provide a test and way to enable/disable the behavior via the API?
We probably won't use this feature on our side so we most likely want it to be disabled by default.

If you prefer I can take over this PR but it will probably take longer ☺️

[bodinsamuel]

Shouldn't it needs to {includeShadowRoots: true}?

[mantou132]

The open mode ShadowDOM can be obtained without passing this parameter.

In order to preserve encapsulation semantics, any closed shadow roots within an element will not be serialized by default.

The default behavior seems to be what we want

[mantou132]

@bodinsamuel Would love to have ShadowDOM support for Docsearch since my docs site uses WebComponents.

If ShadowDOM is not supported by default, is it possible to enable this feature in Crawler config?

Glad you can take over this PR, i don't have much time to perfect it.

Thank you for your team's work

[bodinsamuel]

ah it's for DocSearch, I wasn't aware. In that case we might want to use it indeed ahah
Can you share your website? ☺️

[mantou132]

I have several websites using WebComponents:

• Web framework: https://gemjs.org/
• Documentation website builder: https://book.gemjs.org/
• UI library: https://duoyun-ui.gemjs.org/
• Panel layout: https://panel.gemjs.org/

I used to use fork docsearch-scraper.

[mantou132]

I'm not sure how docsearch retrieves information from the HTML string. If use a parser to analyze the HTML and then query through the DOM API, we need to remove the <template shadowrootmode> strings to prevent them from being re-parsed as ShadowDOM, e.g:

document.body.
  .replaceAll('<template shadowrootmode="open">', '')
  .replaceAll('</template>', '')

Update:

It seems that the Crawler is using Cheerio, so there is no need to remove the <template> tags.

[mantou132]

Page.content(): https://github.com/microsoft/playwright/blob/19a4f15eb67fd82a0b78b12dd94e3564504f83f9/packages/playwright-core/src/server/frames.ts#L866

[mantou132]

@bodinsamuel now use standard method getHTML