Handling Character Encoding Based on Meta Tags in ScrapeWebsite

[AiraNadih]

• I have read this document: https://miniflux.app/opinionated.html#feature-request

Description:

In rare cases, some websites have encoding issues where the response body is garbled. However, browsers can correctly recognize the encoding due to the meta tag in the content, such as:

<meta http-equiv='content-type' content='text/html; charset=gb2312' />

I would like to request the addition of related handling logic in ScrapeWebsite to address this issue. This logic should be implemented before the following code:

htmlDocumentReader, err := charset.NewReader(
    responseHandler.Body(config.Opts.HTTPClientMaxBodySize()),
    responseHandler.ContentType(),
)

This will ensure that the correct character encoding is applied, similar to how browsers handle it.

Thank you for considering this request.