prehistoric-systems/omnivore
1
0
Fork 0
mirror of https://github.com/omnivore-app/omnivore.git synced 2026-03-11 08:54:26 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Skip cleanning nytimes article first few paragraphs which has too many links

Browse source
This commit is contained in:
Hongbo Wu 2022-11-30 11:04:05 +08:00
parent 2564c20f86
commit 548f317607
2 changed files with 14 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
packages/readabilityjs/Readability.js
View file

@ -244,6 +244,9 @@ Readability.prototype = {
"apos": "'",
},
// These are the classes that we skip when cleaning a tag
CLASSES_TO_SKIP: ["post-body", "StoryBodyCompanionColumn"],
/**
* Run any post-process modifications to article content as necessary.
*
@ -2782,6 +2785,7 @@ Readability.prototype = {
return false;
}
var parentClasses = node.parentNode.classList;
var haveToRemove =
!this._isOmnivoreNode(node) && (
(img > 1 && p / img < 0.5 && !this._hasAncestorTag(node, "figure")) ||
@ -2789,7 +2793,7 @@ Readability.prototype = {
(input > Math.floor(p/3)) ||
(!isList && headingDensity < 0.9 && contentLength < 25 && (img === 0 || img > 2) && !this._hasAncestorTag(node, "figure")) ||
// ignores link density for the links inside the .post-body div (the main content)
(!isList && weight < 25 && linkDensity > 0.2 && !(node.parentElement.className.includes("post-body") && linkDensity === 1)) ||
(!isList && weight < 25 && linkDensity > 0.2 && !(this.CLASSES_TO_SKIP.some((c) => parentClasses.contains(c))) )||
// some website like https://substack.com might have their custom styling of tweets
// we should omit ignoring their particular case by checking against "tweet" classname
(weight >= 25 && linkDensity > 0.5 && !(node.className === "tweet" && linkDensity === 1)) ||

9
packages/readabilityjs/test/test-pages/nytimes.com/expected.html
View file

@ -34,6 +34,15 @@
</p>
</header>
<section name="articleBody">
<div>
<p>
<em>Elon Musk closes his purchase of Twitter and fires several top executives. </em><a href="https://www.nytimes.com/live/2022/10/28/business/elon-musk-twitter" title=""><em>Follow for the latest news updates</em></a><em>.</em>
</p>
<p> SAN FRANCISCO — In August 2018, Elon Musk called <a href="https://www.nytimes.com/topic/person/michael-s-dell" title="">Michael Dell</a> for advice. </p>
<p> Mr. Musk, who was <a href="https://www.nytimes.com/2018/08/07/business/tesla-stock-elon-musk-private.html" title="">trying to take Tesla, his electric car company, private</a>, quizzed Mr. Dell, who had done that very thing with <a href="https://archive.nytimes.com/dealbook.nytimes.com/2013/02/05/dell-sets-23-8-billion-deal-to-go-private/" title="">his eponymous computer company in 2013</a>, about the process and the best lawyers to use for the complicated transaction. </p>
<p> “I was really asking him about — did he find being private was good?” Mr. Musk recalled in a <a href="https://storage.courtlistener.com/recap/gov.uscourts.cand.330489/gov.uscourts.cand.330489.403.0.pdf" title="" rel="noopener noreferrer" target="_blank">deposition</a> about the Tesla effort. “Did he regret going private?” </p>
<p> Mr. Dell warned Mr. Musk that it was “a very difficult process” but that he was glad to have done it, according to court filings. </p>
</div>
<div>
<p> Now, Mr. Musk, who did not end up taking Tesla private, is doing so with Twitter. As part of his $44 billion acquisition of the social media service, which closed on Thursday, he is delisting the company’s stock and taking it out of the hands of public shareholders. </p>
<p> Making Twitter a private company gives Mr. Musk some advantages. Unlike publicly traded companies, privately held firms do not have to make quarterly public disclosures about their performance. They are also subject to less regulatory scrutiny and can be more tightly controlled by an owner. That means Mr. Musk can make over Twitter — including tweaking the platform’s content rules, its finances and its priorities — without having to consider the worries of the investing public. </p>