From 2564c20f861215382252c1d58392a8a310f15e68 Mon Sep 17 00:00:00 2001 From: Hongbo Wu Date: Tue, 29 Nov 2022 22:17:46 +0800 Subject: [PATCH 1/4] generate test page for nytimes.com --- .../nytimes.com/expected-metadata.json | 12 + .../test/test-pages/nytimes.com/expected.html | 103 + .../test/test-pages/nytimes.com/source.html | 11853 ++++++++++++++++ .../test/test-pages/nytimes.com/url.txt | 1 + 4 files changed, 11969 insertions(+) create mode 100644 packages/readabilityjs/test/test-pages/nytimes.com/expected-metadata.json create mode 100644 packages/readabilityjs/test/test-pages/nytimes.com/expected.html create mode 100644 packages/readabilityjs/test/test-pages/nytimes.com/source.html create mode 100644 packages/readabilityjs/test/test-pages/nytimes.com/url.txt diff --git a/packages/readabilityjs/test/test-pages/nytimes.com/expected-metadata.json b/packages/readabilityjs/test/test-pages/nytimes.com/expected-metadata.json new file mode 100644 index 000000000..c76bb2d55 --- /dev/null +++ b/packages/readabilityjs/test/test-pages/nytimes.com/expected-metadata.json @@ -0,0 +1,12 @@ +{ + "title": "How Twitter Will Change as a Private Company", + "byline": "Kate Conger", + "dir": null, + "excerpt": "The social media company went public in 2013. But Elon Musk is taking it private as part of his acquisition of the firm. Here’s what that means.", + "siteName": null, + "siteIcon": "http://fakehost/vi-assets/static-assets/favicon-d2483f10ef688e6f89e23806b9700298.ico", + "previewImage": "https://static01.nyt.com/images/2022/10/29/business/00jpTWITTER-PRIVATE2-print/00MUSK-TWITTER-facebookJumbo.jpg", + "publishedDate": "2022-10-28T09:00:25.000Z", + "language": "English", + "readerable": true +} diff --git a/packages/readabilityjs/test/test-pages/nytimes.com/expected.html b/packages/readabilityjs/test/test-pages/nytimes.com/expected.html new file mode 100644 index 000000000..b4d4df352 --- /dev/null +++ b/packages/readabilityjs/test/test-pages/nytimes.com/expected.html @@ -0,0 +1,103 @@ +
+
+
+
+
+
+

The social media company went public in 2013. But Elon Musk is taking it private as part of his acquisition of the firm. Here’s what that means.

+
+
+
+ + + + + Elon Musk plans to change Twitter from a public to a privately held company. + +
+
+ Credit...Jason Henry for The New York Times +
+
+
+

+

+
+
+
+

Now, Mr. Musk, who did not end up taking Tesla private, is doing so with Twitter. As part of his $44 billion acquisition of the social media service, which closed on Thursday, he is delisting the company’s stock and taking it out of the hands of public shareholders.

+

Making Twitter a private company gives Mr. Musk some advantages. Unlike publicly traded companies, privately held firms do not have to make quarterly public disclosures about their performance. They are also subject to less regulatory scrutiny and can be more tightly controlled by an owner. That means Mr. Musk can make over Twitter — including tweaking the platform’s content rules, its finances and its priorities — without having to consider the worries of the investing public.

+

“It’s hard to run a public company if you think you should be the one running it and you’re not open to other views from people, like stockholders,” said Brian J.M. Quinn, a professor at Boston College Law School. “By taking it private, you feel like you have a lot more flexibility.”

+

Here’s how Twitter is set to change as a private company under Mr. Musk.

+ +

As part of buying Twitter, Mr. Musk is merging the social media company with X Holdings, a corporate entity that he established in Delaware to handle the deal. X is buying out all of Twitter’s stock and will control the service, and Mr. Musk will control the holding company.

+
+
+
+
+

Image

+
+
+ Credit...Lucas Jackson/Reuters +
+
+
+
+ +

Twitter will be delisted from the New York Stock Exchange and its shares will no longer trade on public markets as of Nov. 8, according to a securities filing. In September, Twitter’s shareholders approved the company’s sale to Mr. Musk and agreed to sell their stock to him for $54.20 a share. Investors will be able to claim the cash value of their shares.

+
+
+ +

With the deal’s completion, Twitter’s board of directors will dissolve and its nine members will no longer preside over the company’s operations. Mr. Musk will most likely appoint a new board made up of friends and investors who helped fund the acquisition. The new board will be responsible for plotting Twitter’s trajectory as a private company.

+

“It will still be required by law to have a board of directors, and that would probably include Elon Musk and some of the other big equity investors in the company,” said Eric Talley, a professor at Columbia Law School. “I expect Mr. Musk will run it as a somewhat friendly dictatorship.”

+ +

Mr. Musk has already started cleaning house, with several of Twitter’s top executives getting fired on Thursday.

+

The executives who were fired include Parag Agrawal, Twitter’s chief executive, who has clashed publicly and privately with Mr. Musk. When Mr. Musk complained this year that Twitter had an unchecked spam problem, Mr. Agrawal tweeted to rebut his claims. Mr. Musk responded with a poop emoji.

+

At another point, Mr. Agrawal texted Mr. Musk, telling the billionaire that his criticisms were harming Twitter, according to a court filing.

+

“This is a waste of time,” Mr. Musk retorted.

+

Other executives who were fired include Ned Segal, Twitter’s chief financial officer; Vijaya Gadde, the top legal and policy executive; and Sean Edgett, the general counsel.

+
+
+

Under the merger agreement, Mr. Agrawal was potentially set to receive a golden parachute worth about $60 million, with Mr. Segal to receive $46 million, while Ms. Gadde would receive about $20 million. It was not immediately clear whether Mr. Musk intended to make the payments.

+
+
+
+
+

Image

+
+
+ Credit...Kevin Dietsch/Getty Images +
+
+
+
+ +

Twitter has about 7,500 employees. Some of them have been jittery for months about the company’s sale to Mr. Musk. Many could face layoffs or job changes as their new owner takes over.

+

Their compensation is also set to change. Employees typically receive stock options in the company. But, with the delisting of Twitter’s stock, employees are set to be cashed out for shares they already have and to be paid with cash bonuses going forward, instead of the stock options they were scheduled to receive, according to the merger agreement. Some employees have worried that Mr. Musk may not honor the agreement.

+

“Most of these employees have been in a public company and are used to public option grants, which are liquid,” Mr. Quinn said. “They will have to come up with some other Silicon Valley-friendly method of keeping people around.”

+ +

By going private, Twitter will avoid some public scrutiny since it will no longer be required to make quarterly disclosures about the health of its business. This will give Mr. Musk some flexibility as he changes Twitter.

+

But he will face pressure from the banks that lent him $12.5 billion for the deal to begin repaying his debt. The cost of repaying those loans could run as high as $1 billion a year, financial analysts said.

+
+
+

“He has less public pressure, but he has a lot of private pressure from the banks to make the payments,” Mr. Quinn said of Mr. Musk. “Like almost every other private-equity take-private, he’s going to need a manager who is very focused on operations, being lean and being able to pay the bills on a day-to-day basis.”

+

Mr. Musk also took about $7.1 billion from equity investors to push the deal through. He may also face pressure from those investors, who might expect him to take Twitter public again at some point so that they can recoup their investment.

+

In some take-private deals, owners have opted to sell branches of their companies to pay their debts. Mr. Musk could choose to do the same at Twitter.

+

“It’s conceivable that some aspects of Twitter could potentially be carved off, sold off or spun off to raise money that could go toward paying the debt,” Mr. Talley said. “Twitter kind of is pared down to its core mission right now. They would have to get a little creative.”

+
+
+
+
+
+
+
\ No newline at end of file diff --git a/packages/readabilityjs/test/test-pages/nytimes.com/source.html b/packages/readabilityjs/test/test-pages/nytimes.com/source.html new file mode 100644 index 000000000..be8227c41 --- /dev/null +++ b/packages/readabilityjs/test/test-pages/nytimes.com/source.html @@ -0,0 +1,11853 @@ + + + + + + How Twitter Will Change as a Private Company Under Elon Musk - The New + York Times + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+
+
+
+
+
+
+ +
+ +
+ +
+ Skip to contentSkip to site index +
+ +
+ + +
+
+
+ +
+
+
+
+
+
+ +
+
+
+
+ +
+ + + + + + + + +
+ +
+
+
+
+
+

Advertisement

+
+ Continue reading the main story +
+
+
+
+
+
+ +
+
+

+ How Twitter Will Change as a Private Company +

+
+

+ The social media company went public in 2013. But Elon + Musk is taking it private as part of his acquisition of + the firm. Here’s what that means. +

+
+
+
+
    +
  • + +
    +
    + +
    +
    +
    • +
  • +
    +
    + +
    +
    +
    • +
  • + +
    • +
+
+
+
+
+
+
+
+
+ + + + Elon Musk plans to change Twitter from a public to a privately held company. +
+
+ Credit...Jason Henry for The New York Times +
+
+
+
+
+
+
+

+ By Kate Conger +

+
+
+
+
+ +
+
+
+
+
+

+ Elon Musk closes his purchase of Twitter and fires + several top executives. Follow for the latest news updates. +

+

+ SAN FRANCISCO — In August 2018, Elon Musk called + Michael Dell + for advice. +

+

+ Mr. Musk, who was + trying to take Tesla, his electric car company, + private, quizzed Mr. Dell, who had done that very thing with + his eponymous computer company in 2013, about the process and the best lawyers to use for + the complicated transaction. +

+

+ “I was really asking him about — did he find being + private was good?” Mr. Musk recalled in a + deposition + about the Tesla effort. “Did he regret going private?” +

+

+ Mr. Dell warned Mr. Musk that it was “a very difficult + process” but that he was glad to have done it, + according to court filings. +

+
+ +
+
+
+
+

+ Now, Mr. Musk, who did not end up taking Tesla + private, is doing so with Twitter. As part of his $44 + billion acquisition of the social media service, which + closed on Thursday, he is delisting the company’s + stock and taking it out of the hands of public + shareholders. +

+

+ Making Twitter a private company gives Mr. Musk some + advantages. Unlike publicly traded companies, + privately held firms do not have to make quarterly + public disclosures about their performance. They are + also subject to less regulatory scrutiny and can be + more tightly controlled by an owner. That means Mr. + Musk can make over Twitter — including tweaking the + platform’s content rules, its finances and its + priorities — without having to consider the worries of + the investing public. +

+

+ “It’s hard to run a public company if you think you + should be the one running it and you’re not open to + other views from people, like stockholders,” said + Brian J.M. Quinn, a professor at Boston College Law + School. “By taking it private, you feel like you have + a lot more flexibility.” +

+

+ Here’s how Twitter is set to change as a private + company under Mr. Musk. +

+ +

+ As part of buying Twitter, Mr. Musk is merging the + social media company with + X Holdings, a corporate entity that he established in Delaware + to handle the deal. X is buying out all of Twitter’s + stock and will control the service, and Mr. Musk will + control the holding company. +

+
+ +
+
+
+
+
+
+
+ Image +
+
+
+
+
+ Credit...Lucas Jackson/Reuters +
+
+
+
+
+
+
+ +

+ Twitter will be delisted from the New York Stock + Exchange and its shares will no longer trade on public + markets as of Nov. 8, according to + a securities filing. In September, Twitter’s shareholders approved the + company’s sale to Mr. Musk and agreed to sell their + stock to him for $54.20 a share. Investors will be + able to claim the cash value of their shares. +

+
+ +
+
+
+
+ +

+ With the deal’s completion, + Twitter’s board of directors + will dissolve and its nine members will no longer + preside over the company’s operations. Mr. Musk will + most likely appoint a new board made up of friends and + investors who helped fund the acquisition. The new + board will be responsible for plotting Twitter’s + trajectory as a private company. +

+
+ +
+

+ “It will still be required by law to have a board of + directors, and that would probably include Elon Musk + and some of the other big equity investors in the + company,” said Eric Talley, a professor at Columbia + Law School. “I expect Mr. Musk will run it as a + somewhat friendly dictatorship.” +

+ +

+ Mr. Musk has already started cleaning house, with + several of Twitter’s top executives + getting fired on Thursday. +

+

+ The executives who were fired include + Parag Agrawal, Twitter’s chief executive, who has clashed publicly + and privately with Mr. Musk. When Mr. Musk complained + this year that Twitter had an unchecked spam problem, + Mr. Agrawal tweeted to rebut his claims. Mr. Musk + responded with a poop emoji. +

+

+ At another point, Mr. Agrawal texted Mr. Musk, telling + the billionaire that his criticisms were harming + Twitter, according to a + court filing. +

+

+ “This is a waste of time,” Mr. Musk retorted. +

+

+ Other executives who were fired include Ned Segal, + Twitter’s chief financial officer; Vijaya Gadde, the + top legal and policy executive; and Sean Edgett, the + general counsel. +

+
+ +
+
+
+
+

+ Under the merger agreement, Mr. Agrawal was + potentially set to receive a golden parachute worth + about $60 million, with Mr. Segal to receive $46 + million, while Ms. Gadde would receive about $20 + million. It was not immediately clear whether Mr. Musk + intended to make the payments. +

+
+ +
+
+
+
+
+
+
+ Image +
+
+
+
+
+ Credit...Kevin Dietsch/Getty Images +
+
+
+
+
+
+
+ +

+ Twitter has about 7,500 employees. Some of them + have been jittery + for months about the company’s sale to Mr. Musk. Many + could face layoffs or job changes as their new owner + takes over. +

+
+ +
+

+ Their compensation is also set to change. Employees + typically receive stock options in the company. But, + with the delisting of Twitter’s stock, employees are + set to be cashed out for shares they already have and + to be paid with cash bonuses going forward, instead of + the stock options they were scheduled to receive, + according to the merger agreement. Some + employees have worried + that Mr. Musk may not honor the agreement. +

+

+ “Most of these employees have been in a public company + and are used to public option grants, which are + liquid,” Mr. Quinn said. “They will have to come up + with some other Silicon Valley-friendly method of + keeping people around.” +

+ +

+ By going private, Twitter will avoid some public + scrutiny since it will no longer be required to make + quarterly disclosures about the health of its + business. This will give Mr. Musk some flexibility as + he changes Twitter. +

+

+ But he will face pressure from the banks that lent him + $12.5 billion for the deal to begin repaying his debt. + The cost of repaying those loans could run as high as + $1 billion a year, financial analysts said. +

+
+ +
+
+
+
+

+ “He has less public pressure, but he has a lot of + private pressure from the banks to make the payments,” + Mr. Quinn said of Mr. Musk. “Like almost every other + private-equity take-private, he’s going to need a + manager who is very focused on operations, being lean + and being able to pay the bills on a day-to-day + basis.” +

+

+ Mr. Musk also took about $7.1 billion from equity + investors to push the deal through. He may also face + pressure from those investors, who might expect him to + take Twitter public again at some point so that they + can recoup their investment. +

+

+ In some take-private deals, owners have opted to sell + branches of their companies to pay their debts. Mr. + Musk could choose to do the same at Twitter. +

+

+ “It’s conceivable that some aspects of Twitter could + potentially be carved off, sold off or spun off to + raise money that could go toward paying the debt,” Mr. + Talley said. “Twitter kind of is pared down to its + core mission right now. They would have to get a + little creative.” +

+
+ +
+
+
+
+
+
+
+
+
+
+

Advertisement

+
+ Continue reading the main story +
+
+
+
+
+
+
+
+ +
+
+
+
+ + + + + + + + + + + + + + + + + + + diff --git a/packages/readabilityjs/test/test-pages/nytimes.com/url.txt b/packages/readabilityjs/test/test-pages/nytimes.com/url.txt new file mode 100644 index 000000000..3f3e7f47e --- /dev/null +++ b/packages/readabilityjs/test/test-pages/nytimes.com/url.txt @@ -0,0 +1 @@ +https://www.nytimes.com/2022/10/28/technology/twitter-changes.html From 548f317607daed8085a20d7cb1b6305237765ed8 Mon Sep 17 00:00:00 2001 From: Hongbo Wu Date: Wed, 30 Nov 2022 11:04:05 +0800 Subject: [PATCH 2/4] Skip cleanning nytimes article first few paragraphs which has too many links --- packages/readabilityjs/Readability.js | 6 +++++- .../test/test-pages/nytimes.com/expected.html | 9 +++++++++ 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/packages/readabilityjs/Readability.js b/packages/readabilityjs/Readability.js index 1df750d69..409604e9f 100644 --- a/packages/readabilityjs/Readability.js +++ b/packages/readabilityjs/Readability.js @@ -244,6 +244,9 @@ Readability.prototype = { "apos": "'", }, + // These are the classes that we skip when cleaning a tag + CLASSES_TO_SKIP: ["post-body", "StoryBodyCompanionColumn"], + /** * Run any post-process modifications to article content as necessary. * @@ -2782,6 +2785,7 @@ Readability.prototype = { return false; } + var parentClasses = node.parentNode.classList; var haveToRemove = !this._isOmnivoreNode(node) && ( (img > 1 && p / img < 0.5 && !this._hasAncestorTag(node, "figure")) || @@ -2789,7 +2793,7 @@ Readability.prototype = { (input > Math.floor(p/3)) || (!isList && headingDensity < 0.9 && contentLength < 25 && (img === 0 || img > 2) && !this._hasAncestorTag(node, "figure")) || // ignores link density for the links inside the .post-body div (the main content) - (!isList && weight < 25 && linkDensity > 0.2 && !(node.parentElement.className.includes("post-body") && linkDensity === 1)) || + (!isList && weight < 25 && linkDensity > 0.2 && !(this.CLASSES_TO_SKIP.some((c) => parentClasses.contains(c))) )|| // some website like https://substack.com might have their custom styling of tweets // we should omit ignoring their particular case by checking against "tweet" classname (weight >= 25 && linkDensity > 0.5 && !(node.className === "tweet" && linkDensity === 1)) || diff --git a/packages/readabilityjs/test/test-pages/nytimes.com/expected.html b/packages/readabilityjs/test/test-pages/nytimes.com/expected.html index b4d4df352..ceb0aa925 100644 --- a/packages/readabilityjs/test/test-pages/nytimes.com/expected.html +++ b/packages/readabilityjs/test/test-pages/nytimes.com/expected.html @@ -34,6 +34,15 @@

+
+

+ Elon Musk closes his purchase of Twitter and fires several top executives. Follow for the latest news updates. +

+

SAN FRANCISCO — In August 2018, Elon Musk called Michael Dell for advice.

+

Mr. Musk, who was trying to take Tesla, his electric car company, private, quizzed Mr. Dell, who had done that very thing with his eponymous computer company in 2013, about the process and the best lawyers to use for the complicated transaction.

+

“I was really asking him about — did he find being private was good?” Mr. Musk recalled in a deposition about the Tesla effort. “Did he regret going private?”

+

Mr. Dell warned Mr. Musk that it was “a very difficult process” but that he was glad to have done it, according to court filings.

+

Now, Mr. Musk, who did not end up taking Tesla private, is doing so with Twitter. As part of his $44 billion acquisition of the social media service, which closed on Thursday, he is delisting the company’s stock and taking it out of the hands of public shareholders.

Making Twitter a private company gives Mr. Musk some advantages. Unlike publicly traded companies, privately held firms do not have to make quarterly public disclosures about their performance. They are also subject to less regulatory scrutiny and can be more tightly controlled by an owner. That means Mr. Musk can make over Twitter — including tweaking the platform’s content rules, its finances and its priorities — without having to consider the worries of the investing public.

From 4bedfd2abe60224f58358affd2a9b1098c5e1084 Mon Sep 17 00:00:00 2001 From: Hongbo Wu Date: Wed, 30 Nov 2022 11:10:26 +0800 Subject: [PATCH 3/4] Fix test --- .../test/test-pages/electrek/expected-metadata.json | 2 +- packages/readabilityjs/test/test-pages/electrek/expected.html | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/packages/readabilityjs/test/test-pages/electrek/expected-metadata.json b/packages/readabilityjs/test/test-pages/electrek/expected-metadata.json index ec043ebba..d533b49c8 100644 --- a/packages/readabilityjs/test/test-pages/electrek/expected-metadata.json +++ b/packages/readabilityjs/test/test-pages/electrek/expected-metadata.json @@ -4,7 +4,7 @@ "dir": null, "excerpt": "Solar-cell manufacturing giant Q Cells today announced that it's opening a new solar panel manufacturing facility in Dalton, Georgia.", "siteName": "Electrek", - "siteIcon": "/favicon.ico", + "siteIcon": "http://fakehost/favicon.ico", "previewImage": "https://i0.wp.com/electrek.co/wp-content/uploads/sites/3/2022/05/georgia-solar-manufacturing.jpg?resize=1200%2C628&quality=82&strip=all&ssl=1", "publishedDate": "2022-05-26T15:57:59.000Z", "language": "English", diff --git a/packages/readabilityjs/test/test-pages/electrek/expected.html b/packages/readabilityjs/test/test-pages/electrek/expected.html index f8bfb836a..7f53696ef 100644 --- a/packages/readabilityjs/test/test-pages/electrek/expected.html +++ b/packages/readabilityjs/test/test-pages/electrek/expected.html @@ -35,6 +35,9 @@

UnderstandSolar is a free service that links you to top-rated solar installers in your region for personalized solar estimates. Tesla now offers price matching, so it’s important to shop for the best quotes. Click here to learn more and get your quotes. — *ad.

+
+

You’re reading Electrek— experts who break news about Tesla, electric vehicles, and green energy, day after day. Be sure to check out our homepage for all the latest news, and follow Electrek on Twitter, Facebook, and LinkedIn to stay in the loop. Don’t know where to start? Check out our YouTube channel for the latest reviews.

+

Subscribe to Electrek on YouTube for exclusive videos and subscribe to the podcast. From ce44c8e5296c2c44b646c96c7d86be22f2f56cb7 Mon Sep 17 00:00:00 2001 From: Hongbo Wu Date: Wed, 30 Nov 2022 11:59:42 +0800 Subject: [PATCH 4/4] Default parent classes being an empty array --- packages/readabilityjs/Readability.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/readabilityjs/Readability.js b/packages/readabilityjs/Readability.js index 409604e9f..27a159863 100644 --- a/packages/readabilityjs/Readability.js +++ b/packages/readabilityjs/Readability.js @@ -2785,7 +2785,7 @@ Readability.prototype = { return false; } - var parentClasses = node.parentNode.classList; + var parentClasses = node.parentNode?.classList || []; var haveToRemove = !this._isOmnivoreNode(node) && ( (img > 1 && p / img < 0.5 && !this._hasAncestorTag(node, "figure")) ||