prehistoric-systems/removeddit
1
0
Fork 0
mirror of https://github.com/gurnec/removeddit.git synced 2026-03-11 08:54:27 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Removed the need for a back-end, everything is now done front-end

Browse source
This commit is contained in:
Jesper Wrang 2017-09-11 23:54:50 +02:00
parent 1d2f0c4806
commit ea95f47398
16 changed files with 270 additions and 304 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
.gitignore vendored
View file

@ -1,5 +1,2 @@
.secret
access.log
error.log
server
.vscode/settings.json

115
README.md
View file

@ -1,41 +1,100 @@
# Removeddit
[Removeddit](https://removeddit.com) is a site for viewing removed comments from [Reddit](https://www.reddit.com).
Usage: go to any reddit thread and change the "reddit" in the URL to "removeddit".
Usage: go to any reddit thread and change the `reddit` in the URL to `removeddit`.
The site will only display the removed comments and thier parents, not the full thread.
This is a done by comparing the comments found from Reddit API and comments from [Jason Baumgartners](https://pushshift.io/) [Pushshift Reddit API](https://github.com/pushshift/api). The backend of the site is written in Go and frontend with JavaScript (ES6). You can use this content however you want, as long as it's non-commercial.
This is a done by comparing the comments found from Reddit API and comments from [Jason Baumgartners](https://pushshift.io/) [Pushshift Reddit API](https://github.com/pushshift/api). The frontend is written in pure Javascript (ES6). You can use this code however you want, as long as it's non-commercial.
# Quick start
Install [Go](https://golang.org/) and [this](https://godoc.org/golang.org/x/crypto/acme/autocert) Go package with:
```go get golang.org/x/crypto/acme/autocert```
(Note: if you only want to have a local server you could remove all references to this package in *server.go* and it should work if you run it with the *-d* flag)
Create a reddit web app [here](https://www.reddit.com/prefs/apps/).
Open *secret.go* and fill out all the variables:
```go
const (
appName = "Name of your reddit web app"
userName = "Reddit username"
clientID = "Go to the link above and check the string under 'web app'"
clientSecret = "Go to the link above, click the 'edit' link and you'll see the secret"
version = "The version of your reddit web app, doesn't really matter, use e.g. '1.0'"
hostname = "The hostname of your server in production. Not used if you run locally"
)
# The "I just want to get this shit running"
Using [Ubuntu 16.04](http://releases.ubuntu.com/16.04/) and [nginx](https://www.nginx.com/resources/wiki/)
```
sudo git clone git@github.com:JubbeArt/removeddit.git /var/www/
sudo apt install -y nginx
sudo cp /var/www/removeddit/config/basic /etc/nginx/sites-available/default
```
And everything should be set up. Now just build the project with:
Create a reddit app [here](https://www.reddit.com/prefs/apps/), select **installed app**. For "redirect url" it doesn't really matter in this case, you can pick `http://localhost`.
```go build server.go secret.go```
Copy the **client ID** for your app set it as a variable in `id.js`, e.g. with
```
sudo nano /var/www/removeddit/static/id.js
# Insert with ctrl-shift-v
# Save with ctrl-o, exit with ctrl-x
```
And run the program:
Restart nginx and visit "localhost"
```
sudo service nginx restart
```
```./server```
# The "I care about HTTPS and security"
In this part we'll set up [nginx](https://www.nginx.com/resources/wiki/) with SSL and set up a free renewing SSL certificates with [Let's Encrypt](https://letsencrypt.org/). I assume you've already done the guide above.
(just `server` in Windows)
## Nginx.conf
Add the following to `/etc/nginx/nginx.conf` in the **http-block** (you can read about them [here](https://gist.github.com/plentz/6737338))
```
server_tokens off;
add_header X-Frame-Options SAMEORIGIN;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
```
If you want to run it this server on localhost use the *d* flag:
In the same file you also want to change `gzip on` to `gzip off`. (read more [here](https://github.com/h5bp/server-configs-nginx/issues/72))
```./server -d```
## Nginx server config
Copy the ssl config and create a soft link. Create folder for logs and also remove the default config
```
sudo cp /var/www/removeddit/config/ssl /etc/nginx/sites-available/removeddit.com
sudo ln -s /etc/nginx/sites-available/removeddit.com /etc/nginx/sites-enabled/removeddit.com
sudo mkdir /var/log/nginx/removeddit
sudo rm /etc/nginx/sites-enabled/default
```
Change the "server_name" in the config to your domain name.
## SSL with Let's encrypt
Read the full guide [here](https://certbot.eff.org/#ubuntutyakkety-nginx). Start of by installing the Let's Encrypt client [certbot](https://certbot.eff.org/)
```
sudo apt install software-properties-common
sudo add-apt-repository ppa:certbot/certbot
sudo apt update
sudo apt install -y python-certbot-nginx
```
Copy the Let's Encrypt config file for our site
```
sudo mkdir /etc/letsencrypt/configs
cp /var/www/removeddit/config/letsencrypt /etc/letsencrypt/config/removeddit.com.conf
```
In this config file you change the domains you want and a email for when the certificates are close to expiring.
This is when the webmasters start praying to God Almighty, for only He can deside the fate of the certbot.
Forgive me, Father, for I have sinned. Just don't fuck up certs you asshole.
```
sudo certbot --nginx certonly
```
You'll now have a valid SSL certificate (hopefully)! You might have to edit the path for `ssl_certificate` and `ssl_certificate_key` in `/etc/nginx/sites-available/removeddit.com` depending on where the certs are located. (Should be in `/etc/letsencrypt/live/`)
## Automated renewal of certs
The certificate expires after 90 days so we want a way to atomatically update the certs.
There are multiple ways of doing this but I find the easiest to be [cron jobs](https://en.wikipedia.org/wiki/Cron).
First we'll test if renewing atcutally works with
```
sudo certbot renew --dry-run
```
If everything works fine we can create a cron with ```sudo crontab -e``` and select an editor your comfortable with (I like *nano*).
Add the following lines at the bottom
```
# Let's Encrypt cert renewal for all sites (runs every day at 04:30)
30 4 * * * certbot renew --post-hook "systemctl reload nginx"
```
Then just restart nginx and that should do it!
This guide was mostly written for myself, you learn a shitton writing guides, highly recommended. Hopefully you learned something too.

20
config/basic Normal file
View file

@ -0,0 +1,20 @@
server {
listen 80 default_server;
listen [::]:80 default_server;
root /var/www/removeddit;
index index.html;
server_name _;
location / {
try_files /index.html /index.html =404;
}
location /r/ {
try_files /thread.html /thread.html =404;
}
location /static/ {
try_files $uri $uri/ =404;
}
}

9
config/letsencrypt.conf Normal file
View file

@ -0,0 +1,9 @@
# Things to change
domains = removeddit.com, www.removeddit.com
email = removeddit@gmail.com
# These defaults are fine (could use 4096 instead of 2048)
rsa-key-size = 2048
text = True
authenticator = webroot
webroot-path = /var/www/removeddit/

39
config/ssl Normal file
View file

@ -0,0 +1,39 @@
server {
listen 443 ssl http2 default_server;
listen [::]:443 ssl http2 default_server;
root /var/www/removeddit;
index index.html;
server_name removeddit.com;
access_log /var/log/nginx/removeddit/access.log;
error_log /var/log/nginx/removeddit/error.log;
ssl_certificate /etc/letsencrypt/live/removeddit.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/removeddit.com/privkey.pem;
location / {
try_files /index.html /index.html =404;
}
location /r/ {
try_files /thread.html /thread.html =404;
}
location /static/ {
try_files $uri $uri/ =404;
}
}
server {
listen 80 default_server;
listen [::]:80 default_server;
location / {
return 301 https://$host$request_uri;
}
location ~ /.well-known {
allow all;
}
}

58
index.html Normal file
View file

@ -0,0 +1,58 @@
<!DOCTYPE html>
<html>
<head>
<title>Removeddit</title>
<meta charset="utf-8">
<link href="/static/style.css" rel="stylesheet">
<link rel="shotcut icon" href="/static/favicon.ico">
</head>
<body>
<header id="header">
<a href="/" id="header-title-link"><h1 id="header-title">Removeddit [beta]</h1></a>
</header>
<div id="main">
<div id="main-box" class="frontpage-box">
<h2 style="color:#CA302C; margin-top:5px">About</h2>
<p>
This is a site for viewing removed comments from Reddit.
It was created by Jesper Wrang and uses <a href="https://pushshift.io/">Jason Baumgartner</a> service for getting removed comments.</p>
<p>
<b>How to use</b>: go to any Reddit thread and replace the <i>reddit</i> in the URL with <i>removeddit</i>
<br>E.g.
<a href="/r/TwoXChromosomes/comments/6z1hch/">
https://www.removeddit.com/r/TwoXChromosomes/comments/6z1hch/
</a>
</p>
<p>
Note that this site is in it's early stages of development, meaning the site is unstable and will contain bugs.
If you find something that doesn't work please contact me at: <br><b>removeddit (at) gmail.com</b>
</p>
<h2 style="color:#239F2B">TODO</h2>
<ul>
<li>Collapsing comments</li>
<li>Get removed selftext of thread</li>
<li>Get nested "continue this thread"-comments to work</li>
</ul>
<h2 style="color:#1B767A">Acknowledgement</h2>
<p>
<a href="https://pushshift.io/">Jason Baumgartner</a> - for making all this possible with his amazing API
</p>
<p>
<a href="https://github.com/gamefreak">Scott McClaugherty</a> - for his reddit markdown parser
</p>
<p>
<a href="http://reddit.com/r/redditdev">/r/redditdev</a> - for helping with the Reddit API
</p>
<h2 style="color: #CA752C">Links/Contact</h2>
<p>For feedback and bug reports:
<ul>
<li>email: removeddit (at) gmail.com</li>
<li>reddit: <a href="https://www.reddit.com/user/Jubbeart/">/u/JubbeArt</a></li>
</ul>
<p>
<a href="https://github.com/JubbeArt/removeddit">Github for this project</a>
</p>
</div>
</div>
</body>
</html>

10
secret.go
View file

@ -1,10 +0,0 @@
package main
const (
appName = ""
userName = ""
clientID = ""
clientSecret = ""
version = ""
hostname = ""
)

194
server.go
View file

@ -1,194 +0,0 @@
package main
import (
"crypto/tls"
"encoding/json"
"errors"
"flag"
"fmt"
"html/template"
"io/ioutil"
"log"
"net/http"
"net/url"
"os"
"strings"
"time"
"golang.org/x/crypto/acme/autocert"
)
// consts
const (
userAgent = "Javascript:" + appName + ":" + version + "s (by /u/" + userName + ")"
tokenURL = "https://www.reddit.com/api/v1/access_token"
tmplFolder = "templates/"
)
var (
client = &http.Client{Timeout: time.Second * 10}
tokenData = url.Values{"grant_type": []string{"client_credentials"}}
templates = template.Must(template.ParseFiles(
tmplFolder+"header.html",
tmplFolder+"footer.html",
tmplFolder+"thread.html",
tmplFolder+"frontpage.html",
tmplFolder+"error.html",
))
accessLog *log.Logger
errorLog *log.Logger
)
type tokenResponse struct {
AccessToken string `json:"access_token"`
TokenType string `json:"token_type"`
ExpiresIn int `json:"expires_in"`
Scope string `json:"scope"`
}
type threadPageData struct {
Token string
Subreddit string
ThreadID string
CommentIDs []string
}
func main() {
initLogging()
accessLog.Println("Staring server")
var debugMode bool
flag.BoolVar(&debugMode, "d", false, "debug mode: run the server on localhost")
flag.Parse()
http.HandleFunc("/", pageHandler(mainHandler))
http.HandleFunc("/r/", pageHandler(threadHandler))
// Serve static files, in production use cdn.rawgit.com (nvm fuck this, always serve static)
fs := http.FileServer(http.Dir("static"))
http.Handle("/static/", http.StripPrefix("/static/", fs))
// Run locally for debugging/testing
if debugMode {
fmt.Println("Starting server on http://localhost:8080")
if err := http.ListenAndServe(":8080", nil); err != nil {
log.Fatalf("Could not start server: %v", err)
}
} else { // Run in production
certManager := autocert.Manager{
Prompt: autocert.AcceptTOS,
HostPolicy: autocert.HostWhitelist(hostname, "www."+hostname),
Cache: autocert.DirCache("certs"),
}
server := &http.Server{
ReadTimeout: 20 * time.Second,
WriteTimeout: 20 * time.Second,
TLSConfig: &tls.Config{
GetCertificate: certManager.GetCertificate,
},
ErrorLog: errorLog,
}
redirect := &http.Server{
ReadTimeout: 10 * time.Second,
WriteTimeout: 10 * time.Second,
Handler: http.HandlerFunc(redirectTLS),
}
go func() {
if err := redirect.ListenAndServe(); err != nil {
log.Fatalf("Could not start redirect server: %v", err)
errorLog.Printf("Could not start redirect server: %v", err)
}
}()
if err := server.ListenAndServeTLS("", ""); err != nil {
log.Fatalf("Could not start server with SSL/TLS Certificate: %v", err)
errorLog.Printf("Could not start server with SSL/TLS Certificate: %v", err)
}
}
accessLog.Println("Shutting down server")
log.Println("Shutting down server")
}
func initLogging() {
accessFile, _ := os.OpenFile("access.log", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
errorFile, _ := os.OpenFile("error.log", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
accessLog = log.New(accessFile, "", log.Ldate|log.Ltime|log.Lshortfile)
errorLog = log.New(errorFile, "", log.Ldate|log.Ltime|log.Lshortfile)
}
// Wrap page handle function for the access log
func pageHandler(fn func(http.ResponseWriter, *http.Request)) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
accessLog.Println(r.URL.Path)
fn(w, r)
}
}
func renderTemplate(w http.ResponseWriter, pageName string, data interface{}) {
templates.ExecuteTemplate(w, "header.html", nil)
templates.ExecuteTemplate(w, pageName+".html", data)
templates.ExecuteTemplate(w, "footer.html", nil)
}
func handleError(w http.ResponseWriter, msg string) {
renderTemplate(w, "error", msg)
errorLog.Println("Error: " + msg)
}
func mainHandler(w http.ResponseWriter, r *http.Request) {
renderTemplate(w, "frontpage", nil)
}
func threadHandler(w http.ResponseWriter, r *http.Request) {
pathParts := strings.Split(r.URL.Path, "/")
if len(pathParts) < 5 {
handleError(w, "Missing necessary parts of the URL")
return
}
token, err := getAPIToken()
if err != nil {
handleError(w, err.Error())
return
}
data := &threadPageData{
Token: token,
Subreddit: pathParts[2],
ThreadID: pathParts[4],
}
renderTemplate(w, "thread", data)
}
func getAPIToken() (string, error) {
req, _ := http.NewRequest("POST", tokenURL, strings.NewReader(tokenData.Encode()))
req.Header.Add("User-Agent", userAgent)
req.SetBasicAuth(clientID, clientSecret)
resp, err := client.Do(req)
if err != nil {
return "", errors.New("Coulnd't get API token from Reddit")
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return "", errors.New("Coulnd't get API token from Reddit, timeout error")
}
var r tokenResponse
json.Unmarshal(body, &r)
return r.AccessToken, nil
}
func redirectTLS(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, fmt.Sprintf("https://www.%s/%s", hostname, r.RequestURI), http.StatusMovedPermanently)
}

3
static/id.js Normal file
View file

@ -0,0 +1,3 @@
// Change this to your own client ID: https://www.reddit.com/prefs/apps
// The app NEEDS TO BE an installed app and NOT a web apps
const clientID = ""

54
static/script.js
View file

@ -3,6 +3,17 @@
const markdown = SnuOwnd.getParser()
const htmlParser = new DOMParser();
const pathname = window.location.pathname.split("/")
// Just fucking crach if the user is retarded
if(pathname.length <= 4) {
displayError("Missing necessary parts of the URL")
}
const subreddit = pathname[2]
const threadID = pathname[4]
const redditTokenURL = "https://www.reddit.com/api/v1/access_token"
const redditThreadURL = `https://oauth.reddit.com/r/${subreddit}/comments/${threadID}`
const redditMorechildrenURL = `https://oauth.reddit.com/api/morechildren?link_id=t3_${threadID}&children=`
const redditSingleCommentURL = `https://oauth.reddit.com/r/${subreddit}/api/info/?id=`
@ -11,12 +22,12 @@ const pushshiftCommentsURL = "https://api.pushshift.io/reddit/comment/search?ids
const mainDiv = document.getElementById("main")
const loadingText = document.getElementById("loading-text")
const loadingImage = document.getElementById("loading-image")
const loadingImageSrc = "/static/loading.gif"
const statusImage = document.getElementById("loading-image")
const doneImageSrc = "/static/done.png"
const errorImageSrc = "/static/error.png"
// I actually haven't found a better way of doing this...
// Imgur has images-links with no indication that they are actually images
// Imgur has image-links with no indication that they are actually images
const imageHosts = ["i.redd.it", "flickr.com", "i.imgur.com", "imgur.com", "m.imgur.com"]
const commentIDs = []
@ -29,17 +40,30 @@ const deletedCommentIDs = []
let totalComments
const redditInit = {
headers: { "Authorization": "bearer " + token }
const tokenFormData = new FormData();
tokenFormData.append("grant_type", "https://oauth.reddit.com/grants/installed_client");
tokenFormData.append("device_id", "DO_NOT_TRACK_THIS_DEVICE")
tokenFormData.append("scope", "account,edit,history,mysubreddits,privatemessages,report,save,submit,subscribe,vote,wikiedit,wikiread,read,flair,identity,modconfig")
tokenFormData.append("api_type", "json")
const tokenInit = {
headers: { "Authorization": "Basic " + btoa(clientID + ":") },
method: "POST",
body: tokenFormData
}
let token
let redditInit = {
headers: { "Authorization": "" }
}
loadPage()
// Maybe a little to long a function...
async function loadPage() {
loadingImage.src = loadingImageSrc
loadingImage.style.display = "block"
setLoadingText("Loading thread...")
token = await getToken()
redditInit.headers.Authorization = "bearer " + token
// Get thread from reddit and all comment IDs (inc. removed ones) from pushshift
const requests = [
@ -71,7 +95,7 @@ async function loadPage() {
await generateComments(removedComments)
setLoadingText("")
loadingImage.src = doneImageSrc
statusImage.src = doneImageSrc
}
// ------------------------------------------------------------------------------
@ -262,6 +286,7 @@ function createComments() {
function generateThread(data) {
const thread = data[0].data.children[0].data
totalComments = thread.num_comments
document.title = `${thread.title} : ${thread.subreddit}`
const threadDiv = document.createElement("div")
threadDiv.id = "thread"
@ -348,6 +373,11 @@ async function fetchMultiple(url, data, init, jsonWalkdown=[], flattening=false)
return flattening ? flattenArray(json) : json
}
async function getToken() {
const data = await fetch(redditTokenURL, tokenInit).then(json)
return data.access_token
}
// ------------------------------------------------------------------------------
// ---------------------- Other less interesting functions ----------------------
// ------------------------------------------------------------------------------
@ -374,6 +404,12 @@ function json(x) {
return x.json()
}
function displayError(errorMsg) {
statusImage.src = errorImageSrc
setLoadingText("<b>ERROR: " + errorMsg + "</b>")
console.error(errorMsg)
}
// UTC -> "Reddit time format" (5 hours ago, just now, etc...)
function prettyDate(createdUTC) {
const currentUTC = Math.floor((new Date()).getTime() / 1000)

1
static/style.css
View file

@ -18,7 +18,6 @@ p {
}
a {
cursor: auto;
color: #8cb3d9;
}

4
templates/error.html
View file

@ -1,4 +0,0 @@
<div id="main-box" class="error-box">
<h2 class="error-title">Error 500: Internal server error</h2>
<p><b>Details</b>: {{.}}</p>
</div>

3
templates/footer.html
View file

@ -1,3 +0,0 @@
</div>
</body>
</html>

42
templates/frontpage.html
View file

@ -1,42 +0,0 @@
<div id="main-box" class="frontpage-box">
<h2 style="color:#CA302C; margin-top:5px">About</h2>
<p>
This is a site for viewing removed comments from Reddit.
It was created by Jesper Wrang and uses <a href="https://pushshift.io/">Jason Baumgartner</a> service for getting removed comments.</p>
<p>
<b>How to use</b>: go to any Reddit thread and replace the <i>reddit</i> in the URL with <i>removeddit</i>
<br>E.g.
<a href="/r/TwoXChromosomes/comments/6z1hch/">
https://www.removeddit.com/r/TwoXChromosomes/comments/6z1hch/
</a>
</p>
<p>
Note that this site is in it's early stages of development, meaning the site is unstable and will contain bugs.
If you find something that doesn't work please contact me at: <br><b>removeddit (at) gmail.com</b>
</p>
<h2 style="color:#239F2B">TODO</h2>
<ul>
<li>Collapsing comments</li>
<li>Get removed selftext of thread</li>
<li>Get nested "continue this thread"-comments to work</li>
</ul>
<h2 style="color:#1B767A">Acknowledgement</h2>
<p>
<a href="https://pushshift.io/">Jason Baumgartner</a> - for making all this possible with his amazing API
</p>
<p>
<a href="https://github.com/gamefreak">Scott McClaugherty</a> - for his reddit markdown parser
</p>
<p>
<a href="http://reddit.com/r/redditdev">/r/redditdev</a> - for helping with the Reddit API
</p>
<h2 style="color: #CA752C">Links/Contact</h2>
<p>For feedback and bug reports:
<ul>
<li>email: removeddit (at) gmail.com</li>
<li>reddit: <a href="https://www.reddit.com/user/Jubbeart/">/u/JubbeArt</a></li>
</ul>
<p>
<a href="https://github.com/JubbeArt/removeddit">Github for this project</a>
</p>
</div>

8
templates/thread.html
View file

@ -1,8 +0,0 @@
<div id="loading-comments" class="loading-comments"></div>
<script src="https://cdn.rawgit.com/gamefreak/snuownd/533e8dcb/snuownd.js"></script>
<script>
const threadID = "{{ .ThreadID }}";
const token = "{{ .Token }}";
const subreddit = "{{ .Subreddit }}";
</script>
<script src="/static/script.js"></script>

11
templates/header.html → thread.html
View file

@ -11,7 +11,14 @@
<a href="/" id="header-title-link"><h1 id="header-title">Removeddit [beta]</h1></a>
<div id="loading">
<p id="loading-text"></p>
<img style="display:none" id="loading-image"></img>
<img id="loading-image" src="/static/loading.gif"></img>
</div>
</header>
<div id="main">
<div id="main">
<div id="loading-comments" class="loading-comments"></div>
<script src="https://cdn.rawgit.com/gamefreak/snuownd/533e8dcb/snuownd.js"></script>
<script src="/static/id.js"></script>
<script src="/static/script.js"></script>
</div>
</body>
</html>