diff --git a/.gitignore b/.gitignore index 3b8d499..3294e9e 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,2 @@ .secret -access.log -error.log -server .vscode/settings.json diff --git a/README.md b/README.md index 33d9e00..d048538 100644 --- a/README.md +++ b/README.md @@ -1,41 +1,100 @@ # Removeddit [Removeddit](https://removeddit.com) is a site for viewing removed comments from [Reddit](https://www.reddit.com). -Usage: go to any reddit thread and change the "reddit" in the URL to "removeddit". +Usage: go to any reddit thread and change the `reddit` in the URL to `removeddit`. The site will only display the removed comments and thier parents, not the full thread. -This is a done by comparing the comments found from Reddit API and comments from [Jason Baumgartners](https://pushshift.io/) [Pushshift Reddit API](https://github.com/pushshift/api). The backend of the site is written in Go and frontend with JavaScript (ES6). You can use this content however you want, as long as it's non-commercial. +This is a done by comparing the comments found from Reddit API and comments from [Jason Baumgartners](https://pushshift.io/) [Pushshift Reddit API](https://github.com/pushshift/api). The frontend is written in pure Javascript (ES6). You can use this code however you want, as long as it's non-commercial. -# Quick start -Install [Go](https://golang.org/) and [this](https://godoc.org/golang.org/x/crypto/acme/autocert) Go package with: - -```go get golang.org/x/crypto/acme/autocert``` - -(Note: if you only want to have a local server you could remove all references to this package in *server.go* and it should work if you run it with the *-d* flag) - -Create a reddit web app [here](https://www.reddit.com/prefs/apps/). - -Open *secret.go* and fill out all the variables: -```go -const ( - appName = "Name of your reddit web app" - userName = "Reddit username" - clientID = "Go to the link above and check the string under 'web app'" - clientSecret = "Go to the link above, click the 'edit' link and you'll see the secret" - version = "The version of your reddit web app, doesn't really matter, use e.g. '1.0'" - hostname = "The hostname of your server in production. Not used if you run locally" -) +# The "I just want to get this shit running" +Using [Ubuntu 16.04](http://releases.ubuntu.com/16.04/) and [nginx](https://www.nginx.com/resources/wiki/) +``` +sudo git clone git@github.com:JubbeArt/removeddit.git /var/www/ +sudo apt install -y nginx +sudo cp /var/www/removeddit/config/basic /etc/nginx/sites-available/default ``` -And everything should be set up. Now just build the project with: +Create a reddit app [here](https://www.reddit.com/prefs/apps/), select **installed app**. For "redirect url" it doesn't really matter in this case, you can pick `http://localhost`. -```go build server.go secret.go``` +Copy the **client ID** for your app set it as a variable in `id.js`, e.g. with +``` +sudo nano /var/www/removeddit/static/id.js +# Insert with ctrl-shift-v +# Save with ctrl-o, exit with ctrl-x +``` -And run the program: +Restart nginx and visit "localhost" +``` +sudo service nginx restart +``` -```./server``` +# The "I care about HTTPS and security" +In this part we'll set up [nginx](https://www.nginx.com/resources/wiki/) with SSL and set up a free renewing SSL certificates with [Let's Encrypt](https://letsencrypt.org/). I assume you've already done the guide above. -(just `server` in Windows) +## Nginx.conf +Add the following to `/etc/nginx/nginx.conf` in the **http-block** (you can read about them [here](https://gist.github.com/plentz/6737338)) +``` +server_tokens off; +add_header X-Frame-Options SAMEORIGIN; +add_header X-Content-Type-Options nosniff; +add_header X-XSS-Protection "1; mode=block"; +``` -If you want to run it this server on localhost use the *d* flag: +In the same file you also want to change `gzip on` to `gzip off`. (read more [here](https://github.com/h5bp/server-configs-nginx/issues/72)) -```./server -d``` +## Nginx server config +Copy the ssl config and create a soft link. Create folder for logs and also remove the default config +``` +sudo cp /var/www/removeddit/config/ssl /etc/nginx/sites-available/removeddit.com +sudo ln -s /etc/nginx/sites-available/removeddit.com /etc/nginx/sites-enabled/removeddit.com +sudo mkdir /var/log/nginx/removeddit +sudo rm /etc/nginx/sites-enabled/default +``` + +Change the "server_name" in the config to your domain name. + +## SSL with Let's encrypt +Read the full guide [here](https://certbot.eff.org/#ubuntutyakkety-nginx). Start of by installing the Let's Encrypt client [certbot](https://certbot.eff.org/) +``` +sudo apt install software-properties-common +sudo add-apt-repository ppa:certbot/certbot +sudo apt update +sudo apt install -y python-certbot-nginx +``` + +Copy the Let's Encrypt config file for our site +``` +sudo mkdir /etc/letsencrypt/configs +cp /var/www/removeddit/config/letsencrypt /etc/letsencrypt/config/removeddit.com.conf +``` + +In this config file you change the domains you want and a email for when the certificates are close to expiring. + +This is when the webmasters start praying to God Almighty, for only He can deside the fate of the certbot. +Forgive me, Father, for I have sinned. Just don't fuck up certs you asshole. +``` +sudo certbot --nginx certonly +``` + +You'll now have a valid SSL certificate (hopefully)! You might have to edit the path for `ssl_certificate` and `ssl_certificate_key` in `/etc/nginx/sites-available/removeddit.com` depending on where the certs are located. (Should be in `/etc/letsencrypt/live/`) + +## Automated renewal of certs +The certificate expires after 90 days so we want a way to atomatically update the certs. +There are multiple ways of doing this but I find the easiest to be [cron jobs](https://en.wikipedia.org/wiki/Cron). + +First we'll test if renewing atcutally works with + +``` +sudo certbot renew --dry-run +``` + +If everything works fine we can create a cron with ```sudo crontab -e``` and select an editor your comfortable with (I like *nano*). +Add the following lines at the bottom + +``` +# Let's Encrypt cert renewal for all sites (runs every day at 04:30) +30 4 * * * certbot renew --post-hook "systemctl reload nginx" +``` + +Then just restart nginx and that should do it! + +This guide was mostly written for myself, you learn a shitton writing guides, highly recommended. Hopefully you learned something too. \ No newline at end of file diff --git a/config/basic b/config/basic new file mode 100644 index 0000000..ac2ac86 --- /dev/null +++ b/config/basic @@ -0,0 +1,20 @@ +server { + listen 80 default_server; + listen [::]:80 default_server; + + root /var/www/removeddit; + index index.html; + server_name _; + + location / { + try_files /index.html /index.html =404; + } + + location /r/ { + try_files /thread.html /thread.html =404; + } + + location /static/ { + try_files $uri $uri/ =404; + } +} \ No newline at end of file diff --git a/config/letsencrypt.conf b/config/letsencrypt.conf new file mode 100644 index 0000000..58c6c71 --- /dev/null +++ b/config/letsencrypt.conf @@ -0,0 +1,9 @@ +# Things to change +domains = removeddit.com, www.removeddit.com +email = removeddit@gmail.com + +# These defaults are fine (could use 4096 instead of 2048) +rsa-key-size = 2048 +text = True +authenticator = webroot +webroot-path = /var/www/removeddit/ \ No newline at end of file diff --git a/config/ssl b/config/ssl new file mode 100644 index 0000000..9362847 --- /dev/null +++ b/config/ssl @@ -0,0 +1,39 @@ +server { + listen 443 ssl http2 default_server; + listen [::]:443 ssl http2 default_server; + + root /var/www/removeddit; + index index.html; + server_name removeddit.com; + + access_log /var/log/nginx/removeddit/access.log; + error_log /var/log/nginx/removeddit/error.log; + + ssl_certificate /etc/letsencrypt/live/removeddit.com/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/removeddit.com/privkey.pem; + + location / { + try_files /index.html /index.html =404; + } + + location /r/ { + try_files /thread.html /thread.html =404; + } + + location /static/ { + try_files $uri $uri/ =404; + } +} + +server { + listen 80 default_server; + listen [::]:80 default_server; + + location / { + return 301 https://$host$request_uri; + } + + location ~ /.well-known { + allow all; + } +} diff --git a/index.html b/index.html new file mode 100644 index 0000000..f9b414d --- /dev/null +++ b/index.html @@ -0,0 +1,58 @@ + + + + Removeddit + + + + + + +
+
+

About

+

+ This is a site for viewing removed comments from Reddit. + It was created by Jesper Wrang and uses Jason Baumgartner service for getting removed comments.

+

+ How to use: go to any Reddit thread and replace the reddit in the URL with removeddit +
E.g. + + https://www.removeddit.com/r/TwoXChromosomes/comments/6z1hch/ + +

+

+ Note that this site is in it's early stages of development, meaning the site is unstable and will contain bugs. + If you find something that doesn't work please contact me at:
removeddit (at) gmail.com +

+

TODO

+ +

Acknowledgement

+

+ Jason Baumgartner - for making all this possible with his amazing API +

+

+ Scott McClaugherty - for his reddit markdown parser +

+

+ /r/redditdev - for helping with the Reddit API +

+

Links/Contact

+

For feedback and bug reports: +

+

+ Github for this project +

+
+
+ + \ No newline at end of file diff --git a/secret.go b/secret.go deleted file mode 100644 index d162de9..0000000 --- a/secret.go +++ /dev/null @@ -1,10 +0,0 @@ -package main - -const ( - appName = "" - userName = "" - clientID = "" - clientSecret = "" - version = "" - hostname = "" -) diff --git a/server.go b/server.go deleted file mode 100644 index e39b4dc..0000000 --- a/server.go +++ /dev/null @@ -1,194 +0,0 @@ -package main - -import ( - "crypto/tls" - "encoding/json" - "errors" - "flag" - "fmt" - "html/template" - "io/ioutil" - "log" - "net/http" - "net/url" - "os" - "strings" - "time" - - "golang.org/x/crypto/acme/autocert" -) - -// consts -const ( - userAgent = "Javascript:" + appName + ":" + version + "s (by /u/" + userName + ")" - tokenURL = "https://www.reddit.com/api/v1/access_token" - tmplFolder = "templates/" -) - -var ( - client = &http.Client{Timeout: time.Second * 10} - tokenData = url.Values{"grant_type": []string{"client_credentials"}} - templates = template.Must(template.ParseFiles( - tmplFolder+"header.html", - tmplFolder+"footer.html", - tmplFolder+"thread.html", - tmplFolder+"frontpage.html", - tmplFolder+"error.html", - )) - - accessLog *log.Logger - errorLog *log.Logger -) - -type tokenResponse struct { - AccessToken string `json:"access_token"` - TokenType string `json:"token_type"` - ExpiresIn int `json:"expires_in"` - Scope string `json:"scope"` -} - -type threadPageData struct { - Token string - Subreddit string - ThreadID string - CommentIDs []string -} - -func main() { - initLogging() - accessLog.Println("Staring server") - - var debugMode bool - flag.BoolVar(&debugMode, "d", false, "debug mode: run the server on localhost") - flag.Parse() - - http.HandleFunc("/", pageHandler(mainHandler)) - http.HandleFunc("/r/", pageHandler(threadHandler)) - - // Serve static files, in production use cdn.rawgit.com (nvm fuck this, always serve static) - fs := http.FileServer(http.Dir("static")) - http.Handle("/static/", http.StripPrefix("/static/", fs)) - - // Run locally for debugging/testing - if debugMode { - fmt.Println("Starting server on http://localhost:8080") - - if err := http.ListenAndServe(":8080", nil); err != nil { - log.Fatalf("Could not start server: %v", err) - } - } else { // Run in production - certManager := autocert.Manager{ - Prompt: autocert.AcceptTOS, - HostPolicy: autocert.HostWhitelist(hostname, "www."+hostname), - Cache: autocert.DirCache("certs"), - } - - server := &http.Server{ - ReadTimeout: 20 * time.Second, - WriteTimeout: 20 * time.Second, - TLSConfig: &tls.Config{ - GetCertificate: certManager.GetCertificate, - }, - ErrorLog: errorLog, - } - - redirect := &http.Server{ - ReadTimeout: 10 * time.Second, - WriteTimeout: 10 * time.Second, - Handler: http.HandlerFunc(redirectTLS), - } - - go func() { - if err := redirect.ListenAndServe(); err != nil { - log.Fatalf("Could not start redirect server: %v", err) - errorLog.Printf("Could not start redirect server: %v", err) - } - }() - - if err := server.ListenAndServeTLS("", ""); err != nil { - log.Fatalf("Could not start server with SSL/TLS Certificate: %v", err) - errorLog.Printf("Could not start server with SSL/TLS Certificate: %v", err) - } - } - - accessLog.Println("Shutting down server") - log.Println("Shutting down server") -} - -func initLogging() { - accessFile, _ := os.OpenFile("access.log", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666) - errorFile, _ := os.OpenFile("error.log", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666) - - accessLog = log.New(accessFile, "", log.Ldate|log.Ltime|log.Lshortfile) - errorLog = log.New(errorFile, "", log.Ldate|log.Ltime|log.Lshortfile) -} - -// Wrap page handle function for the access log -func pageHandler(fn func(http.ResponseWriter, *http.Request)) http.HandlerFunc { - return func(w http.ResponseWriter, r *http.Request) { - accessLog.Println(r.URL.Path) - fn(w, r) - } -} - -func renderTemplate(w http.ResponseWriter, pageName string, data interface{}) { - templates.ExecuteTemplate(w, "header.html", nil) - templates.ExecuteTemplate(w, pageName+".html", data) - templates.ExecuteTemplate(w, "footer.html", nil) -} - -func handleError(w http.ResponseWriter, msg string) { - renderTemplate(w, "error", msg) - errorLog.Println("Error: " + msg) -} - -func mainHandler(w http.ResponseWriter, r *http.Request) { - renderTemplate(w, "frontpage", nil) -} - -func threadHandler(w http.ResponseWriter, r *http.Request) { - pathParts := strings.Split(r.URL.Path, "/") - - if len(pathParts) < 5 { - handleError(w, "Missing necessary parts of the URL") - return - } - - token, err := getAPIToken() - if err != nil { - handleError(w, err.Error()) - return - } - - data := &threadPageData{ - Token: token, - Subreddit: pathParts[2], - ThreadID: pathParts[4], - } - - renderTemplate(w, "thread", data) -} - -func getAPIToken() (string, error) { - req, _ := http.NewRequest("POST", tokenURL, strings.NewReader(tokenData.Encode())) - req.Header.Add("User-Agent", userAgent) - req.SetBasicAuth(clientID, clientSecret) - - resp, err := client.Do(req) - if err != nil { - return "", errors.New("Coulnd't get API token from Reddit") - } - defer resp.Body.Close() - body, err := ioutil.ReadAll(resp.Body) - if err != nil { - return "", errors.New("Coulnd't get API token from Reddit, timeout error") - } - - var r tokenResponse - json.Unmarshal(body, &r) - return r.AccessToken, nil -} - -func redirectTLS(w http.ResponseWriter, r *http.Request) { - http.Redirect(w, r, fmt.Sprintf("https://www.%s/%s", hostname, r.RequestURI), http.StatusMovedPermanently) -} diff --git a/static/id.js b/static/id.js new file mode 100644 index 0000000..55206f2 --- /dev/null +++ b/static/id.js @@ -0,0 +1,3 @@ +// Change this to your own client ID: https://www.reddit.com/prefs/apps +// The app NEEDS TO BE an installed app and NOT a web apps +const clientID = "" \ No newline at end of file diff --git a/static/script.js b/static/script.js index 0b2f989..a3d941a 100644 --- a/static/script.js +++ b/static/script.js @@ -3,6 +3,17 @@ const markdown = SnuOwnd.getParser() const htmlParser = new DOMParser(); +const pathname = window.location.pathname.split("/") + +// Just fucking crach if the user is retarded +if(pathname.length <= 4) { + displayError("Missing necessary parts of the URL") +} + +const subreddit = pathname[2] +const threadID = pathname[4] + +const redditTokenURL = "https://www.reddit.com/api/v1/access_token" const redditThreadURL = `https://oauth.reddit.com/r/${subreddit}/comments/${threadID}` const redditMorechildrenURL = `https://oauth.reddit.com/api/morechildren?link_id=t3_${threadID}&children=` const redditSingleCommentURL = `https://oauth.reddit.com/r/${subreddit}/api/info/?id=` @@ -11,12 +22,12 @@ const pushshiftCommentsURL = "https://api.pushshift.io/reddit/comment/search?ids const mainDiv = document.getElementById("main") const loadingText = document.getElementById("loading-text") -const loadingImage = document.getElementById("loading-image") -const loadingImageSrc = "/static/loading.gif" +const statusImage = document.getElementById("loading-image") const doneImageSrc = "/static/done.png" +const errorImageSrc = "/static/error.png" // I actually haven't found a better way of doing this... -// Imgur has images-links with no indication that they are actually images +// Imgur has image-links with no indication that they are actually images const imageHosts = ["i.redd.it", "flickr.com", "i.imgur.com", "imgur.com", "m.imgur.com"] const commentIDs = [] @@ -29,17 +40,30 @@ const deletedCommentIDs = [] let totalComments -const redditInit = { - headers: { "Authorization": "bearer " + token } +const tokenFormData = new FormData(); +tokenFormData.append("grant_type", "https://oauth.reddit.com/grants/installed_client"); +tokenFormData.append("device_id", "DO_NOT_TRACK_THIS_DEVICE") +tokenFormData.append("scope", "account,edit,history,mysubreddits,privatemessages,report,save,submit,subscribe,vote,wikiedit,wikiread,read,flair,identity,modconfig") +tokenFormData.append("api_type", "json") + +const tokenInit = { + headers: { "Authorization": "Basic " + btoa(clientID + ":") }, + method: "POST", + body: tokenFormData +} + +let token +let redditInit = { + headers: { "Authorization": "" } } loadPage() +// Maybe a little to long a function... async function loadPage() { - - loadingImage.src = loadingImageSrc - loadingImage.style.display = "block" setLoadingText("Loading thread...") + token = await getToken() + redditInit.headers.Authorization = "bearer " + token // Get thread from reddit and all comment IDs (inc. removed ones) from pushshift const requests = [ @@ -71,7 +95,7 @@ async function loadPage() { await generateComments(removedComments) setLoadingText("") - loadingImage.src = doneImageSrc + statusImage.src = doneImageSrc } // ------------------------------------------------------------------------------ @@ -262,6 +286,7 @@ function createComments() { function generateThread(data) { const thread = data[0].data.children[0].data totalComments = thread.num_comments + document.title = `${thread.title} : ${thread.subreddit}` const threadDiv = document.createElement("div") threadDiv.id = "thread" @@ -348,6 +373,11 @@ async function fetchMultiple(url, data, init, jsonWalkdown=[], flattening=false) return flattening ? flattenArray(json) : json } +async function getToken() { + const data = await fetch(redditTokenURL, tokenInit).then(json) + return data.access_token +} + // ------------------------------------------------------------------------------ // ---------------------- Other less interesting functions ---------------------- // ------------------------------------------------------------------------------ @@ -374,6 +404,12 @@ function json(x) { return x.json() } +function displayError(errorMsg) { + statusImage.src = errorImageSrc + setLoadingText("ERROR: " + errorMsg + "") + console.error(errorMsg) +} + // UTC -> "Reddit time format" (5 hours ago, just now, etc...) function prettyDate(createdUTC) { const currentUTC = Math.floor((new Date()).getTime() / 1000) diff --git a/static/style.css b/static/style.css index d851f75..904b27b 100644 --- a/static/style.css +++ b/static/style.css @@ -18,7 +18,6 @@ p { } a { - cursor: auto; color: #8cb3d9; } diff --git a/templates/error.html b/templates/error.html deleted file mode 100644 index eca8550..0000000 --- a/templates/error.html +++ /dev/null @@ -1,4 +0,0 @@ -
-

Error 500: Internal server error

-

Details: {{.}}

-
\ No newline at end of file diff --git a/templates/footer.html b/templates/footer.html deleted file mode 100644 index 92f6532..0000000 --- a/templates/footer.html +++ /dev/null @@ -1,3 +0,0 @@ - - - \ No newline at end of file diff --git a/templates/frontpage.html b/templates/frontpage.html deleted file mode 100644 index 85f1445..0000000 --- a/templates/frontpage.html +++ /dev/null @@ -1,42 +0,0 @@ -
-

About

-

- This is a site for viewing removed comments from Reddit. - It was created by Jesper Wrang and uses Jason Baumgartner service for getting removed comments.

-

- How to use: go to any Reddit thread and replace the reddit in the URL with removeddit -
E.g. - - https://www.removeddit.com/r/TwoXChromosomes/comments/6z1hch/ - -

-

- Note that this site is in it's early stages of development, meaning the site is unstable and will contain bugs. - If you find something that doesn't work please contact me at:
removeddit (at) gmail.com -

-

TODO

- -

Acknowledgement

-

- Jason Baumgartner - for making all this possible with his amazing API -

-

- Scott McClaugherty - for his reddit markdown parser -

-

- /r/redditdev - for helping with the Reddit API -

-

Links/Contact

-

For feedback and bug reports: -

-

- Github for this project -

-
\ No newline at end of file diff --git a/templates/thread.html b/templates/thread.html deleted file mode 100644 index 7f77509..0000000 --- a/templates/thread.html +++ /dev/null @@ -1,8 +0,0 @@ -
- - - \ No newline at end of file diff --git a/templates/header.html b/thread.html similarity index 53% rename from templates/header.html rename to thread.html index ca87d17..e990955 100644 --- a/templates/header.html +++ b/thread.html @@ -11,7 +11,14 @@

Removeddit [beta]

- +
-
\ No newline at end of file +
+
+ + + +
+ + \ No newline at end of file