Home / Use cases / How discourse.org uses social data + build your own with…
How discourse.org uses social data + build your own with Scrapall
discourse.org is a customizable, scalable forum/community SaaS platform for building online communities, as stated on its website. Products like discourse.org depend on public social data. With Scrapall you can pull the same kind of public data from Facebook Ad Library, Reddit through 8 REST endpoints and build your own version.
Updated · by the Scrapall team
Platforms and endpoints
| Product | Platform | Endpoint | What it returns |
|---|---|---|---|
| discourse.org | /v1/reddit/post | Post | |
| discourse.org | /v1/reddit/post_comments | Post Comments | |
| discourse.org | /v1/reddit/post_captions | Post Transcript | |
| discourse.org | /v1/reddit/search | Search | |
| discourse.org | /v1/reddit/subreddit | Subreddit Posts | |
| discourse.org | /v1/reddit/subreddit_details | Subreddit Details | |
| discourse.org | /v1/reddit/subreddit_search | Subreddit Search | |
| discourse.org | Facebook Ad Library | /v1/facebook/ads_ad | Ad Details |
Example request
Every endpoint is a plain HTTPS GET or POST with your key in the X-Scapi-Key header. Responses are JSON.
curl "https://api.scrapall.bipsport.com/scapi/v1/reddit/post?url=https://www.reddit.com/r/webscraping/comments/1q6pxwn/just_started_web_scraping_is_this_a_good_start/" \
-H "X-Scapi-Key: YOUR_API_KEY"What you can build
To build a tool in the same space as discourse.org, collect public social data from Facebook Ad Library, Reddit with the endpoints below, keep the responses in your own database, and add the analysis or workflow your users pay for. The facts about discourse.org on this page come from the sources listed at the bottom.
How to build it, step by step
- Get an API key on the Buy page; new accounts start with 20 free credits.
- Call /v1/reddit/post first to check the response fields you need.
- Add the other endpoints (/v1/reddit/post_comments, /v1/reddit/post_captions, /v1/reddit/search) and store each JSON response with a timestamp.
- Schedule the collector (cron or a queue) and build the view or alert on top of the stored data.
FAQ
Which platforms does this use case need?
It uses public data from Facebook Ad Library, Reddit, all reachable through one Scrapall API key.
How do I authenticate the requests?
Send your key in the X-Scapi-Key header on every request. Responses are JSON.
How much does it cost to start?
New accounts get 20 free credits. After that you pay per request in USDT; see the Pricing page.
Can I run it on a schedule?
Yes. Each endpoint is a plain HTTPS call, so a cron job or queue worker can collect fresh data every day.
Sources
- https://discourse.org/
- https://github.com/discourse/discourse
- https://www.discourse.org/
- https://www.discourse.org/jobs
- https://www.discourse.org/pricing
- https://x.com/paulofonseca__/status/1892637930100809822