The foundation

Every ambitious project starts with a single, foundational question. For us, it was: what if getting data from any corner of the web was as simple and reliable as querying a database?

Not just sometimes. Every time. Not just from simple, static sites. From the complex, JavaScript-heavy applications that power the modern web. Not just for a single page, but for an entire website.

This question is the foundation of Supacrawler. And today, we're excited to share the first version of our answer with you.

The Vision

Our vision is to build an all-in-one data extraction platform that can handle any target, including social media, and make it affordable for everyone from a student with an idea to a growing business. But a grand vision is meaningless without a rock-solid foundation. We are launching with three core, powerful APIs that solve the most immediate and painful problems in web data extraction. This is our foundation, built with the craft and obsession of an indie developer.

The Scrape API: Content Made Simple

First, there is the Scrape API. This is your go-to tool for grabbing the content of a single page. Its power is in what you don't have to do. Unlike other APIs that require you to toggle browser rendering with confusing parameters, our Scrape API uses a real, hardened browser for every request, automatically. Modern websites run on JavaScript, so we believe browser rendering should be the default, not an option.

Here's how simple it is:

curl -G https://api.supacrawler.com/api/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d url="https://example.com" \
  -d format="markdown"

You get the final, fully-rendered HTML without ever having to think about it. You get the data as a user sees it, in clean Markdown or raw HTML, with a single API call.

The response is clean and structured:

{
  "success": true,
  "url": "https://example.com",
  "content": "# Example Domain\n\nThis domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.\n\n[More information...](https://www.iana.org/domains/example)",
  "title": "Example Domain",
  "metadata": {
    "status_code": 200
  }
}

The Screenshots API: Pixel-Perfect Captures

Second, we built the Screenshots API. Getting a pixel-perfect screenshot of a website should be trivial, but it rarely is. Our API solves this. You provide a URL, and we return a beautiful, high-resolution screenshot, customized to any viewport. It's a simple utility that saves hours of frustration wrestling with browser dependencies and font rendering issues on a server.

curl -G https://api.supacrawler.com/api/v1/screenshots \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d url="https://example.com" \
  -d device="desktop" \
  -d format="png"

The Jobs API: Scale Without Limits

Third, and perhaps most importantly, is the Jobs API. What if you need to scrape not one page, but an entire website? This is where the real infrastructure nightmare begins for most developers. Our Jobs API is the answer. You give it a starting URL, and it kicks off an asynchronous process to crawl and scrape the entire site up to your specified depth.

curl -X POST https://api.supacrawler.com/api/v1/jobs \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "depth": 3,
    "format": "markdown",
    "maxPages": 100
  }'

We handle the discovery, the queueing, and the concurrent scraping of every page, with built-in safeguards to manage memory and resources effectively. You get a Job ID to check the status, and when it's complete, the full dataset is ready for you. It turns a massive, unreliable task into a simple, fire-and-forget operation.

Link Discovery: Map Any Website

Our Scrape API also doubles as a powerful link discovery tool. Set the format to links and discover all the connections within a website:

curl -G https://api.supacrawler.com/api/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d url="https://example.com" \
  -d format="links" \
  -d depth=2 \
  -d maxLinks=100

Built for Modern Web

For websites that rely heavily on JavaScript (e.g., Single Page Applications built with React, Vue, or Svelte), our APIs handle the complexity automatically. Here's how to scrape a modern SPA:

curl -G https://api.supacrawler.com/api/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d url="https://spa-example.com" \
  -d format="markdown" \
  -d render=true \
  -d wait=3000

The Foundation for What's Next

These three tools are our starting line. They are the stable, well-engineered core upon which we will build everything else. Our roadmap is ambitious, but we know that trust is earned through reliability and great engineering.

We invite you to build on our foundation. Show us what you can create when the undifferentiated heavy lifting of data extraction is finally solved. This is just the beginning, and we want you to be a part of it.

Ready to get started? Check out our API documentation and start building with our free tier today.