Everything you need to integrate universal web scraping into your applications.
The OmniCrawl API provides a simple and powerful way to extract structured data from any website. It uses advanced AI techniques to understand and navigate web pages, extracting all content including dynamic elements that traditional scrapers miss.
Scrapes the provided URL and returns the content in either JSON or Markdown format.
| Parameter | Type | Required | Description |
|---|---|---|---|
| url | string | Required | The URL of the website to scrape. |
| response_format | string | Optional | The format of the response. Either "json" or "text" (Markdown). |
| api_key | string | Required | Your OmniCrawl API key |
Returns either a JSON object or Markdown text depending on the specified response_format.
{
"content": "Welcome to Example.com! This domain is established to be used for illustrative examples in documents. You may use this domain in examples without prior coordination or asking for permission.\n\nWhen using this domain in examples:\n- The content should be clear and educational\n- No real user data should be included\n- Examples should be self-contained\n- Documentation should reference this domain appropriately",
"section": "Introduction",
"sections": [
{
"content": "Example.com is reserved for use in documentation and testing. This domain is specifically set aside for demonstrating how websites and documentation work. When you need a domain name placeholder in examples, this is the recommended choice.",
"section": "Purpose and Usage",
"title": "Purpose and Usage"
},
{
"content": "When using Example.com in your documentation:\n1. Always use it as a placeholder\n2. Never include sensitive information\n3. Keep examples clear and concise\n4. Reference it appropriately in your docs",
"section": "Best Practices",
"title": "Best Practices"
},
{
"content": "Example.com can be used in various documentation scenarios:\n- API documentation\n- Web development tutorials\n- Network configuration examples\n- Security testing documentation\n- Educational materials",
"section": "Use Cases",
"title": "Use Cases"
}
],
"title": "Example.com Documentation Guide"
}{
"detail": "Error message describing what went wrong"
}Check if the API is up and running properly.
{
"status": "ok"
}Here are examples showing how to use the OmniCrawl API in various programming languages:
import requests
response = requests.post(
"https://omnicrawl.onrender.com/scrape",
headers={
"Content-Type": "application/json",
"Authorization": f"Bearer <your_api_key>"
},
json={
"url": "https://example.com",
"response_format": "json"
}
)The OmniCrawl API supports two response formats: JSON and text (Markdown).
The JSON response provides structured data that's easy to parse and use in applications.
{
"content": "Example.com is a domain reserved for use in documentation and examples. It provides a safe, standardized way to demonstrate web concepts without using real domains.\nWhen using example.com:\n- Always use it as a placeholder\n- Never include sensitive information\n- Keep examples clear and concise\n- Reference appropriately in documentation\n\n> **Note:** This domain is managed by IANA and is guaranteed to remain reserved for documentation purposes.",
"section": "Example.com Usage",
"sections": [
{
"content": "Example.com is specifically reserved for documentation and testing purposes. It provides a reliable domain name that won't conflict with real-world usage.",
"section": "What is Example.com?",
"title": "What is Example.com?"
},
{
"content": "Example.com can be used in any documentation that requires a domain name placeholder. This includes API documentation, tutorials, and technical specifications.",
"section": "Use Cases",
"title": "Use Cases"
},
{
"content": "When using example.com in documentation:\n- Use it consistently throughout your docs\n- Don't mix it with other example domains\n- Ensure examples are self-contained\n- Keep the context educational",
"section": "Best Practices",
"title": "Best Practices"
}
],
"title": "Example.com Documentation"
}The text response provides clean, formatted Markdown that's perfect for documentation or content display.
# Example.com Documentation Guide Welcome to Example.com! This domain is established to be used for illustrative examples in documents. You may use this domain in examples without prior coordination or asking for permission. When using this domain in examples: - The content should be clear and educational - No real user data should be included - Examples should be self-contained - Documentation should reference this domain appropriately ## Purpose and Usage Example.com is reserved for use in documentation and testing. This domain is specifically set aside for demonstrating how websites and documentation work. When you need a domain name placeholder in examples, this is the recommended choice. ## Best Practices When using Example.com in your documentation: 1. Always use it as a placeholder 2. Never include sensitive information 3. Keep examples clear and concise 4. Reference it appropriately in your docs ## Use Cases Example.com can be used in various documentation scenarios: - API documentation - Web development tutorials - Network configuration examples - Security testing documentation - Educational materials
Try the OmniCrawl API now and see how easy it is to extract structured data from any website.