Getting Started with Extractor API

The Extractor API allows you to extract clean text, title, author and other relevant metadata from articles, blogs, press releases, and other long-form pages. To get started you just need an API key and a target URL. If you don't have an API key, sign up for one of our plans. Please see the full documentation for a complete overview of the API.

Extractor Endpoint

Base URL

This is the base URL for the Extractor endpoint:

https://extractorapi.com/api/v1/extractor/

Parameters

The query string only requires two parameters - apikey and url.

curl "https://extractorapi.com/api/v1/extractor/?apikey=YOUR_API_KEY&url=TARGET_URL"

Example

So if your API key was 123456789 and your target url was nytimes.com/investigative-article, you'd structure your request this way:

curl "https://extractorapi.com/api/v1/extractor/?apikey=123456789&url=nytimes.com/investigative-article"

Output

If you entered your API key and URL correctly, you should see a JSON output like this:

			
			// OUTPUT

			{
			   "url": "https://nytimes.com/investigative-article", // Your target URL
			   "status": "COMPLETE", // Status of request - will display ERROR if there was an issue crawling the URL
			   "domain": "nytimes.com", // The domain associated with the target URL
			   "date_published": "2020-03-13T00:00:00Z",
			   "images": [
				  "nytimes.com/image1.png",
				  "nytimes.com/image2.png"
			   ],
			   "videos": [],
			   "title": "Engrossing NY Times Article", // The title of the content in the target URL
			   "author": [ // Author candidates
				 "S. King",
				 "S. King on Twitter"
			   ],
			   "text": "Gluten-free locavore kale chips." // The relevant article, blog, etc. text, minus boilerplate
			   "html": "<html><head><title>Title</title></head><body>Text</body></html>" // The page's HTML
			}
			

HTML & Raw Text

You can add the fields parameter to specify the fields you'd like to see in your response. This includes raw_text, which isn't included in responses by default. Note that if you specify the fields parameter, only the URL, status, text and chosen fields will be displayed.

curl "https://extractorapi.com/api/v1/extractor/?apikey=YOUR_API_KEY&fields=raw_text&url=YOUR_URL"
			
			// RAW TEXT

			{
			   "url": "https://nytimes.com/investigative-article", // Your target URL
			   "status": "COMPLETE", // Status of request - will display ERROR if there was an issue crawling the URL
			   "text": "Gluten-free locavore kale chips.", // The relevant article, blog, etc. text, minus boilerplate
			   "raw_text": "ADVERTISEMENT. Gluten-free locavore kale chips." // Text including boilerplate
			}
			

JavaScript Rendering & Waiting

If you are subscribed to any of the paid plans, you can add the js parameter to load any JavaScript on the page. Note that adding this parameter uses 5 requests instead of 1 request. If you're using the js parameter in your GET request, you can also the wait parameter to wait for the page to load for any number of seconds. Adding wait and a value of 3-4 is highly recommended is you're using js (wait doesn't cost any addition requests).

curl "https://extractorapi.com/api/v1/extractor/?apikey=YOUR_API_KEY&js=true&wait=3&url=YOUR_URL"

Text Translation

All paid plans include the ability to translated your extracted text from and to over 50 languages. To start translating simply add the translate_from and translate_to parameters with the appropriate language codes (see the dictionary below). Each call that uses these parameters will take an additional 10 requests per 500 characters. Do note that translations use deep learning and take time - the longer the extracted text, the longer it will take to translate.

So an article that's 3,000 characters (not words) would take 10 * (3,000 / 500) requests + 1 request (for the original extract call), or + 5 requests if the js parameter is used (in this example, that equals to 61 requests without the js parameter and 65 with). For instance, to translate from English to French, you would input the following:

curl "https://extractorapi.com/api/v1/extractor/?apikey=YOUR_API_KEY&translate_from=en&translate_to=fr&url=YOUR_URL"
			
			// LANGUAGE CODE DICTIONARY

			{
			   "Afrikaans": "af",
			   "Albanian": "sq",
			   "Amharic": "am",
			   "Arabic": "ar",
			   "Azerbaijani": "az",
			   "Bengali": "bn",
			   "Bosnian": "bs",
			   "Bulgarian": "bg",
			   "Chinese (Simplified)": "zh",
			   "Chinese (Traditional)": "zh-TW",
			   "Croatian": "hr",
			   "Czech": "cs",
			   "Danish": "da ",
			   "Dari": "fa-AF",
			   "Dutch": "nl ",
			   "English": "en",
			   "Estonian": "et",
			   "Finnish": "fi",
			   "French": "fr",
			   "French (Canada)": "fr-CA",
			   "Georgian": "ka",
			   "German": "de",
			   "Greek": "el",
			   "Hausa": "ha",
			   "Hebrew": "he ",
			   "Hindi": "hi",
			   "Hungarian": "hu",
			   "Indonesian": "id ",
			   "Italian": "it",
			   "Japanese": "ja",
			   "Korean": "ko",
			   "Latvian": "lv",
			   "Malay": "ms",
			   "Norwegian": "no",
			   "Pashto": "ps",
			   "Persian": "fa",
			   "Polish": "pl",
			   "Portuguese": "pt",
			   "Romanian": "ro",
			   "Russian": "ru",
			   "Serbian": "sr",
			   "Slovak": "sk",
			   "Slovenian": "sl",
			   "Somali": "so",
			   "Spanish": "es",
			   "Spanish (Mexico)": "es-MX",
			   "Swahili": "sw",
			   "Swedish": "sv",
			   "Tagalog": "tl",
			   "Tamil": "ta",
			   "Thai": "th",
			   "Turkish": "tr",
			   "Ukrainian": "uk",
			   "Urdu": "ur",
			   "Vietnamese": "vi"
			}
			

Visual Extractor

All plans include the option to extract article text and other data using our online visual tool. Check out the video below to see how to get started.

Creating Jobs

Think of a Job as a way to bulk-extract data from many URLs at once. When you use the visual extractor, you paste or upload the URLs you want to extract data from, give them a job name, and hit Extract (see the video above for an example).

Similarly, you can assign the URLs to a job name using the API. You will be able to view your jobs online (via the Jobs page, once you're registered), or using the API to check the status of each of your jobs. Once any job is finished (all the data is extracted), you can then download the results on the Jobs page or retrieve them programmatically. Learn more about creating jobs with the API here.

Plan Features

Depending on your Extractor API plan, you'll have different request limits per second and per month. You can find a more detailed comparison on the Features page.

News Search Endpoint

Base URL

This is the base URL for the Search endpoint:

https://extractorapi.com/api/v1/search/

Parameters

The query string only requires two parameters - apikey and search_term.

curl "https://extractorapi.com/api/v1/search/?apikey=YOUR_API_KEY&search_term=SEARCH_TERM"

Example

So if your API key was 123456789 and your search term was otters, you'd structure your request this way:

curl "https://extractorapi.com/api/v1/search/?apikey=123456789&search_term=otters"

Output

Every request can have up to 100 results, sorted by date published. If you entered your API key correctly and our system found results for your search term, you should see a JSON output like this:

			
			// SEARCH OUTPUT

			[
			   {
			     "title": "Otters are too cute, says Fauci",
			     "url": "https://otterville.com/fauci-otters-too-cute",
			     "summary": "In an unusual change of tone, NIAID Director Anthony Fauci took time out of his restless schedule to warn the American public on the cuteness of otters.",
			     "site_name": "OtterVille",
			     "date_published": "2020-03-13T00:00:00Z"
			   },
			   {
			     "title": "Otters protest zoo's filthy pond",
			     "url": "https://otterlyoutrageous.com/otters-zoo-protest",
			     "summary": "Otters today staged a non-cute protest against a local zoo's dirty pond water.",
			     "site_name": "Otterly Outrangeous",
			     "date_published": "2020-03-12T00:00:00Z"
			   }
			]
			

News Locations

You can specify both language and location in your news search using the location parameter. You can find usable language-country codes below.

curl "https://extractorapi.com/api/v1/search/?apikey=YOUR_API_KEY&search_term=otters&location=es-AR"
			
			// LANGUAGE-COUNTRY CODE DICTIONARY

			{
			  "Argentina": {
			    "Spanish": "es-AR"
			  },
			  "Australia": {
			    "English": "en-AU"
			  },
			  "Austria": {
			    "German": "de-AT"
			  },
			  "Belgium": {
			    "Dutch": "nl-BE"
			  },
			  "Brazil": {
			    "Portuguese": "pt-BR"
			  },
			  "Canada": {
			    "English": "en-CA",
			    "French": "fr-CA"
			  },
			  "Chile": {
			    "Spanish": "es-CL"
			  },
			  "Denmark": {
			    "Danish": "da-DK"
			  },
			  "English": {
			    "general": "en-XA"
			  },
			  "Finland": {
			    "Finnish": "fi-FI"
			  },
			  "France": {
			    "French": "fr-FR"
			  },
			  "Germany": {
			    "German": "de-DE"
			  },
			  "Hong Kong SAR": {
			    "Traditional Chinese": "zh-HK"
			  },
			  "India": {
			    "English": "en-IN"
			  },
			  "Indonesia": {
			    "English": "en-ID"
			  },
			  "Ireland": {
			    "English": "en-IE"
			  },
			  "Italy": {
			    "Italian": "it-IT"
			  },
			  "Japan": {
			    "Japanese": "ja-JP"
			  },
			  "Korea": {
			    "Korean": "ko-KR"
			  },
			  "Malaysia": {
			    "English": "en-MY"
			  },
			  "Mexico": {
			    "Spanish": "es-MX"
			  },
			  "Netherlands": {
			    "Dutch": "nl-NL"
			  },
			  "New Zealand": {
			    "English": "en-NZ"
			  },
			  "People's republic of China": {
			    "Chinese": "zh-CN"
			  },
			  "Poland": {
			    "Polish": "pl-PL"
			  },
			  "Republic of the Philippines": {
			    "English": "en-PH"
			  },
			  "Russia": {
			    "Russian": "ru-RU"
			  },
			  "Singapore": {
			    "English": "en-SG"
			  },
			  "South Africa": {
			    "English": "en-ZA"
			  },
			  "Spain": {
			    "Spanish": "es-ES"
			  },
			  "Spanish": {
			    "general": "es-XL"
			  },
			  "Sweden": {
			    "Swedish": "sv-SE"
			  },
			  "Switzerland": {
			    "French": "fr-CH",
			    "German": "de-CH"
			  },
			  "Taiwan": {
			    "Traditional Chinese": "zh-TW"
			  },
			  "Turkey": {
			    "Turkish": "tr-TR"
			  },
			  "United Kingdom": {
			    "English": "en-GB"
			  },
			  "United States": {
			    "English": "en-US",
			    "Spanish": "es-US"
			  }
			}