1 / 5

Extracting Job Postings Data From Indeed com Using Python

Job hunting is one of the most tactical tasks. It requires special attention to each job posting. If you are applying for different roles, it requires a unique application for every position. These are a few of the many hassles in the job-hunting process. One more such task is to obtain data for job postings from different companies. A manual approach to finding these posts from different companies would not be easy, especially when you have other important stuff to take care of, like preparing for an interview.

BotScrapers
Télécharger la présentation

Extracting Job Postings Data From Indeed com Using Python

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Extracting Job Postings Data From Indeed.com Using Python Job hunting is one of the most tactical tasks. It requires special attention to each job posting. If you are applying for different roles, it requires a unique application for every position. These are a few of the many hassles in the job-hunting process. One more such task is to obtain data for job postings from different companies. A manual approach to finding these posts from different companies would not be easy, especially when you have other important stuff to take care of, like preparing for an interview. There is a solution, and it's called web scraping. Few of the best web scraping services USA help you scrape job websites like Indeed using Python. Here's a guide to how Indeed search operations will help us mimic the same in our scraper. Introduction To begin with, you will require an HTTP client library to scrape Indeed. For your web scraper, you can install the httpx library by the pip console command. $ pip install httpx httpx is the most preferred proposal to be used due to several reasons. One of them being that httpx supports the http2 protocol. Apart from httpx, many HTTP clients in Python exist. For example, there is requests, aiohttp, and more! Moreover, httpx is also a very fast option as it supports the asynchronous Python. Job Hunt This is when we dive into the official web page of Indeed, where we can scrape the job profiles. Once you land on the home screen, you will notice that your search query, once inserted, is redirected to the search URL, including a few functions. https://www.indeed.com/jobs?q=python&l=Texas Right now we are looking for jobs specifically in Texas. To do so, we need to create a request with l=Texas with you q=Python. The URL goes like: import httpx HEADERS = {

  2. "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36", "Accept-Encoding": "gzip, deflate, br", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*; q=0.8", "Connection": "keep-alive", "Accept-Language": "en-US,en;q=0.9,lt;q=0.8,et;q=0.7,de;q=0.6", } response = httpx.get("https://www.indeed.com/jobs?q=python&l=Texas", headers=HEADERS) print(response) An important point to remember: There is a possibility of witnessing a response code 403. In this case, you must be blocked. Moving on, the response here has brought us 10+ job listings already. There are more pages to explore. But before that, we need to parse job data from this specific response. We have a simple way to unearth the job listings data deep from the HTML as a JSON document. There is another tedious method by parsing the HTML document by using CSS or XPath selectors. view-source : https : //www.indeed.com/jobs?q=python&l=Texas <script id="mosaic-data" type="text/javascript"> … window.mosaic.providerData["mosaic-provider-jobcards"]={ "metaData" : { "mosaic ProviderJobCardsModel" : { "results" : [ { company" : "Coding with Kids", "jobkey" : "a82cf0bd2092efa3", "salarySnippet" : { "salaryTextFormatted" : false, "source" : "EXTRACTION", "text" : "$15 - $25 an hour" }, }; …

  3. </script> Therefore, parsing data would be better in this regular expression. import httpx HEADERS = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36", "Accept-Encoding": "gzip, deflate, br", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*; q=0.8", "Connection": "keep-alive", "Accept-Language": "en-US,en;q=0.9,lt;q=0.8,et;q=0.7,de;q=0.6", } def parse_search_page(html: str): data = re.findall(r'window.mosaic.providerData\["mosaic-provider- jobcards"\]=(\{.+?\});', html) data = json.loads(data[0]) return { "results": data['metaData']['mosaicProviderJobCardsModel']['results'], "meta": data['metaData']['mosaicProviderJobCardsModel']['tierSummaries'], } response = httpx.get("https://www.indeed.com/jobs?q=python&l=Texas", headers=HEADERS) print(parse_search_page(response.text)) This code is using a regular expression to select mosaic-provider-jobcards value. Now, Python dictionary loading will parse the meta-datqa and parsing. This is how we can now access the remaining page data. import asyncio import json import re from typing import List from urllib.parse import urlencode import httpx def parse_search_page(html: str): data = re.findall(r'window.mosaic.providerData\["mosaic-provider- jobcards"\]=(\{.+?\});', html)

  4. data = json.loads(data[0]) return { "results": data["metaData"]["mosaicProviderJobCardsModel"]["results"], "meta": data["metaData"]["mosaicProviderJobCardsModel"]["tierSummaries"], } async def scrape_search(client: httpx.AsyncClient, query: str, location: str): def make_page_url(offset): parameters = {"q": query, "l": location, "filter": 0, "start": offset} return "https://www.indeed.com/jobs?" + urlencode(parameters) print(f"scraping first page of search: {query=}, {location=}") response_first_page = await client.get(make_page_url(0)) data_first_page = parse_search_page(response_first_page.text) results = data_first_page["results"] total_results = sum(category["jobCount"] for category in data_first_page["meta"]) # there's a page limit on indeed.com if total_results > 1000: total_results = 1000 print(f"scraping remaining {total_results - 10 / 10} pages") other_pages = [make_page_url(offset) for offset in range(10, total_results + 10, 10)] for response in await asyncio.gather(*[client.get(url=url) for url in other_pages]): results.extend(parse_search_page(response.text)) return results At this point, we have scraped large data banks. We now move forward to access comprehensive and detailed data of job listings. Scrape Jobs The full description jobs are found only with a job id. This id can be discovered in the jobkey section of the search results. { "jobkey": "a82cf0bd2092efa3", } With jobkey, we can incorporate data instead of HTML and request for a full job details page.

  5. def parse_job_page(html): """parse job data from job listing page""" data = re.findall(r"_initialData=(\{.+?\});", html) data = json.loads(data[0]) return data["jobInfoWrapperModel"]["jobInfoModel"] async def scrape_jobs(client: httpx.AsyncClient, job_keys: List[str]): """scrape job details from job page for given job keys""" urls = [f"https://www.indeed.com/m/basecamp/viewjob?viewtype=embedded&jk={job_key }" for job_key in job_keys] scraped = [] for response in await asyncio.gather(*[client.get(url=url) for url in urls]): scraped.append(parse_job_page(response.text)) return scraped This final component completes our scraper's functionality. There are ways to scrape Indeed.com using Python without getting blocked. For more information, stay tuned! Conclusion If you need the services of web scraping services USA to extract job postings data from Indeed.com using Python, feel free to contact us. Our team of experts can provide you with customized solutions that are cost and time efficient. We can help you get the information you need so that you can make informed decisions. Our services are quick, secure, and reliable, so you can trust us to get the job done. So, get in touch today to get started!

More Related