Tag: crawler

Scraper – Automatic Content Crawl and Post Plugin for WordPress (Utilities)

Scraper is a WordPress plugin that copies content and posts automatically from any web site. With tons of useful and unique features, Scraper takes content creating process to another level.

Offer : Use A.I. Spinning Power on Scraping

Get A.I. Spinner on Codecanyon and use it as spinner on Scraper. Your posts will be unique on web with A.I. Spinner’s results. Scraper supports A.I. Spinner and you could enable it easily.

Get A.I. Spinner Now!

F.A.Q.

Yes, it works. We both have functions for web login or form fill and authentication function that can be used with cookies. So you could easily get data from any website that has login.

+ Is there any feature to translate content?

Yes, it translates any content with Google Translate API. It supports all languages that supported by Google.

+ Is it possible to update prices on product?

Yes, with transform function, you could apply some regex expressions or mathematical functions for numbers. With this way, you could update numbers with yours or change the values with math functions.

+ Is it possible to scrape meta tags?

Yes, that’s possible. You could scrap HTML tags, meta tags, attributes or even JSON parts of page with visual editor.

+ Is it possible to get meta tags or JSON values on page?

Yes, that’s possible. You could scrap HTML tags, meta tags, attributes or even JSON parts of page with visual editor.

Visit Frequenly Asked Questions

Changelog

08/17/2018 (v1.0.0)

        + Cron issues fixed
        + Image download feature added
        + JSON Parsing Method Added
        + A.I. Spinner Service Enabled
        + Translation Service Enabled
        + First Release

Purchase $27.00

September 4, 2018

wordpress Crawler – A crawler/bot/ software, which crawls the required URL, and creates the content material (Utilities)
Purchase $20.00

<!DOCTYPE html PUBLIC ”-//W3C//DTD XHTML 1.0 Transitional//EN” “http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd”>
<html xmlns=”http://www.w3.org/1999/xhtml”>
<head>
<meta http-equiv=”Content-Type” content=”text/html; charset=utf-8” />
<title>WP Crawler – By Pluginology.Space</title>
</head>

WP Crawler

It’s a crawler/bot/spider application, which crawls the specified URL, and creates the table of contents with the findings. The findings include:
- Generator
  The version of the WordPress that the site is running.
- WordPress Theme
  All the information about the theme: name, description, author, etc.
- Links
  All the internal and external links on the page.
- Images
  All the images on the page, with the actual image and the URL to it.
- Post Content
  The entire content of the post/page is retrieved and displayed using the style of your active theme.
- Post Content as Plain Text
  The content with the tags removed. That’s how Google and other search spiders see your page.
- Top 25 Keywords
  The software analyses the page SEO to identify what the page is all about, and displays the top 25 keywords identified.
Here’s the sample output (only the Generator and WordPress Theme are shown):

Using the Plugin

The plugin adds the spider icon to the WordPress Visual Editor, which inserts the shortcode with the URL to crawl:

Another way to use the plugin is to provide the crawl form on a page:

Purchase $20.00
April 17, 2018
Crawlomatic Multisite Scraper Submit Generator Plugin for WordPress (Miscellaneous)
Purchase $19.00

What Can You Do With This Plugin?

Crawlomatic Multisite Scraper Post Generator Plugin for WordPress is a breaking edge (made in 2017) website crawling and scraping, post generator autoblogging plugin that uses website crawling and scraping to turn your website into a autoblogging or even a money making machine!
Get content from almost any webpage! You no longer need API’s which requires registration and provides limited access, also you can retrieve data from non API providing websites. Schedule it for once and let it autopilot your posts 7/24 for you like a master!

You can scrape content from almost every web site that you open in your browser. Just be sure that the content is not generated dynamically by javascript, such as load more buttons or lazy loading content by scrolling down.

Also, you can automatically generate unlimited number of custom website crawling and scraping.

Other plugin features:
- Google Translate support – select the language in which you want to post your articles
- Text Spinner support – automatically modify generated text, changing words with their synonyms – great SEO value!
- customizable generated post status (published, draft, pending, private, trash)
- shortcode to list all posts generated by this plugin: [crawlomatic-list-posts type => ‘any’, order => ‘ASC’, ‘orderby’ => ‘date’, ‘posts’ => 50, ‘category’ => ’’, ‘ruleid’ => ’’]
- automatically generate post categories or tags from marketplace items
- manually add post categories or tags to items
- choose if you want to update post if it is already posted
- send custom cookies with the request to the crawled webpage (authentification)
- generate post or page or any custom post type
- embeds videos from YouTube, Vimeo, Flickr, IGN, Ustream.tv and DailyMotion using website crawling and scraping
- define publishing constrains: do not publish posts that do not have images, posts with short/long title/content
- automatically generate a featured image for the post
- enable/disable comments, pingbacks or trackbacks for the generated post
- customize post title and content (with the included wide variety of relevant post shortcodes)
- ‘Keyword Replacer Tool’ – It’s purpose is to define keywords that are substituted automatically with your affiliate links, anywhere they appear in the content of your site. For example, you can define a keyword ‘codecanyon’ and have it substituted by a link to http://www.codecanyon.net/?ref=user_name anywhere it appears in your site’s content.
- ‘Random Sentence Generator Tool’ (relevant sentences – as you define them)
- option to automatically delete generated posts after a period of time
- detailed plugin activity logging
- scheduled rule runs
- custom field support for generated posts
- option to copy or not images locally
- Maximum/minimum title length post limitation
- Maximum/minimum content length post limitation
- Add post only if predefined required keywords found in title/content
- Add post only if predefined banned keywords are not found in the title/content
- Save and restore plugin rule list from file
Testing this plugin
- You can test the plugin’s functionality using the ‘Test Site Generator’. Here you can try the plugin’s full functionality. Note that the generated testing blog will be deleted automatically after 24 hours.
- This blog is dedicated fully to this plugin! Try it out now!
Plugin Requirements
- PHP DOM -> how to install it (if you don’t have it, but probably you already have it): http://php.net/manual/en/dom.setup.php
- PHP 5.0 or higher
- dom, mbstring, iconv and json extensions (enabled by default)
Need support?

Just email me at kisded@yahoo.com and I will respond as soon as I can.

Changelog:

Version 1.0 Release Date 2017-08-20
```
First version released!
```
Are you already a customer?

If you already bought this and you have tried it out, please contact me in the item’s comment section and give me feedback, so I can make it a better WordPress plugin!

WordPress 4.8 Tested!

Disclaimer
Through this plugin you are able to grab content from various websites that does not necessary belong to you or which are not under your control. If you grab copyrighted material without the author’s permission, the plugin’s developer does not assume any responsibility for your actions. Also, the plugin’s developer has no control over the nature, content and availability of those sites.

Do you like our work and want more of it?
Check out this MEGA plugin bundle.

Purchase $19.00
August 16, 2017
Frisk Crawler – Crawl and Obtain (Miscellaneous)
Purchase $12.00

Dvee Cart

Frisk is a web crawler application that searches websites for content. With a beautiful Bootstrap design, the application prodives an easy and convinient way to browse and download the search results.

Frisk Features
- Preview media files (images, audio and video)
- Download content as Zip archives or json files
- Browser tool for displaying search results
- Cache system provides a smooth experience
- Displays each search result as soon as it finds it
- Ignores robots.txt rules that prevent access
DEMOS

FRISK DEMO

Known Issues
- Frisk may fail to crawl some websites
- Downloading content may take some time
Purchase $12.00
December 6, 2016
Scrapes – Net scraper plugin for WordPress (Utilities)
Purchase $25.00

Features

Ready to go.
It is designed to use without any coding. Upload it to your WordPress site, activate and use, that’s all.

Multiple tasks.
Run multiple tasks which have different scraping features simultaneously and without limitation.

Native content.
Scraped content supports all WordPress inputs like title, excerpt, content, date, author one by one.

Featured image.
Choose featured image of the content from the gallery or use the source content image automatically.

Custom fields.
Scrape the required custom field values for your needs from the source content except WordPress default inputs.

Generated categories.
Add the scraped content to your existing categories or create the same category to your web site from the source.

Custom post types.
It supports custom post types and custom taxonomies which are created by your plugins or themes, adjusts the content to your structure.

HTML support.
Scrape cleaned text content from the source or whole HTML area with all the tags.

Unique check.
If you wish, prevent duplicate content entry and only update the existing content.

Plan B.
If anything goes wrong because of an error, plugin may stop the current process and skips next if you want.

Detailed logs.
Every action from the plugin is logged in details and ready whenever you want to view.

High performance.
Coded neatly, aims to achieve the high performance with the minimum system requirements.

3 Easy steps
- Enter source URL that you would like to scrape content from.
- Select the scraper type and set the options as needs
- Save and go!
Use case ideas
- Track and archive any single web page.
- Scrape latest currency information inside your post.
- Scrape current weather forecast.
- Scrape live scores to any of your posts.
- Scrape job listings and bulletin boards.
- Create an affiliate site by scraping products.
- Create an e-commerce site by scraping products as WooCommerce products
- Create an image gallery by scraping images.
- Create a blog or magazine site by scraping blog posts from different sources.
- Create a music, video or movie site by scraping embed codes from services.
- Create a news site by scraping RSS or Atom feeds form feed URLs
- And much more up to your imagination.
Can I scrape from any web site?

You can scrape content from almost every web site that you open in your browser. Just be sure that the content is not generated dynamically by javascript (Such as load more buttons or with lazy load by scrolling down).

You can check your website that you would like to scrape by disabling javascript from your browser settings and check if your content is still visible to you. Besides for serial type scraping make sure that a next page button is available and the content is not loading automatically on the current page.

Purchase $25.00
November 22, 2016
wordpress Put up Instruments – Totally computerized search engine marketing helper on your auto blogging plugin! (Utilities)
Purchase $24.00

Idea

There are several auto blogging plugins that can be used with WordPress. An auto blogging plugin, such as an RSS plugin or WP Content Crawler, is used to automatically get content from other sites to your WordPress site. You can get content from other sites, but, most of the time, you cannot automatically edit those automatically saved posts. WP Post Tools offers a few tools that can be used to automatically edit those posts. The plugin can work with any auto blogging plugin.

The auto blog plugin has to use WordPress’ wp_insert_post function to save the posts or trigger the save_post hook after it saves a post. Since this is the WordPress way to create a post, most of the plugins probably do it that way. If you have concerns about this, please ask the developer of the auto blog plugin you use about this to make sure.

How it works

When your auto blogging plugin creates a post on your site, WP Post Tools interrupts the save operation by making the post’s status ‘draft’. Then, it applies the edits you configured, and turns the status of the post back to its original, which is most probably ‘publish’. So, it edits automatically saved posts and publish them, automatically.

Video tutorials

The video tutorials that you can find in the playlist below cover almost every aspect of the plugin and they will help you understand how the plugin works.

Available in two languages, ready for translation

Existing tools

There are several tools which can be grouped as templates, images, videos, social media, and advertisements.

1. Templates

A template is a text that can contain short codes that the plugin offers or any other text or HTML you want. For example, you can use a post title template to create/change a post title, or post content template to create/change a post content. Inside those templates, you can use several short codes. You can define more than one template for a, let’s say, post title. In that case, the plugin will use one of the templates you defined randomly. So, you can define, for example, 10 post title templates.

1.1 Short codes

The plugin makes it possible to include almost any post data into almost any template. For example, you can use post categories in the title, post content in meta description, post tags in meta keywords, or even post content in post title and so on. The short codes can have several attributes. For example, you can make the post title all uppercase, make only the first chars of the words uppercase, cut the text by words, word percentage or even sentence percentage, add ellipsis to cut texts, limit the number of categories a post can have and a lot more.

One of the most powerful short codes is “conditional phrase” short codes. You can use those to conditionally show something. For example, you can show “code canyon” when the post content contains “wordpress”. Again, you can use that short code in any template. So, you can add “wordpress” tag to a post when the post content contains “envato”. The only limit is your imagination when you use this short code.

Another powerful short code is “random phrase” short code. You can define several random phrases for a short code. Then, when you include the short code in a template, it will be replaced with one of the phrases you defined for that short code. Again, how you will use this depends on your imagination.

2. Images

The plugin can add a related image for a post. You can define image folders and relate them with a keyword or phrase. For example, you can create an image folder named as “health”, put some apple or doctor photos in it, and use that folder when a post’s title contains “health”. So, in case that there is a post whose title contains “health”, the plugin can add one of the images in “health” folder.

It can also write some texts onto the image. You can use the short codes to define the text that will be written onto the image. There are several styling options available as well. Please watch the video tutorial to understand the image creation better.

3. Videos

The plugin can create a video for a post and upload it to YouTube. For this to work, you need to install FFmpeg library to your server and include a few libraries for it as well. You can make sure the libraries are activated properly using “system check” page of the plugin.

So, the plugin can use the images you defined in “images” section along with the images inside a post and make them a slideshow video by adding a background audio and some text. Again, you can define the texts using the short codes. There are several styling options available as well. Please watch the video tutorial to understand the video creation better.

4. Social Media

The plugin is able to share the posts on Facebook and Twitter. You can define the share texts using the templates and short codes. Since you might not want to share every post on social media, the plugin offers you a “popular posts” option, using which you can define the minimum number of views that a post should have so that it can be published. Please watch the video tutorial to understand the social media options better.

5. Advertisements

This tool can be used to show related affiliate products or any other advertisements for a post. The plugin can show manually created advertisement codes, or it can show products from a web site. For example, you can configure the plugin to show “android figures” from Amazon when the post content contains “android”. Please watch the video tutorial to understand the advertisements better.

Other things

The plugin can also
- spin texts
- find and replace (it is possible to use regex as well) in post content, excerpt, title, tags, meta title, description and keywords
- add links to defined phrases
- remove unwanted phrases
- boost related posts
- change post category depending on phrases
- add tags depending on phrases
- extract tags from post title
- use custom RSS feed to export edited content to another site
- show bonus videos
- make different configurations for different categories
and do several other handy things.

Also
- Ready to translate (.po file)
- Get updates from your WP admin panel
Final notes

This plugin makes it possible to fully automate post editing. It offers several tools to help you implement your SEO techniques. How you can use the provided tools depends on your imagination. With combination of WP Post Tools and an auto blog plugin, you can fully automate your blog.

For more information on how the plugin works or how it is used, you can watch the video tutorials, read the documentation or check out the demo.

System Requirements

Must

PHP >= 5.6, mbstring, allow_url_fopen

Optional for video creation to work

exec, FFmpeg library (with libraries enabled: gpl, libass, libfdk-aac, libfreetype, libfontconfig, libfribidi, libmp3lame, libopus, libtheora, libvorbis, libvpx, libx264, libx265, nonfree)

Feature list

Below are the main but not all features of the plugin.
- Import/export almost every setting.
- Test almost every setting.
1. General
- Activate/deactivate auto editing
- Select suitable/unsuitable categories for auto editing
- Extract tags from post title
- Set maximum meta title/description length
- Add ellipsis to long meta titles/descriptions
- Set maximum number of meta keywords
- Set maximum number of tags
- Set maximum number of tags that can be extracted from the title
- Set maximum number of post categories
- Set maximum number of phrases that can be linked in the post content
- Link phrases on run time or hard link them
- Set minimum and maximum tag/meta keyword length
- Set post meta keys for meta title, meta description and meta keywords so that you can set meta information of the post for any SEO plugin you have
- Set suitable post types for auto editing
- Activate the RSS feed that you can get detailed post information to export auto edited posts to other sites
- Set post categories depending on the phrases the post has in its title or content.
- Create templates for post title, excerpt, content, tags, meta title, meta description and meta keywords. You can create unlimited number of templates for each one to randomize.
- Find and replace in post title, excerpt, content, tags, meta title, meta description and meta keywords.
- Find and replace using regular expressions.
- Use “find and replace” options for text-spinning.
- Set post tags depending on the phrases the post has in its title or content.
- Add links to phrases in the post content.
- Create phrases that you can use randomly in the templates.
- Create phrases for the short code that will show what you want only if some phrases exist in post title, excerpt, content, tags, meta title, meta description or meta keywords.
- Set unwanted phrases so that they are removed from the content.
- Boost related posts.
- Create your own boost post template and add your custom CSS.
2. Images
- Activate/deactivate auto image creation.
- Add a random related image to the posts.
- Add the image as a featured image.
- Add the image to the post content.
- Delete old featured images automatically to use your disk space effectively.
- Write texts onto the image using templates and short codes.
- Change color of the texts and text backgrounds, add border to the images, set border color, set the font of the texts.
- Add a related image to the post depending on the phrases the post has in its title or content.
3. Videos
- Activate/deactivate auto video creation.
- Set suitable/unsuitable categories for auto video creation.
- Upload videos to YouTube channels.
- Delete videos after they are uploaded to use the disk space effectively.
- Set minimum view count for the posts for which a video can be created.
- Set maximum number of popular posts that can be added to video creation queue each time.
- Set video title, description and tags using the templates and short codes.
- Set a text template that will be used to create the text that will be written onto the video.
- Set minimum display duration for an image.
- Set duration per word, to make sure the texts are shown long enough for the viewer to read.
- Add intro/outro videos.
- Set video size, background color, font, percentage of the area that will be used to show the texts, text color, text background color, font size, image fade color.
- Add a related background audio to the video depending on the existence of some phrases in the post’s title or content.
- Upload to several YouTube channels.
- Set video category.
- Add bonus YouTube videos to the posts.
4. Social Media
- Activate/deactivate automatic sharing.
- Share on several Facebook and/or Twitter pages/accounts.
- Set minimum view count for the posts that can be shared.
- Set maximum number of popular posts that can be added to share queue each time.
- Create share texts using templates and short codes.
5. Adverisements
- Activate/deactivate advertisements
- Set maximum ad title and text length.
- Open ads in new window.
- Add ad links “nofollow”.
- Add additional CSS to ad codes.
- Create ad groups.
- Activate/deactivate an ad group.
- Set an ad group as fallback.
- Add additional CSS to ad groups.
- Show ads of an ad group in specific posts.
- Show ads depending on the phrases the post’s title, excerpt, content, tags, categories, meta title, meta description and/or meta keywords contain.
- Show ads/affiliate products from other web sites.
- Add custom URL parameters to the URLs of the ads retrieved from other web sites.
- Get an ad from target web site if it contains/starts with/matches the phrases you define.
- Remove unnecessary elements from the ads retrieved from other web sites.
- Find and replace in codes of the ads retrieved from other web sites.
- Add manual ad codes to an ad group.
- Show an ad full width and single.
- Import/export ad group settings.
- Define custom general fallback ad codes.
- Make custom general fallback ad codes full width and single.
6. Tester
- Test auto edit, video creation, advertisements and bonus videos for a post.
- See before and after the edit side by side in the tester.
7. Tools
- Edit/revert a post.
- Remove boost post data.
- Create a video for a post.
- Remove the video from a post.
- Remove bonus video data from a post.
- Remove saved Facebook pages, Twitter accounts and YouTube channels.
- Create manual ad codes.
- Invalidate caches for the URLs used to retrieve advertisements.
8. Logs
- See almost everything happening in the background.
- Set a refresh interval to see changes while they are happening.
- Download logs.
9. System Check
- See if your system satisfies the requirements for the plugin.
Changelog
```
v1.0.0 - First release
```
Purchase $24.00
November 15, 2016
wordpress Content material Crawler (Utilities)
Purchase $24.00

Get content from almost any site to your WordPress blog, automatically!

For what it can be used
- Create a personal site which collects news, posts, etc. from your favorite sites to see them in one place
- Use it with WooCommerce to collect products from shopping sites
- Collect products from affiliate programs to make money
- Collect posts to create a test environment for your plugin/theme
- Collect plugins, themes, apps, images from other sites to create a collection of them
- Keep track of competitors
How it works

It’s all about CSS selectors and you can learn how to use them in minutes by watching the introduction tutorial.

See it in action, learn in minutes

Available in two languages, ready for translation

Requirements:

PHP >= 5.5

Tested with WP versions:

4.5.2, 4.4.2, 4.3.3, 4.2.7, 4.1.10, 4.0.10, 3.9.11

Main features:
- Save post’s title, excerpt and content.
- Save meta keywords and description.
- Save images in the posts to your site.
- Save featured image of target post as the featured image of the post created in your site.
- Save the list in posts (extract the list from the post).
- Save paginated posts. If target post is paginated, the post in your site can be paginated as well.
- Save anything from target page as post meta.
- Find and replace anything in the target page’s HTML before saving its content.
- Find and replace using regular expressions.
- Find and replace in image URLs before saving them.
- Manipulate target page’s HTML before saving to make it suitable for your needs.
- Remove unnecessary elements from target page’s HTML.
- Map categories of your site to target site’s categories to keep posts organized.
- Automatically get category URLs for category mapping.
- Add unlimited categories for a site and automatically check for new posts uniformly.
- Set alternative CSS selectors for each setting. By this way, get content from the pages of the same site which are designed differently.
- Automatically save posts.
- Automatically check for new posts.
- Set maximum number of category pages to be checked automatically.
- Add unlimited sites.
- Import/export site settings.
- Set post type. For instance, you can add posts as products if you use WooCommerce.
- Set a template for the post.
- Set a template for each list item.
- Set passwords for created posts.
- Keep notes for each site to keep a changelog or anything you want.
- Test any CSS selector and find-and-replace setting right away.
- Test a site before activating it for automatic crawling.
- Manually create posts by providing the URL of the posts.
- If you want, use settings different than general settings for each site.
- Activate/deactive each site any time to start/stop automatic crawling.
- Either directly publish the posts or keep them as drafts to make changes before publishing.
- Online documentation
- Video tutorials for a quick start
- Ready to translate (.po file)
- Get updates from your WordPress admin panel.
Purchase $24.00
May 8, 2016
WordPress Hyperlink Standing Professional – Restore Damaged Hyperlinks & Take care of Redirections (Utilities)
Purchase $20.00

WP Link Status Pro is a WordPress plugin to check HTTP status response codes of all your content links and images.

It works crawling your content, extracting links and images, looking for broken links, redirections, nofollow links, etc.

Features
- Create multiple scans, each one containing its own configuration and results.
- Run the crawler in background mode, no WordPress session needed.
- Advanced filters criteria and search by URL or anchor text.
- Edit content directly from crawler results without having to enter the post editor.
- Change URLs, anchor text, manage nofollow links, replace redirections, etc.
- Extra URL tools to perform massive changes matching post URLs.
- Any crawler parameter configurable via extended settings page.
Try a demo version

Download and install a demo version of this plugin with almost all premium features.

Support
- Contact through the WP Link Status Pro plugin page or this item comments.
- Step by step detailed user guide documentation.
Purchase $20.00
February 8, 2016

Tag: crawler

Offer : Use A.I. Spinning Power on Scraping

F.A.Q.

+ Does it work with login or authentication?

+ Is there any feature to translate content?

+ Is it possible to update prices on product?

+ Is it possible to scrape meta tags?

+ Is it possible to get meta tags or JSON values on page?

Changelog

WP Crawler

Using the Plugin

What Can You Do With This Plugin?

Testing this plugin

Plugin Requirements

Need support?

Changelog:

Are you already a customer?

Dvee Cart

Frisk Features

DEMOS

Known Issues

Features

3 Easy steps

Use case ideas

Can I scrape from any web site?

Idea

How it works

Video tutorials

Available in two languages, ready for translation

Existing tools

1. Templates

1.1 Short codes

2. Images

3. Videos

4. Social Media

5. Advertisements

Other things

Also

Final notes

System Requirements

Must

Optional for video creation to work

Feature list

1. General

2. Images

3. Videos

4. Social Media

5. Adverisements

6. Tester

7. Tools

8. Logs

9. System Check

Changelog

For what it can be used

How it works

See it in action, learn in minutes

Available in two languages, ready for translation

Requirements:

Tested with WP versions:

Main features:

Features

Try a demo version

Support