Search entire projects with the crawler

koality.io was primarily built to continuously monitor websites for multiple projects simultaneously. We have developed a unique component model for the selection of websites to be monitored. The idea behind it, each critical error appears in all pages of one type. As a simple example we can take an online store. If the button to the shopping cart is missing on one product page, there is a good chance that it will be missing on all of them. In technical jargon, this is called equivalence class testing.

Continuous monitoring is therefore particularly efficient when it comes to systematic errors. Since these are the most critical in projects, this is the best start to professional quality assurance. For editorial errors, however, this type of monitoring does not score very well.

For this reason, koality.io offers, in addition to the conventional checks, a crawler that can be used to search entire projects. This includes up to 500 subpages, which are automatically accessed and examined for the desired criteria.

Implemented crawls

But how exactly does the integrated crawler work? koality.io provides a constantly growing number of standard crawls. The goal here is to be able to check everything that is recurrently important on many websites. Currently our service offers the following crawls:

• Large files - Large images or videos slow down the website immensely and may also make it unusable for users who come to the site via smartphone or those who have slow internet connections due to more rural environments. The implemented crawler renders up to 500 pages of a project in the browser, checking all loaded files for their size. If an image, a video, a CSS file or similar, included on the page is larger than 1 MB, the crawler will find it.
• VAT check - We search the deposited project for outdated VAT rates. This check is especially used in e-commerce. Here the VAT rates were adjusted due to the Corona crisis. Various patterns have been deposited in the crawler that react to 19 or 7% VAT or sales tax.

Planned crawls

The goal is to implement all crawls that have generality and thus add value for all customers. Currently planned are:

• Lorem Ipsum: Is there still a placeholder text like "Lorem ipsum" somewhere in the project?
• Dead Link: On which pages are there links pointing to nothing or broken pages?
• Google Analytics: Where was the Google Analytics pixel forgotten?
• IVW/Infonline**: Do all pages in a project transfer the data cleanly to IVW?
• Missing link to imprint: Does every page link to the imprint as required by the data protection law?
• JavaScript errors: On which pages are there JavaScript errors?

Vision

The crawler is already very mature and offers great added value. Nevertheless, it will be further developed and not only in terms of the number of possible crawls.

• Export as CSV: Crawls can produce any amount of data. To be able to process them even more cleanly afterwards, there will be a CSV export, with which you can get the data into Excel or other spreadsheets.
• Returning: It will be possible to run crawls recurrently without being able to trigger them manually. For example, you will be able to configure weekly crawls of the web project for dead links. As a result, you will get one report per week, with all the errors found, without having to do any manual work.
• Adjustable Crawl Depth: Currently, the number of URLs that are checked is limited to 500. This limit should be removed in the future under certain circumstances.

For more ideas on where koality.io is headed, we've summarized our article about 2021.