Java Web Crawler

A web crawler, or spider, is a type of bot that's typically operated by search engines like Google and Bing. Their purpose is to index the content of websites all across the Internet so that those websites can appear in search engine results.

Selenium

docker run --detach \ --publish 4444:4444 \ --hostname firefox \ --name firefox \ --shm-size 2g \ selenium/standalone-firefox:80.0 --OR-- docker run --detach \ --publish 4444:4444 \ --hostname firefox \ --name firefox \ --volume /dev/shm:/dev/shm \ selenium/standalone-firefox:80.0 docker exec -it firefox cat /etc/hosts http://localhost:4444/wd/hub

docker run --detach \ --publish 4444:4444 \ --hostname chrome \ --name chrome \ --shm-size 2g \ selenium/standalone-chrome:85.0 --OR-- docker run --detach \ --publish 4444:4444 \ --hostname chrome \ --name chrome \ --volume /dev/shm:/dev/shm \ selenium/standalone-chrome:85.0 docker exec -it chrome cat /etc/hosts http://localhost:4444/wd/hub

docker run --detach \ --publish 4444:4444 \ --hostname opera \ --name opera \ --shm-size 2g \ selenium/standalone-opera:71.0 --OR-- docker run --detach \ --publish 4444:4444 \ --hostname opera \ --name opera \ --volume /dev/shm:/dev/shm \ selenium/standalone-opera:71.0 docker exec -it opera cat /etc/hosts http://localhost:4444/wd/hub

References

Connect to SQLite using Spring Boot Spring Boot Data Redis for caching Selenium web driver without a GUI Introduction to Spring Data Redis Java Web Crawler Libraries What Is a Web Crawler? Liquibase with SQLite Selenium Projects Spring Data Redis SQLite Dialect

@Cacheable Key on Multiple Method Arguments Docker images for the Selenium Grid Server EhCache overflow to disk at specific path @EnableCaching Annotation in Spring Selenium Webdriver Remote Setup Using EhCache 3 with Spring boot Using Selenium-Server on Docker Spring Boot Ehcache Example Cache Eviction in Spring Boot Chrome Headless with Java

Spring Boot Web Application using Selenium WebDriver Automated Testing With JUnit & Selenium for Browser Maximum length of a URL for the Browsers Spring @CacheEvict using wildcards Spring Cache Annotations Tips & Tricks Selenium Jupiter Quick Reference Disable RobotServer in Crawler4j Selenium Jupiter Crawler4j

---

Docker Image selenium/standalone-firefox Docker Image selenium/standalone-chrome Docker Image selenium/standalone-opera Baeldung Crawler4j

Find XPath in Chrome Browser XPath Iteration in Java