161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fix bug in redirect handling that caused the crawler to not index some documents.

Browse Source
This commit is contained in:
vlofgren 2022-08-17 00:51:07 +02:00
parent 8f2485870d
commit 0bac422091
1 changed files with 4 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
marginalia_nu/src/main/java/nu/marginalia/wmsa/edge/crawling/retreival/CrawlerRetreiver.java
View File

@ -168,10 +168,11 @@ public class CrawlerRetreiver {
var d = doc.get();
crawledDomainWriter.accept(d);
try {
visited.add(new EdgeUrl(d.url));
if (d.url != null) {
try {
visited.add(new EdgeUrl(d.url));
} catch (URISyntaxException ex) {}
}
catch (URISyntaxException ex) {}
}