161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
414 Commits 19 Branches 14 Tags 14 MiB
13c8305dc2
Commit Graph

414 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
vlofgren
3f2854a5e9 WIP n-gram loader 2022-08-27 20:30:18 +02:00
vlofgren
0282156979 WIP n-gram loader 2022-08-27 19:19:16 +02:00
vlofgren
c865d6c6b2 Change TF-IDF normalization to reduce the amount of not-so-relevant matches. 2022-08-27 11:38:29 +02:00
vlofgren
f4ad7aaf33 Remove accidental import of an unused library, 
fix build on jdk18-systems.
 2022-08-26 20:48:44 +02:00
vlofgren
3200c36072 Experimental changes for 22-08/09 update. 2022-08-26 16:08:46 +02:00
vlofgren
db056be06a WIP logic for detecting significant images in the body of a website. 2022-08-24 22:05:32 +02:00
Viktor Lofgren
be56852c19 Merge pull request 'Crawler fixes, better stylesheet for search' (#104) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/104
 2022-08-24 19:30:50 +02:00
Viktor Lofgren
c8bbb37de1 Merge branch 'release' into master 2022-08-24 19:30:40 +02:00
vlofgren
c6db2aad48 Fixed stylesheet for search to make random websites button more prominent. 2022-08-24 19:29:00 +02:00
vlofgren
69b9f93dc6 Fixed stylesheet for search to make random websites button more prominent. 2022-08-24 19:28:06 +02:00
vlofgren
9cf78d6929 Bugfixes for the crawler: Better charset support, better 429 handling, better error handling, fixed resource leak. 2022-08-24 19:27:46 +02:00
Viktor Lofgren
fd9083c61f Merge pull request 'Use links index for site suggestions.' (#103) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/103
 2022-08-24 04:42:03 +02:00
Viktor Lofgren
6c6b2ca19f Merge branch 'release' into master 2022-08-24 04:41:54 +02:00
vlofgren
407ec39c0c Use links index for site suggestions. 2022-08-24 04:41:26 +02:00
Viktor Lofgren
be1f5f076b Merge pull request 'Use links index for site suggestions.' (#102) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/102
 2022-08-24 03:50:39 +02:00
Viktor Lofgren
e6549ed30d Merge branch 'release' into master 2022-08-24 03:50:30 +02:00
vlofgren
e1a726babf Use links index for site suggestions. 2022-08-24 03:50:08 +02:00
Viktor Lofgren
556aa31452 Merge pull request 'Use links index for site suggestions.' (#101) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/101
 2022-08-24 03:45:41 +02:00
Viktor Lofgren
8f3eedf0ee Merge branch 'release' into master 2022-08-24 03:45:32 +02:00
vlofgren
4c8c8f5140 Use links index for site suggestions. 2022-08-24 03:45:09 +02:00
Viktor Lofgren
204635e1c7 Merge pull request 'Serve assets from search service instead of resource-store,' (#100) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/100
 2022-08-24 00:41:51 +02:00
Viktor Lofgren
54a24294b9 Merge branch 'release' into master 2022-08-24 00:41:42 +02:00
vlofgren
961ef2a930 Serve assets from search service instead of resource-store, 
dynamically render index for future goodies,
css tweaks.
 2022-08-24 00:41:20 +02:00
Viktor Lofgren
2f8cab7f0e Merge pull request 'master' (#99) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/99
 2022-08-24 00:39:10 +02:00
Viktor Lofgren
5e7c706802 Merge branch 'release' into master 2022-08-24 00:39:00 +02:00
vlofgren
ee0580273e Serve assets from search service instead of resource-store, 
dynamically render index for future goodies,
css tweaks.
 2022-08-24 00:35:22 +02:00
vlofgren
db4cf70784 Reduce resource consumption during crawling, 
reduce TIME_WAIT sockets with a custom socket
factory.
 2022-08-23 13:26:37 +02:00
vlofgren
6fc72b3eb8 Clean up feature extraction, fix misidentification of 'application/ld+json' as javascript. 2022-08-23 00:48:48 +02:00
vlofgren
6e2fdb7a77 Reduce crawling memory consumption, 
Increase crawling threads,
Dynamically adjust crawling rate.
 2022-08-23 00:35:45 +02:00
Viktor Lofgren
c97cdfcc25 Merge pull request 'Revert the previous change as my IP got kicked back to ol' reliable '81.170.128.52'' (#98) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/98
 2022-08-22 17:34:22 +02:00
Viktor Lofgren
f48e92630e Merge branch 'release' into master 2022-08-22 17:34:13 +02:00
vlofgren
fc9d9d1bad And revert the previous change as my IP got kicked back to ol' reliable '81.170.128.52' 2022-08-22 17:32:56 +02:00
Viktor Lofgren
f498b55301 Merge pull request 'Update crawler IP file to reflect the fact that the IP changed.' (#97) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/97
 2022-08-22 13:04:50 +02:00
Viktor Lofgren
e9dc01dfc5 Merge branch 'release' into master 2022-08-22 13:04:40 +02:00
vlofgren
087ad0124d Update crawler IP file to reflect the fact that the IP changed. 2022-08-22 13:04:07 +02:00
Viktor Lofgren
bbcedb7e9d Merge pull request 'Tweak CSS a tiny bit to add more padding to the right of info cells.' (#95) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/95
 2022-08-19 16:07:52 +02:00
Viktor Lofgren
475b7fe5e0 Merge branch 'release' into master 2022-08-19 16:07:44 +02:00
vlofgren
095ed7c6c4 Tweak CSS a tiny bit to add more padding to the right of info cells. 2022-08-19 16:07:26 +02:00
Viktor Lofgren
7cf2be356c Merge pull request 'Add recipe selector to list of updates.' (#94) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/94
 2022-08-19 15:55:49 +02:00
Viktor Lofgren
09f3e0c97e Merge branch 'release' into master 2022-08-19 15:55:38 +02:00
vlofgren
2adbe5f74c Update publicity roll. 2022-08-19 15:55:01 +02:00
Viktor Lofgren
a671341c5c Merge pull request 'Update publicity roll with Deutschlandfunk article' (#93) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/93
 2022-08-19 15:51:11 +02:00
Viktor Lofgren
8de6b52e1e Merge branch 'release' into master 2022-08-19 15:51:01 +02:00
vlofgren
56987f6664 Update publicity roll. 2022-08-19 15:50:15 +02:00
vlofgren
7567890708 Update publicity roll. 2022-08-19 15:49:52 +02:00
Viktor Lofgren
f708fa643b Merge pull request 'master' (#92) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/92
 2022-08-18 20:45:28 +02:00
Viktor Lofgren
2a72f1da78 Merge branch 'release' into master 2022-08-18 20:45:19 +02:00
vlofgren
ede62f2515 Retain cookies for domain. 2022-08-18 20:44:44 +02:00
vlofgren
a1eb8375a2 Exclude wp-content/uploads from crawling 2022-08-18 19:05:07 +02:00
Viktor Lofgren
014068ace5 Merge pull request 'Don't try to fetch text/css and text/javascript-files. Refactor fetcher to separate content type sniffing logic. Clean up crawler a smidge.' (#91) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/91
 2022-08-18 18:41:22 +02:00