161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
404 Commits 19 Branches 14 Tags 14 MiB
2e740bb7bd
Commit Graph

404 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
vlofgren
9cf78d6929 Bugfixes for the crawler: Better charset support, better 429 handling, better error handling, fixed resource leak. 2022-08-24 19:27:46 +02:00
Viktor Lofgren
fd9083c61f Merge pull request 'Use links index for site suggestions.' (#103) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/103
 2022-08-24 04:42:03 +02:00
Viktor Lofgren
6c6b2ca19f Merge branch 'release' into master 2022-08-24 04:41:54 +02:00
vlofgren
407ec39c0c Use links index for site suggestions. 2022-08-24 04:41:26 +02:00
Viktor Lofgren
be1f5f076b Merge pull request 'Use links index for site suggestions.' (#102) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/102
 2022-08-24 03:50:39 +02:00
Viktor Lofgren
e6549ed30d Merge branch 'release' into master 2022-08-24 03:50:30 +02:00
vlofgren
e1a726babf Use links index for site suggestions. 2022-08-24 03:50:08 +02:00
Viktor Lofgren
556aa31452 Merge pull request 'Use links index for site suggestions.' (#101) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/101
 2022-08-24 03:45:41 +02:00
Viktor Lofgren
8f3eedf0ee Merge branch 'release' into master 2022-08-24 03:45:32 +02:00
vlofgren
4c8c8f5140 Use links index for site suggestions. 2022-08-24 03:45:09 +02:00
Viktor Lofgren
204635e1c7 Merge pull request 'Serve assets from search service instead of resource-store,' (#100) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/100
 2022-08-24 00:41:51 +02:00
Viktor Lofgren
54a24294b9 Merge branch 'release' into master 2022-08-24 00:41:42 +02:00
vlofgren
961ef2a930 Serve assets from search service instead of resource-store, 
dynamically render index for future goodies,
css tweaks.
 2022-08-24 00:41:20 +02:00
Viktor Lofgren
2f8cab7f0e Merge pull request 'master' (#99) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/99
 2022-08-24 00:39:10 +02:00
Viktor Lofgren
5e7c706802 Merge branch 'release' into master 2022-08-24 00:39:00 +02:00
vlofgren
ee0580273e Serve assets from search service instead of resource-store, 
dynamically render index for future goodies,
css tweaks.
 2022-08-24 00:35:22 +02:00
vlofgren
db4cf70784 Reduce resource consumption during crawling, 
reduce TIME_WAIT sockets with a custom socket
factory.
 2022-08-23 13:26:37 +02:00
vlofgren
6fc72b3eb8 Clean up feature extraction, fix misidentification of 'application/ld+json' as javascript. 2022-08-23 00:48:48 +02:00
vlofgren
6e2fdb7a77 Reduce crawling memory consumption, 
Increase crawling threads,
Dynamically adjust crawling rate.
 2022-08-23 00:35:45 +02:00
Viktor Lofgren
c97cdfcc25 Merge pull request 'Revert the previous change as my IP got kicked back to ol' reliable '81.170.128.52'' (#98) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/98
 2022-08-22 17:34:22 +02:00
Viktor Lofgren
f48e92630e Merge branch 'release' into master 2022-08-22 17:34:13 +02:00
vlofgren
fc9d9d1bad And revert the previous change as my IP got kicked back to ol' reliable '81.170.128.52' 2022-08-22 17:32:56 +02:00
Viktor Lofgren
f498b55301 Merge pull request 'Update crawler IP file to reflect the fact that the IP changed.' (#97) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/97
 2022-08-22 13:04:50 +02:00
Viktor Lofgren
e9dc01dfc5 Merge branch 'release' into master 2022-08-22 13:04:40 +02:00
vlofgren
087ad0124d Update crawler IP file to reflect the fact that the IP changed. 2022-08-22 13:04:07 +02:00
Viktor Lofgren
bbcedb7e9d Merge pull request 'Tweak CSS a tiny bit to add more padding to the right of info cells.' (#95) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/95
 2022-08-19 16:07:52 +02:00
Viktor Lofgren
475b7fe5e0 Merge branch 'release' into master 2022-08-19 16:07:44 +02:00
vlofgren
095ed7c6c4 Tweak CSS a tiny bit to add more padding to the right of info cells. 2022-08-19 16:07:26 +02:00
Viktor Lofgren
7cf2be356c Merge pull request 'Add recipe selector to list of updates.' (#94) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/94
 2022-08-19 15:55:49 +02:00
Viktor Lofgren
09f3e0c97e Merge branch 'release' into master 2022-08-19 15:55:38 +02:00
vlofgren
2adbe5f74c Update publicity roll. 2022-08-19 15:55:01 +02:00
Viktor Lofgren
a671341c5c Merge pull request 'Update publicity roll with Deutschlandfunk article' (#93) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/93
 2022-08-19 15:51:11 +02:00
Viktor Lofgren
8de6b52e1e Merge branch 'release' into master 2022-08-19 15:51:01 +02:00
vlofgren
56987f6664 Update publicity roll. 2022-08-19 15:50:15 +02:00
vlofgren
7567890708 Update publicity roll. 2022-08-19 15:49:52 +02:00
Viktor Lofgren
f708fa643b Merge pull request 'master' (#92) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/92
 2022-08-18 20:45:28 +02:00
Viktor Lofgren
2a72f1da78 Merge branch 'release' into master 2022-08-18 20:45:19 +02:00
vlofgren
ede62f2515 Retain cookies for domain. 2022-08-18 20:44:44 +02:00
vlofgren
a1eb8375a2 Exclude wp-content/uploads from crawling 2022-08-18 19:05:07 +02:00
Viktor Lofgren
014068ace5 Merge pull request 'Don't try to fetch text/css and text/javascript-files. Refactor fetcher to separate content type sniffing logic. Clean up crawler a smidge.' (#91) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/91
 2022-08-18 18:41:22 +02:00
Viktor Lofgren
4e3a977049 Merge branch 'release' into master 2022-08-18 18:41:13 +02:00
vlofgren
340d80f6c7 Don't try to fetch text/css and text/javascript-files. Refactor fetcher to separate content type sniffing logic. Clean up crawler a smidge. 2022-08-18 18:40:34 +02:00
Viktor Lofgren
a915b2d37a Merge pull request 'Don't try to fetch ftp://, webcal://, etc.' (#90) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/90
 2022-08-18 18:27:15 +02:00
vlofgren
6b6cd56e3a Don't try to fetch text/css and text/javascript-files. Refactor fetcher to separate content type sniffing logic. Clean up crawler a smidge. 2022-08-18 18:25:12 +02:00
Viktor Lofgren
e5d63d8a61 Merge branch 'release' into master 2022-08-18 17:26:18 +02:00
vlofgren
4afccdc536 Don't try to fetch ftp://, webcal://, etc. 2022-08-18 17:25:22 +02:00
Viktor Lofgren
4435334ebe Merge pull request 'Fix bug where url fragments were considered path elements' (#89) from master into release 
Reviewed-on: https://git.marginalia.nu/marginalia/marginalia.nu/pulls/89
 2022-08-18 16:48:48 +02:00
Viktor Lofgren
579037db05 Merge branch 'release' into master 2022-08-18 16:48:32 +02:00
vlofgren
5cd552458a Fix fragment bug. 2022-08-18 16:47:59 +02:00
vlofgren
2bc81e8e9a Fix fragment bug. 2022-08-18 16:45:51 +02:00