161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,674 Commits 19 Branches 14 Tags 14 MiB
acc2b4e10f
Commit Graph

1674 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Viktor Lofgren
fc30da0d48 (converter) Add academia recognition to DomainProcessor 
The code now includes an additional function in the DomainProcessor class that checks if a domain is associated with academia. An academic domain is identified by the ".edu" TLD, or fits a specific regex pattern matching domains like *.ac.ccTld or *.edu.ccTld.

 If these conditions are met, the search term "special:academia" is added to the domain.

 The existing academia search filter uses personalized pagerank to select academia-adjacent domains, but it isn't working very well.  The hope is that filtering on domain names will be more effective, and that it can supplant the ranking-based approach.
 2023-12-08 20:31:34 +01:00
Viktor Lofgren
e6a1052ba7 Simplify CrawlerMain, removing the CrawlerLimiter and using a global HttpFetcher with a virtual thread pool dispatcher instead of the default. 2023-12-08 20:24:01 +01:00
Viktor Lofgren
968dce50fc (crawler) Refactored IpInterceptingNetworkInterceptor for clarity. 2023-12-08 17:45:46 +01:00
Viktor Lofgren
3bbffd3c22 (crawler) Refactor HttpFetcher to integrate WarcRecorder 
Partially hook in the WarcRecorder into the crawler process.  So far it's not read, but should record the crawled documents.

The WarcRecorder and HttpFetcher classes were also refactored and broken apart to be easier to reason about.
 2023-12-08 17:12:51 +01:00
Viktor Lofgren
072b5fcd12 Implement Warc-recording wrapper for OkHttp3 client 
This is a first step of using WARC as an intermediate flight recorder style step in the crawler, ultimately aimed at being able to resume crawls if the crawler is restarted.  This component is currently not hooked into anything.

The OkHttp3 client wrapper class 'WarcRecordingFetcherClient' was implemented for web archiving. This allows for the recording of HTTP requests and responses. New classes were introduced, 'WarcDigestBuilder', 'IpInterceptingNetworkInterceptor', and 'WarcProtocolReconstructor'.

The JWarc dependency was added to the build.gradle file, and relevant unit tests were also introduced. Some HttpFetcher-adjacent structural changes were also done for better organization.
 2023-12-08 13:49:16 +01:00
Viktor Lofgren
fabffa80f0 (warc) Integrate the crawler's content type parsing and charset logic into the WarcSideloader 2023-12-07 15:26:01 +01:00
Viktor Lofgren
064265b0b9 (crawler) Move content type/charset sniffing to a separate microlibrary 
This functionality needs to be accessed by the WarcSideloader, which is in the converter.  The resultant microlibrary is tiny, but I think in this case it's justifiable.
 2023-12-07 15:16:37 +01:00
Viktor Lofgren
2d5d11645d (warc) Refactor WarcSideloaderTest to not rely on specific test files on the computer 2023-12-06 19:00:29 +01:00
Viktor Lofgren
cc813a5624 (convert) Add basic support for Warc file sideloading 
This update includes the integration of the jwarc library and implements support for Warc file sideloading, as a first trial integration with this library.
 2023-12-06 18:43:55 +01:00
Viktor Lofgren
156c067f79 (search) Fix mobile issues with browse feature 2023-12-05 21:28:50 +01:00
Viktor Lofgren
b33b013d41 (search) Fix broken script tag 
Apparently it can't be called suggestions.js...?
 2023-12-05 20:29:13 +01:00
Viktor Lofgren
e74e2f705f (search) Fix broken script tag 
suggestions.js became something else.
 2023-12-05 20:20:07 +01:00
Viktor Lofgren
2e438847fc (search) Optimize related domains queries 
In the future this logic probably needs to move into a separate
service, as it's still quite slow to load.  But this fixes response
times and DOS potential of previous version.
 2023-12-05 20:12:03 +01:00
Viktor Lofgren
9301c47d93 (search) Optimize related domains queries 2023-12-05 14:42:03 +01:00
Viktor Lofgren
20ec58b07f (search) Remove layout-breakingly long URLs from the similar domains view. 
They're almost all .onion URLs anyway, not really the space we're looking to peer into.
 2023-12-05 13:58:15 +01:00
Viktor Lofgren
98983c1015 (search) Hopefully fix race condition that leaves the response with no Content-type header 2023-12-05 13:52:36 +01:00
Viktor Lofgren
67195592c6 (search) Hopefully fix race condition that leaves the response with no Content-type header 2023-12-05 13:48:42 +01:00
Viktor
21abfc6424
Merge pull request #61 from MarginaliaSearch/new-look 
Design Revamp For search.marginalia.nu
 2023-12-05 13:28:54 +01:00
Viktor Lofgren
d1e88df71e (search) Cleaning up the code a bit 2023-12-05 13:26:05 +01:00
Viktor Lofgren
f36cfe34ab (search) Hackery to get a more balanced view 2023-12-04 22:50:39 +01:00
Viktor Lofgren
8a1934008c (search) Merge similar sites results with the info view. 
WIP: This commit needs to be cleaned up.
 2023-12-04 22:10:24 +01:00
Viktor Lofgren
b41bb9cfcf (search) Use a Ξ for mobile button title instead of "Filters". 
Makes it easier to distinguish form the search button.
 2023-12-03 16:33:25 +01:00
Viktor Lofgren
d58324bbef (search) Clean up filters menu a bit, improve accessibility. 2023-12-02 18:05:30 +01:00
Viktor Lofgren
cbbd45d3e5 (search) Clean up filters menu a bit, improve accessibility. 2023-12-02 18:01:03 +01:00
Viktor Lofgren
b89633ae4b (search) Don't render a filter button on mobile when there are no filters to be presented. 2023-12-02 17:23:45 +01:00
Viktor Lofgren
96357e9bfd (search) Fix typeahead suggestions, as well as improve mobile and desktop UX in small ways. 2023-12-02 17:06:40 +01:00
Viktor Lofgren
d530c3096f (search) GUI tweaks to make the new interface not fall apart on mobile/chrome 2023-12-02 17:06:40 +01:00
Viktor Lofgren
ae0c1c3f2d (control) Adjust search result margins for better visual density 2023-12-02 17:06:40 +01:00
Viktor Lofgren
0cc2564380 (search) CSS tweaks 2023-12-02 17:06:40 +01:00
Viktor Lofgren
38d20022ad (search) Fix script loading for mobile support 2023-12-02 17:06:40 +01:00
Viktor Lofgren
280132dad0 (search) Fix script loading for mobile support 2023-12-02 17:06:40 +01:00
Viktor Lofgren
61de4e2789 (search) Retain filter options when performing a new search from the input field 2023-12-02 17:06:40 +01:00
Viktor Lofgren
f9d3455320 (search) Reduce visual weight of search results 2023-12-02 17:06:40 +01:00
Viktor Lofgren
2ff64c3c12 (search) New toggle for reducing tracking 2023-12-02 17:06:40 +01:00
Viktor Lofgren
902f235b5b (search) Integrate 'similar' tab in site info. 2023-12-02 17:06:40 +01:00
Viktor Lofgren
97d43a6fa2 (search) Revamp browse results with new look. 2023-12-02 17:06:40 +01:00
Viktor Lofgren
9bc65ff0ca (search) Desaturate search result titles according to rank 2023-12-02 17:06:40 +01:00
Viktor Lofgren
6cd6a615fd (search) Add data-filter to body as a data attribute 
For future shenanigans ;D
 2023-12-02 17:06:40 +01:00
Viktor Lofgren
5639f0653d (search) Rename SearchProfile.name into filterId 
Avoid foot-gun caused by name clash with the Enumeration method name(), which returns the Java name of the enumeration value.
 2023-12-02 17:06:40 +01:00
Viktor Lofgren
251174c9a2 (search) Update front page with new look 2023-12-02 17:06:40 +01:00
Viktor Lofgren
42ea87d637 (search) Update conversion results, error page, and dictionary results with new CSS. 2023-12-02 17:06:40 +01:00
Viktor Lofgren
7c8a60b8cf (search) Site info view is mostly done 
Also optimize the rendering a bit to avoid having to allocate huge string buffers, writing directly to Spark's response instead.
 2023-12-02 17:06:40 +01:00
Viktor Lofgren
2f4500be5a (search) New frontend look 2023-12-02 17:06:40 +01:00
Viktor Lofgren
fa7534a362 (search) Remove dead code 2023-12-02 17:06:40 +01:00
Viktor Lofgren
a258f0af7a (search) Refactor search parameters to include query 2023-12-02 17:06:40 +01:00
Viktor Lofgren
01621c6344 (renderer) Make helpers configurable on a by-service basis. 2023-12-02 17:06:40 +01:00
Viktor Lofgren
c7934342a6 (control) Automatic recrawl 2023-12-02 17:06:24 +01:00
Viktor Lofgren
f5c324c06b (minor) Fix broken test 2023-12-01 17:44:39 +01:00
Viktor Lofgren
f615cf2391 (convert) Loosen up the rules enforcement for documents that have external links. 2023-12-01 17:44:29 +01:00
Viktor Lofgren
c984a97262 (docs) Update crawling.md 2023-11-30 21:53:56 +01:00