Viktor Lofgren
fabffa80f0
(warc) Integrate the crawler's content type parsing and charset logic into the WarcSideloader
2023-12-07 15:26:01 +01:00
Viktor Lofgren
064265b0b9
(crawler) Move content type/charset sniffing to a separate microlibrary
...
This functionality needs to be accessed by the WarcSideloader, which is in the converter. The resultant microlibrary is tiny, but I think in this case it's justifiable.
2023-12-07 15:16:37 +01:00
Viktor Lofgren
2d5d11645d
(warc) Refactor WarcSideloaderTest to not rely on specific test files on the computer
2023-12-06 19:00:29 +01:00
Viktor Lofgren
cc813a5624
(convert) Add basic support for Warc file sideloading
...
This update includes the integration of the jwarc library and implements support for Warc file sideloading, as a first trial integration with this library.
2023-12-06 18:43:55 +01:00
Viktor Lofgren
156c067f79
(search) Fix mobile issues with browse feature
2023-12-05 21:28:50 +01:00
Viktor Lofgren
b33b013d41
(search) Fix broken script tag
...
Apparently it can't be called suggestions.js...?
2023-12-05 20:29:13 +01:00
Viktor Lofgren
e74e2f705f
(search) Fix broken script tag
...
suggestions.js became something else.
2023-12-05 20:20:07 +01:00
Viktor Lofgren
2e438847fc
(search) Optimize related domains queries
...
In the future this logic probably needs to move into a separate
service, as it's still quite slow to load. But this fixes response
times and DOS potential of previous version.
2023-12-05 20:12:03 +01:00
Viktor Lofgren
9301c47d93
(search) Optimize related domains queries
2023-12-05 14:42:03 +01:00
Viktor Lofgren
20ec58b07f
(search) Remove layout-breakingly long URLs from the similar domains view.
...
They're almost all .onion URLs anyway, not really the space we're looking to peer into.
2023-12-05 13:58:15 +01:00
Viktor Lofgren
98983c1015
(search) Hopefully fix race condition that leaves the response with no Content-type header
2023-12-05 13:52:36 +01:00
Viktor Lofgren
67195592c6
(search) Hopefully fix race condition that leaves the response with no Content-type header
2023-12-05 13:48:42 +01:00
Viktor
21abfc6424
Merge pull request #61 from MarginaliaSearch/new-look
...
Design Revamp For search.marginalia.nu
2023-12-05 13:28:54 +01:00
Viktor Lofgren
d1e88df71e
(search) Cleaning up the code a bit
2023-12-05 13:26:05 +01:00
Viktor Lofgren
f36cfe34ab
(search) Hackery to get a more balanced view
2023-12-04 22:50:39 +01:00
Viktor Lofgren
8a1934008c
(search) Merge similar sites results with the info view.
...
WIP: This commit needs to be cleaned up.
2023-12-04 22:10:24 +01:00
Viktor Lofgren
b41bb9cfcf
(search) Use a Ξ for mobile button title instead of "Filters".
...
Makes it easier to distinguish form the search button.
2023-12-03 16:33:25 +01:00
Viktor Lofgren
d58324bbef
(search) Clean up filters menu a bit, improve accessibility.
2023-12-02 18:05:30 +01:00
Viktor Lofgren
cbbd45d3e5
(search) Clean up filters menu a bit, improve accessibility.
2023-12-02 18:01:03 +01:00
Viktor Lofgren
b89633ae4b
(search) Don't render a filter button on mobile when there are no filters to be presented.
2023-12-02 17:23:45 +01:00
Viktor Lofgren
96357e9bfd
(search) Fix typeahead suggestions, as well as improve mobile and desktop UX in small ways.
2023-12-02 17:06:40 +01:00
Viktor Lofgren
d530c3096f
(search) GUI tweaks to make the new interface not fall apart on mobile/chrome
2023-12-02 17:06:40 +01:00
Viktor Lofgren
ae0c1c3f2d
(control) Adjust search result margins for better visual density
2023-12-02 17:06:40 +01:00
Viktor Lofgren
0cc2564380
(search) CSS tweaks
2023-12-02 17:06:40 +01:00
Viktor Lofgren
38d20022ad
(search) Fix script loading for mobile support
2023-12-02 17:06:40 +01:00
Viktor Lofgren
280132dad0
(search) Fix script loading for mobile support
2023-12-02 17:06:40 +01:00
Viktor Lofgren
61de4e2789
(search) Retain filter options when performing a new search from the input field
2023-12-02 17:06:40 +01:00
Viktor Lofgren
f9d3455320
(search) Reduce visual weight of search results
2023-12-02 17:06:40 +01:00
Viktor Lofgren
2ff64c3c12
(search) New toggle for reducing tracking
2023-12-02 17:06:40 +01:00
Viktor Lofgren
902f235b5b
(search) Integrate 'similar' tab in site info.
2023-12-02 17:06:40 +01:00
Viktor Lofgren
97d43a6fa2
(search) Revamp browse results with new look.
2023-12-02 17:06:40 +01:00
Viktor Lofgren
9bc65ff0ca
(search) Desaturate search result titles according to rank
2023-12-02 17:06:40 +01:00
Viktor Lofgren
6cd6a615fd
(search) Add data-filter to body as a data attribute
...
For future shenanigans ;D
2023-12-02 17:06:40 +01:00
Viktor Lofgren
5639f0653d
(search) Rename SearchProfile.name into filterId
...
Avoid foot-gun caused by name clash with the Enumeration method name(), which returns the Java name of the enumeration value.
2023-12-02 17:06:40 +01:00
Viktor Lofgren
251174c9a2
(search) Update front page with new look
2023-12-02 17:06:40 +01:00
Viktor Lofgren
42ea87d637
(search) Update conversion results, error page, and dictionary results with new CSS.
2023-12-02 17:06:40 +01:00
Viktor Lofgren
7c8a60b8cf
(search) Site info view is mostly done
...
Also optimize the rendering a bit to avoid having to allocate huge string buffers, writing directly to Spark's response instead.
2023-12-02 17:06:40 +01:00
Viktor Lofgren
2f4500be5a
(search) New frontend look
2023-12-02 17:06:40 +01:00
Viktor Lofgren
fa7534a362
(search) Remove dead code
2023-12-02 17:06:40 +01:00
Viktor Lofgren
a258f0af7a
(search) Refactor search parameters to include query
2023-12-02 17:06:40 +01:00
Viktor Lofgren
01621c6344
(renderer) Make helpers configurable on a by-service basis.
2023-12-02 17:06:40 +01:00
Viktor Lofgren
c7934342a6
(control) Automatic recrawl
2023-12-02 17:06:24 +01:00
Viktor Lofgren
f5c324c06b
(minor) Fix broken test
2023-12-01 17:44:39 +01:00
Viktor Lofgren
f615cf2391
(convert) Loosen up the rules enforcement for documents that have external links.
2023-12-01 17:44:29 +01:00
Viktor Lofgren
c984a97262
(docs) Update crawling.md
2023-11-30 21:53:56 +01:00
Viktor Lofgren
a02c06a837
(docs) Update sideloading-howto.md
2023-11-30 21:51:03 +01:00
Viktor Lofgren
21d6aa421c
(docs) Update setup instructions
2023-11-30 21:44:29 +01:00
Viktor Lofgren
e5d274fe1c
(docs) Improve architectural documentation
2023-11-30 21:38:57 +01:00
Viktor Lofgren
166a391eae
(docs) Improve architectural documentation for the crawler.
2023-11-30 21:30:57 +01:00
Viktor Lofgren
5fb24bb27f
(docs) Improve architectural documentation for the converter.
2023-11-30 20:43:22 +01:00
