Viktor Lofgren
b07752fa9b
(search) Toggle for showing recent results
...
Will by default show results from the last 2 years. May need to tune this later.
2024-01-10 09:37:39 +01:00
Viktor Lofgren
68fd0efbde
(search) Clean up search results template
...
Rendering is very slow. Let's see if this has a measurable effect on latency.
2024-01-10 09:37:39 +01:00
Viktor Lofgren
c80d3eb812
(search) Remove dead code
2024-01-10 09:37:35 +01:00
Viktor Lofgren
f9320995d6
(search) When clicking asn-links, show results from the unfiltered view...
2024-01-10 09:37:13 +01:00
Viktor Lofgren
302c53a8e7
(build) Enable reproducible builds in build.gradle
...
Settings for enabling reproducible builds for all subprojects were added to improve build consistency. This includes preserving file timestamps and ordering files reproducibly.
This is primarily of help for docker, since it uses hashes to determine if a file or image layer has changed.
2024-01-05 13:22:13 +01:00
Viktor Lofgren
ef02b712ad
(build) Remove false depdencency between icp and index-service
...
This dependency causes the executor service docker image to change when the index service docker image changes.
2024-01-05 13:22:13 +01:00
Viktor Lofgren
aca217cf9a
(qs) Better metrics for QS
2024-01-05 13:22:13 +01:00
Viktor Lofgren
9e3386dbbb
(search) Fetch fewer results per page
...
This is a test to evaluate how this impacts load times.
2024-01-05 13:22:13 +01:00
Viktor Lofgren
fdec565b34
(converter) Add upper 128KB limit to how much HTML we'll parse
2024-01-05 13:22:13 +01:00
Viktor Lofgren
33c2188c87
(feature) More trackers
2024-01-05 13:22:13 +01:00
Viktor Lofgren
b3c8fa74cc
(feature) Add another doubleclick variant to the adtech trackers
2024-01-05 13:22:13 +01:00
Viktor Lofgren
e53bb70bef
(converter) Penalize chatgpt content farm spam
2024-01-05 13:22:13 +01:00
Viktor Lofgren
109bec372c
(index) Adjust BM25 parameters
2024-01-05 13:21:52 +01:00
Viktor Lofgren
5c2561d05d
(search) Add query strategy requiring link
2024-01-05 13:21:52 +01:00
Viktor Lofgren
0e970b8037
(valuation) Tweaking penalties a bit
2024-01-05 13:21:52 +01:00
Viktor Lofgren
1694b4d6ef
(valuation) Increase the penalty for adtech a bit
2024-01-05 13:21:34 +01:00
Viktor Lofgren
396299c1db
(index) Reduce the value of site and site-adjacent in BM25P calculations
2024-01-05 13:21:33 +01:00
Viktor Lofgren
71d789aab0
(index) Tweak result valuation renormalization
2024-01-05 13:21:33 +01:00
Viktor Lofgren
c70f508ae8
(prometheus) Saner histogram buckets
2024-01-02 17:13:14 +01:00
Viktor Lofgren
72b773f06d
(search) fix search metrics labeling
2024-01-02 15:46:14 +01:00
Viktor Lofgren
57a4f92722
(api) fix missing metrics label in api service
2024-01-02 15:41:38 +01:00
Viktor
7920c67a48
Merge pull request #71 from MarginaliaSearch/metrics
...
Add Prometheus Instrumentation
2024-01-02 15:13:53 +01:00
Viktor Lofgren
192e356169
(prometheus) Add instrumentation to the api service
2024-01-02 15:12:44 +01:00
Viktor Lofgren
31232e49fb
(prometheus) Add instrumentation to the search, qs and index services.
2024-01-02 15:02:29 +01:00
Viktor Lofgren
116595d218
(prometheus) Add in-docker prometheus instance to exfiltrate metrics from the docker-based services
2024-01-02 14:28:53 +01:00
Viktor Lofgren
9f7df59945
(sideload) Reduce quality assessment.
...
This will make these sideloaded results rank much better as there is a pretty harsh penalty for large low-q websites.
2024-01-02 12:35:59 +01:00
Viktor Lofgren
d2418521a7
(index) Further ranking adjustments
2024-01-02 12:35:59 +01:00
Viktor Lofgren
9330b5b1d9
(index) Adjust rank weightings to fix bad wikipedia results
...
There was as bug where if the input of ResultValuator.normalize() was negative, it was truncated to zero. This meant that "bad" results always rank the same. The penalty factor "overallPart" was moved outside of the function and was re-weighted to accomplish a better normalization.
Some of the weights were also re-adjusted based on what appears to produce better results. Needs evaluation.
2024-01-02 12:35:44 +01:00
Viktor Lofgren
faa50bf578
(sideload) Just index based on first paragraph
...
This seems like it would make the wikipedia search result worse, but it drastically improves the result quality!
This is because wikipedia has a lot of articles that each talk about a lot of irrelevant concepts, and indexing the entire document means tangentially relevant results tend to displace the most relevant results.
2024-01-02 12:35:44 +01:00
Viktor Lofgren
f6fa8bd722
(search) Fetch fewer linking and similar domains.
...
Showing a total of 200 connected domains is not very informative.
2023-12-29 16:37:00 +01:00
Viktor Lofgren
6aee27a3f1
(*) Fix bug in EdgeDomain where it would permit domains with a trailing period, DNS style.
2023-12-29 16:36:01 +01:00
Viktor Lofgren
ea73be6831
(search) Remove the ugly placeholder screenshots from the site info view.
2023-12-29 15:55:46 +01:00
Viktor Lofgren
a065040323
(search) Don't inject arbitrary HTML into the site info view xD
2023-12-29 14:04:26 +01:00
Viktor Lofgren
70f338c3de
(search) Fix NPE in layout selection
2023-12-28 19:34:46 +01:00
Viktor
775974d5ec
Merge pull request #67 from MarginaliaSearch/rss-feeds-in-site-info
...
Add RSS Feeds to site info (WIP)
2023-12-28 13:25:38 +01:00
Viktor Lofgren
c7af40c368
(search) Change layout balance when feeds/samples are present
2023-12-28 13:16:10 +01:00
Viktor Lofgren
00a974a721
(crawler) Fix bug in resynchronizer where it would fail to capture expected exceptions
...
This commit also improves the test coverage for this part of the code.
2023-12-27 20:02:17 +01:00
Viktor Lofgren
f811a29f87
(crawler) Fix resource leak in crawler
...
A 10 MB thread local buffer wasn't static. Oops.
2023-12-27 16:32:17 +01:00
Viktor Lofgren
9707366348
(test) Fix a few slow tests that broke due to domainCount
2023-12-27 13:29:59 +01:00
Viktor Lofgren
9e5fe71f5b
(crawler) Switch hash function in crawler
...
Guava's hashers are a bit allocation hungry, and a big driver of GC churn in the crawler. This switches to the modified Murmur hash function used throughout Marginalia.
2023-12-27 13:29:00 +01:00
Viktor Lofgren
5d1b7da728
Updated site info feed and search service
...
Modified site info feed template to secure the description field against injected code. Also adjusted search service by extracting samples within the correct scope and including them in the returned site info. This improves the quality and security of the displayed information.
2023-12-26 22:06:01 +01:00
Viktor Lofgren
3ea1ddae22
(crawler) Roll back switch to virtual thread pool in crawler
...
This seems to cause a resource leak, it seems the http library uses thread locals?
2023-12-26 19:37:34 +01:00
Viktor Lofgren
1694e9c78c
(search) Add RSS Feeds to site info
...
This change integrates the Feedlot RSS Bot with Marginalia's site info view to offer a preview of the latest updates.
The change introduces a new tiny feature that is a feedlot-client based on Java's HttpClient.
2023-12-26 16:21:40 +01:00
Viktor Lofgren
4763077b76
(search/index) Add a new keyword "count"
...
This is for filtering results on how many times the term appears on the domain. The intent is to be beneficial in creating e.g. a domain search feature. It's also very helpful when tracking down spammy domains.
2023-12-25 20:38:29 +01:00
Viktor Lofgren
c0eaca220c
(search) Add convenient link for AS search to the search view
2023-12-25 15:07:58 +01:00
Viktor Lofgren
25d086c4e1
(crawler) Clean up stale warc files
...
We should probably have an option to keep them, but not by default!
2023-12-25 15:07:36 +01:00
Viktor Lofgren
88551043cd
(crawler) Even more lenient resyncing
2023-12-25 01:48:11 +01:00
Viktor Lofgren
f779f760c4
(crawler) Even more lenient resyncing
2023-12-25 01:44:18 +01:00
Viktor Lofgren
f18f82e229
(crawler) Write etags and last-modified on reference copy
...
This commit also fixes a test that broke with a previous change.
2023-12-25 01:40:13 +01:00
Viktor Lofgren
67ef2b45fa
(crawler) Reduce logging
2023-12-25 01:10:03 +01:00
