Viktor Lofgren
d304c10641
Merge branch 'master' into converter-optimizations
2024-01-05 13:22:46 +01:00
Viktor Lofgren
302c53a8e7
(build) Enable reproducible builds in build.gradle
...
Settings for enabling reproducible builds for all subprojects were added to improve build consistency. This includes preserving file timestamps and ordering files reproducibly.
This is primarily of help for docker, since it uses hashes to determine if a file or image layer has changed.
2024-01-05 13:22:13 +01:00
Viktor Lofgren
ef02b712ad
(build) Remove false depdencency between icp and index-service
...
This dependency causes the executor service docker image to change when the index service docker image changes.
2024-01-05 13:22:13 +01:00
Viktor Lofgren
aca217cf9a
(qs) Better metrics for QS
2024-01-05 13:22:13 +01:00
Viktor Lofgren
9e3386dbbb
(search) Fetch fewer results per page
...
This is a test to evaluate how this impacts load times.
2024-01-05 13:22:13 +01:00
Viktor Lofgren
fdec565b34
(converter) Add upper 128KB limit to how much HTML we'll parse
2024-01-05 13:22:13 +01:00
Viktor Lofgren
33c2188c87
(feature) More trackers
2024-01-05 13:22:13 +01:00
Viktor Lofgren
b3c8fa74cc
(feature) Add another doubleclick variant to the adtech trackers
2024-01-05 13:22:13 +01:00
Viktor Lofgren
e53bb70bef
(converter) Penalize chatgpt content farm spam
2024-01-05 13:22:13 +01:00
Viktor Lofgren
109bec372c
(index) Adjust BM25 parameters
2024-01-05 13:21:52 +01:00
Viktor Lofgren
5c2561d05d
(search) Add query strategy requiring link
2024-01-05 13:21:52 +01:00
Viktor Lofgren
0e970b8037
(valuation) Tweaking penalties a bit
2024-01-05 13:21:52 +01:00
Viktor Lofgren
1694b4d6ef
(valuation) Increase the penalty for adtech a bit
2024-01-05 13:21:34 +01:00
Viktor Lofgren
396299c1db
(index) Reduce the value of site and site-adjacent in BM25P calculations
2024-01-05 13:21:33 +01:00
Viktor Lofgren
71d789aab0
(index) Tweak result valuation renormalization
2024-01-05 13:21:33 +01:00
Viktor Lofgren
41ca50ff0e
(build) Enable reproducible builds in build.gradle
...
Settings for enabling reproducible builds for all subprojects were added to improve build consistency. This includes preserving file timestamps and ordering files reproducibly.
This is primarily of help for docker, since it uses hashes to determine if a file or image layer has changed.
2024-01-05 13:19:59 +01:00
Viktor Lofgren
6d2e14a656
(build) Remove false depdencency between icp and index-service
...
This dependency causes the executor service docker image to change when the index service docker image changes.
2024-01-05 13:17:29 +01:00
Viktor Lofgren
4078708aea
(qs) Better metrics for QS
2024-01-04 13:27:14 +01:00
Viktor Lofgren
343ea9c6d8
(search) Fetch fewer results per page
...
This is a test to evaluate how this impacts load times.
2024-01-04 13:18:07 +01:00
Viktor Lofgren
60361f88ed
(converter) Add upper 128KB limit to how much HTML we'll parse
2024-01-03 23:14:03 +01:00
Viktor Lofgren
f7560cb1d8
(feature) More trackers
2024-01-03 17:31:02 +01:00
Viktor Lofgren
1f66568d59
(feature) More trackers
2024-01-03 17:27:25 +01:00
Viktor Lofgren
7af07cef95
(feature) Add another doubleclick variant to the adtech trackers
2024-01-03 17:21:12 +01:00
Viktor Lofgren
41a540a629
(converter) Penalize chatgpt content farm spam
2024-01-03 17:04:38 +01:00
Viktor Lofgren
f599944942
(converter) Penalize chatgpt content farm spam
2024-01-03 16:51:26 +01:00
Viktor Lofgren
1e06aee6a2
(index) Adjust BM25 parameters
2024-01-03 16:30:46 +01:00
Viktor Lofgren
7bbaedef97
(search) Add query strategy requiring link
2024-01-03 16:23:00 +01:00
Viktor Lofgren
87048511fe
(valuation) Tweaking penalties a bit
2024-01-03 16:02:25 +01:00
Viktor Lofgren
c770f0b68b
(valuation) Tweaking penalties a bit
2024-01-03 15:59:21 +01:00
Viktor Lofgren
78c00ad512
(valuation) Tweaking penalties a bit
2024-01-03 15:52:57 +01:00
Viktor Lofgren
a19879d494
(valuation) Tweaking penalties a bit
2024-01-03 15:32:33 +01:00
Viktor Lofgren
ac1aca36b0
(valuation) Increase the penalty for adtech a bit
2024-01-03 15:20:38 +01:00
Viktor Lofgren
1f3b89cf28
(index) Reduce the value of site and site-adjacent in BM25P calculations
2024-01-03 15:20:18 +01:00
Viktor Lofgren
f732f6ae6f
(index) Tweak result valuation renormalization
2024-01-03 14:53:53 +01:00
Viktor Lofgren
0b9f3d1751
(*) Remove accidental commit of debug logging
2024-01-03 14:32:00 +01:00
Viktor Lofgren
0806aa6dfe
(language-processing) Add maximum length limit for text input in SentenceExtractor
...
Added a new constant, MAX_TEXT_LENGTH, to the SentenceExtractor class. If the length of the text input exceeds this limit, the text is truncated to fit within the limit. This modification is designed to prevent excessive resource usage for unusually long text inputs.
2024-01-03 14:27:47 +01:00
Viktor Lofgren
32436d099c
(language-processing) Add maximum length limit for text input in SentenceExtractor
...
Added a new constant, MAX_TEXT_LENGTH, to the SentenceExtractor class. If the length of the text input exceeds this limit, the text is truncated to fit within the limit. This modification is designed to prevent excessive resource usage for unusually long text inputs.
2024-01-03 14:27:47 +01:00
Viktor Lofgren
4ce692ccaf
(converter) Use SimpleBlockingThreadPool in ProcessingIterator
2024-01-03 14:27:47 +01:00
Viktor Lofgren
3caa4eed75
Merge branch 'master' into converter-optimizations
2024-01-02 17:13:25 +01:00
Viktor Lofgren
c70f508ae8
(prometheus) Saner histogram buckets
2024-01-02 17:13:14 +01:00
Viktor Lofgren
9e64d7aaf9
Merge branch 'master' into converter-optimizations
2024-01-02 15:46:24 +01:00
Viktor Lofgren
72b773f06d
(search) fix search metrics labeling
2024-01-02 15:46:14 +01:00
Viktor Lofgren
5f978b865b
Merge branch 'master' into converter-optimizations
2024-01-02 15:41:48 +01:00
Viktor Lofgren
57a4f92722
(api) fix missing metrics label in api service
2024-01-02 15:41:38 +01:00
Viktor Lofgren
87351e89ca
Merge branch 'master' into converter-optimizations
2024-01-02 15:17:02 +01:00
Viktor
7920c67a48
Merge pull request #71 from MarginaliaSearch/metrics
...
Add Prometheus Instrumentation
2024-01-02 15:13:53 +01:00
Viktor Lofgren
192e356169
(prometheus) Add instrumentation to the api service
2024-01-02 15:12:44 +01:00
Viktor Lofgren
31232e49fb
(prometheus) Add instrumentation to the search, qs and index services.
2024-01-02 15:02:29 +01:00
Viktor Lofgren
116595d218
(prometheus) Add in-docker prometheus instance to exfiltrate metrics from the docker-based services
2024-01-02 14:28:53 +01:00
Viktor Lofgren
9d93a31755
Merge branch 'master' into converter-optimizations
2024-01-02 12:36:16 +01:00
