Viktor Lofgren
7aa2f80117
(domain) id.au should be treated as a TLD
2023-11-06 19:07:47 +01:00
Viktor
d29f9c4ffd
Merge pull request #59 from MarginaliaSearch/atags
...
Support for anchor tag keywords
* Added new (optional) model file in $WMSA_HOME/data/atags.parquet. Due to size limitations on github, this is available at https://downloads.marginalia.nu/exports
* Converter gets a component for creating a projection of its domains onto the full atags parquet file
* New WordFlag ExternalLink
* These terms are also for now flagged as title words
* The ranking algorithm was tweaked to make better use of ngram information as well as weighting the priority BM25
* Fixed a bug where Title words aliased with UrlDomain words
* Fixed a bug in the encyclopedia sideloader that gave everything too high topology ranking
* Crawler will also use the anchor tag file to prioritize crawling documents with external links.
2023-11-06 19:02:52 +01:00
Viktor Lofgren
7617b4cbc2
(crawler) Fix NPE in crawler caused by not having fetched the domains list yet
2023-11-06 18:16:38 +01:00
Viktor Lofgren
e0c769fd19
(converter) Integrate atags.parquet with the encyclopedia sideloader
...
Also clean up stackexchange and dirtree a bit.
2023-11-06 18:03:01 +01:00
Viktor Lofgren
ebd10a5f28
(crawler) Integrate atags.parquet with the crawler so that "important" URLs are prioritized
2023-11-06 16:14:58 +01:00
Viktor Lofgren
2b77184281
(converter) Integrate atags with the topology field
2023-11-06 13:46:44 +01:00
Viktor Lofgren
e23976f6c4
(search) Fix card title overflow
2023-11-06 13:25:39 +01:00
Viktor Lofgren
0b8dc02eba
(result-ranking) Nudge up results with ngram matches a tiny bit
2023-11-06 13:14:22 +01:00
Viktor Lofgren
fde1d0677e
(search) Remove unnecessary dependencies
2023-11-06 12:56:32 +01:00
Viktor Lofgren
48986574ae
(result-ranking) Use a weighted calculation of priority term importance
2023-11-06 12:56:21 +01:00
Viktor Lofgren
c7a6a71d07
(result-ranking) Use a weighted calculation of priority term importance
2023-11-06 12:48:23 +01:00
Viktor Lofgren
1847845151
Revert "(loader) Optimize INSERT statements"
...
This reverts commit 7cb92195d1
2023-11-04 19:32:02 +01:00
Viktor Lofgren
7cb92195d1
(loader) Optimize INSERT statements
...
INSERT IGNORE is too slow.
2023-11-04 17:43:55 +01:00
Viktor Lofgren
72afa0341f
duckdb connection may need to be synchronized?
2023-11-04 14:30:25 +01:00
Viktor Lofgren
0152004c42
Initial Commit Anchor Tags
...
* Added new (optional) model file in $WMSA_HOME/data/atags.parquet
* Converter gets a component for creating a projection of its domains onto the full atags parquet file
* New WordFlag ExternalLink
* These terms are also for now flagged as title words
* Fixed a bug where Title words aliased with UrlDomain words
* Fixed a bug in the encyclopedia sideloader that gave everything too high topology ranking
2023-11-04 14:24:17 +01:00
Viktor Lofgren
30ca5046b5
(docker) Route screenshots to dating as well
2023-11-02 15:47:18 +01:00
Viktor Lofgren
8e9698c9a0
(control/search) Add ability to suggest removing a site from random exploration
...
This is what most complaints have been about.
2023-11-02 15:29:49 +01:00
Viktor Lofgren
3047e2dd7c
(screenshot-capture-tool) Make screenshot-capture-tool cooperate with docker
2023-11-01 16:38:55 +01:00
Viktor Lofgren
a8b9d21f2d
(executor) Refine atag export logic
...
* Remove obviously uninteresting tags
* Omit URL schema for more sensible sorting
* Change the column order to put the source domain last
2023-11-01 13:23:14 +01:00
Viktor Lofgren
c77a5b7cb6
(control) GUI for atags export
2023-10-31 17:55:47 +01:00
Viktor Lofgren
23f2068e33
(executor) Actor for exporting anchor tag data from crawl data
2023-10-31 17:32:34 +01:00
Viktor Lofgren
ffadfb4149
(control) Use a partial template for the storage types tabs.
2023-10-31 17:12:14 +01:00
Viktor Lofgren
b7e38cfbae
(control) Add exports view
2023-10-31 17:08:48 +01:00
Viktor Lofgren
659743b39c
(executor) Export Data actor allocates its own storage
2023-10-31 17:04:07 +01:00
Viktor
cbac42bdd1
Merge pull request #55 from MarginaliaSearch/multinode-index
...
Multinode index, control GUI redesign
2023-10-31 16:37:00 +01:00
Viktor Lofgren
69758c5859
(control) Nicer redirects acknowledging actions
2023-10-31 16:26:29 +01:00
Viktor Lofgren
81bfd7e5fb
(experiment) Utility for exporting atags
2023-10-31 16:10:21 +01:00
Viktor Lofgren
fd8a5e695d
(build) Upgrade dependencies with CVEs
2023-10-31 16:09:58 +01:00
Viktor Lofgren
8f74dbdbb4
(crawler) Set more lenient parameters for recrawl
2023-10-30 11:35:30 +01:00
Viktor Lofgren
fd5a7eac87
(crawler) Exit crawler retriever on thread interrupted
2023-10-30 11:34:16 +01:00
Viktor Lofgren
6bac3c75cb
(api) API documentation
2023-10-29 16:13:21 +01:00
Viktor Lofgren
5d6e0e3790
(log) Clean up logging
...
Don't log the PROCESS stream to executor's logs, as it will also be logged in the spawned process' log files.
Also tell the spawned process which "service" it is so that it gets a log file with a name that makes sense.
2023-10-29 15:52:17 +01:00
Viktor Lofgren
2871a326e6
(ctrl/exe) Clean up UX and code
2023-10-29 14:09:39 +01:00
Viktor Lofgren
abb42f0f36
(crawler) Fix bug in SQL statement
...
Arguments were in the wrong order in inserting fetching sites submitted to be crawled
2023-10-29 13:19:17 +01:00
Viktor Lofgren
f6fcb04817
(experiment) Repair the experiment runner
2023-10-27 16:16:50 +02:00
Viktor Lofgren
b8796d825d
(docs) Update documentation
2023-10-27 13:24:49 +02:00
Viktor Lofgren
e97259aca3
(docs) Update documentation
2023-10-27 13:22:11 +02:00
Viktor Lofgren
88f49834fd
(docs) Update documentation
2023-10-27 12:45:39 +02:00
Viktor Lofgren
4415f52e18
(keyword-extraction) Fix broken test
2023-10-27 12:19:33 +02:00
Viktor Lofgren
98d742d634
(actor) Code cleanup
2023-10-27 12:19:20 +02:00
Viktor Lofgren
6c1ca10be7
(minor) code cleanup
2023-10-27 11:38:37 +02:00
Viktor Lofgren
aeaf2d546a
(search) Fix broken redirect for flagging problems with websites
2023-10-27 11:20:49 +02:00
Viktor Lofgren
c7cb6664b4
(control) Indicate missing services with danger-color instead of having a distracting and constantly updating last-seen number
2023-10-26 18:05:22 +02:00
Viktor Lofgren
79adba9284
(index) Fix bug in dealing with quoted search terms
2023-10-26 16:28:23 +02:00
Viktor Lofgren
37b7f52f2c
(minor) Reduce log severity for getTermMeta miss
2023-10-26 15:41:52 +02:00
Viktor Lofgren
c89e0ab255
(minor) Disable ~vlofgren specific debug test
2023-10-26 15:27:59 +02:00
Viktor Lofgren
f613f4f2df
(array) Fix spurious search results
...
This was caused by a bug in the binary search algorithm causing it to sometimes return positive values when encoding a search miss.
It was also necessary to get rid of the vestiges of the old LongArray and IntArray classes to make this fix doable.
2023-10-26 15:27:02 +02:00
Viktor Lofgren
a497e4c920
(crawler) Terminate crawler after a few hours of no progress
2023-10-26 12:49:28 +02:00
Viktor Lofgren
0f637fb722
(logging) Better logging configurations
2023-10-26 12:48:10 +02:00
Viktor Lofgren
ba48c8e25b
(docker-compose) Add health check to mariadb
2023-10-26 11:10:53 +02:00
