161sh/CatgirlIntelligenceAgency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,204 Commits 19 Branches 14 Tags 14 MiB
23f0c79fba
Commit Graph

1204 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Viktor Lofgren
23f0c79fba (control) GUI for data sets/domain types. 2023-10-19 17:48:34 +02:00
Viktor Lofgren
81dd3809e9 (*) WIP Add node affinity to EC_DOMAIN 
Very messy commit due to fractalline yak shaving
 2023-10-19 17:48:34 +02:00
Viktor Lofgren
2bf0c4497d (*) Tool for unfcking old crawl data so that it aligns with the new style IDs 2023-10-19 17:48:34 +02:00
Viktor Lofgren
93122bdd18 (run) Add two nodes to the demo setup 2023-10-16 17:37:26 +02:00
Viktor Lofgren
978550f809 (executor-service) Retire features-convert and move the corresponding packages into the executor service. 2023-10-16 15:43:46 +02:00
Viktor Lofgren
84fea0fd05 (node) Nodes auto-start their monitor actors. 2023-10-16 15:33:22 +02:00
Viktor Lofgren
2df3e0f881 (node) Nodes auto-configure on start-up instead of requiring manual configuration. 2023-10-16 14:46:35 +02:00
Viktor Lofgren
c98117f69d (actor) FS monitor should pick up stuff in BACKUP as well. 2023-10-16 14:37:36 +02:00
Viktor Lofgren
ede5d1f890 (actor) Give process spawners more easily recognizable names. 2023-10-16 14:19:00 +02:00
Viktor Lofgren
39911e3acd (control) Fix incorrect storage base and clean up GUI for data 2023-10-16 13:30:26 +02:00
Viktor Lofgren
3d1c15ef99 (client) Refactor liveness monitor 2023-10-16 12:34:01 +02:00
Viktor Lofgren
f718482e98 (client) Fix tests 2023-10-16 12:12:16 +02:00
Viktor Lofgren
8dafd13cd7 (client) Fix executor tests 2023-10-16 12:02:57 +02:00
Viktor Lofgren
0b19b28a64 (file-storage) Delete unused code 2023-10-16 12:02:57 +02:00
Viktor Lofgren
c245f7ce3a (control) Bootstrapify review-domains and search-to-ban views. 2023-10-15 22:04:23 +02:00
Viktor Lofgren
607d647483 (control) Remove services listing view 2023-10-15 21:48:55 +02:00
Viktor Lofgren
9a38a455c9 (control/exec) File listings in control GUI 2023-10-15 19:15:44 +02:00
Viktor Lofgren
16e0738731 (*) Get multi-node routing working. 2023-10-15 18:38:30 +02:00
Viktor Lofgren
eacbf87979 (control) New list and form for index nodes. 2023-10-14 21:46:52 +02:00
Viktor Lofgren
108b4cb648 (service) Keep disabled multi-noded services dormant when they are configured to be disabled. 2023-10-14 20:58:55 +02:00
Viktor Lofgren
a9dff407a1 (config/db) Clean up migrations 2023-10-14 20:34:03 +02:00
Viktor Lofgren
9e26109e36 (reverse-index) Don't always POST 2023-10-14 16:48:29 +02:00
Viktor Lofgren
6308a8dfcd (control) Node configuration 2023-10-14 16:47:52 +02:00
Viktor Lofgren
4baf9527d7 (*) WIP Control GUI redesign, executor-service, multi-node mq 
This turned out to be very difficult to do in small isolated steps.

* Design overhaul of the control gui using bootstrap
* Move the actors out of control-service into to a new executor-service, that can be run on multiple nodes
* Add node-affinity to message queue
 2023-10-14 12:08:43 +02:00
Viktor Lofgren
199c459697 (*) Add node-affinity to services, processes and file storage. 2023-10-10 12:32:22 +02:00
Viktor Lofgren
61288c5e68 (service, client) First steps towards multiple nodedness 2023-10-09 22:13:27 +02:00
Viktor Lofgren
8375237de5 (converter) Add special keyword for websites with a tilde url. 2023-10-09 17:02:32 +02:00
Viktor
c8d820c17b
Merge pull request #53 from MarginaliaSearch/standalone-index 
Move ranking to the index-service, and query parsing to a new query-service; separate out the search-service
 2023-10-09 15:42:06 +02:00
Viktor Lofgren
6319b8ef51 (api-service) Improved testability, always set content type to application/json 2023-10-09 15:39:34 +02:00
Viktor Lofgren
397a85eaa4 (query-service) Apply blacklisting to search results 2023-10-09 15:18:53 +02:00
Viktor Lofgren
3889c4bdd9 (refactor) Remove features-search and update documentation 2023-10-09 15:12:30 +02:00
Viktor Lofgren
c899f1cb85 (docs) Update documentation to reflect new query service 2023-10-09 14:56:59 +02:00
Viktor Lofgren
d8956c51d0 (refactor) Remove api:search-api 
Application services should not have an API, but purely act as clients
to the core services (which should always have an API).
 2023-10-09 14:42:33 +02:00
Viktor Lofgren
5dd55c7cad (refactor) Rename satellite services to application services 
This is a better descriptor, since they now all implement different applications on top of the core services' APIs.
 2023-10-09 13:45:45 +02:00
Viktor Lofgren
c0e61d4c87 (refactor) Move search service into services-satellite 2023-10-09 13:40:01 +02:00
Viktor Lofgren
97e17282ab (query-service) Move query parsing from search-service to the new query service. 2023-10-09 13:27:44 +02:00
Viktor Lofgren
94c882af7d (query-service) Provide delegate of IndexApi's query functionality. 
This is an intermediate step in the process of introducing the query-service as a proxy between search and index.
 2023-10-08 22:22:26 +02:00
Viktor Lofgren
89c6d85f2f (query-service) Create new empty 'query-service' service 2023-10-08 17:31:50 +02:00
Viktor Lofgren
cf366c602f (search) Refactor SearchQueryIndexService in preparation for feature extraction. 
Prefer working on DecoratedSearchResultItem in favor of UrlDetails.
 2023-10-08 17:15:41 +02:00
Viktor Lofgren
77ccab7d80 (index) Move linkdb to index from search. 
This makes index complete in the sense that you can deploy an index instance and build a complete separate application on top of it, without having to go through the Marginalia-laden search service.
 2023-10-08 16:48:35 +02:00
Viktor Lofgren
f51ba63742 (search) Remove dead file 2023-10-07 21:05:06 +02:00
Viktor Lofgren
9044518be5 (search) Fix broken link to git repo 2023-10-07 19:43:22 +02:00
Viktor Lofgren
9e0367eef4 (search) Filter blacklisted items in API query service as well 2023-10-07 16:16:04 +02:00
Viktor Lofgren
235bb6c1b9 (control) Administrative QOL improvement, GUI for banning spam 2023-10-07 15:45:50 +02:00
Viktor Lofgren
49344d7ea8 (control) Administrative QOL improvement, GUI for banning spam 2023-10-07 15:43:18 +02:00
Viktor Lofgren
1b418d77ff (search) We got some new IP ranges to work with for the crawler 2023-10-07 13:41:55 +02:00
Viktor Lofgren
80cc302627 (search) We can't in claim to be on PC hardware anymore... 2023-10-07 11:49:29 +02:00
Viktor
8e1abc3f10
(index-reverse) Parallel construction of the reverse indexes. (#52) 
* (index-reverse) Parallel construction of the reverse indexes.

* (array) Remove wasteful calculation of numDistinct before merging two sorted arrays.

* (index-reverse)  Force changes to disk on close, reduce logging.

* (index-reverse)  Clean up merging process and add back logging

* (run)  Add a conservative default for INDEX_CONSTRUCTION_PROCESS_OPTS's parallelism as it eats a lot of RAM

* (index-reverse)  Better logging during processing

* (array) 2GB+ compatible write() function

* (array) 2GB+ compatible write() function

* (index-reverse) We are logging like Bolsonaro and I will not have it.

* (reverse-index) Self-diagnostics

* (btree) Fix bug in btree reader to do with large data sizes
 2023-10-07 10:00:00 +02:00
Viktor Lofgren
e498c6907a (forward-index) Don't leak off heap memory 2023-10-05 21:22:13 +02:00
Viktor Lofgren
08e8fc6736 (index-journal) Thread safe IndexJournalReadEntry 2023-10-05 19:39:09 +02:00