Commit Graph

1667 Commits

Author SHA1 Message Date
Viktor Lofgren
825fd10efa (control) Clean up the MQ ui a bit 2023-07-13 15:14:04 +02:00
Viktor Lofgren
1ec6f9cde2 (mq) More robust resume and recovery logic, protection against spurious state changes, minor bugfixes 2023-07-13 14:55:45 +02:00
Viktor Lofgren
a5118fe8f1 (minor) clean-up 2023-07-12 22:46:14 +02:00
Viktor Lofgren
6c88f00a9d (mqsm) guard against spurious transitions from unexpected messages 2023-07-12 22:44:05 +02:00
Viktor Lofgren
bf783dad7a (converter) NPE fix 2023-07-12 20:13:01 +02:00
Viktor Lofgren
8a53e107fa (mq) Synchronous and Asynchronous inboxes. 2023-07-12 20:12:52 +02:00
Viktor Lofgren
0ed938545b (mq) Add single-shot inbox 2023-07-12 18:41:27 +02:00
Viktor Lofgren
480abfe966 (minor) Add limit to pol count in MqPersistence, fix test 2023-07-12 18:16:23 +02:00
Viktor Lofgren
89e4343fdb (minor) Fix test 2023-07-12 18:15:50 +02:00
Viktor Lofgren
8c16a2aede (work-log, minor) Clean up code 2023-07-12 18:10:05 +02:00
Viktor Lofgren
5deec63667 (work-log) Better tests 2023-07-12 18:04:06 +02:00
Viktor Lofgren
363368b150 (converter) Remove auto-refresh. 2023-07-12 17:48:37 +02:00
Viktor Lofgren
74caf9e38a (processes) Remove forEach-constructs in favor of iterators. 2023-07-12 17:47:36 +02:00
Viktor Lofgren
7087ab5f07 (run) Reduce nginx access log noise for local setup 2023-07-11 23:11:34 +02:00
Viktor Lofgren
0b0cf48849 (control) Better looking UUIDs 2023-07-11 23:11:02 +02:00
Viktor Lofgren
00d9773b44 (control) Better looking progress bar 2023-07-11 21:37:32 +02:00
Viktor Lofgren
ac2d7034db (minor) Bugfix in Path handling 2023-07-11 21:24:29 +02:00
Viktor Lofgren
88b9ec70c6 (control, WIP) Run reconvert-load from converter :D 2023-07-11 18:05:37 +02:00
Viktor Lofgren
77261a38cd (control, WIP) MQFSM and ProcessService are sitting in a tree
We're spawning processes from the MSFSM in control service now!
2023-07-11 17:08:43 +02:00
Viktor Lofgren
3c7c77fe21 (minor) Bugfix in Path handling 2023-07-11 17:06:52 +02:00
Viktor Lofgren
4ee3f6ba3f (minor) Refactor ControlService 2023-07-11 14:51:51 +02:00
Viktor Lofgren
4c016b0318 Process monitoring
* Also refactored the SQL tables a bit
2023-07-11 14:46:21 +02:00
Viktor Lofgren
f59cab300e (minor) Javadoc comments for MqPersistance and MqMessageState 2023-07-10 21:59:51 +02:00
Viktor Lofgren
ec7826659a (minor) Javadoc comments for MqPersistance and MqMessageState 2023-07-10 21:52:25 +02:00
Viktor Lofgren
98b5f22104 (control) WIP control service
* Set messages to OK when received so they're cleaned up properly.
2023-07-10 21:33:57 +02:00
Viktor Lofgren
2283ceb77d (control) WIP control service 2023-07-10 18:58:43 +02:00
Viktor Lofgren
fba466d6e2 (crawler) Update URL blocklist
* Don't crawl MDN mirrors
* More mailing list variants
2023-07-10 18:58:43 +02:00
Viktor
cbbf60a599 Better fingerprinting (#35)
* Better fingerprinting for server tech
* Many more features in FeatureExtractor
* Blog specialization
* SiteType table
2023-07-10 18:58:43 +02:00
Viktor Lofgren
c125d8ab48 (search) Fix a bug where space-like characters weren't normalized in query processing. 2023-07-10 18:58:43 +02:00
Viktor Lofgren
f03146de4b (crawler) Fix bug poor handling of duplicate ids
* Also clean up the code a bit
2023-07-10 18:58:43 +02:00
Viktor Lofgren
dbb758d1a8 Minor: Better error handling in crawled domain reader 2023-07-10 18:58:43 +02:00
Viktor Lofgren
da8bcc6e24 Minor: Don't blow up the reader on a corrupted file 2023-07-10 18:58:43 +02:00
Viktor Lofgren
96eecc6ea5 Minor: Readability. 2023-07-10 18:58:43 +02:00
Viktor Lofgren
74644d59f3 (crawler) Update URL blocklist
* Don't crawl MDN mirrors
* More mailing list variants
2023-07-10 18:04:43 +02:00
Viktor
0f9b90eb1c
Better fingerprinting (#35)
* Better fingerprinting for server tech
* Many more features in FeatureExtractor
* Blog specialization
* SiteType table
2023-07-10 17:36:12 +02:00
Viktor Lofgren
ae9537b68e (search) Fix a bug where space-like characters weren't normalized in query processing. 2023-07-07 20:02:05 +02:00
Viktor Lofgren
2619d196bb (crawler) Fix bug poor handling of duplicate ids
* Also clean up the code a bit
2023-07-07 19:56:14 +02:00
Viktor Lofgren
17db23c2c1 Minor: Better error handling in crawled domain reader 2023-07-07 19:48:32 +02:00
Viktor Lofgren
040bea1f75 Minor: Don't blow up the reader on a corrupted file 2023-07-07 19:48:11 +02:00
Viktor Lofgren
dc8277223a Minor: Readability. 2023-07-06 19:50:13 +02:00
Viktor Lofgren
98d1898610 Bugfix: Don't run the xenforo specialization on phpBB. 2023-07-06 18:12:26 +02:00
Viktor Lofgren
1400fb4a9b Bugfix: Don't run the xenforo specialization on phpBB. 2023-07-06 18:11:19 +02:00
Viktor Lofgren
647bbfa617 Fix so that crawler tests don't sometimes fetch real sitemaps when they're run. 2023-07-06 18:05:23 +02:00
Viktor Lofgren
b73fcc19fe Fix so that crawler tests don't sometimes fetch real sitemaps when they're run. 2023-07-06 18:05:03 +02:00
Viktor Lofgren
d9e6c4f266 Trial integration of MQ-FSM into index service. 2023-07-06 18:04:16 +02:00
Viktor Lofgren
34653f03a2 Temporary bugfix, need to find source 2023-07-06 14:13:03 +02:00
Viktor Lofgren
f0a8ca440f MQFSM Usability WIP 2023-07-06 13:33:11 +02:00
Viktor Lofgren
d89db10645 MQFSM Usability WIP 2023-07-06 13:02:16 +02:00
Viktor
413dc6ced4 Update FUNDING.yml 2023-07-05 18:03:36 +02:00
Adrthegamedev
78f21dd19a (an attempt to) Add wikidot to wiki generators list 2023-07-05 18:03:36 +02:00