squatter fails on search for strings with 3 characters or less

[paulraines68]

We recently upgraded our old CentOS7 mail server to one based on Rocky 9 with cyrus-imapd-3.4.8-1.el9.x86_64.

Everything seemed to be working fine as before though we noticed a peculiar behavior with searches (such as in alpine client doing a select -> text -> subject). Any search on a string that is 3 chars or less just fails to find any matches. Matches are found on any mail received since the squatter index was last updated.

We have a cron job that builds the squatter index nightly looping over all users $u running

runuser -s /bin/bash - cyrus -c "/usr/sbin/squatter -d -v -r -i -p user/$u

This is usually complete by 4am

If I search for a 3 letter string in the subject like 'DMZ' it finds no matches for any email in my mailbox before 4am, but finds them in mail I received after 4am. If I search for 4 letter string " DMZ" I do find the email from before 4am with " DMZ" in the subject.

I searched the squatter man page but could not find anything about a minimal search string size. Is this expected behavior?

In imapd.conf I have:

search_engine: squat
search_maxtime: 120
search_index_headers: 0

[elliefm]

If I recall correctly, the squat search engine does not index substrings below some length threshold, probably because doing so would make search indexes huge. So short search strings can only be found by a (slow) full body search. But we usually only use a full body search on messages that haven't been indexed yet.

From the behaviour you describe, it sounds like we're not giving short strings any special treatment at search time. They will never be in the index because they're too short, but we only examine full bodies of unindexed messages, so once a message has been indexed it will no longer be found by searching for a short string.

Ahh, here we go, in imap/squat.h:

/* Don't change this unless you're SURE you know what you're doing.
   Its only effect on the API is that searches for strings that are
   shorter than SQUAT_WORD_SIZE are not allowed.
   In SQUAT, a 'word' simply refers to a string of SQUAT_WORD_SIZE
   arbitrary bytes.
*/
#define SQUAT_WORD_SIZE 4

So this is kind of "operating as designed". The squat backend is not well-tested, nor actively maintained, so this probably won't change unless you change it...

You might be able to patch the behaviour of cmd_search such that, if the search string is shorter than SQUAT_WORD_SIZE, apply the full body search step to all messages, not just the unindexed ones. But I suspect this would be prohibitively slow on large mailboxes, and not worth the effort in the end. I'll review a PR if you submit one.

3.4 contains a fix for #2598, in which the squat search index was never being used (all searches were full body searches). That would explain the change of behaviour you've noticed after upgrading: before, your indexes were worthless, so the fact that short strings weren't indexed didn't matter; now, your indexes are being used, and you can't reliably search for things that aren't indexable. I suppose, if the prior search performance without indexing was acceptable, and 3 string searches are important to you, you could just turn off indexing...

[paulraines68]

I doubt I would ever do full mail body searches with just a 3 character search. 95%+ of the time my searches in pine are of the subject or participants. I only do full mail body searches as a "narrow" of a previous subject/participant/date search because, yes, they take too long otherwise.

I looked at the code for a while imap/search_squat.c and could not figure it out enough to know what to do exactly. I was thinking of trying to make it treat everything as unindexed if the search string was < SQUAT_WORD_SIZE but didn't really see how. I will just figure out how to live with the behavior.